GSOC

Introduction

I wanted to contribute to CRIU for GSoC, but i never ended up writing a proposal.

Partly because I underestimated the complexity of the project, and partly because I kept feeling that I needed to understand everything before contributing.

Instead of contributing code, I spent months trying to understand what CRIU actually does.

This post is a record of that journey. so what is CRIU, CRIU is cheak point restore tool, god knows how that shorts to CRIU, but what it does is closely work with the kernel, to get the image of any process running, all the memory, all the flags, register states, and stop the process and store the state as an .image file, we can then use this image to restore/restart this process

CRIU must dump and restore every meaningful field there. When you restore a process, you’re reconstructing a fake/new task_struct from scratch. some fields are ephemeral (scheduler state) vs. persistent (credentials, namespaces) vs. arch-dependent (registers) some of these are saved some are not.

The flow:

    Running Process
       v
    Freeze Tasks
       v
    Collect Metadata
       +--> Registers
       +--> Memory Maps
       +--> FDs
       +--> Namespaces
       +--> Cgroups
       v
    Write Image Files

Logic behind capture of one process

dump_one_task() 
	parse_pid_stat inside proc/pid/stat
	collect mapping
		parse smap -> reads /proc/pid/smaps
		for file backed VMA -> dump_filemap saves the backing file
	collect fds
		opeddir-> read every symlink??
		build list of fd number for parasite to drain??
	parse_posix_timers
		read/proc/pid/timers
	dump_task_signals

verbose and noisy eh? i get it, one has to understand many things to get the process CRIU dump, since CRIU works very closly to kernel

To understand CRIU, I had to understand:

1. Process tracing (ptrace)
2. Process freezing
3. Namespaces
4. Cgroups
5. Memory management
6. IPC (pipes, sockets, shared memory)
7. TCP connection restoration

Each of these solves a specific problem in checkpoint/restore.

so lets start with the fundamental process which evens allows us to read some other process’s memory, Ptrace

Ptrace

CRIU needs a consistent snapshot, so the entire process tree must be stopped atomically. It walks /proc to discover all PIDs in the tree, then calls ptrace(PTRACE_SEIZE, pid, …) on each one. PTRACE_SEIZE is better than the older PTRACE_ATTACH because it doesn’t send SIGSTOP (which would disturb signal state), and it allows PTRACE_INTERRUPT to stop a running thread cleanly.

The tricky part: there’s a race. New threads can be spawned between when CRIU reads the PID list and when it attaches. CRIU handles this by repeatedly rescanning /proc/[pid]/task/ until the count stabilizes — a technique called the “freeze loop.”

How ptrace Works

The fundamental concept of ptrace Historically ptrace was designed around a parent-child model, but modern ptrace can also attach to existing processes.

1. Establishing a Connection

A process can be traced in two ways:

• PTRACE_TRACEME: A child process calls this to tell the kernel, “some entity is going to trace me.” This is usually followed by an execve() to start the program you want to debug.
• PTRACE_ATTACH: A tracer attaches to an already running process (if it has the right permissions).

2. The Stop-and-Go Mechanism

When a process is being traced, it stops every time a “significant event” occurs. Significant events include:

• Receiving a signal.
• Entering a system call.
• Exiting a system call.
• Hitting a breakpoint.

When the tracee stops, it enters a TASK_TRACED state. The tracer is notified via waitpid(). While the tracee is stopped, the tracer can inspect its “guts.”

Key Operations

The ptrace system call signature looks like this: long ptrace(enum __ptrace_request request, pid_t pid, void *addr, void *data);

Common requests include:

The Workflow of a Debugger

To understand ptrace in action, look at how a debugger like GDB uses it to hit a breakpoint:

1. Search: The tracer finds the memory address where the user wants a breakpoint.

2. Save & Replace: The tracer reads the original instruction at that address (PTRACE_PEEKTEXT) and saves it. It then replaces it with a special trap instruction (like INT 3 on x86).

3. Run: The tracer tells the tracee to continue (PTRACE_CONT).

4. Trap: When the tracee hits INT 3, the CPU raises an exception, the kernel catches it, stops the tracee, and sends a SIGTRAP signal.

5. Handle: The tracer (notified by waitpid) sees the SIGTRAP. It then replaces the trap instruction with the original instruction so the program can eventually continue normally.

due to its powerfullness, modern kernel use YAMA LSM to restrict scope of ptrace, typically, a process can only trace its own descendants.

Parasite

This is the most unusual and beautiful part of CRIU and the hardest to grasp at first. The problem is: CRIU needs to make syscalls from the address space of the target process — for example, to dump the contents of a Unix socket or read memory that’s only accessible from inside. But CRIU is a separate process.

The solution: CRIU allocates a small region of memory in the target with ptrace-driven mmap, then writes a compiled blob of shellcode (the “parasite”) into it using process_vm_writev. It then uses PTRACE_SETREGS to redirect the target’s instruction pointer into the parasite, and PTRACE_CONT to let it run. The parasite communicates back to CRIU via a Unix socket pair. After the dump, CRIU runs the parasite again to clean itself up, then restores registers and detaches.

Before CRIU can ask the parasite to do real work, they need a communication channel. CRIU redirects RIP into the parasite’s entry point, which calls socketpair(AF_UNIX, SOCK_STREAM, 0, fds) using raw syscalls. One end is passed back to CRIU via SCM_RIGHTS (file descriptor passing over Unix sockets). From this point, CRIU sends commands to the parasite and receives results over this socket

CRIU sends protobuf-encoded command messages to the parasite over the socket. The parasite executes them using raw inline syscalls — everything happens inside the target’s PID, address space, and file descriptor table

When all data is collected, CRIU sends a QUIT command. The parasite calls munmap() on its own mapping — deleting itself from the target’s address space — then exits. CRIU uses PTRACE_SETREGS to restore the exact registers it saved in step 1 (including the original RIP, RSP, and all flags). Then PTRACE_DETACH releases the target. The process resumes exactly at the instruction it was at when frozen. It has no idea any of this happened — its memory, registers, and FDs are completely unmodified.

Dump memory pages

CRIU reads /proc/[pid]/pagemap via parasite to build a bitmap of which physical pages are actually present (vs swapped out, or never dirtied). This avoids dumping gigabytes of zero pages. Then it reads the actual page content from /proc/[pid]/mem. For live migration, CRIU can do a “pre-dump” pass that uses userfaultfd or soft-dirty PTEs: it marks all pages clean, lets the process run briefly, then re-dumps only the pages that were written during that window. This shrinks the final stop-and-copy pause dramatically.

At this point CRIU has captured process memory and CPU state.

Unfortunately a process is more than memory.

It also exists inside namespaces, cgroups, mount trees, and network configurations.

These kernel resources must be restored as well.

kernel resources

Namespaces

Namespaces are the hardest part of CRIU dump because each type has a completely different dump mechanism. PID namespace — CRIU must preserve the exact PID numbering. During restore, it uses clone(CLONE_NEWPID) to create a new PID namespace, then uses set_tid (a kernel feature CRIU itself helped land) to restore processes with their original PIDs. Without this, a process that had PID 42 inside a container would come back as a different PID and all its self-references would break.

Network namespace — The richest dump. CRIU reads the entire network state via netlink sockets: all interfaces (RTM_GETLINK), IP addresses (RTM_GETADDR), routes (RTM_GETROUTE), routing rules (RTM_GETRULE), and iptables/nftables rules via iptables-save. It also handles veth pairs, bridge devices, macvlan, and tun/tap devices — each needs special handling.

Mount namespace — CRIU reads /proc/[pid]/mountinfo and reconstructs the exact mount tree. Bind mounts, overlay mounts, and tmpfs all need special handling. This interacts deeply with the filesystem layer.

Cgroups

Cgroups provide:

1. Hierarchical grouping of processes into a tree
2. Per-group resource accounting (how much CPU/memory is this group using?)
3. Per-group resource limits (this group may not exceed X)
4. Per-group resource control (throttle this group to Y bandwidth)

In cgroups v1 (the orginals version), each resource controller is a separate hierarchy:

The fundamental v1 problem: A process can be in different positions in different hierarchies. nginx might be under /cpu/high-priority/ for CPU but /memory/web-servers/ for memory. This created confusion, inconsistency, and made hierarchical delegation nearly impossible.

Cgroups v2: Unified Hierarchy

v2 has one tree with all controllers:

/sys/fs/cgroup/           ← unified root
├── cgroup.controllers    ← "cpu memory io pids"
├── cgroup.procs          ← PIDs in root cgroup
├── cpu.stat
├── memory.current
├── system.slice/
│   ├── cgroup.controllers
│   ├── cgroup.procs
│   ├── nginx.service/
│   │   ├── cgroup.procs
│   │   ├── cpu.max           ← "200000 1000000" = 20% CPU
│   │   ├── memory.max        ← "536870912" = 512MB
│   │   ├── memory.current    ← current usage
│   │   └── io.max
│   └── postgres.service/
└── user.slice/

/sys/fs/cgroup/mycontainer/ └── cgroup.freeze — 0 (thawed) | 1 (frozen) cgroup.events — contains “frozen 1” when freeze complete Freeze: echo 1 > /sys/fs/cgroup/mycontainer/cgroup.freeze Wait for completion (poll cgroup.events): while ! grep -q “frozen 1” /sys/fs/cgroup/mycontainer/cgroup.events; do sleep 0.01 done Thaw: echo 0 > /sys/fs/cgroup/mycontainer/cgroup.freeze

cgroup freezer is better then SIGTOP for CRIU

SIGSTOP:

• signals are delivered asynchronously - race window between finding all PIDs and stopping all of them
• new thread can spawn between enumeration and stop loop
• SIGCONT delivery is not guaranteed to be ordered
• Threads in TASK_UNINTERRUPTIBLE dont respond to signal at all (parasite code needed)

freezer approach

• Move all process into a freezer cgroup(atomically) before freezing
• frozen, the kernel freezer all current the future atomically
• The kernel handles the TASK_UNINTERRUPTIBLE case: it waits for those processes to exit the uninterruptible state before freezing
• After freeze is confirmed, every process is in TASK_FROZEN state — guaranteed consistent
• CRIU can now safely enumerate and dump

CRIU’s actual freeze strategy uses a combination:

1. Cgroup freezer to atomically stop the process tree
2. ptrace attachment to individual threads for register access and memory reading
3. The cgroup freeze ensures no new children spawn between enumeration steps

Pipes

pipe() creates a pipe, a unidirectional data channel that can be used for interprocess communication. The array pipefd is used to return two file descriptors referring to the ends of the pipe. pipefd[0] refers to the read end of the pipe. pipefd[1] refers to the write end of the pipe. Data written to the write end of the pipe is buffered by the kernel until it is read from the read end of the pipe

The pipe is a circular buffer of pages — default 16 pages = 64KB. Each pipe_buffer entry points to a struct page and tracks offset/length within it.

CRIU and Pipes

When CRIU dumps both processes:

1. Save the pipe’s current buffer contents (data in flight)
2. Save which process has which end
3. Save pipe capacity (F_GETPIPE_SZ)
4. On restore: create pipe, write saved data back into it, connect ends to correct processes

If a pipe has data buffered but neither process has read it, that data must survive the checkpoint. CRIU reads it from the kernel using a trick: splice the pipe data to a temporary file, save the file, restore by splicing back.

Sockets

Sockets are a method of IPC that allow data to be exchanged between applications, either on the same host (computer) or on different hosts connected by a network

• Each application creates a socket. A socket is the “apparatus” that allows com- munication, and both applications require one.
• The server binds its socket to a well-known address (name) so that clients can locate it.

Before checkpoint

Client <--TCP--> Server

Checkpoint
     |
     v
Save:
- seq numbers
- receive queue
- send queue

Restore
     |
     v
Recreate identical TCP state

// Enable TCP repair mode — socket acts as if it’s in maintenance int repair = 1; setsockopt(sock, SOL_TCP, TCP_REPAIR, &repair, sizeof(repair));

// Now you can: // 1. Set sequence numbers (normally read-only) setsockopt(sock, SOL_TCP, TCP_REPAIR_QUEUE, &queue, sizeof(queue)); // queue = TCP_SEND_QUEUE or TCP_RECV_QUEUE

// 2. Inject data into send/receive queues without actually sending send(sock, data, len, 0); // in repair mode: injects into queue, no packet sent

// 3. Get/set TCP window parameters struct tcp_repair_window wnd; getsockopt(sock, SOL_TCP, TCP_REPAIR_WINDOW, &wnd, &len);

// 4. Read connection metadata struct tcp_info info; getsockopt(sock, SOL_TCP, TCP_INFO, &info, &len); // info.tcpi_snd_nxt, tcpi_rcv_nxt, tcpi_snd_una, etc.

// To restore a TCP connection on a different machine: // 1. Create socket // 2. Enable TCP_REPAIR // 3. bind() source address, connect() destination (no SYN sent in repair mode!) // 4. Restore send/receive queues with saved data // 5. Set sequence numbers to match checkpoint state // 6. Disable TCP_REPAIR — connection is now live repair = 0; setsockopt(sock, SOL_TCP, TCP_REPAIR, &repair, sizeof(repair)); // Remote end never knew there was an interruption!

CRIU and Shared Memory

Shared memory requires special handling: if two processes share the same physical memory, CRIU must:

1. Detect the sharing (pages with same physical frame number in /proc/[pid]/pagemap for different processes)
2. Save the content once (not twice)
3. On restore: create the shmem region, restore content, mmap it into both processes

For shm_open regions: save the name, size, and content. Restore by recreating with same name.

Write in process A → instantly visible to process B (same physical page). No copy, no syscall for the communication itself.

MAP_ANONYMOUS | MAP_SHARED — shared memory without a file:

void *mem = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
                 MAP_ANONYMOUS | MAP_SHARED, -1, 0);
// Works across fork() — parent and child share this mapping
// Not accessible to unrelated processes (no name)

Image Files

CRIU does not create a single giant snapshot.

Instead it creates multiple image files:

• pstree.img
• mm.img
• pages.img
• fdinfo.img
• pipes.img
• sockets.img

Each contains a specific part of process state.

What I Learned

Before geting into CRIU, I had a vague idea of process and kernel foundations only as concepts and ideas.

all this helped me learn and even work closely with:

• Registers
• Open files
• Namespaces
• Cgroups
• TCP connections
• Shared memory
• Process trees

CRIU works because it reconstructs all of these pieces together.

Even though I never submitted a GSoC proposal, understanding CRIU taught me more about Linux internals than any project I had before.