GSOC

A Deep Dive into CRIU: How Linux Checkpoint/Restore Works Under the Hood

Introduction

One of my first blog and public posting attempts.

I originally wanted to contribute to CRIU (Checkpoint/Restore in Userspace) for Google Summer of Code (GSoC), but I never ended up writing a proposal. Partly because I underestimated the complexity and magnitude of the project, and partly because I kept feeling that I needed to understand everything before contributing.

Instead of writing code right away, I spent months trying to understand what CRIU actually does. This blog is an attempt to record that journey.

What is CRIU?

At its core, CRIU is a checkpoint and restore tool. It works closely with the Linux kernel to capture the exact image of a running process—including all its memory, flags, register states, and open file descriptors—stops the process, and stores the state as a set of .image files. You can later use these image files to restore or restart the process from where it left off.

To successfully checkpoint a process, CRIU must dump every meaningful field within the kernel’s process representation (task_struct). When you restore a process, you are essentially reconstructing a fake/new task_struct from scratch.

Some fields are ephemeral (like scheduler state), while others are persistent (credentials, namespaces) or architecture-dependent (registers). A simplified version of what the kernel manages looks like this:

struct task_struct {
    /* Scheduling */
    volatile long        state;        // TASK_RUNNING, TASK_INTERRUPTIBLE, etc.
    unsigned int         flags;        // PF_EXITING, PF_KTHREAD, etc.
    int                  prio;
    
    /* Identity */
    pid_t                pid;          // thread ID (what the kernel calls pid)
    pid_t                tgid;         // thread group ID (what userspace calls pid)
    struct task_struct  *group_leader; // points to main thread
    
    /* Memory & Filesystem */
    struct mm_struct    *mm;           // userspace memory descriptor
    struct files_struct *files;        // open file descriptors
    
    /* Credentials & Namespaces */
    const struct cred   *cred;
    struct nsproxy      *nsproxy;      // mount, uts, ipc, pid, net namespaces
    
    /* Thread-local storage */
    struct thread_struct thread;       // arch-specific: registers, FPU state, TLS
};

The Basic Dump Flow

    Running Process
       v
    Freeze process and its threads/tasks (via Cgroups)
       v
    Collect Metadata
       +--> Registers
       +--> Memory Maps
       +--> FDs & Namespaces
       v
    Write Image Files

1. Process Tracing (ptrace)

CRIU needs a consistent snapshot, meaning the entire process tree must be stopped atomically. It walks /proc to discover all PIDs, then calls ptrace(PTRACE_SEIZE, pid, ...) on each one. PTRACE_SEIZE is preferred over the older PTRACE_ATTACH because it doesn’t send an intrusive SIGSTOP and allows PTRACE_INTERRUPT to stop a running thread cleanly.

• The Race Condition: New threads can spawn between when CRIU reads the PID list and when it attaches. CRIU handles this via a “freeze loop”, repeatedly re-scanning /proc/[pid]/task/ until the count stabilizes.
• The Workflow: Debuggers and tools like CRIU use ptrace to inspect registers (PTRACE_GETREGS), read/write memory (PTRACE_PEEKTEXT/PTRACE_POKETEXT), and control execution. Due to its power, modern kernels use the YAMA LSM to restrict ptrace scope (typically allowing a process to trace only its direct descendants).

2. The Parasite Code

CRIU needs to execute system calls inside the address space of the target process (e.g., to dump socket states or read restricted memory), but CRIU is an external process.

To solve this, CRIU uses a clever mechanism called the Parasite:

1. It allocates a small memory region in the target process via ptrace-driven mmap.
2. It writes a compiled binary blob (the parasite code) into it using process_vm_writev.
3. It uses PTRACE_SETREGS to redirect the target’s instruction pointer (RIP) into the parasite, letting it run via a Unix socket pair for communication.
4. Once data collection is complete, the parasite cleans up after itself (munmap), and CRIU restores the original register states, detaching cleanly so the process resumes without noticing any interruption.

3. Dumping Memory Pages

Instead of blindly dumping gigabytes of zero-pages or unallocated space, CRIU queries /proc/[pid]/pagemap via the parasite to build a bitmap of active physical pages. It then reads actual page contents from /proc/[pid]/mem.

For live migration, CRIU can perform a “pre-dump” using userfaultfd or soft-dirty page tracking (PTEs). It marks pages clean, lets the process run briefly, and re-dumps only the modified pages, drastically shrinking the final cutover pause.

4. Handling Kernel Resources

A process is more than just memory and registers; it lives inside complex kernel subsystems:

• Namespaces: CRIU handles PID namespaces using clone(CLONE_NEWPID) and set_tid to restore processes with their exact original PIDs. Network namespaces are reconstructed via netlink sockets (RTM_GETLINK, RTM_GETADDR, RTM_ROUTE), alongside mount namespace mapping via /proc/[pid]/mountinfo.
• Cgroups v2: CRIU utilizes modern unified cgroups and the cgroup.freeze interface (echo 1 > cgroup.freeze) to atomically freeze process trees safely, bypassing the race conditions and signal-handling issues inherent to traditional SIGSTOP methods.
• Pipes & Sockets: CRIU reads buffered pipe data (often via splice), and handles active TCP streams using TCP Repair Mode (TCP_REPAIR), allowing it to capture and restore sequence numbers, windows, and queues without dropping live network connections.
• Shared Memory (shm): It detects shared physical memory frames across multiple processes, saves the content once, and accurately maps them back on restore.

5. The Image Files

CRIU does not output a monolithic snapshot file. Instead, it organizes the state into specialized image components:

• pstree.img (Process hierarchy)
• mm.img (Memory maps)
• pages.img (Actual memory contents)
• fdinfo.img & pipes.img & sockets.img (Inter-process communication and files)

What I Learned

Before diving into CRIU, my understanding of Linux internals was mostly theoretical. Studying how CRIU pieces together registers, open files, namespaces, cgroups, TCP streams, and shared memory gave me a profound, hands-on appreciation for how the Linux kernel operates under the hood.

Even though I never submitted a GSoC proposal, exploring CRIU taught me more about operating systems than any project I had tackled before.