Moving a Running AI Agent Sandbox Between Nodes

An agent's workspace contains only part of its state. A Python process may hold intermediate results in memory. A subprocess may still be working. An open file may no longer even have a pathname.

When a worker needs to be drained, preserving its files does not preserve all that work. We wanted the running workload to continue on another worker, with its memory and supported open files intact.

Sandbox0 now has a system-managed migration path for compatible workers. In six sequential migrations of a Python workload holding 128 MiB of random memory, the time from the drain trigger to the first authenticated command response on the destination ranged from 3.40 to 3.64 seconds, with a 3.52-second median.

The current implementation uses stop-and-copy: the workload pauses while its execution state is captured and transferred, then resumes on the target. External connections may need to reconnect. The result is process continuity across a planned node move, with a measurable interruption.

This post explains how that handoff works, why copy-on-write storage already solves much of the disk-transfer problem, and how we shortened the remaining path without removing its durability requirements.

Files Are Only Part of a Running Workspace#

In our Nomad architecture, a sandbox has a durable identity. Its compute allocation is one runtime generation attached to that identity. PostgreSQL records lifecycle state and writer authority; encrypted block-COW objects hold committed filesystem data in regional object storage.

That separation already lets a sandbox resume on a different worker. Ordinary pause/resume, however, starts a new runtime from the committed filesystem. Variables that existed only in memory, live subprocesses, and open file descriptors do not survive it.

For an interactive Python session, the distinction is concrete. Reopening its working directory does not recover an intermediate result held in a variable. For an open file that has already been unlinked, there may be no pathname to reopen at all. Retaining the original file descriptor and offset requires preserving execution state.

Start with Stock gVisor#

gVisor provides checkpoint and restore primitives. Sandbox0 uses the official stock runsc runtime; the release qualified for this migration path is release-20260914.0.

Those primitives save and restore the guest's execution state. A complete migration system also has to coordinate the writable filesystem, resource reservations, network policy, node identity, routing, and recovery after an interrupted operation.

Before disturbing the source, manager reserves compatible destination capacity. Eligibility checks cover runtime architecture, the runsc execution shape, and CPU features. A target must be able to restore the source's saved state. Having free memory alone does not make it eligible.

The sandbox ID survives the move. The physical allocation changes and the runtime generation advances. Guest processes retain their identities, including guest PIDs; host processes and host PIDs are a different matter.

Inside the sandbox, the preserved procd process rebinds to the new runtime generation. It must not go through an ordinary activation that starts the workload again. A successful restore must preserve the original process instance, rather than make a restarted entrypoint look like continuity.

COW Already Removes the Full-Disk Copy#

A natural question is: if RootFS is copy-on-write, why transfer a whole disk?

We do not. Immutable base blocks already live in regional object storage. The source seals changed RootFS blocks and their mapping into a new immutable generation. The destination attaches that exact generation, reuses cached blocks, and reads missing data on demand.

There are two kinds of state to move:

Anonymous process memory is outside the RootFS block graph. A Python object held in RAM still needs to be represented in the execution checkpoint, even when the process has barely written to disk.

The two representations must describe the same point in execution. Blocking new API commands is insufficient: existing processes and background threads can still change memory or write files. The source stops execution before sealing the matching filesystem state, and the filesystem barrier must cover host-side writes as well.

COW therefore reduces disk movement while leaving a separate memory-transfer and consistency problem to solve.

Peer Transfer Starts Before Everything Is Published#

A serial transfer path would wait for the complete checkpoint, publish it to regional storage, and then have the destination download it. That leaves the destination idle while useful bytes already exist at the source.

Our peer path lets the target receive checkpoint data as it becomes available. Transfer over pinned mutual TLS overlaps with publication to the encrypted regional object store. The destination retains a bounded temporary cache; when the final image is known, it verifies reusable chunks and receives any missing or changed data.

This chunk reuse belongs to one migration. It avoids retransmitting matching checkpoint data that the target has already received. It does not implement incremental memory dirty-page tracking across successive migrations, and the guest is not continuing to run through iterative memory pre-copy rounds.

The key distinction is that receiving bytes does not authorize execution. The destination still needs the final committed manifest, verified image contents, and an explicit grant to restore.

The parallel branches show work that can overlap, not equal-duration stages. Regional durability remains on the critical path: we do not start the target merely because its peer cache is full.

That cache is an optimization with a recovery path. If it cannot be reused, the destination can fall back to the published regional image. Invalid authorization or mismatched identity must remain errors; treating them as ordinary cache misses would erase an important security boundary.

Only One Copy May Execute#

Filesystem writer fencing prevents two owners from publishing divergent disk state. Execution ownership needs its own protection. Two copies of a process could still send requests to an external service even if only one could write the durable filesystem.

Our central handoff rule is therefore:

Before the target is authorized to execute, the source must have lost both execution authority and writer access.

PostgreSQL owns the migration intent and the exact source and destination identities. Node journals preserve the local evidence needed to recover an interrupted step. A checkpoint directory or a successful file copy cannot become a second authority for starting a runtime.

The simplified sequence is:

1. Reserve a compatible target and prepare its network policy and storage.
2. Stop the source and capture matching execution and filesystem state.
3. Commit the durable image and prove source execution and writer access have been revoked.
4. Authorize the exact destination and restore the preserved workload.
5. Verify command readiness, commit the new routing, and finish source cleanup.

If no compatible target is available, the source keeps running while the system waits. Once a consistency cut is held, failures must be reconciled against that same operation rather than starting unrelated copies.

For example, if regional storage is temporarily unavailable, the source cut is retained and publication is retried. The target's early peer receipt does not let it bypass that wait. If the target may already have executed but its response was lost, recovery must inspect and reconcile that target. Blindly restoring the image elsewhere could run the workload twice.

This protects execution ownership. It does not provide exactly-once semantics for arbitrary external API requests, and requests already sent to external services are not automatically replayed.

What Survived the Move#

A fast response from a new container would be weak evidence of migration. Our acceptance workload checked state that a restarted entrypoint could not reconstruct merely by reopening its workspace:

• the original guest PID and a continuously advancing counter;
• the SHA-256 of the complete 128 MiB random-memory payload;
• an open, unlinked file and its current offset;
• shared mmap contents;
• runtime /tmp contents and persistent RootFS files.

All six moves in the final default-configuration cohort passed the state checks and reused the prefetched image. Cleanup checks also verified release of source resources and temporary staging, with capture scopes collected after the cohort.

Separate fault tests exercised manager restart, regional storage interruption, and failures around source capture and destination restore. Their outcomes depend on the failure boundary: some continued the same migration; others safely converged to a paused sandbox at a committed filesystem head. They are evidence for specific recovery paths, not a claim that every host failure preserves volatile memory.

The implementation and acceptance record contains those boundaries and the historical experiments.

Measuring Until the Target Is Usable#

For the final cohort, we measured from the source drain trigger to the first authenticated command response through the regional entry point from the committed destination generation. This includes the handoff; it is not just the duration of a runsc restore call.

The fixture used cluster-gateway as its single-cluster regional entry point, without a separate regional-gateway process. Source cleanup was checked separately and can finish after the destination becomes usable. Full workload validation supports the continuity result but is not the timing endpoint.

The payload size describes the workload's random memory, not the exact size of the complete checkpoint. We kept the CPU quota and shared-memory THP setting unchanged for this cohort.

The result is a repeatable 3.x-second observation for this workload. None of these six moves completed within three seconds. Six serial samples do not establish production percentiles or a latency bound for larger memory, heavier disk writes, cold caches, or concurrent migrations.

There is still substantial work in capture and restore. In this cohort, checkpoint and restore had medians of about 667 ms and 987 ms, respectively. Those are useful optimization targets, but stage medians cannot be added into a complete waterfall: work overlaps, and there are other stages in the end-to-end path.

Earlier experiments also used different timing endpoints, including final source cleanup. Comparing those directly with target usability would produce a misleading speedup number. We keep the historical observations in the engineering record with their original conditions.

The Current Operating Boundary#

Migration is a system lifecycle operation selected during eligible planned node evacuation. Users do not choose a destination node or call a public sandbox.migrate() API. The current scope is compatible nodes within one cluster and region.

External network continuity remains a separate problem. Host-side egress proxy connections, SSH connections, and gateway streaming sessions are not contained in the guest execution image. An internal guest-loopback TCP test passed, but it does not establish continuity for external connections. Applications must be prepared to reconnect and handle connection errors.

Planned migration also requires the source to remain available long enough to produce a complete durable checkpoint. A filesystem snapshot cannot recover memory that was lost before it was saved. Explicit runtime-upgrade maintenance continues to use its own audited pause/resume flow; this feature does not turn every upgrade into process-preserving migration.

Our next performance work is to characterize larger working sets, cold-cache behavior, dirty RootFS workloads, concurrency, and tail latency. Memory pre-copy and external connection continuity would require additional designs and validation.

The practical gain is already concrete: an eligible running workload can continue on a different worker with its process state preserved. Durable workspace storage made compute replaceable. Execution-state migration extends that separation to the work still happening in memory.

For the surrounding architecture, read A Sandbox Is Not a Container. For protocol details, measured conditions, and failure evidence, see the versioned migration engineering record.