A Sandbox Is Not a Container: Inside Sandbox0's New Nomad Architecture

The most important object in a sandbox platform is not the container.

It is the sandbox identity that survives after one container, process tree, or worker is gone.

That distinction now shapes the entire Sandbox0 architecture. The current data plane runs on dedicated Nomad nodes, starts sandboxes with stock gVisor runsc, and persists writable files as encrypted block copy-on-write generations in S3-compatible object storage. PostgreSQL records the lifecycle truth. Compute nodes are deliberately disposable.

The result is a simpler rule:

a sandbox is a durable identity; compute is one replaceable runtime generation attached to it.

This post explains why we rebuilt the runtime around that rule, how a claim moves through the system, what survives pause or failure, and where we draw the line between scheduler state and physical cleanup.

Why We Changed the Boundary#

Sandbox0's previous data plane was organized around Kubernetes objects, containerd, a custom SandboxVolume abstraction, and S0FS. That architecture gave us a productive first version, but its boundaries became less natural as the product moved toward long-running agent workloads.

An agent sandbox may live for days while its compute allocation lives for minutes. It may pause to release compute, resume on another node, produce a snapshot while running, fork into several attempts, or recover after a worker disappears. The durable object is therefore not any individual Pod or runtime process.

We also wanted one warm pool to serve sandboxes with different CPU and memory requests. Encoding those requests into pre-created workload shapes fragments a pool: capacity can be free on a node but unavailable in the exact shape a claim needs. A generic warm carrier is more useful if the platform can assign the real resource lease when it knows the claim.

Finally, cleanup needed a stronger definition. An orchestrator saying that an allocation disappeared is important evidence, but it does not prove that a stale runsc process, RootFS writer, mount, network rule, or cgroup is absent from the node.

This is not an argument that Nomad is inherently safer or faster than Kubernetes. It is an architecture decision about ownership. Nomad gives us a small placement surface for dedicated workers and one-shot carrier allocations. Sandbox0 owns the state machine that is specific to persistent agent sandboxes.

The Architecture at a Glance#

The system separates global routing, regional routing, data-plane control, node authority, and guest execution.

In a single-cluster installation, cluster-gateway can run in full mode and serve as the regional endpoint, so regional-gateway and scheduler are not required. In a multi-cluster region, the scheduler selects a cluster for a new claim using live capacity, weight, and template compatibility. Existing sandboxes continue to route to their recorded cluster.

All clusters in one region share that region's PostgreSQL writer authority and S3-compatible RootFS store. Regions do not share these stores. The optional global gateway knows which region owns a team, but it does not know the Nomad nodes or clusters inside that region.

That keeps the failure domains explicit: global routing is not runtime truth, regional routing is not node authority, and the scheduler is not a RootFS writer.

Four Ownership Rules#

The redesign follows four rules.

1. Sandbox identity is durable#

PostgreSQL owns the sandbox lifecycle, current runtime generation, RootFS head, writer epoch, template association, policy state, and resource lease records. A runtime allocation is attached to that identity; it does not define it.

Pause seals the current writable RootFS generation and releases compute. Resume creates a new runtime generation, potentially on a different worker, for the same sandbox identity.

2. Workers are disposable#

A worker's local NBD attachment, XFS or OverlayFS branch, downloaded blocks, mounts, journal, and caches must be reconstructible. They can accelerate the runtime, but they cannot be the only copy of durable sandbox state.

The durable RootFS is a graph of immutable, application-encrypted block-COW objects in regional object storage. PostgreSQL records which committed head belongs to the sandbox. Losing a worker should lose compute and possibly an uncommitted dirty tail, not the last committed filesystem generation.

3. There is one transactional authority#

Manager replicas are active-active, but they converge through one PostgreSQL writer authority. Claim, capacity, runtime generation, RootFS writer, and terminal-release transitions are transactions rather than independently maintained copies of state.

The same boundary applies to usage. Open-source Sandbox0 records metering truth. PostgreSQL stores the current metering projection state and a transactional outbox; ClickHouse can build the historical query projection asynchronously. Pricing, invoices, and payments remain a Sandbox0 Cloud concern, not a second usage ledger in the runtime.

4. Cleanup fails closed#

A carrier slot, resource lease, or RootFS writer cannot be reused just because the Nomad task or task driver is gone. A plugin-independent reconciler must prove the old writer, runsc instance, mount, network state, and cgroup are physically absent before the terminal transaction returns their capacity.

This is conservative by design. Temporarily withholding capacity is safer than assigning the same CPU, mount, or writer authority to two runtime generations.

One Shared Warm Pool#

Nomad places a system job with one-shot, resource-neutral carriers on dedicated Sandbox0 workers. Before claim, a carrier has a network namespace, a control socket, and a small amount of driver overhead. It does not have a guest runtime or a sandbox-sized CPU and memory reservation.

Compatibility is based on inputs that cannot change at claim time: CPU architecture, task-driver and runsc versions, gVisor platform, file-access mode, RootFS format, the fixed procd contract, and security class.

CPU, memory, PID limits, weights, quotas, and cpusets are different. Manager leases those exact resources from physical node capacity during claim. They do not create separate carrier pools.

This lets compatible templates with different resource requests draw from the same warm supply. It also keeps metering honest: resource usage is associated with the sandbox's real lease, not the carrier's small Nomad overhead reservation.

Carriers are single-use. After a successful claim, pause, crash, stop, or delete, Nomad creates a fresh allocation to replenish the pool. We do not restart another guest inside the old allocation and assume all privileged node state was reset correctly.

What Happens During a Claim#

A claim crosses several authorities, so its publication point matters.

Manager first reserves a compatible ready carrier and exact CPU, memory, PID, and RootFS attachment capacity in PostgreSQL. The current ctld primary on the selected node then creates the cgroup v2 lease, attaches the writable RootFS, and applies the network policy before guest startup.

The task driver receives a committed assignment. It writes the OCI bundle with the existing network namespace, RootFS mount, and exact resource lease, then invokes stock runsc create and runsc start.

The generation is not published merely because a process was started. Manager waits for an authenticated readiness or first-command proof from procd that is bound to the expected allocation and runtime generation. Late proof from an older attempt cannot publish a newer sandbox.

Node Authority Lives Outside the Guest#

Each dedicated Nomad client runs ctld-a and ctld-b directly on the host. They share a state directory and use a kernel lock to elect one primary. The standby can take over after the old primary releases the lock or exits.

The active ctld owns the node operations that should not be delegated to the guest or inferred from Nomad alone:

• physical capacity reporting and per-lease cgroup v2 setup
• RootFS writer attachment, checkpointing, fencing, and node journals
• default-deny network policy, L4/L7 controls, transparent proxying, and supported credential injection
• identity-bound proof that mounts, processes, network state, and cgroups were removed

The A/B design improves process availability on one node. It is not node-level high availability, and it does not make the worker durable. If the node is lost, the control plane recovers from PostgreSQL and object storage and creates a replacement runtime generation elsewhere.

Persistence Without Preserving a Process#

Sandbox0 persists filesystem state, not a frozen machine.

A planned pause checkpoints the writable branch, atomically publishes the new RootFS head, and releases compute. Resume creates a new writer epoch and runtime generation from that head.

A named RootFS snapshot uses the same generation boundary. A snapshot can now be created from a running sandbox: Sandbox0 briefly barriers the active writer, publishes a committed checkpoint, and lets the source continue. The snapshot does not contain process memory, open sockets, PID state, or live sessions.

Fork, restore, and rebase also operate on committed RootFS state. This is useful for initializing one agent workspace and branching many attempts, but it is not live-process migration.

An unplanned worker crash has a different meaning from a successful pause. The system can recover the last durable generation, while an uncommitted dirty tail is abandoned. Calling both outcomes a "snapshot" would hide an important data boundary.

Scaling the Disposable Layer#

The architecture supports a fixed worker pool plus provider-managed elastic workers. The current production provider integration is Aliyun ESS; it is not a claim that every cloud provider is already implemented.

Scale-out is more than creating a VM. A node becomes eligible only after its identity, runtime compatibility, ctld channel, physical capacity, carriers, and provider lifecycle admission are ready. Scale-in must drain claims and complete terminal cleanup before provider release.

Keeping provider lifecycle outside the carrier compatibility model lets the sandbox runtime remain stable while infrastructure integrations evolve. Fixed and elastic nodes present the same admitted capacity boundary to manager.

What Changed for Sandbox0 Users#

The public sandbox model remains intentionally familiar:

• claim a sandbox from a template
• run commands, files, services, SSH, and supervised sessions
• pause and resume while preserving committed files
• snapshot, restore, fork, or rebase filesystem state
• control network destinations and scoped egress credentials

The larger change is operational. Self-hosted installations now run the Sandbox0 data plane on dedicated Nomad clients with host-level ctld A/B, stock gVisor runsc, PostgreSQL, and an S3-compatible RootFS store. The retired Kubernetes Operator, SandboxTemplate CRD, ReplicaSet warm pool, SandboxVolume, S0FS, storage proxy, and containerd sandbox runtime are not part of this data plane.

That cutover removed code as well as adding it. A new architecture is only an improvement if its old authority paths disappear; leaving two lifecycle or storage truths would make failure handling harder, not safer.

The Larger Lesson#

Orchestrating an agent sandbox is different from keeping a container running.

Agents accumulate files, caches, tools, credentials, browser state, and recovery points over time. Their compute should be replaceable without making their durable identity ambiguous. At the same time, the platform must be honest about what it did not preserve: a committed filesystem is not a live process image, and scheduler state is not physical cleanup proof.

The new Sandbox0 architecture makes those distinctions explicit:

PostgreSQL owns the transaction, object storage owns durable RootFS blocks, Nomad places disposable carriers, ctld owns node truth, and gVisor isolates the guest.

That is the foundation we want for long-running agent infrastructure: durable where state must survive, disposable where compute should be replaceable, and fail-closed at every boundary where the two meet.

Explore the Sandbox0 architecture on GitHub, read the self-hosting architecture, or see how RootFS snapshots, restore, and fork use the same durable generation model.