01Documentation
Self-hosted Overview
Sandbox0 self-hosting uses dedicated Nomad client nodes, stock gVisor runsc,
direct node services, regional PostgreSQL, and S3-compatible RootFS storage.
Architecture#
Service Boundaries#
| Scope | Components | Responsibility |
|---|---|---|
| Global, optional | global-gateway | Team-to-region directory and global identity entrypoint. It does not understand cluster topology. |
| Region control | regional-gateway, optional scheduler, ssh-gateway | External API, identity, internal routing, and multi-cluster selection. |
| Data-plane control | cluster-gateway, active-active manager | Sandbox/template APIs, lifecycle truth, RootFS metadata, node capacity, and claim transactions. |
| Dedicated Nomad node | Nomad client, ctld-a, ctld-b, nomad-driver-sandbox0, stock runsc | Resource leases, cgroups, RootFS mounts, network policy, runtime creation, and terminal proof. |
| Sandbox | procd inside gVisor | Commands, contexts, files, services, and events. |
| Regional storage | PostgreSQL, S3-compatible object storage, optional ClickHouse | Transactional truth, encrypted RootFS blocks, and asynchronous metering/observability read models. |
A region may have several data-plane clusters. They share that region's
PostgreSQL and S3 authorities. scheduler selects a cluster from live
PostgreSQL capacity for new claims; existing sandbox requests route to the
sandbox's recorded cluster.
For a single-cluster installation, cluster-gateway can run in full mode and
act as the external regional entrypoint. scheduler is optional in that
topology.
Unified Resource Pool#
Nomad schedules dedicated Sandbox0 nodes and resource-neutral warm carrier allocations. Carrier CPU and memory reserve only allocation overhead. Manager and PostgreSQL atomically lease exact sandbox CPU and memory from ctld-advertised node capacity.
Compatibility classes contain only immutable runtime properties. CPU quota, CPU weight, cpuset, memory, and PIDs are dynamic claim properties and do not split the pool. Production nodes must be dedicated to Sandbox0 so Nomad cannot allocate the same physical capacity to unrelated jobs.
Stateless Node Pools#
Dedicated workers are disposable compute, not storage authorities. PostgreSQL owns resource leases, enrollment, fences, and lifecycle state; S3-compatible storage owns RootFS and volume blocks. Node-local disks contain reconstructible cache, runtime journals, mounts, and short-lived credentials only.
A production deployment can combine an always-running fixed baseline with a provider autoscaling pool whose minimum is zero. Manager derives desired capacity from live fixed carriers, active leases, and short-lived failed-claim demand. New nodes remain claim-fenced until identity, ctld capacity, all warm carriers, and the provider scale-out lifecycle action are complete. Scale-in removes a node only after leases, allocations, routes, and runtime state are absent.
RootFS Boundary#
Manager owns regional RootFS metadata and the durable OCI importer. ctld owns
node-local block attachment and the exact active writer. Writable generations,
snapshots, forks, template captures, and rebase operations are block-COW state
stored in S3-compatible storage.
A terminal resource lease is released only after a plugin-independent reconciler proves the old writer, runsc process, mount, network state, and cgroup absent. Driver disappearance alone is never terminal proof.
Metering Boundary#
Open-source Sandbox0 owns usage truth. PostgreSQL's metering schema stores
projection state and transactional projection_outbox. ClickHouse is an
asynchronous query projection for history, quota aggregates, and export. The
normal direction is PostgreSQL to ClickHouse; the two stores are not separate
usage authorities.
Pricing, invoices, payments, and billing remain outside this repository.
Deployment Assets#
deploy/nomad/control: direct systemd boundary for gateways, manager, scheduler, and SSH gateway.deploy/nomad/ctld: node installer, ctld A/B units, cgroup setup, NBD configuration, and Nomad plugin configuration.nomad-driver-sandbox0/example: runtime-class catalog, trusted Nomad endpoint catalog, and warm-carrier job.