# Install This guide describes the production Nomad boundary. It intentionally does not create a local orchestration cluster or install a development-only alternate runtime. ## Prerequisites Prepare: - a Nomad server cluster and Linux Nomad clients dedicated to Sandbox0; - cgroup v2 with `cpu`, `cpuset`, `memory`, and `pids` controllers; - stock `runsc`, a compatible Linux kernel, OverlayFS, XFS, NBD, TPROXY, ipset, and required netfilter modules on every sandbox node; - a regional PostgreSQL writer endpoint; - an access-controlled S3-compatible bucket with conditional-create behavior; - private DNS and mutually authenticated TLS between manager, ctld, and Nomad; - an OCI registry reachable by manager's durable RootFS importer; and - optional ClickHouse for metering and per-sandbox historical query projections. The Nomad clients must use the `sandbox0` node pool and metadata `sandbox0_dedicated=true`. Do not schedule general workloads on these nodes. ## 1. Build And Pin Artifacts Build the control and node binaries from one commit: ```bash make build-local-all cd nomad-driver-sandbox0 go build -o ../bin/nomad-driver-sandbox0 . cd .. ``` Acquire stock `runsc` from a pinned gVisor release. Record the exact Sandbox0 commit, driver version, `runsc --version`, platform, DirectFS/file-access mode, and RootFS format. Do not mix versions within one runtime compatibility class. `procd` is imported into RootFS artifacts by manager. Configure its exact path, protocol, and SHA-256 digest in manager; do not fetch a mutable procd image on the claim path. ## 2. Provision Regional Stores Create a database and configure all regional/data-plane services with the same PostgreSQL writer endpoint. Manager applies its schemas and migrations at startup. The endpoint must fail over to a read-write server; manager readiness fails closed when a pooled connection becomes read-only. Create a private RootFS bucket. Use installation-specific credentials and application-layer object encryption. Keep encryption keys identical across manager replicas and ctld A/B, but outside service YAML where possible. ClickHouse is optional for initial bring-up. PostgreSQL remains lifecycle and metering producer truth even when ClickHouse is enabled. ## 3. Install Control Services Use `deploy/nomad/control/sandbox0-control@.service` or equivalent Nomad service jobs for: - `regional-gateway`; - `scheduler` only for multi-cluster routing; - `cluster-gateway`; - at least two `manager` replicas; and - `ssh-gateway` when SSH access is enabled. ```bash sudo install -o root -g root -m 0644 \ deploy/nomad/control/sandbox0-control@.service \ /etc/systemd/system/ sudo install -o root -g root -m 0644 \ deploy/nomad/control/sandbox0-control.target \ /etc/systemd/system/ sudo install -d -o root -g sandbox0 -m 0750 \ /etc/sandbox0 /var/lib/sandbox0 ``` Create `/etc/sandbox0/.yaml` and optional mode-0600 `/etc/sandbox0/.env` for each enabled service. The files support environment expansion. The checked-in `deploy/nomad/control/manager.yaml.example` is a boundary example with placeholders, not a production-ready config. For self-hosted identity, configure `regional-gateway` with `edition: self-hosted`, `auth_mode: self_hosted`, signing keys, and either built-in auth or OIDC. `built_in_auth.init_user` creates the first admin and team only when the identity database is empty. Store the bootstrap password in a protected environment/file workflow and rotate it after first login. Start manager before admitting sandbox nodes: ```bash sudo systemctl daemon-reload sudo systemctl enable --now sandbox0-control@manager.service ``` Every manager replica must use identical region/cluster IDs, RootFS encryption keys, writer-token key, runtime-class catalog, node CA, node identity mapping, and trusted Nomad endpoint catalog. ## 4. Configure Node Authority Provision these root-owned manager inputs: | Directory | Files | | --- | --- | | `/etc/sandbox0/node-authority/tls` | `tls.crt`, `tls.key`, `client-ca.crt` | | `/etc/sandbox0/node-authority/claim` | `runtime-classes.json`, 32-byte `writer-token.key` | | `/etc/sandbox0/node-authority/control` | `nomad-endpoints.json` and referenced CA, client certificate, key, and ACL-token files | Start from: - `nomad-driver-sandbox0/example/runtime-classes.example.json`; and - `nomad-driver-sandbox0/example/nomad-endpoints.example.json`. Map every node certificate common name to one exact cluster ID, Nomad node ID, durable node UID, and agent UID. The endpoint catalog must pin one HTTPS server endpoint per Nomad cluster and one exact HTTPS client endpoint per node. Redirects and ambient proxy routing are rejected. Enable manager's terminal worker. It is the plugin-independent owner of stop, client GC, physical-absence observation, and final capacity release. ## 5. Install Each Dedicated Node Copy and replace every placeholder in: - `deploy/nomad/ctld/ctld.yaml.example`; - `deploy/nomad/ctld/ctld-networking.yaml.example`; - `deploy/nomad/ctld/ctld.env.example`; and - `deploy/nomad/ctld/nomad-plugin.hcl.example`. Set `SANDBOX0_RESOURCE_*` from the dedicated cpuset after reserving host, Nomad, and ctld overhead. These values are physical sandbox capacity. Do not copy the warm job's small carrier resource values. Then run as root: ```bash sudo ./deploy/nomad/ctld/install-node.sh \ --ctld ./bin/ctld \ --driver ./bin/nomad-driver-sandbox0 \ --runsc ./bin/runsc \ --config ./node/ctld.yaml \ --network-config ./node/ctld-networking.yaml \ --nomad-config ./node/nomad-plugin.hcl \ --env ./node/ctld.env \ --start ``` The installer configures the NBD pool, cgroup subtree, kernel modules, networking sysctls, ctld A/B systemd units, and the Nomad task-driver plugin. Nomad has a hard dependency on a healthy ctld target. Verify that one ctld instance is primary-ready and its peer is synchronized-standby-ready before enabling claims. The A/B services must share the host mount, device, and network namespaces; do not add systemd isolation that creates private mount or device namespaces. ## 6. Submit The Unified Carrier Job Review and submit the warm-slot job: ```bash nomad job run \ -var='datacenter=' \ nomad-driver-sandbox0/example/warm-slot.nomad ``` Keep `restart { attempts = 0 }`. A carrier is one-shot and must be replaced by a fresh allocation and network namespace after consumption. Keep every carrier on `mode = "cni/sandbox0"`. The control-plane-rendered `sandbox0` CNI network uses stock CNI `ptp` and binds the node's allocated CIDR. Its directly routed host veth is required for TCP and UDP policy enforcement; bridge peers are rejected before carrier admission. Nomad's built-in `bridge` mode uses its separate default network and is not interchangeable. Migrating an existing CNI bridge requires a durable node drain and removal of all its carrier ports before replacing the configuration. Confirm manager sees: - live node capacity and current boot IDs; - connected authenticated node channels; - ready resource-neutral slots with the expected compatibility digest; - default-deny warm networking; and - enough NBD devices and replacement carriers. Authenticated node streams rotate periodically to refresh credentials and capacity. In-flight requests allow up to five seconds for the same node boot to reconnect, within the caller's deadline, and retain their exact request identity. Longer outages still fail closed; this grace does not relax node authentication or permit live operations on another boot. ## 7. Start The API Path Start cluster-gateway, regional-gateway, optional scheduler, and optional ssh-gateway after manager and node readiness succeed. In single-cluster mode, cluster-gateway can provide the external full-mode endpoint. In multi-cluster mode, expose regional-gateway and keep cluster endpoints private. Install `s0`, point it at the external endpoint, and log in: ```bash export SANDBOX0_BASE_URL=https://sandbox0.example.com s0 auth login ``` Self-hosted installs do not contain a hidden builtin-template seeder. Publish an image that manager can pull, resolve its manifest digest, and create the initial team-owned template explicitly: ```yaml # default-template.yaml spec: mainContainer: image: registry.example.com/sandbox0/default@sha256:<64-hex-digest> resources: memory: 4Gi ephemeralStorage: 8Gi network: mode: block-all ``` ```bash s0 template create --id default --spec-file default-template.yaml s0 template list ``` Watch manager until template import discovery has ensured the operation and the durable importer logs a `ready` result. Then validate the public path: ```bash s0 sandbox create --template default ``` An earlier claim must fail as data-plane-not-ready; it must never fall back to pulling a mutable image in the Nomad allocation. ## 8. Acceptance Before production traffic: 1. run the stock-runsc cgroup integration test documented in `deploy/nomad/ctld/README.md`; 2. run the DirectFS/Gofer/runc semantic and isolation corpus as a compatibility test, while keeping stock runsc DirectFS as the production lane; 3. exercise RootFS attach, writes, pause, resume, snapshot, running fork, rebase, and cross-node restore against real S3-compatible storage; 4. inject task-driver, ctld primary, manager, Nomad server/client, PostgreSQL writer, node reboot, and object-store failures; and 5. run the public regional-gateway-to-procd SLO tool. Production acceptance requires at least eight carriers, eight truthful dedicated physical CPU cores, sufficient non-oversubscribed memory, serial 1000 and synchronized concurrency, multi-node fault injection, and security gates. A smaller host validates only its actual width. ## Next Steps Review immutable identities, capacity, storage, and rollout rules.