# Configuration Sandbox0 configuration is split across direct service YAML, root-owned credential/identity files, Nomad client configuration, and the warm-carrier job. Keep these inputs under versioned infrastructure management and pin them to one tested release. ## Configure In This Order 1. region and cluster identity; 2. PostgreSQL writer and RootFS object storage; 3. manager node authority and active-active terminal reconciliation; 4. dedicated node identity and truthful capacity; 5. immutable runtime-class catalog; 6. resource-neutral carrier count; 7. gateway identity, routing, registry, network, quota, and telemetry; and 8. rollout and acceptance policy. ## Region And Cluster Identity A provider/region has one independent PostgreSQL and S3 boundary. Several data-plane clusters may share it. Use stable values for: - `region_id` on regional and data-plane services; - `cluster_id`/`default_cluster_id` for each data plane; - Nomad server cluster identity; - exact Nomad client node IDs; - a durable node UID that survives reboot; and - a new boot ID for every host boot. `global-gateway` routes teams to regions only. It must not contain a region's Nomad or data-plane cluster topology. For one data-plane cluster, run cluster-gateway in full mode and omit scheduler. For several clusters, regional-gateway calls scheduler, and scheduler selects a cluster from live PostgreSQL capacity for new claims. ## PostgreSQL All replicas in one region use the same PostgreSQL writer endpoint. Manager, scheduler, and gateways must not point at independently maintained lifecycle stores. Manager validates that new and pooled connections are read-write. During a failover, readiness and new capacity-changing work fail closed until the writer endpoint resolves to the new primary. PostgreSQL is authoritative for: - sandbox/template lifecycle and runtime generations; - node capacity, carrier slots, and exact resource leases; - RootFS heads, writer epochs, snapshots, forks, imports, and deletion queues; - network and credential policy state; and - metering projection state and transactional outbox. ## RootFS Object Storage Configure `rootfs_object_storage` on every manager and ctld with one regional bucket and endpoint. Use conditional object creation and keep credentials private. ```yaml rootfs_object_storage: type: s3 bucket: sandbox0-rootfs region: us-east-1 endpoint: https://s3.internal.example.com access_key: ${SANDBOX0_S3_ACCESS_KEY} secret_key: ${SANDBOX0_S3_SECRET_KEY} object_encryption_enabled: true object_encryption_key_path: /etc/sandbox0/rootfs/object-key ``` The object-encryption key is installation state. Losing it loses access to encrypted RootFS objects; changing it without a migration makes existing objects unreadable. RootFS caches are demand-filled after the artifact is selected; warm carriers do not preload tenant RootFS data. Before exposing its runtime, a ctld primary resolves initial S3-compatible provider credentials into the SDK's existing credential cache. This does not read tenant objects or require OSS bucket lifecycle permissions. Initial credential failures prevent runtime readiness; later expiry/refresh still follows the provider and can affect subsequent reads. This initialization is not proof of object permissions or a latency guarantee. The session manager shares a 128 MiB verified range LRU, including decoded mapping pages and their accounted memory. Parent bindings are still checked on mapping-cache hits. A cold encrypted-object header load normally combines its fixed prefix and JSON header in one bounded 1 KiB range GET. For bounded offset-zero demands, including mapping pages, the first GET can also include demanded ciphertext, up to 256 KiB total. That ciphertext belongs only to the initiating read, not the shared header cache; other waiters share only parsed crypto state. For cold nonzero offsets, the store may overlap the header load with a bounded ciphertext probe using current writer geometry. There are at most eight private probes of up to 256 KiB per store, including completed probes waiting for a header. At most 1 KiB of incidental bytes can precede the demanded frame; the intervening pack prefix is never downloaded. The stored header must validate the hinted layout before use. Old frame sizes, large headers or failed probes fall back to the ordinary exact demanded range; probe saturation never queues extra speculative work. Cancellation is per caller, and slow private probes do not delay shared-header publication. Larger valid headers require one exact remainder GET, up to the existing 1 MiB header limit. Stored frame geometry determines any continuation range, including old objects whose frames differ from current writer defaults. Demanded frames still authenticate separately, and plaintext ranges still require checksum validation. This reduces request round trips without changing persisted objects or requiring tenant RootFS prewarming. Each encrypted RootFS store also bounds cached envelope headers and crypto state to 1,024 entries / 8 MiB. If GC recreates an immutable plaintext key with a new encryption envelope, a reader can refresh the header once before any frame authenticates; following an authentication failure it retries data only when the header changed. These caches do not change persisted formats or replace PostgreSQL/S3 authority. Cache budgets do not include active read buffers, live reader state, kernel caches, or total process RSS. New encrypted RootFS objects use independently authenticated 16 KiB frames. Existing objects keep the frame size recorded in their header, including older 1 MiB and 64 KiB frames; changing the writer does not rewrite existing immutable keys. Block-map checksums and encryption frames are separate boundaries: reducing only one does not remove overfetch imposed by the other. Keys, algorithms and stored format versions are unchanged. For mapping entries smaller than 128 KiB, a bulk read of at least 128 KiB can combine logically and physically adjacent entries from the same verified leaf and object into a bounded 1 MiB window. Every entry is checksum-verified before use or cache admission. Small reads retain the selected entry's granularity. With less than 2 MiB of read cache, coalescing stays within the actual demand; otherwise a bulk demand can fill the remainder of its 1 MiB window. It does not fetch extra mapping pages or cross holes, object boundaries, or coarse entries. Verified cache hits at a window's edges are excluded from its transport range. Interior hits stay inside the single contiguous request, avoiding extra serial GETs; their verified cached bytes are reused when assembling the result. The whole-window flight identity and source-buffer/concurrency limits are unchanged. This is claim-time demand-driven reading, not tenant RootFS prewarming. Set dirty-tail and materializer limits from measured storage latency and node disk headroom. The regional composite-tail backlog is bounded: an S3 outage must apply backpressure instead of growing unbounded local or PostgreSQL data. Do not release a writer or delete its local branch before immutable publication or an explicit crash-abandon terminal transaction. The shared retirement reserve remains charged to its writer group across WAL close and reopen, until every charged journal is reclaimed. Reopening a journal alone does not authorize retirement writes or enlarge ordinary write admission. ## Durable OCI Importer Manager's RootFS importer is required for the Nomad runtime. Configure: ```yaml rootfs_importer: work_root: /var/lib/sandbox0/rootfs-importer procd_path: /usr/local/bin/procd procd_protocol: procd_digest: sha256: discovery_interval: 5s discovery_page_size: 100 format_generation: 2 # Require the default 64 KiB decoded geometry for new image-based claims. data_range_bytes: 65536 data_layout_policy: "" mapping_group_policy: "" ``` The work root must have enough space for bounded OCI extraction and block-image construction. Use `registry.pull_credentials_file` for private source images. All manager replicas run bounded template discovery and may run the importer. The discovery cursor is expendable; deterministic PostgreSQL import operations are the durable authority. PostgreSQL leases and conditional object publication provide active-active ownership and retry safety. The importer identity includes the image digest, OCI platform, block format, exact RootFS size, and pinned `procd` protocol and digest. Before changing `rootfs_importer.procd_path` and `procd_digest`, upgrade every manager to executable-filtered import leasing. Older workers can otherwise abandon a new version's pending import. Prepare matching ready artifacts for all image templates before switching claim selection; installing host binaries alone neither updates importer configuration nor replaces embedded `/procd`. Snapshots and captured templates keep the executable bound by their immutable artifact, provided its procd protocol remains compatible. Updating the importer digest does not rewrite existing sandbox generations or snapshots. `rootfs_importer.format_generation` selects the durable format for new image imports and image-based claims. Omitted or `0` selects format `1`; `2` opts into independently compressed, directly addressed data and mapping ranges. Other values are rejected. Discovery, durable operation reconstruction, and claim selection use the same format binding; there is no fallback to a different format when the requested artifact is not yet ready. `rootfs_importer.data_range_bytes` controls the checksummed data ranges of new imported base artifacts. An omitted value or `0` uses the selected format's build default and does not filter compatible artifacts by their import geometry. | Import format | Default data range | Maximum data range | | --- | --- | --- | | `1` | 8 MiB | 8 MiB | | `2` | 64 KiB decoded | 64 KiB decoded | An explicit value must be a positive multiple of 4 KiB that evenly divides the default 64 MiB pack size. `1048576` selects 1 MiB ranges for format `1`, but is rejected for format `2`. These decoded/checksummed ranges are distinct from the 16 KiB encryption frames of newly written objects. Smaller ranges reduce overfetch for sparse demand, but can increase object GET counts. Measure both bytes and first-command latency against the actual storage backend before enabling this policy. Explicit geometry is shared by template discovery and ordinary image-based claims. A ready artifact with different or unknown import geometry cannot satisfy that policy. Discovery uses the existing durable importer to ensure a matching artifact, and new claims remain unavailable until one is selectable. For a legacy artifact, retained ready-import records can supply missing geometry only when their complete source and build-contract provenance agrees. Conflicting or unverifiable history stays unknown and is reported as unavailable, not guessed from the old default or accepted merely because an import says `ready`. Source selection also matches the exact digest-pinned image reference, not just its digest. Import geometry provenance stays with immutable artifact metadata after completed import operations are garbage-collected. Changing `data_range_bytes` alone does not rewrite existing objects, change the block or encryption format, or add a runtime compatibility class. Existing generations, captured templates, snapshots, forks, and resume continue to use their exact artifact identity; the new-import selector is not applied to those lookups. It does not change incremental checkpoint or materializer geometry. Quiesce old import and import-GC workers before applying the geometry migration or recovering legacy provenance: old workers cannot preserve conflicting evidence under the new rules. Upgrade all participating managers, then enable the same value on every manager in the region. Already-collected history cannot be recovered. Do not prewarm tenant RootFS contents or increase request timeouts to make an acceptance test pass. `rootfs_importer.data_layout_policy` is an optional import optimization, not a runtime compatibility class. Empty retains legacy global-grid segmentation and selects legacy-policy artifacts for new image claims. `xfs-file-ranges-v1` requires format `2` and 64 KiB data ranges (explicit or the format-2 default). It aligns complete data ranges relative to files after read-only XFS extent discovery; it does not change filesystem placement, read tenant data at claim in advance, or cache a tenant RootFS in the generic warm carrier. The requested policy participates in the immutable import operation identity. Version-2 artifact attestations additionally bind policy, range size and any whole-image fallback. Existing legacy operation/attestation bytes do not change. Discovery and new image claims select the exact policy; snapshots, captured templates, forks and resume remain bound to their committed artifact digest. Upgrade all managers and apply the layout migration before enabling the same policy across the region. The database rejects a pre-policy worker attempting to complete a new-policy operation with a legacy artifact. Downgrading the migration is refused while policy-bearing rows exist; do not erase provenance to force a downgrade. Discovery is bounded to 10,000,000 entries, 1,048,576 extents, hardlink identities and preferred spans, depth 256, and 4096-byte paths. Exhausting an optimization budget discards the entire preferred plan and publishes the original global segmentation; the attestation and successful import log record `scan-budget-exceeded`. This preserves ordinary OCI input support and is not an optimization success. Unsafe or malformed metadata, filesystem boundary changes, I/O failures and cancellation still fail closed. A successful import or absence of fallback is not a cold-start guarantee: verify full-image cost, fresh-node claim plus the first real command, and actual density/reclaim before rollout. `rootfs_importer.mapping_group_policy` defaults to empty and applies only to new image imports. The opt-in `contiguous-mapping-v1` requires format `2` and packs adjacent mapping pages into bounded immutable objects (up to four pages, 240 KiB stored and 1 MiB decoded per group). It can coexist with `xfs-file-ranges-v1`; it does not alter data-range geometry or carrier identity. Readers can fetch contiguous sibling mappings together, verify each page, and parse entries only on demand using the existing bounded shared cache and source admission. This reduces some request counts but can amplify bytes for partial reads; it is not a cold-start latency guarantee. The policy is part of the durable import identity and version-3 Ready artifact attestation. Empty-policy operation and version-1/2 attestation identities remain unchanged. Upgrade all managers and apply the mapping migration before enabling the same policy region-wide. PostgreSQL prevents old workers from completing a grouped operation with an ungrouped artifact. Rollback refuses policy-bearing rows rather than discarding their provenance. New image selection matches the exact policy; committed snapshots, captures, forks and resume retain their exact artifact identity. Incremental and batch publishers do not accept this import setting. Validate fresh-node claim plus the first real command and occupied production width before rollout; neither tenant prewarming nor a larger timeout is part of this policy. After the staging XFS image is unmounted, the importer reads it through an exclusively owned read-only descriptor. On Linux, filesystem-reported holes can be skipped without changing the data-unit boundaries, immutable objects or descriptor. Unsupported hole discovery falls back to a complete scan; actual I/O failures or detected file changes still fail the import. No extra setting, format migration, tenant prewarming or timeout increase is involved. A sparse logical image is not equivalent to the same size of populated data, and faster offline import does not prove faster sandbox claim or first-command execution. Before selecting format `2`, upgrade every participating manager, ctld instance, and task driver to readers that support it. The upgraded runtime still uses the same generic carrier catalog; do not split pools by image or tenant. Do not enable format `2` while old readers can receive claims. Apply the same import policy to all managers in the region. Existing snapshot claims and captured templates use their attested generation's format, and resume retains its committed generation, even if the new-image policy changes in either direction. Reverting that policy does not permit rolling readers back to binaries that cannot read already published format-two generations. No automatic migration or object rewrite is performed. Format-two opt-in is not a production cold-start acceptance result; validate empty-node-cache and cached-new-sandbox claims, first real commands, mixed images and sustained node occupancy before rollout. ## Manager Node Authority Use the manager example in `deploy/nomad/control/manager.yaml.example` as the field reference. Production requires `node_authority.enabled: true` and the plugin-independent terminal worker enabled. Every manager replica needs identical: - server certificate/key and node client CA; - certificate-CN to cluster/node/node-UID/agent-UID mappings; - 32-byte writer-token key; - runtime-class catalog; - trusted Nomad endpoint catalog and referenced credentials; and - claim/heartbeat/writer lease policies. Node channels use mTLS and exact identities. Bind the authority endpoint to a private address and configure exact DNS/SPIFFE SANs. Do not permit redirects, ambient HTTP proxies, wildcard node identity, or an unpinned Nomad client endpoint. The terminal worker must remain independent of the task-driver process. It owns exact Nomad stop and client GC, node cleanup requests, physical absence observation, RootFS fencing, and the final PostgreSQL capacity release. Nomad allocation observations use exact-ID, indexed summaries without embedding the carrier job. Incomplete, unauthorized, or mismatched catalog responses do not establish absence; physical cleanup still requires the exact node proof. Direct-client allocation observations have a five-second upper bound, even when the endpoint permits longer cleanup requests. If an observation fails, the worker defers other slots on that exact node incarnation until its next pass. Their leases and physical cleanup requirements remain intact. This prevents a retired node's carrier inventory from repeatedly consuming the same pass deadline while healthy nodes wait for capacity to be reclaimed. The worker resumes each bounded batch after the previous candidate and wraps for retries at the end of the traversal. This in-memory cursor prevents an unresolved full batch from permanently hiding later cleanup work; it does not change the required authority checks or physical absence proofs. Compact external-crash proofs are replay history for exact writer bindings already verified terminal regionally and reclaimed on the node. Local expiry requires the matching durable binding and physical proof, `BranchRemoved`, absent branch/mount paths, no live owner, and released device reservations where applicable. It deletes history only, never physical artifacts, and supplies no new terminal release authority. Missing records, expired regional proofs, and HTTP 403 responses are not terminal authority; reclamation outside this history-only path still requires positive regional verification. The compact proof stays quiet until 48 hours after the latest of its crash `RequestedAt`, proof `ObservedAt`, and record `UpdatedAt`, not just the crash request. After that deadline, eligible history can be forgotten without a fresh regional grant/proof lookup, including after a node is offline beyond PostgreSQL's 72-hour terminal-proof retention window. ## Runtime Classes The catalog file uses outer version `3`; each compatibility record uses version `2`. Start from `nomad-driver-sandbox0/example/runtime-classes.example.json`. A class contains only immutable execution properties: - artifact OS/architecture and host architecture; - driver and stock runsc versions; - gVisor platform; - overlay, file-access, and DirectFS modes; - `/procd`, its fixed port, and static runtime control mode; and - security class or immutable device requirements. Configure `procd_internal_jwt_public_key_file` with the region data-plane Ed25519 public key. The driver bind-mounts that public key read-only at `/config/internal_jwt_public.key`; it must not be baked into a base image or copied into an immutable RootFS artifact because regional key rotation is independent of image import. Never include CPU quota, CPU weight, cpuset, memory, PIDs, or template memory in the compatibility digest. Those values belong to the exact claim-time resource lease. Until an explicit public class selector exists, configure exactly one class for each requested cluster. Ambiguous or missing class selection fails closed. ## Dedicated Node Capacity Configure the Nomad client with: - `client.node_pool = "sandbox0"`; and - node metadata `sandbox0_dedicated=true`. The `SANDBOX0_RESOURCE_*` values in `ctld.env` describe allocatable sandbox capacity after host, Nomad, ctld, NBD, and network overhead is reserved: ```sh SANDBOX0_RESOURCE_CPU_MILLICORES=8000 SANDBOX0_RESOURCE_MEMORY_BYTES=17179869184 SANDBOX0_RESOURCE_CPUSET_CPUS=0-7 SANDBOX0_RESOURCE_CPUSET_MEMS=0 ``` Use only CPUs truly dedicated to this node pool. Do not advertise SMT threads as independent physical-core acceptance capacity. Carrier-job resources are small overhead reservations and must not be copied into these values. `ctld-resource-cgroup-setup` provisions `/sys/fs/cgroup/sandbox0` and enables `cpu`, `cpuset`, `memory`, and `pids` for per-lease children. Existing active lease children survive ctld A/B restart. A lease returns to capacity only after its cgroup and all other runtime state are physically absent. Lightly loaded pools can opt into admission budgets larger than physical capacity using `nomad_runtime.admission_cpu_millicores` and `nomad_runtime.admission_memory_bytes`. Omitted or zero values preserve physical capacity admission. These budgets bound the sum of individual sandbox limits; they do not change a sandbox's CPU quota, memory limit, metering, or isolation. Include the `150m` CPU floor for every sandbox when sizing this budget. For example, 420 minimum-memory sandboxes require at least 63000 millicores of CPU admission, plus the leases of any other running sandboxes. A larger admission budget does not add physical CPUs or guarantee simultaneous peak CPU usage. Each individual request must still fit the physical node. CPU budgets are bounded to 16 times physical capacity and memory budgets to twice physical capacity; these are validation ceilings, not recommended operating ratios. Before enabling overcommit, provision the parent resource cgroup with `cpu.max` equal to the physical CPU budget over a 100000-microsecond period, `memory.max` equal to the physical memory budget, and `memory.swap.max=0`. ctld rejects an unbounded parent when overcommit is enabled. These exact physical bounds are also accepted without overcommit and can remain in place when a node reboots with reduced admission budgets. With memory overcommit, new cgroups are rejected when current aggregate usage reaches 90% of the physical budget or the requested memory limit exceeds remaining physical headroom. Exact retries of existing cgroups remain available. Guest growth after admission can still reach the aggregate limit, so simultaneous peak usage is not guaranteed by an overcommitted profile. Validate resident workload memory and tail latency before increasing density. Admission budgets cannot change within the same node boot. Drain expanded-budget nodes before rolling back the database migration, and keep autoscaler per-node admission budgets consistent with the enrolled runtime profile. ## NBD And Local Disk The canonical warm job provisions privileged carriers. Additional groups are eligible only on nodes with matching `sandbox0_privileged_carriers` Nomad metadata. Keep the node profile, usable private IPs, and RootFS devices consistent; drain affected carriers before reducing the bound. `nomad_runtime.nbd_devices` is the actual concurrent RootFS attachment bound. The kernel's `nbds_max` alone does not make devices available to ctld. Keep the configured list wider than the largest synchronized claim batch plus replacement/retirement headroom. The host installer derives the NBD module device count from the configured `SANDBOX0_ROOTFS_NBD_DEVICES` inventory, with a floor of 64 devices. It refuses to reload an in-use module to grow the pool; apply a larger pool on a fresh or drained, rebooted node. Each NBD connection admits up to 64 independent read requests, with a separate byte budget of eight maximum-sized responses. This wider small-read window helps expose independent ranges while callers wait on coalesced range loads. Both limits remain held through the complete reply; writes, flushes, trims, zeroing, and disconnects wait for earlier reads and replies. With the default 8 MiB request limit, outstanding read-response buffers remain bounded to 64 MiB per attached device, before object decryption, verified-range caches, and kernel memory. Each generation reader separately limits distinct source range loads to eight; callers coalescing on the same immutable range share one load slot. Reserve host memory for the configured attachment count, not just warm carrier overhead. Clean branch reads retain verified ranges for the current request and stop at journal overrides; this is demand coalescing, not speculative RootFS prefetch. `state_path`, `runtime_slot_journal_path`, and branch state under `/var/lib/sandbox0/ctld` must survive process restarts. Runtime mounts and control sockets under `/run/sandbox0` are boot-scoped. On reboot, only the new authenticated boot for the same durable node UID may reconcile old state, and only after proving the old runtime absent. Periodic RootFS recovery has bounded aggregate concurrency, a smaller allowance for proof housekeeping, per-attempt deadlines, and capped exponential backoff for unchanged sessions. Physical-writer recovery takes priority; authenticated regional cleanup is not held behind this speculative retry budget. Retry scheduling is disposable boot-local/process-local state, rebuilt after restart or an observed recovery transition, not durable ownership or terminal truth. The durable journal and regional authority remain the recovery inputs. An optional node-wide disk tier reuses verified immutable RootFS ranges across sandboxes, memory-cache eviction, and ctld process replacement. Configure both `nomad_runtime.read_cache_directory` and `nomad_runtime.read_disk_cache_bytes`; `nomad_runtime.read_cache_bytes` controls the memory tier and defaults to 128 MiB. Use a dedicated node-private local directory. The disk cache contains decoded plaintext, validates every hit against its content checksum, and never caches a sandbox's private writable branch. Missing or corrupt entries fall back to object storage. Bounded asynchronous fills and eviction keep cache failures off the source-read success path. This cache is disposable and does not replace durable branch state or object storage. The primary ctld metrics endpoint exposes `ctld_rootfs_read_cache_disk_*` hit, miss, fill, error, drop, and occupancy metrics, plus `ctld_rootfs_read_cache_queued_write_bytes`. Compare these with object-storage reads and end-to-end command readiness. A new sandbox on a cached node does not represent an empty-node-cache start. ## Network Runtime Run ctld A/B directly in host mount, network, cgroup, and device namespaces. The primary owns warm default-deny, claim-time policy compilation, TPROXY/ipset state, and the exact network-namespace incarnation. The standby follows durable state and takes over after promotion. Configure cluster DNS and platform-allowed CIDRs narrowly. Sandbox policy is applied before runsc creation, and manager exposes the generation only after the applied policy digest is part of the launch proof. Cloud metadata destinations are always denied for sandbox traffic: Aliyun `100.100.100.200`, IPv4 link-local `169.254.0.0/16`, and the IPv6 metadata endpoint `fd00:ec2::254`. Tenant rules, platform allow lists, DNS bypass lists, and SOCKS5 proxy selection cannot override this boundary. The IPv4 kernel drop precedes configured bypasses and applies only to registered sandbox source IPs, preserving host-side enrollment and credential renewal. Never enable a second task-driver-local network-policy path. The driver consumes ctld's durable policy proof; it does not install namespace-local fallback rules. ## Warm Carrier Job `nomad-driver-sandbox0/example/warm-slot.nomad` is the resource-neutral `system` carrier job. The current job has eight task groups, so each admitted dedicated node receives exactly eight one-shot carriers. Change the reviewed group inventory—not a per-template pool or task resource profile—only when changing node carrier width. Keep: ```hcl restart { attempts = 0 mode = "fail" } ``` Each task group must also keep `network { mode = "cni/sandbox0" }`. That name selects the rendered per-node stock CNI `ptp` configuration and its leased allocation CIDR. A directly routed host veth is required for UDP TPROXY; ctld rejects bridged carrier peers. Nomad `bridge` mode selects a different built-in network and must not be used for these carriers. Drain an existing node and wait for every old bridge port to disappear before migrating its CNI configuration; existing namespaces retain the topology they were created with. Each carrier is one-shot. Nomad replaces the allocation after consumption. Pool width, node capacity, and NBD count are independent constraints and should be monitored together. Autoscaler workload demand counts resource leases for claiming, starting, and active slots. Quiescing and orphaned slots retain their resource leases until physical cleanup is proven, but do not create replacement workload demand. Cleanup backlog therefore cannot by itself purchase additional worker capacity. Unsatisfied claims and configured headroom still contribute to scale-out demand; scale-in still requires the normal stabilization and node drain protocol. Before a scale-out write, manager reads the PostgreSQL capacity snapshot and provider desired capacity again. If the target, fixed-node availability, workload, unsatisfied demand, node readiness, or provider capacity changed, it defers the purchase to the next bounded pass. A pressureless fixed-node replacement is additionally debounced for two reconcile intervals —20 seconds with the default interval—so a normal adaptive carrier refill cannot be mistaken for the fixed node disappearing. Aliyun route cleanup deletes only the verified immutable route entry ID and waits for route absence before releasing node identity or the allocation subnet. An accepted asynchronous delete request alone does not complete scale-in. Autoscaled clients must start with `sandbox0_admitted=false`. After exact node identity and a live ctld capacity heartbeat are established, bootstrap may set it to `true` so the system job creates carriers. PostgreSQL must retain a separate `warming` claim fence until the provider confirms scale-out; Nomad metadata alone is not provider admission. PostgreSQL lifecycle actions are also the recovery source after a provider lifecycle token disappears. Every nonterminal action carries a bounded recovery deadline, a fenced recovery owner and epoch, and a terminal convergence receipt. When a successful provider enumeration no longer observes the token and every child instance is absent from the exact scaling-group inventory, manager waits a bounded observation grace period, retires only expired unclaimed warm carriers with a provider-absence proof, rejects nodes that still own leases or non-warm Nomad allocations, removes the route, purges the Nomad identity, revokes the node, and closes the durable action. The provider queue can accelerate this path, but it can no longer be the only way to resume cleanup. ## Gateway Identity And Routing For self-hosted regional-gateway, configure: - `edition: self-hosted` and `auth_mode: self_hosted`; - asymmetric JWT signing keys in files; - built-in auth and/or OIDC; - one stable `region_id` and external `base_url`; - the default cluster-gateway URL for single-cluster mode; or - scheduler URL and license inputs for multi-cluster mode. Keep internal service endpoints private and use internal signed tokens for regional-to-cluster and cluster-to-manager calls. Team-owned runtime access routes directly through the regional endpoint; global-gateway is not in that path. ## Registry Credentials Registry providers are configured directly on regional-gateway/manager. Prefer root-owned file fields over inline secret values. Manager's durable importer uses a Docker credential file for private pulls; runtime allocations receive no registry credentials. Configure different `push_registry`, `pull_registry`, and `internal_registry` endpoints when public CI and private regional traffic require different routes. Use the pull image reference returned by the credentials API. ## Quotas And Metering Manager's `default_team_quotas` declaratively reconciles regional defaults. Team-specific PostgreSQL policy overrides the same dimension. The claim transaction checks concurrency and resource dimensions before leasing capacity. Metering records the actual resource lease rather than deriving CPU/memory from a static carrier or compatibility profile. Enable the PostgreSQL producer even when ClickHouse is temporarily unavailable; ClickHouse consumers replay the transactional outbox. See [Quotas](/docs/self-hosted/quotas) and [Observability](/docs/self-hosted/observability). ## Rollouts Use `deploy/nomad/ctld/rollout-node.sh` to restart B then A and wait for each instance to become primary-ready or synchronized-standby-ready before touching its peer. Drain and roll nodes one at a time when changing runsc, driver, compatibility digest, RootFS format, NBD pool, or network policy format. Never downgrade across an on-disk journal version unsupported by the previous binary. Keep the prior tested binary/config set for rollback and preserve writer-token and encryption keys across ordinary rollouts. ## Next Steps Monitor claims, leases, node channels, RootFS, and metering projections.