Configuration

This page explains how to think about Sandbox0Infra configuration and includes a generated field reference sourced from the operator CRD.

The goal is to keep the page readable for operators while still exposing the full supported config surface.

How to Read Sandbox0Infra#

A Sandbox0Infra spec is easier to reason about when you split it into six layers:

Recommended Reading Order#

1. Pick your topology with spec.services.*.
2. Decide whether database/storage/Redis/registry stay builtin or move external.
3. Configure public exposure and cluster identity.
4. Seed templates and initial admin user.
5. Only then tune service and data-plane runtime configuration.

spec.initUser is consumed by the gateway runtime, not created by the operator itself. In local-password mode it bootstraps the first admin credentials; in OIDC-only mode it pre-creates the admin user and initial team so the first OIDC login with the same email lands on the intended admin account.

When spec.builtinTemplates is omitted, infra-operator seeds public builtin templates for default, openclaw, hermes, browser, and coding-agent. Set spec.builtinTemplates explicitly when you want to pin images, adjust pools, disable one of the builtins, or provide a full template spec for a runtime-specific setup. Set it to [] to remove all operator-managed builtin templates.

Pin Builtin Images for Predictable Startup#

The self-hosted openclaw, hermes, and browser presets use upstream mutable latest tags by default. Sandbox0 sets the effective main-container pull policy to IfNotPresent unless a system template overrides it. A claim on a node that already has the image does not download it again, but a new or cache-miss node resolves the tag and pulls the current image. That pull can add substantial cold-start latency for large images, and different nodes can keep different digests for the same mutable tag.

For production self-hosted deployments, replace the complete builtin template list and pin each mutable image to a reviewed digest:

yaml
spec:
    builtinTemplates:
        - templateId: default
        - templateId: openclaw
          image: ghcr.io/openclaw/openclaw@sha256:<openclaw-digest>
        - templateId: hermes
          image: nousresearch/hermes-agent@sha256:<hermes-digest>
        - templateId: browser
          image: ghcr.io/steel-dev/steel-browser@sha256:<browser-digest>

spec.builtinTemplates has replacement semantics, so omitting an entry removes that operator-managed builtin. A version tag is easier to read than a digest but remains mutable if its publisher retags it. A digest is the immutable choice.

Pinning makes the runtime version deterministic; it does not eliminate the first pull on an empty node. If image-pull latency is part of your cold-start budget, preload the pinned digests into node images or use a node-level pre-puller before admitting sandbox workloads.

Sandbox0 Cloud handles the mutable-tag policy at the platform layer: managed-region configuration pins these images by digest and rolls upstream digest changes through deployment automation. Cloud users therefore do not maintain these pins themselves. A genuinely cold node can still need a missing pinned image, but cache and prewarming operations remain the managed platform's responsibility rather than part of the customer template configuration.

Supported Topologies#

The single-cluster sample is always fullmode. manager hosts lifecycle control and the storage API runtime. Each sandbox node runs the ctld-a and ctld-b DaemonSets; the elected primary owns rootfs persistence, volume portals, and the network runtime, and the synchronized standby takes over after promotion. Configure storage behavior under spec.storage.runtime, network behavior under spec.network, and shared sandbox-node placement under spec.sandboxNodePlacement.

Sandbox Claim Admission#

Team Quota is the claim admission boundary. The sandbox_claims rate policy controls sustained claim rate and immediate burst at the external regional or full-mode gateway, while active_sandboxes limits the number of running sandboxes for a team. Manager does not maintain a separate cluster-wide claim-start budget.

After quota admission, manager claims a ready idle pod or creates a cold pod directly. Warm-pool ReplicaSets also scale directly to each template's configured minIdle. In multi-cluster mode, scheduler routing prefers ready idle capacity, then estimated cluster headroom, configured weight, and deterministic fallback order.

Metering Boundary#

Metering is disabled by default. Set spec.metering.enabled: true and configure spec.clickHouse.type as builtin or external to enable the region usage ledger.

Metering export is eventually consistent:

A ClickHouse outage increases the delay without discarding committed outbox operations. After projection-state bootstrap, producers can also restart while ClickHouse is unavailable.

Export and upgrade rules:

• complete_before is the greatest globally ordered producer watermark delivered to ClickHouse; every older committed batch has also been delivered
• new rows expose their PostgreSQL outbox sequence; rows created before that column use sequence: 0
• cursor pagination is the canonical export contract across upgrades
• signed Unix nanosecond timestamps preserve DateTime64(9) boundaries and cursor precision

Monitor manager_metering_outbox_pending_operations and manager_metering_outbox_oldest_pending_age_seconds for delivery backlog.

Sandbox compute uses a serverless-style contract:

Public template idle pools are platform cost. Team-owned template warm pools are attributed to the owning team with subject_type=template.

Object-store request windows are usage facts; Sandbox0 does not attach prices.

• manager and ctld aggregate provider HTTP attempts for one minute before writing them to the PostgreSQL outbox
• volume requests are attributed from sandboxvolumes/<team_id>/<volume_id>
• rootfs requests are attributed from sandbox-rootfs/<team_id>/<sandbox_id>
• window data records provider, bucket, billing_item, operation, cost_scope, and prefix_class
• requests to customer-owned external S3 volumes are excluded

Reconcile these windows with provider access and metering logs before using them for cost reporting.

Platform Observability#

Sandbox0 separates three data paths:

See Self-hosted Observability for collector setup, ClickHouse examples, audit persistence, retention, key management, and monitoring. See Sandbox Observability for the user-facing query and watch APIs.

For cold-start diagnosis, collect manager_pod_lifecycle_stage_duration_seconds. Use kubelet kubelet_runtime_operations_duration_seconds{operation_type=~"run_podsandbox|create_container|start_container"} when you need individual CRI operation latency.

What Usually Changes First#

External PostgreSQL#

Move spec.database.type from builtin to external when you want managed PostgreSQL durability, backup policy, and operational separation.

yaml
spec:
    database:
        type: external
        external:
            host: your-db.rds.amazonaws.com
            port: 5432
            database: sandbox0
            username: sandbox0
            passwordSecret:
                name: db-credentials
                key: password

S3 or OSS Backing Storage#

Configure the object-storage backend and manager storage runtime together under spec.storage.

yaml
spec:
    storage:
        type: s3
        s3:
            bucket: sandbox0-prod
            region: us-east-1
            endpoint: https://s3.amazonaws.com
            credentialsSecret:
                name: aws-credentials
                accessKeyKey: accessKeyId
                secretKeyKey: secretAccessKey
        runtime:
            objectEncryptionEnabled: true

Sandbox RuntimeClass#

Use services.manager.config.sandboxRuntimeClassName when sandbox Pods should run with a non-default runtime such as runc, gvisor, or a custom gvisor-rootfs handler.

yaml
spec:
    services:
        manager:
            config:
                sandboxRuntimeClassName: gvisor-rootfs

This field only controls the runtimeClassName written onto sandbox Pods. The named Kubernetes RuntimeClass must already exist, and its handler must already be configured in the node CRI runtime on every eligible sandbox node.

infra-operator does not install node runtime handlers. It does not install runsc, write /etc/containerd/config.toml, restart containerd, or create a gvisor-rootfs handler. Do that through your node image, node bootstrap automation, or a dedicated privileged node setup process before scheduling sandbox workloads there.

Procd Binary Image Volume#

Sandbox Pods mount Sandbox0 procd binaries from a small OCI artifact image at /procd-image by default. This avoids a per-Pod init container on the sandbox startup path.

When procdBinImageRef is omitted, infra-operator derives it from the deployed infra image tag by appending -procd-bin. For example, sandbox0ai/infra:v0.8.0 uses sandbox0ai/infra:v0.8.0-procd-bin.

Set procdBinImageRef only when you publish the procd binary artifact image separately:

yaml
spec:
    services:
        manager:
            config:
                procdBinImageRef: registry.example.com/sandbox0/procd-bin:v0.8.0

Kubernetes image volume support is part of the Sandbox0 cluster baseline. Unsupported runtimes fail sandbox Pod startup instead of falling back inside the Pod. Sandbox0 mounts the whole image volume and starts /procd-image/usr/local/bin/procd, so clusters do not need container-runtime support for image volume subPath.

Sandbox Memory Limits#

Sandbox claims and runtime updates can set config.resources.memory. The minimum accepted value is 128Mi. The platform maximum defaults to 32Gi. Manager derives CPU from the configured memory-per-CPU ratio and enforces a minimum CPU limit of 150m for claimed sandboxes.

The minimum applies when a sandbox is claimed, resumed, or resized. Upgrading manager does not resize sandbox pods that were already running before the upgrade.

Use services.manager.config.sandboxMaxMemory to lower or raise the platform maximum:

yaml
spec:
    services:
        manager:
            config:
                sandboxMaxMemory: 16Gi

Sandbox0 derives CPU from memory using services.manager.config.teamTemplateMemoryPerCpu, so callers do not set CPU directly on a sandbox claim or update.

For gVisor rootfs checkpoint and restore support on self-managed clusters, use a dedicated containerd handler that points to runsc with shared file access and overlay disabled, then create a matching RuntimeClass:

toml
# /etc/containerd/config.toml
version = 2

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor-rootfs]
    runtime_type = "io.containerd.runsc.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor-rootfs.options]
    TypeUrl = "io.containerd.runsc.v1.options"
    ConfigPath = "/etc/containerd/runsc-rootfs.toml"

toml
# /etc/containerd/runsc-rootfs.toml
[runsc_config]
    overlay2 = "none"
    file-access = "shared"

yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
    name: gvisor-rootfs
handler: gvisor-rootfs

sandboxRuntimeClassName applies only to sandbox Pods. Ctld is a node-local host component and must use a runtime that supports its required hostNetwork, hostPath, FUSE, and network-programming access; do not run ctld under gvisor, gvisor-rootfs, or kata.

ctld assumes the node's containerd data root is /var/lib/containerd by default. If your nodes move containerd data to another host path, configure the same path so rootfs pause/resume can use the overlayfs fast path:

yaml
spec:
    services:
        ctld:
            containerdHostDataRoot: /var/lib/containerd/sandbox0

ctld also keeps a disposable node-local rootfs object cache under its host data directory. This cache accelerates pause/resume when the restoring ctld already has the needed rootfs layer objects. The default cap is 20Gi per ctld node. Set rootfsObjectCacheMaxBytes: "0" to disable it, or tune the GC fields to fit your node disk:

yaml
spec:
    services:
        ctld:
            rootfsObjectCacheMaxBytes: 20Gi
            rootfsObjectCacheMinFreeBytes: 5Gi
            rootfsObjectCacheMaxAge: 24h
            rootfsObjectCacheSweepInterval: 1m

Cache contents are safe to delete. If a cached object is missing, corrupt, too old, or evicted to preserve free space, ctld downloads the rootfs object from configured object storage.

Shared Redis Rate Limiting#

Gateway request rate limiting defaults to process-local memory. Configure spec.redis so api_requests and network bandwidth Team Quotas share admission state across gateway replicas, ctld nodes, and data-plane clusters in the region.

yaml
spec:
    redis:
        type: builtin
        builtin:
            enabled: true

For an external Redis instance, store the URL in a Secret and reference it from spec.redis.

yaml
apiVersion: v1
kind: Secret
metadata:
    name: redis-url
    namespace: sandbox0-system
stringData:
    url: rediss://:[email protected]:6379/0
---
apiVersion: infra.sandbox0.ai/v1alpha1
kind: Sandbox0Infra
metadata:
    name: fullmode
    namespace: sandbox0-system
spec:
    redis:
        type: external
        external:
            urlSecret:
                name: redis-url
                key: url
        keyPrefix: sandbox0:ratelimit:prod
        operationTimeout: 100ms
        failOpen: true

Credential Source Storage#

Credential sources are stored in encrypted PostgreSQL by default. infra-operator creates and mounts a stable per-installation encryption key for manager. During upgrade, legacy plaintext rows remain readable and are not rewritten automatically.

Configure spec.credentialVault when new credential source material should live in a Vault-compatible backend. Builtin mode installs OpenBao:

yaml
spec:
    credentialVault:
        type: builtin
        builtin:
            enabled: true
            mount: secret

External mode connects manager to an existing HashiCorp Vault-compatible endpoint:

yaml
apiVersion: v1
kind: Secret
metadata:
    name: sandbox0-vault-token
    namespace: sandbox0-system
stringData:
    token: <vault-token>
---
apiVersion: infra.sandbox0.ai/v1alpha1
kind: Sandbox0Infra
metadata:
    name: fullmode
    namespace: sandbox0-system
spec:
    credentialVault:
        type: external
        external:
            address: https://vault.example.com
            tokenSecret:
                name: sandbox0-vault-token
                key: token
            mount: secret
            allowedPathPrefixes:
                - sandbox0/credential-sources/{{teamID}}/

When spec.credentialVault is enabled, manager stores newly written credential sources in the configured Vault backend. Existing rows keep their recorded storage backend.

Egress Auth Resolve TTL#

The ctld network runtime caches resolved egress auth material until the expiresAt timestamp returned by manager. Manager does not keep an additional resolved-material cache. Credential source rotation actively invalidates affected ctld cache entries, so already-claimed sandboxes normally use the rotated source on their next matching egress resolve. The default lifetime is 5 minutes, and the TTL remains the fallback consistency bound if realtime invalidation delivery is interrupted.

Set spec.services.manager.config.egressAuthDefaultResolveTtl when you want a different platform default. Individual bindings can override it with network.credentialBindings[*].cachePolicy.ttl.

yaml
spec:
    services:
        manager:
            config:
                egressAuthDefaultResolveTtl: 1m

Public Routing and Region Identity#

spec.publicExposure decides the public host pattern used for sandbox URLs. Keep it consistent with your region and DNS plan.

yaml
spec:
    publicExposure:
            enabled: true
            rootDomain: sandbox0.example.com
            regionId: aws-us-east-1

SSH Gateway#

Enable spec.services.sshGateway when you want a region-scoped SSH entrypoint that bridges standard ssh and scp clients into sandbox procd sessions.

The routing model is intentionally simple:

• One ssh-gateway per region
• One fixed DNS-only host or TCP proxy address per region, for example aws-us-east-1.ssh.sandbox0.app
• SSH username is the target sandbox ID
• User authentication uses SSH public keys uploaded to the gateway API

In Kubernetes terms, spec.services.sshGateway.service.port is the Kubernetes Service port and spec.services.sshGateway.config.sshPort is the container listen port. You usually keep the internal port at 2222 and expose 22 or a NodePort externally.

When an external load balancer listens on a different public port than the Kubernetes Service or NodePort, set spec.services.sshGateway.endpointPort to the public port that clients should use.

The operator also manages a persistent Ed25519 host key Secret for the service. As long as that Secret is retained, clients continue to see the same SSH host identity across pod restarts.

yaml
spec:
    services:
        sshGateway:
            enabled: true
            replicas: 1
            service:
                type: LoadBalancer
                port: 22
            config:
                sshPort: 2222

For local labs or kind-style environments, a NodePort is often simpler:

yaml
spec:
    services:
        sshGateway:
            enabled: true
            replicas: 1
            service:
                type: NodePort
                port: 30222
            endpointPort: 22

After exposing the service through DNS or a load balancer, users can upload SSH public keys with POST /users/me/ssh-keys and connect with standard clients. See SSH for the user-facing flow.

AWS LoadBalancer TLS#

When you expose regionalGateway or globalGateway through a cloud load balancer instead of Kubernetes Ingress, declare provider-specific Service annotations under spec.services.<service>.service.annotations. On AWS, this is the supported place to attach ACM certificates and choose the load balancer mode.

Also set spec.services.<service>.config.baseUrl to the final browser-facing URL. That keeps auth callbacks, generated links, and projected status endpoints aligned with the hostname you put behind Cloudflare.

yaml
spec:
    services:
        regionalGateway:
            enabled: true
            service:
                type: LoadBalancer
                port: 443
                annotations:
                    service.beta.kubernetes.io/aws-load-balancer-type: nlb
                    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
                    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <acm-certificate-arn>
                    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            config:
                baseUrl: https://your-gateway.example.com

Multi-Cluster Data Plane Registration#

Data-plane clusters need control-plane connection details and a stable cluster identity.

yaml
spec:
    controlPlane:
        url: https://api.sandbox0.example.com
        internalAuthPublicKeySecret:
            name: control-plane-public-key
            key: public.key
    cluster:
        id: cluster-001
        name: production-use1-a
    services:
        clusterGateway:
            enabled: true
            config:
                authMode: internal
        manager:
            enabled: true

Service and Runtime Config#

Use spec.services.<service>.config for deployable services, spec.storage.runtime for manager storage behavior, and spec.network.config for the network runtime in ctld.

Examples:

• services.clusterGateway.config.authMode switches between public, internal, and both
• services.manager.config.autoscaler.* controls warm-pool scaling behavior
• services.manager.config.k8sClientQps and k8sClientBurst tune the manager Kubernetes client token bucket rate limit; defaults are 50 and 100
• services.manager.config.sandboxMaxMemory caps per-sandbox memory overrides; the default is 32Gi
• services.manager.config.procdBinImageRef overrides the derived procd artifact image used by sandbox Pods
• services.manager.config.allowColdStartWithoutReadyDataPlane lets cold claims create Pending pods for external node autoscaler scale-from-zero deployments; keep it disabled unless the cluster autoscaler can provision matching sandbox nodes
• services.manager.config.autoscalerSafeToEvictAnnotationKeys lists the platform autoscaler's pod annotation keys; manager writes true on idle pods and false on claimed pods without embedding a vendor-specific key
• services.manager.config.podTeardown.* bounds warm-pool rollout and repair work per node and across the cluster
• storage.runtime.filesystem* tunes the manager storage runtime; services.manager.config.procdConfig.volume* tunes mounted volume cache sizing
• storage.runtime.s0fsSegmentTargetSize sets the target S0FS segment size for materialized volume data; the default is 4Mi
• storage.runtime.s0fsStateFormatVersion selects the format used for new S0FS state writes; readers accept versions 1 and 2, and the default remains 1
• storage.runtime.s0fsHotCacheMaxSize bounds the conservatively estimated memory used by clean node-local S0FS engines retained after an intentional pause; the cache has no age-based expiry, uses probation/protected admission to resist short-lived churn, defaults to 1Gi, and is disabled by setting the value to 0
• storage.runtime.s0fsCompaction* tunes background S0FS compaction for RWO volume owners; s0fsCompactionInterval accepts 0, off, or disabled to turn it off
• storage.runtime.cacheSizeLimit and logSizeLimit cap storage cache and logs in the manager Pod; volumePortalCacheSizeLimit and volumePortalRootMinFree tune the volume portals used by ctld
• storage.runtime.objectEncryptionEnabled controls application-layer encryption for persisted rootfs checkpoint objects, S0FS Volume objects, and node-local S0FS volume cache files; it is enabled by default
• network.config.egressBandwidthBytesPerSecond, ingressBandwidthBytesPerSecond, and bandwidthBurstBytes apply per-sandbox bandwidth throttling when set above zero
• services.manager.config.defaultTeamQuotas configures region-wide capacity, API request, and network bandwidth defaults; see Team Quotas
• network.config.* also controls proxy ports, policy enforcement, and node-level networking behavior

For autoscaled sandbox node pools, configure the annotation contract and teardown budgets together:

yaml
spec:
    services:
        manager:
            config:
                autoscalerSafeToEvictAnnotationKeys:
                    - cluster-autoscaler.kubernetes.io/safe-to-evict
                podTeardown:
                    maxConcurrentPerNode: 4
                    maxConcurrentPerDegradedNode: 1
                    maxConcurrentReplacements: 40

maxConcurrentPerNode is applied independently to each healthy node, so cleanup can progress in parallel across nodes. A Ready node reporting memory, disk, PID, or network pressure uses maxConcurrentPerDegradedNode. Manager starts no new teardown on a NotReady or Unknown node, but stale force-deletes still make bounded progress with the degraded-node limit. All templates and cleanup paths share the same cluster-wide replacement budget.

Manager also creates a PodDisruptionBudget for each idle pool. It protects ready capacity from autoscalers that use the Eviction API, while the node-aware manager budget controls direct rollout and repair deletes.

Monitor the S0FS resume cache with ctld_s0fs_hot_cache_requests_total, ctld_s0fs_hot_cache_admissions_total, ctld_s0fs_hot_cache_evictions_total, and the entry, estimated-byte, segment, and budget gauges under the ctld_s0fs_hot_cache_ prefix. Admission and segment labels are intentionally low-cardinality; sandbox and volume IDs appear only in structured logs.

S0FS State Format Rollout#

Treat S0FS state format 2 as a two-phase rollout. First deploy the dual-reader build to every manager and both ctld DaemonSets while leaving s0fsStateFormatVersion: 1. After every old reader is gone, set the field to 2. Once any committed head or snapshot has been written as version 2, roll back only to a build that can read both versions; changing the field back to 1 changes future writes but does not rewrite existing version 2 objects.

yaml
spec:
    redis:
        type: builtin
        builtin:
            enabled: true
    storage:
        runtime:
            cacheSizeLimit: 20Gi
            logSizeLimit: 1Gi
            s0fsSegmentTargetSize: 4Mi
            s0fsStateFormatVersion: 1
            s0fsHotCacheMaxSize: 1Gi
            s0fsCompactionInterval: 1m
            s0fsCompactionMinDeadRatio: "0.5"
            s0fsCompactionMinReclaimSize: 1Mi
            volumePortalCacheSizeLimit: 20Gi
            volumePortalRootMinFree: 5Gi
    network:
        config:
            egressBandwidthBytesPerSecond: 10485760
            ingressBandwidthBytesPerSecond: 10485760
            bandwidthBurstBytes: 10485760
    services:
        manager:
            config:
                defaultTeamQuotas:
                    - dimension: network_egress_bytes
                      limitValue: 104857600
                      intervalMs: 1000
                      burstValue: 104857600
                    - dimension: network_ingress_bytes
                      limitValue: 104857600
                      intervalMs: 1000
                      burstValue: 104857600

Use spec.sandboxNodePlacement.nodeSelector and tolerations for the hard placement shared by sandbox template Pods and the ctld HA pair. infra-operator owns sandbox0.ai/data-plane-ready and adds it to sandbox Pod placement after the required node-local components are Ready.

preferredNodeSelector is different: it is a soft preference used only for manager-created sandbox Pods. The manager also uses the same selector when a hot claim chooses among already scheduled idle Pods. This lets a fixed-capacity pool absorb claims first while preserving an autoscaled pool as fallback:

yaml
spec:
    sandboxNodePlacement:
        nodeSelector:
            sandbox0.ai/node-role: sandbox
        preferredNodeSelector:
            sandbox0.ai/capacity-type: fixed
        tolerations:
            - key: sandbox0.ai/sandbox
              operator: Equal
              value: "true"
              effect: NoSchedule

Do not put a fallback capacity label such as sandbox0.ai/capacity-type: fixed in nodeSelector; that would turn the preference into a hard requirement and prevent scheduling onto burst nodes.

Full Reference#

The reference below is generated from the Sandbox0Infra CRD schema produced by controller-gen, not manually maintained MDX.