#Configuration

This page explains how to think about Sandbox0Infra configuration and includes a generated field reference sourced from the operator CRD.

The goal is to keep the page readable for operators while still exposing the full supported config surface.

How to Read Sandbox0Infra#

A Sandbox0Infra spec is easier to reason about when you split it into six layers:

Recommended Reading Order#

1. Pick your topology with spec.services.*.
2. Decide whether database/storage/Redis/registry stay builtin or move external.
3. Configure public exposure and cluster identity.
4. Seed templates and initial admin user.
5. Only then tune service and data-plane runtime configuration.

spec.initUser is consumed by the gateway runtime, not created by the operator itself. In local-password mode it bootstraps the first admin credentials; in OIDC-only mode it pre-creates the admin user and initial team so the first OIDC login with the same email lands on the intended admin account.

When spec.builtinTemplates is omitted, infra-operator seeds public builtin templates for default, openclaw, hermes, browser, and coding-agent. Set spec.builtinTemplates explicitly when you want to pin images, adjust pools, disable one of the builtins, or provide a full template spec for a runtime-specific setup. Set it to [] to remove all operator-managed builtin templates.

Pin Builtin Images for Predictable Startup#

The self-hosted openclaw, hermes, and browser presets use upstream mutable latest tags by default. Sandbox0 sets the effective main-container pull policy to IfNotPresent unless a system template overrides it. A claim on a node that already has the image does not download it again, but a new or cache-miss node resolves the tag and pulls the current image. That pull can add substantial cold-start latency for large images, and different nodes can keep different digests for the same mutable tag.

For production self-hosted deployments, replace the complete builtin template list and pin each mutable image to a reviewed digest:

yaml
spec:
    builtinTemplates:
        - templateId: default
        - templateId: openclaw
          image: ghcr.io/openclaw/openclaw@sha256:<openclaw-digest>
        - templateId: hermes
          image: nousresearch/hermes-agent@sha256:<hermes-digest>
        - templateId: browser
          image: ghcr.io/steel-dev/steel-browser@sha256:<browser-digest>

spec.builtinTemplates has replacement semantics, so omitting an entry removes that operator-managed builtin. A version tag is easier to read than a digest but remains mutable if its publisher retags it. A digest is the immutable choice.

Pinning makes the runtime version deterministic; it does not eliminate the first pull on an empty node. If image-pull latency is part of your cold-start budget, preload the pinned digests into node images or use a node-level pre-puller before admitting sandbox workloads.

Sandbox0 Cloud handles the mutable-tag policy at the platform layer: managed-region configuration pins these images by digest and rolls upstream digest changes through deployment automation. Cloud users therefore do not maintain these pins themselves. A genuinely cold node can still need a missing pinned image, but cache and prewarming operations remain the managed platform's responsibility rather than part of the customer template configuration.

Supported Topologies#

The single-cluster sample is always fullmode. manager hosts lifecycle control and the storage API runtime. Each sandbox node runs the ctld-a and ctld-b DaemonSets; the elected primary owns rootfs persistence, volume portals, and the network runtime, and the synchronized standby takes over after promotion. Configure storage behavior under spec.storage.runtime, network behavior under spec.network, and shared sandbox-node placement under spec.sandboxNodePlacement.

Sandbox Claim Admission#

Team Quota is the claim admission boundary. The sandbox_claims rate policy controls sustained claim rate and immediate burst at the external regional or full-mode gateway, while active_sandboxes limits the number of running sandboxes for a team. Manager does not maintain a separate cluster-wide claim-start budget.

After quota admission, manager claims a ready idle pod or creates a cold pod directly. Warm-pool ReplicaSets also scale directly to each template's configured minIdle. In multi-cluster mode, scheduler routing prefers ready idle capacity, then estimated cluster headroom, configured weight, and deterministic fallback order.

Metering Boundary#

Sandbox0 records usage truth in the region usage ledger and exports it from the region-scoped metering endpoints. Metering is disabled by default. When spec.metering.enabled is true, metering requires spec.clickHouse.type to be builtin or external. ClickHouse is the long-term usage ledger and the only metering query/export backend. PostgreSQL holds only a compact transactional delivery outbox and producer projection state; it is not a second queryable usage ledger. Producers capture stable, idempotent ClickHouse operations in that outbox, storage mutations use the same PostgreSQL business transaction, and manager retries the oldest transaction batch until ClickHouse accepts it.

Metering export is therefore eventually consistent. Manager lifecycle events and storage mutations are normally visible after the projector poll (about two seconds). Network byte windows from ctld add meteringReportInterval (10 seconds by default), while periodic storage byte-hour windows add up to one minute. A ClickHouse outage increases this delay without discarding committed outbox operations; after the one-time projection-state bootstrap has completed, producers can also restart while ClickHouse is unavailable. The outbox is globally ordered, so the exported complete_before value is the greatest producer watermark that has reached ClickHouse; projecting it proves that every older committed batch has also been delivered, and historical producer instances do not pin the frontier. Monitor manager_metering_outbox_pending_operations and manager_metering_outbox_oldest_pending_age_seconds for delivery backlog. Billing systems should consume the exported watermark and rate usage windows outside the open-source data plane.

New event and window rows expose their PostgreSQL outbox operation sequence. Rows projected before the sequence column was introduced retain sequence: 0; cursor-based pagination is the canonical export contract across upgrades. Metering timestamps are written to ClickHouse as signed Unix nanoseconds so DateTime64(9) window boundaries and cursors preserve their full precision.

Sandbox compute uses a serverless-style contract:

Public template idle pools are platform cost. Team-owned template warm pools are attributed to the owning team with subject_type=template.

OSS request windows are usage facts only; Sandbox0 does not attach prices. The manager and ctld aggregate provider HTTP attempts for one minute before writing them to the PostgreSQL metering outbox. Successful volume requests are attributed from sandboxvolumes/<team_id>/<volume_id>, and rootfs requests from sandbox-rootfs/<team_id>/<sandbox_id>. Window data records provider, bucket, billing_item, operation, cost_scope, and prefix_class. Requests to customer-configured external S3 volumes are excluded because the customer owns that object-store account. Reconcile these windows with provider access and metering logs before using them for cost reporting.

Platform Observability#

Sandbox0 platform services emit logs to container stdout and expose Prometheus-compatible metrics on their metrics ports. Configure spec.observability when you want the operator to export sandbox0-owned platform telemetry to your observability stack.

Platform telemetry export is disabled by default. The operator owns collection and export integration only: it can point platform services at an external collector, or install OpenTelemetry Collectors that export to your OTLP endpoint. The per-sandbox historical observability APIs are separate from this platform telemetry layer; they require a configured query backend and return 503 unavailable when that backend is disabled or unavailable. The historical observability tables are separate from the metering usage ledger even when both use the same region ClickHouse component.

For cold-start diagnosis, manager exposes manager_pod_lifecycle_stage_duration_seconds. Its stage label separates scheduling, the PodReadyToStartContainers sandbox/network boundary, procd container start, first PodIP observation, and Sandbox0 startup/readiness probe completion. The sandbox boundary is a Kubernetes lifecycle timestamp, not the duration of an individual CRI call. Collect kubelet kubelet_runtime_operations_duration_seconds{operation_type=~"run_podsandbox|create_container|start_container"} when exact node-level CRI operation latency is required.

ClickHouse is configured once on the Sandbox0Infra CR under spec.clickHouse. The infra-operator owns builtin provisioning or external DSN resolution, the same way it owns built-in or external PostgreSQL and Redis wiring. Feature configs such as spec.sandboxObservability and spec.metering consume this region component. Do not configure separate ClickHouse backends on cluster-gateway, manager, or ctld when deploying through Sandbox0Infra.

Historical logs and runtime metrics are available whenever sandbox observability is enabled. Centralized sandbox audit is an enterprise feature: set spec.sandboxObservability.audit.enabled to true and provide a signed enterprise license containing the sandbox_audit feature. The license is mounted only into cluster-gateway, which authorizes audit ingestion and the public observability-event query and watch API.

ClickHouse is the canonical audit store. Audit facts are written to sandbox_audit_events with stable event IDs, trusted actor and producer identity, a canonical payload digest, and an Ed25519 gateway signature. PostgreSQL is not a second audit store. The infra-operator creates separate network-producer and audit-signing key pairs: the active ctld process receives only the producer private key, while cluster-gateway receives its public key and the independent signing key. Fsync-backed buffers are delivery state, not competing audit ledgers or query sources; the public observability-event list and watch responses read only canonical ClickHouse rows and include signature_status, with event-ID payload conflicts reported separately as event_id_conflict. A buffered event is therefore not visible to audit readers until ClickHouse acknowledges it. Cluster-gateway uses a persistent volume claim for pending audit events, while each node-local ctld network runtime keeps its flow buffer on that node. Delivery records are deleted only after ClickHouse acknowledges the exact event.

The v2 audit schema deliberately uses a new sandbox_audit_events table. Older sandbox_events rows do not have trustworthy actor or signature data and are retained as legacy observability history rather than being retroactively signed. Existing CRs whose persisted table value is the old default are routed to the new canonical table. At startup, audit-enabled cluster-gateway validates the actual ClickHouse engine, partition key, sorting key, and required columns, and refuses to serve audit operations against an old or incompatible custom table.

Use builtin ClickHouse when you want the infra-operator to install ClickHouse for the region:

yaml
spec:
    enterpriseLicense:
        secretRef:
            name: sandbox0-enterprise-license
            key: license.lic
    clickHouse:
        type: builtin
        builtin:
            image: clickhouse/clickhouse-server:24.8
            persistence:
                size: 50Gi
        databases:
            observability: sandbox0_observability
            metering: sandbox0_metering
        schemaMigration:
            enabled: true
    sandboxObservability:
        enabled: true
        backend: clickhouse
        audit:
            enabled: true
            deliveryMode: durable_async
            deliveryPersistence:
                size: 1Gi
        retention:
            auditDays: 90
            logDays: 7
            runtimeSampleDays: 30
        ingest:
            queueSize: 1024
            batchSize: 100
            flushInterval:
                duration: 1s
            requestTimeout:
                duration: 2s
            maxRetries: 3
            retryBackoff:
                duration: 100ms
    metering:
        enabled: true
        clickHouse:
            eventsTable: usage_events
            windowsTable: usage_windows
            watermarksTable: producer_watermarks
            sandboxStateTable: sandbox_projection_state
            storageStateTable: storage_projection_state

Audit retention is measured from the cluster-gateway-owned ingested_at timestamp, not the producer's potentially delayed occurred_at. This timestamp marks when cluster-gateway creates or normalizes the canonical event; it is not a ClickHouse acknowledgement timestamp. An event delayed in a delivery buffer may therefore have less than the configured retention period remaining when it becomes query-visible.

spec.sandboxObservability.audit.deliveryMode selects the admission boundary for non-mutating Sandbox API requests, public-exposure requests, and new network flows handled by ctld:

• durable_async is the default. The operation may proceed after its attempt or event is fsynced to the local delivery buffer; ClickHouse ingestion and audit-query visibility follow asynchronously. On a healthy deployment, visibility is driven by immediate delivery wakeups plus the default one-second flush/replay cycles, so it is not a read-after-write guarantee.
• canonical_sync waits for ClickHouse to acknowledge the admission event before the operation or flow proceeds. This puts a ClickHouse round trip on the critical path and fails closed when canonical acknowledgement is unavailable.

State-changing Sandbox API requests are always strict regardless of this setting. Cluster-gateway requires canonical acknowledgement for the attempt before execution, buffers the downstream response, and requires canonical acknowledgement for the signed result before returning that response. The delivery buffer is still used to preserve a pending fact for replay, but it never turns a mutation into a successful response before ClickHouse acknowledges the fact. If canonical admission fails after the attempt is durably buffered, cluster-gateway does not execute the downstream handler and attempts to append a correlated signed failed result with HTTP 503, execution_started=false, operation_executed=false, failure_stage=canonical_audit_admission, and failure_code=canonical_ack_unavailable. When result custody succeeds, both facts become query-visible after replay. If the result buffer and canonical fallback both fail, the operation remains unexecuted, the response reports audit_result=unrecorded, and the surviving orphan attempt is an audit-completeness incident that requires investigation.

During a ClickHouse outage, durable_async traffic can continue only while the local delivery buffer remains writable and durable; audit readers will lag until replay succeeds, with no fixed visibility bound during the outage. If the buffer cannot accept an event, Sandbox0 tries a synchronous canonical fallback and fails closed when neither path can take custody. canonical_sync traffic and all mutations fail closed on canonical timeout or outage. Choose canonical_sync when immediate canonical admission is worth the added latency and availability coupling; choose durable_async when durable capture with eventual canonical visibility is the preferred default.

Use the external backend when you already operate ClickHouse:

yaml
spec:
    enterpriseLicense:
        secretRef:
            name: sandbox0-enterprise-license
            key: license.lic
    clickHouse:
        type: external
        external:
            dsnSecret:
                name: sandbox0-clickhouse
                key: dsn
            connectTimeout:
                duration: 10s
        databases:
            observability: sandbox0_observability
            metering: sandbox0_metering
    sandboxObservability:
        enabled: true
        backend: clickhouse
        audit:
            enabled: true
            deliveryMode: durable_async
            deliveryPersistence:
                size: 1Gi
        retention:
            auditDays: 90
            logDays: 7
            runtimeSampleDays: 30
    metering:
        enabled: true

Set spec.sandboxObservability.enabled to false or omit the field to keep all historical APIs disabled. Leave spec.sandboxObservability.audit.enabled unset or false to keep canonical audit ingestion and event queries disabled while retaining historical logs and runtime metrics. Set spec.metering.enabled to false or omit spec.metering to keep metering producers, quota usage reads, and metering export readers disabled. Legacy spec.sandboxObservability.type fields are still accepted for compatibility, but new installs should use spec.clickHouse for ClickHouse provisioning. Disabling audit stops new facts but does not delete existing rows. retention.auditDays is the retention policy for the canonical ClickHouse audit table; set it to the period required by your organization before enabling audit in production. Metering remains a separate ledger.

When the backend is enabled, infra-operator injects bounded internal ingest endpoints into ctld for sandbox-wide CRI runtime samples and manager for historical logs. When licensed audit is also enabled, it injects the selected delivery mode and durable event-buffer configuration into cluster-gateway and the ctld network runtime. In durable_async, ctld opens an allowed upstream connection after its producer attempt is fsynced locally; cluster-gateway later normalizes and signs that event during canonical ingestion. In canonical_sync, ctld waits for cluster-gateway and ClickHouse to acknowledge the attempt. Gateway mutation responses are always held until their signed result is acknowledged; pending results return 503 and replay from disk.

Audit currently requires spec.services.clusterGateway.replicas: 1. Infra-operator mounts a ReadWriteOnce audit-event delivery PVC and uses the Recreate deployment strategy so the same buffer is detached and reattached before a replacement gateway starts. In production, set audit.deliveryPersistence.storageClass to durable, encrypted storage that supports cross-node volume reattachment; the cluster default is used when it is omitted. Ctld network flow buffers remain node-local and survive same-node Pod restarts, but not permanent node loss. All delivery records are removed only after ClickHouse acknowledgement. Preserve the generated audit-signing Secret for the full retention period. Inline verification currently uses only the active public key, so events signed by a retained historical key report signature_status=unavailable; destructive key replacement therefore requires retaining previous public keys outside the cluster for historical offline verification.

The ctld network runtime exports sandbox0_netd_audit_ingest_events_total and sandbox0_netd_audit_ingest_batches_total; alert on persist_failed, corrupt, retrying, replay_failed, and ack_failed results. Only canonical_sync admission adds a ClickHouse round trip to each new audited flow; the default durable_async path pays for local fsync and gains ClickHouse visibility asynchronously. HTTP requests multiplexed over one existing connection remain protocol operations beneath that flow rather than independent network connections.

The built-in controls provide durable delivery, authenticated producer identity, tenant scoping, append-only application behavior, and tamper-evident row signatures. Regulatory certification still depends on deployment controls outside the process: highly available ClickHouse storage, restricted ClickHouse mutation credentials, encrypted backups, tested restore procedures, retention and legal-hold policy, clock synchronization, and independent preservation of signing public keys or exported checkpoints. A database administrator who can rewrite the complete table and its backups is outside the protection of a per-row signature alone.

In managedCollector mode sandbox0 installs collectors for logs, metrics, and traces, then exports to your OTLP endpoint:

yaml
spec:
    observability:
        backend:
            type: external
            external:
                mode: managedCollector
                otlp:
                    endpoint: otel-gateway.example.com:4317
                    insecure: false
        resourceAttributes:
            deployment.environment: production
        collection:
            logs:
                enabled: true
            metrics:
                enabled: true
            traces:
                enabled: true

If your platform already runs collectors, use existingCollector. In this mode sandbox0 does not install collectors; it only injects standard OTEL trace environment variables into platform services:

yaml
spec:
    observability:
        backend:
            type: external
            external:
                mode: existingCollector
                otlp:
                    endpoint: http://otel-collector.observability.svc.cluster.local:4317
                    headersSecret:
                        name: otel-export-headers
                        key: headers
                    insecure: true

The legacy direct trace exporter config remains supported and overrides export-derived trace endpoint defaults:

yaml
spec:
    observability:
        traces:
            enabled: true
            exporter: otlp
            endpoint: http://otel-collector.observability.svc.cluster.local:4317
            insecure: true
            timeout: 10s
            sampleRate: "0.25"

The operator adds built-in resource attributes such as service.name, sandbox0.region.id, sandbox0.cluster.id, k8s.namespace.name, k8s.pod.name, and k8s.node.name. Built-in attributes win over user-provided keys so dashboards and trace queries can rely on stable names.

When traces are not enabled through collection.traces or the legacy traces block, services keep the default noop trace exporter. Existing direct environment overrides still work; standard OpenTelemetry variables such as OTEL_TRACES_EXPORTER, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_HEADERS, and OTEL_RESOURCE_ATTRIBUTES take precedence over operator-injected defaults.

Official sample manifests:

• single-cluster/fullmode.yaml
• single-cluster/fullmode-gke-gvisor.yaml
• multi-cluster/control-plane.yaml
• multi-cluster/data-plane.yaml

What Usually Changes First#

External PostgreSQL#

Move spec.database.type from builtin to external when you want managed PostgreSQL durability, backup policy, and operational separation.

yaml
spec:
    database:
        type: external
        external:
            host: your-db.rds.amazonaws.com
            port: 5432
            database: sandbox0
            username: sandbox0
            passwordSecret:
                name: db-credentials
                key: password

S3 or OSS Backing Storage#

Configure the object-storage backend and manager storage runtime together under spec.storage.

yaml
spec:
    storage:
        type: s3
        s3:
            bucket: sandbox0-prod
            region: us-east-1
            endpoint: https://s3.amazonaws.com
            credentialsSecret:
                name: aws-credentials
                accessKeyKey: accessKeyId
                secretKeyKey: secretAccessKey
        runtime:
            objectEncryptionEnabled: true

Sandbox RuntimeClass#

Use services.manager.config.sandboxRuntimeClassName when sandbox Pods should run with a non-default runtime such as runc, gvisor, or a custom gvisor-rootfs handler.

yaml
spec:
    services:
        manager:
            config:
                sandboxRuntimeClassName: gvisor-rootfs

This field only controls the runtimeClassName written onto sandbox Pods. The named Kubernetes RuntimeClass must already exist, and its handler must already be configured in the node CRI runtime on every eligible sandbox node.

infra-operator does not install node runtime handlers. It does not install runsc, write /etc/containerd/config.toml, restart containerd, or create a gvisor-rootfs handler. Do that through your node image, node bootstrap automation, or a dedicated privileged node setup process before scheduling sandbox workloads there.

Procd Binary Image Volume#

Sandbox Pods mount Sandbox0 procd binaries from a small OCI artifact image at /procd-image by default. This avoids a per-Pod init container on the sandbox startup path.

When procdBinImageRef is omitted, infra-operator derives it from the deployed infra image tag by appending -procd-bin. For example, sandbox0ai/infra:v0.8.0 uses sandbox0ai/infra:v0.8.0-procd-bin.

Set procdBinImageRef only when you publish the procd binary artifact image separately:

yaml
spec:
    services:
        manager:
            config:
                procdBinImageRef: registry.example.com/sandbox0/procd-bin:v0.8.0

Kubernetes image volume support is part of the Sandbox0 cluster baseline. Unsupported runtimes fail sandbox Pod startup instead of falling back inside the Pod. Sandbox0 mounts the whole image volume and starts /procd-image/usr/local/bin/procd, so clusters do not need container-runtime support for image volume subPath.

Sandbox Memory Limits#

Sandbox claims and runtime updates can set config.resources.memory. The minimum accepted value is 128Mi. The platform maximum defaults to 32Gi. Manager derives CPU from the configured memory-per-CPU ratio and enforces a minimum CPU limit of 150m for claimed sandboxes.

The minimum applies when a sandbox is claimed, resumed, or resized. Upgrading manager does not resize sandbox pods that were already running before the upgrade.

Use services.manager.config.sandboxMaxMemory to lower or raise the platform maximum:

yaml
spec:
    services:
        manager:
            config:
                sandboxMaxMemory: 16Gi

Sandbox0 derives CPU from memory using services.manager.config.teamTemplateMemoryPerCpu, so callers do not set CPU directly on a sandbox claim or update.

For gVisor rootfs checkpoint and restore support on self-managed clusters, use a dedicated containerd handler that points to runsc with shared file access and overlay disabled, then create a matching RuntimeClass:

toml
# /etc/containerd/config.toml
version = 2

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor-rootfs]
    runtime_type = "io.containerd.runsc.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor-rootfs.options]
    TypeUrl = "io.containerd.runsc.v1.options"
    ConfigPath = "/etc/containerd/runsc-rootfs.toml"

toml
# /etc/containerd/runsc-rootfs.toml
[runsc_config]
    overlay2 = "none"
    file-access = "shared"

yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
    name: gvisor-rootfs
handler: gvisor-rootfs

sandboxRuntimeClassName applies only to sandbox Pods. Ctld is a node-local host component and must use a runtime that supports its required hostNetwork, hostPath, FUSE, and network-programming access; do not run ctld under gvisor, gvisor-rootfs, or kata.

ctld assumes the node's containerd data root is /var/lib/containerd by default. If your nodes move containerd data to another host path, configure the same path so rootfs pause/resume can use the overlayfs fast path:

yaml
spec:
    services:
        ctld:
            containerdHostDataRoot: /var/lib/containerd/sandbox0

ctld also keeps a disposable node-local rootfs object cache under its host data directory. This cache accelerates pause/resume when the restoring ctld already has the needed rootfs layer objects. The default cap is 20Gi per ctld node. Set rootfsObjectCacheMaxBytes: "0" to disable it, or tune the GC fields to fit your node disk:

yaml
spec:
    services:
        ctld:
            rootfsObjectCacheMaxBytes: 20Gi
            rootfsObjectCacheMinFreeBytes: 5Gi
            rootfsObjectCacheMaxAge: 24h
            rootfsObjectCacheSweepInterval: 1m

Cache contents are safe to delete. If a cached object is missing, corrupt, too old, or evicted to preserve free space, ctld downloads the rootfs object from configured object storage.

Shared Redis Rate Limiting#

Gateway request rate limiting defaults to process-local memory. Configure spec.redis so api_requests and network bandwidth Team Quotas share admission state across gateway replicas, ctld nodes, and data-plane clusters in the region.

yaml
spec:
    redis:
        type: builtin
        builtin:
            enabled: true

For an external Redis instance, store the URL in a Secret and reference it from spec.redis.

yaml
apiVersion: v1
kind: Secret
metadata:
    name: redis-url
    namespace: sandbox0-system
stringData:
    url: rediss://:[email protected]:6379/0
---
apiVersion: infra.sandbox0.ai/v1alpha1
kind: Sandbox0Infra
metadata:
    name: fullmode
    namespace: sandbox0-system
spec:
    redis:
        type: external
        external:
            urlSecret:
                name: redis-url
                key: url
        keyPrefix: sandbox0:ratelimit:prod
        operationTimeout: 100ms
        failOpen: true

Credential Source Storage#

Credential sources are stored in encrypted PostgreSQL by default. infra-operator creates and mounts a stable per-installation encryption key for manager. During upgrade, legacy plaintext rows remain readable and are not rewritten automatically.

Configure spec.credentialVault when new credential source material should live in a Vault-compatible backend. Builtin mode installs OpenBao:

yaml
spec:
    credentialVault:
        type: builtin
        builtin:
            enabled: true
            mount: secret

External mode connects manager to an existing HashiCorp Vault-compatible endpoint:

yaml
apiVersion: v1
kind: Secret
metadata:
    name: sandbox0-vault-token
    namespace: sandbox0-system
stringData:
    token: <vault-token>
---
apiVersion: infra.sandbox0.ai/v1alpha1
kind: Sandbox0Infra
metadata:
    name: fullmode
    namespace: sandbox0-system
spec:
    credentialVault:
        type: external
        external:
            address: https://vault.example.com
            tokenSecret:
                name: sandbox0-vault-token
                key: token
            mount: secret
            allowedPathPrefixes:
                - sandbox0/credential-sources/{{teamID}}/

When spec.credentialVault is enabled, manager stores newly written credential sources in the configured Vault backend. Existing rows keep their recorded storage backend.

Egress Auth Resolve TTL#

The ctld network runtime caches resolved egress auth material until the expiresAt timestamp returned by manager. Manager does not keep an additional resolved-material cache. Credential source rotation actively invalidates affected ctld cache entries, so already-claimed sandboxes normally use the rotated source on their next matching egress resolve. The default lifetime is 5 minutes, and the TTL remains the fallback consistency bound if realtime invalidation delivery is interrupted.

Set spec.services.manager.config.egressAuthDefaultResolveTtl when you want a different platform default. Individual bindings can override it with network.credentialBindings[*].cachePolicy.ttl.

yaml
spec:
    services:
        manager:
            config:
                egressAuthDefaultResolveTtl: 1m

Public Routing and Region Identity#

spec.publicExposure decides the public host pattern used for sandbox URLs. Keep it consistent with your region and DNS plan.

yaml
spec:
    publicExposure:
            enabled: true
            rootDomain: sandbox0.example.com
            regionId: aws-us-east-1

SSH Gateway#

Enable spec.services.sshGateway when you want a region-scoped SSH entrypoint that bridges standard ssh and scp clients into sandbox procd sessions.

The routing model is intentionally simple:

• One ssh-gateway per region
• One fixed DNS-only host or TCP proxy address per region, for example aws-us-east-1.ssh.sandbox0.app
• SSH username is the target sandbox ID
• User authentication uses SSH public keys uploaded to the gateway API

In Kubernetes terms, spec.services.sshGateway.service.port is the Kubernetes Service port and spec.services.sshGateway.config.sshPort is the container listen port. You usually keep the internal port at 2222 and expose 22 or a NodePort externally.

When an external load balancer listens on a different public port than the Kubernetes Service or NodePort, set spec.services.sshGateway.endpointPort to the public port that clients should use.

The operator also manages a persistent Ed25519 host key Secret for the service. As long as that Secret is retained, clients continue to see the same SSH host identity across pod restarts.

yaml
spec:
    services:
        sshGateway:
            enabled: true
            replicas: 1
            service:
                type: LoadBalancer
                port: 22
            config:
                sshPort: 2222

For local labs or kind-style environments, a NodePort is often simpler:

yaml
spec:
    services:
        sshGateway:
            enabled: true
            replicas: 1
            service:
                type: NodePort
                port: 30222
            endpointPort: 22

After exposing the service through DNS or a load balancer, users can upload SSH public keys with POST /users/me/ssh-keys and connect with standard clients. See SSH for the user-facing flow.

AWS LoadBalancer TLS#

When you expose regionalGateway or globalGateway through a cloud load balancer instead of Kubernetes Ingress, declare provider-specific Service annotations under spec.services.<service>.service.annotations. On AWS, this is the supported place to attach ACM certificates and choose the load balancer mode.

Also set spec.services.<service>.config.baseUrl to the final browser-facing URL. That keeps auth callbacks, generated links, and projected status endpoints aligned with the hostname you put behind Cloudflare.

yaml
spec:
    services:
        regionalGateway:
            enabled: true
            service:
                type: LoadBalancer
                port: 443
                annotations:
                    service.beta.kubernetes.io/aws-load-balancer-type: nlb
                    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
                    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <acm-certificate-arn>
                    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            config:
                baseUrl: https://your-gateway.example.com

Multi-Cluster Data Plane Registration#

Data-plane clusters need control-plane connection details and a stable cluster identity.

yaml
spec:
    controlPlane:
        url: https://api.sandbox0.example.com
        internalAuthPublicKeySecret:
            name: control-plane-public-key
            key: public.key
    cluster:
        id: cluster-001
        name: production-use1-a
    services:
        clusterGateway:
            enabled: true
            config:
                authMode: internal
        manager:
            enabled: true

Service and Runtime Config#

Use spec.services.<service>.config for deployable services, spec.storage.runtime for manager storage behavior, and spec.network.config for the network runtime in ctld.

Examples:

• services.clusterGateway.config.authMode switches between public, internal, and both
• services.manager.config.autoscaler.* controls warm-pool scaling behavior
• services.manager.config.k8sClientQps and k8sClientBurst tune the manager Kubernetes client token bucket rate limit; defaults are 50 and 100
• services.manager.config.sandboxMaxMemory caps per-sandbox memory overrides; the default is 32Gi
• services.manager.config.procdBinImageRef overrides the derived procd artifact image used by sandbox Pods
• services.manager.config.allowColdStartWithoutReadyDataPlane lets cold claims create Pending pods for external node autoscaler scale-from-zero deployments; keep it disabled unless the cluster autoscaler can provision matching sandbox nodes
• storage.runtime.filesystem* tunes the manager storage runtime; services.manager.config.procdConfig.volume* tunes mounted volume cache sizing
• storage.runtime.s0fsSegmentTargetSize sets the target S0FS segment size for materialized volume data; the default is 4Mi
• storage.runtime.s0fsCompaction* tunes background S0FS compaction for RWO volume owners; s0fsCompactionInterval accepts 0, off, or disabled to turn it off
• storage.runtime.cacheSizeLimit and logSizeLimit cap storage cache and logs in the manager Pod; volumePortalCacheSizeLimit and volumePortalRootMinFree tune the volume portals used by ctld
• storage.runtime.objectEncryptionEnabled controls application-layer encryption for persisted rootfs checkpoint objects, S0FS Volume objects, and node-local S0FS volume cache files; it is enabled by default
• network.config.egressBandwidthBytesPerSecond, ingressBandwidthBytesPerSecond, and bandwidthBurstBytes apply per-sandbox bandwidth throttling when set above zero
• services.manager.config.defaultTeamQuotas configures region-wide capacity, API request, and network bandwidth defaults; see Team Quotas
• network.config.* also controls proxy ports, policy enforcement, and node-level networking behavior

yaml
spec:
    redis:
        type: builtin
        builtin:
            enabled: true
    storage:
        runtime:
            cacheSizeLimit: 20Gi
            logSizeLimit: 1Gi
            s0fsSegmentTargetSize: 4Mi
            s0fsCompactionInterval: 1m
            s0fsCompactionMinDeadRatio: "0.5"
            s0fsCompactionMinReclaimSize: 1Mi
            volumePortalCacheSizeLimit: 20Gi
            volumePortalRootMinFree: 5Gi
    network:
        config:
            egressBandwidthBytesPerSecond: 10485760
            ingressBandwidthBytesPerSecond: 10485760
            bandwidthBurstBytes: 10485760
    services:
        manager:
            config:
                defaultTeamQuotas:
                    - dimension: network_egress_bytes
                      limitValue: 104857600
                      intervalMs: 1000
                      burstValue: 104857600
                    - dimension: network_ingress_bytes
                      limitValue: 104857600
                      intervalMs: 1000
                      burstValue: 104857600

Use spec.sandboxNodePlacement for the shared node placement consumed by sandbox template Pods and the ctld HA pair. infra-operator owns sandbox0.ai/data-plane-ready and adds it to sandbox Pod placement after the required node-local components are Ready.

Full Reference#

The reference below is generated from the Sandbox0Infra CRD schema produced by controller-gen, not manually maintained MDX.