#Configuration

This page explains how to think about Sandbox0Infra configuration and includes a generated field reference sourced from the operator CRD.

The goal is to keep the page readable for operators while still exposing the full supported config surface.

How to Read Sandbox0Infra#

A Sandbox0Infra spec is easier to reason about when you split it into five layers:

Recommended Reading Order#

1. Pick your topology with spec.services.*.
2. Decide whether database/storage/Redis/registry stay builtin or move external.
3. Configure public exposure and cluster identity.
4. Seed templates and initial admin user.
5. Only then tune per-service config blocks.

spec.initUser is consumed by the gateway runtime, not created by the operator itself. In local-password mode it bootstraps the first admin credentials; in OIDC-only mode it pre-creates the admin user and initial team so the first OIDC login with the same email lands on the intended admin account.

When spec.builtinTemplates is omitted, infra-operator seeds public builtin templates for default, dins, openclaw, and hermes. Set spec.builtinTemplates explicitly when you want to pin images, adjust pools, disable one of the builtins, or provide a full template spec for a runtime-specific setup. Set it to [] to remove all operator-managed builtin templates.

Deployment Profiles#

Metering Boundary#

Sandbox0 records usage truth in the region database and exports it from the region-scoped metering endpoints. Billing systems should rate these exported usage windows outside the open-source data plane.

Sandbox compute uses a serverless-style contract:

Idle template pool pods are capacity, not billable sandbox runtime. Public template idle pools are platform cost, and team template idle pools should be controlled by plan or entitlement limits rather than usage metering.

Platform Observability#

Sandbox0 platform services emit logs to container stdout and expose Prometheus-compatible metrics on their metrics ports. Configure spec.observability when you want the operator to export sandbox0-owned platform telemetry to your observability stack.

Observability is disabled by default. The operator owns collection and export integration only: it can point platform services at an external collector, or install OpenTelemetry Collectors that export to your OTLP endpoint. Sandbox0 does not install or manage an observability storage/query backend. User-facing sandbox observability APIs, query adapters, dashboards, and retention policy remain outside this layer.

In managedCollector mode sandbox0 installs collectors for logs, metrics, and traces, then exports to your OTLP endpoint:

yaml
spec:
    observability:
        backend:
            type: external
            external:
                mode: managedCollector
                otlp:
                    endpoint: otel-gateway.example.com:4317
                    insecure: false
        resourceAttributes:
            deployment.environment: production
        collection:
            logs:
                enabled: true
            metrics:
                enabled: true
            traces:
                enabled: true

If your platform already runs collectors, use existingCollector. In this mode sandbox0 does not install collectors; it only injects standard OTEL trace environment variables into platform services:

yaml
spec:
    observability:
        backend:
            type: external
            external:
                mode: existingCollector
                otlp:
                    endpoint: http://otel-collector.observability.svc.cluster.local:4317
                    headersSecret:
                        name: otel-export-headers
                        key: headers
                    insecure: true

The legacy direct trace exporter config remains supported and overrides export-derived trace endpoint defaults:

yaml
spec:
    observability:
        traces:
            enabled: true
            exporter: otlp
            endpoint: http://otel-collector.observability.svc.cluster.local:4317
            insecure: true
            timeout: 10s
            sampleRate: "0.25"

The operator adds built-in resource attributes such as service.name, sandbox0.region.id, sandbox0.cluster.id, k8s.namespace.name, k8s.pod.name, and k8s.node.name. Built-in attributes win over user-provided keys so dashboards and trace queries can rely on stable names.

When traces are not enabled through collection.traces or the legacy traces block, services keep the default noop trace exporter. Existing direct environment overrides still work; standard OpenTelemetry variables such as OTEL_TRACES_EXPORTER, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_HEADERS, and OTEL_RESOURCE_ATTRIBUTES take precedence over operator-injected defaults.

Official sample manifests:

• single-cluster/minimal.yaml
• single-cluster/fullmode.yaml
• single-cluster/volumes.yaml
• single-cluster/network-policy.yaml
• multi-cluster/control-plane.yaml
• multi-cluster/data-plane.yaml

What Usually Changes First#

External PostgreSQL#

Move spec.database.type from builtin to external when you want managed PostgreSQL durability, backup policy, and operational separation.

yaml
spec:
    database:
        type: external
        external:
            host: your-db.rds.amazonaws.com
            port: 5432
            database: sandbox0
            username: sandbox0
            passwordSecret:
                name: db-credentials
                key: password

S3 or OSS Backing Storage#

If you need persistent volume features in production, configure spec.storage and enable services.storageProxy.

yaml
spec:
    storage:
        type: s3
        s3:
            bucket: sandbox0-prod
            region: us-east-1
            endpoint: https://s3.amazonaws.com
            credentialsSecret:
                name: aws-credentials
                accessKeyKey: accessKeyId
                secretKeyKey: secretAccessKey
    services:
        storageProxy:
            enabled: true

Sandbox RuntimeClass#

Use services.manager.config.sandboxRuntimeClassName when sandbox Pods should run with a non-default runtime such as runc, gvisor, or a custom gvisor-rootfs handler.

yaml
spec:
    services:
        manager:
            config:
                sandboxRuntimeClassName: gvisor-rootfs

This field only controls the runtimeClassName written onto sandbox Pods. The named Kubernetes RuntimeClass must already exist, and its handler must already be configured in the node CRI runtime on every eligible sandbox node.

infra-operator does not install node runtime handlers. It does not install runsc, write /etc/containerd/config.toml, restart containerd, or create a gvisor-rootfs handler. Do that through your node image, node bootstrap automation, or a dedicated privileged node setup process before scheduling sandbox workloads there.

For gVisor rootfs checkpoint and restore support on self-managed clusters, use a dedicated containerd handler that points to runsc with shared file access and overlay disabled, then create a matching RuntimeClass:

toml
# /etc/containerd/config.toml
version = 2

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor-rootfs]
    runtime_type = "io.containerd.runsc.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor-rootfs.options]
    TypeUrl = "io.containerd.runsc.v1.options"
    ConfigPath = "/etc/containerd/runsc-rootfs.toml"

toml
# /etc/containerd/runsc-rootfs.toml
[runsc_config]
    overlay2 = "none"
    file-access = "shared"

yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
    name: gvisor-rootfs
handler: gvisor-rootfs

Keep services.netd.runtimeClassName unset, or set it only to a host-compatible runtime such as runc. netd should not run under gvisor, gvisor-rootfs, or kata.

ctld assumes the node's containerd data root is /var/lib/containerd by default. If your nodes move containerd data to another host path, configure the same path so rootfs pause/resume can use the overlayfs fast path:

yaml
spec:
    services:
        ctld:
            containerdHostDataRoot: /var/lib/sandbox0-worker/containerd

Shared Redis Rate Limiting#

Gateway request rate limiting defaults to process-local memory. Configure spec.redis when gateway replicas need a shared Redis backend. The same Redis backend is also used by netd when cluster-scoped team bandwidth limits are configured.

yaml
spec:
    redis:
        type: builtin
        builtin:
            enabled: true

For an external Redis instance, store the URL in a Secret and reference it from spec.redis.

yaml
apiVersion: v1
kind: Secret
metadata:
    name: redis-url
    namespace: sandbox0-system
stringData:
    url: rediss://:[email protected]:6379/0
---
apiVersion: infra.sandbox0.ai/v1alpha1
kind: Sandbox0Infra
metadata:
    name: fullmode
    namespace: sandbox0-system
spec:
    redis:
        type: external
        external:
            urlSecret:
                name: redis-url
                key: url
        keyPrefix: sandbox0:ratelimit:prod
        operationTimeout: 100ms
        failOpen: true

Credential Source Storage#

Credential sources are stored in encrypted PostgreSQL by default. infra-operator creates and mounts a stable per-installation encryption key for manager. During upgrade, legacy plaintext rows remain readable and are not rewritten automatically.

Configure spec.credentialVault when new credential source material should live in a Vault-compatible backend. Builtin mode installs OpenBao:

yaml
spec:
    credentialVault:
        type: builtin
        builtin:
            enabled: true
            mount: secret

External mode connects manager to an existing HashiCorp Vault-compatible endpoint:

yaml
apiVersion: v1
kind: Secret
metadata:
    name: sandbox0-vault-token
    namespace: sandbox0-system
stringData:
    token: <vault-token>
---
apiVersion: infra.sandbox0.ai/v1alpha1
kind: Sandbox0Infra
metadata:
    name: fullmode
    namespace: sandbox0-system
spec:
    credentialVault:
        type: external
        external:
            address: https://vault.example.com
            tokenSecret:
                name: sandbox0-vault-token
                key: token
            mount: secret
            allowedPathPrefixes:
                - sandbox0/credential-sources/{{teamID}}/

When spec.credentialVault is enabled, manager stores newly written credential sources in the configured Vault backend. Existing rows keep their recorded storage backend.

Public Routing and Region Identity#

spec.publicExposure decides the public host pattern used for sandbox URLs. Keep it consistent with your region and DNS plan.

yaml
spec:
    publicExposure:
            enabled: true
            rootDomain: sandbox0.example.com
            regionId: aws-us-east-1

SSH Gateway#

Enable spec.services.sshGateway when you want a region-scoped SSH entrypoint that bridges standard ssh and scp clients into sandbox procd sessions.

The routing model is intentionally simple:

• One ssh-gateway per region
• One fixed DNS-only host or TCP proxy address per region, for example aws-us-east-1.ssh.sandbox0.app
• SSH username is the target sandbox ID
• User authentication uses SSH public keys uploaded to the gateway API

In Kubernetes terms, spec.services.sshGateway.service.port is the Kubernetes Service port and spec.services.sshGateway.config.sshPort is the container listen port. You usually keep the internal port at 2222 and expose 22 or a NodePort externally.

When an external load balancer listens on a different public port than the Kubernetes Service or NodePort, set spec.services.sshGateway.endpointPort to the public port that clients should use.

The operator also manages a persistent Ed25519 host key Secret for the service. As long as that Secret is retained, clients continue to see the same SSH host identity across pod restarts.

yaml
spec:
    services:
        sshGateway:
            enabled: true
            replicas: 1
            service:
                type: LoadBalancer
                port: 22
            config:
                sshPort: 2222

For local labs or kind-style environments, a NodePort is often simpler:

yaml
spec:
    services:
        sshGateway:
            enabled: true
            replicas: 1
            service:
                type: NodePort
                port: 30222
            endpointPort: 22

After exposing the service through DNS or a load balancer, users can upload SSH public keys with POST /users/me/ssh-keys and connect with standard clients. See SSH for the user-facing flow.

AWS LoadBalancer TLS#

When you expose regionalGateway or globalGateway through a cloud load balancer instead of Kubernetes Ingress, declare provider-specific Service annotations under spec.services.<service>.service.annotations. On AWS, this is the supported place to attach ACM certificates and choose the load balancer mode.

Also set spec.services.<service>.config.baseUrl to the final browser-facing URL. That keeps auth callbacks, generated links, and projected status endpoints aligned with the hostname you put behind Cloudflare.

yaml
spec:
    services:
        regionalGateway:
            enabled: true
            service:
                type: LoadBalancer
                port: 443
                annotations:
                    service.beta.kubernetes.io/aws-load-balancer-type: nlb
                    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
                    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <acm-certificate-arn>
                    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            config:
                baseUrl: https://your-gateway.example.com

Multi-Cluster Data Plane Registration#

Data-plane clusters need control-plane connection details and a stable cluster identity.

yaml
spec:
    controlPlane:
        url: https://api.sandbox0.example.com
        internalAuthPublicKeySecret:
            name: control-plane-public-key
            key: public.key
    cluster:
        id: cluster-001
        name: production-use1-a
    services:
        clusterGateway:
            enabled: true
            config:
                authMode: internal
        manager:
            enabled: true

Service-Level Config#

spec.services.<service>.config is where you tune component-specific behavior after the topology is already correct.

Examples:

• services.clusterGateway.config.authMode switches between public, internal, and both
• services.manager.config.autoscaler.* tunes pool scale behavior
• services.manager.config.allowColdStartWithoutReadyDataPlane lets cold claims create Pending pods for external node autoscaler scale-from-zero deployments; keep it disabled unless the cluster autoscaler can provision matching sandbox nodes
• services.storageProxy.config.filesystem* tunes filesystem behavior; services.manager.config.procdConfig.volume* tunes mounted volume cache sizing
• services.storageProxy.config.s0fsSegmentTargetSize sets the target S0FS segment size for materialized volume data; the default is 4Mi
• services.storageProxy.config.s0fsCompaction* tunes background S0FS compaction for RWO volume owners; s0fsCompactionInterval accepts 0, off, or disabled to turn it off
• services.storageProxy.config.cacheSizeLimit, logSizeLimit, volumePortalCacheSizeLimit, and volumePortalRootMinFree cap node-local storage used by storage-proxy cache, logs, and mounted volume portals
• services.storageProxy.config.objectEncryptionEnabled controls application-layer encryption for S0FS object storage and node-local volume cache files; it is enabled by default
• services.netd.config.egressBandwidthBytesPerSecond, ingressBandwidthBytesPerSecond, and bandwidthBurstBytes apply per-sandbox bandwidth throttling when set above zero
• services.netd.config.teamEgressBandwidthBytesPerSecond, teamIngressBandwidthBytesPerSecond, and teamBandwidthBurstBytes apply cluster-scoped per-team aggregate throttling when set above zero and spec.redis is configured
• services.netd.config.* also controls proxy ports, policy enforcement, and node-level networking behavior

yaml
spec:
    redis:
        type: builtin
        builtin:
            enabled: true
    services:
        storageProxy:
            config:
                cacheSizeLimit: 20Gi
                logSizeLimit: 1Gi
                s0fsSegmentTargetSize: 4Mi
                s0fsCompactionInterval: 1m
                s0fsCompactionMinDeadRatio: "0.5"
                s0fsCompactionMinReclaimSize: 1Mi
                volumePortalCacheSizeLimit: 20Gi
                volumePortalRootMinFree: 5Gi
        netd:
            config:
                egressBandwidthBytesPerSecond: 10485760
                ingressBandwidthBytesPerSecond: 10485760
                bandwidthBurstBytes: 10485760
                teamEgressBandwidthBytesPerSecond: 104857600
                teamIngressBandwidthBytesPerSecond: 104857600
                teamBandwidthBurstBytes: 104857600

Use spec.sandboxNodePlacement for the shared node placement consumed by sandbox template Pods, netd, and ctld. infra-operator owns sandbox0.ai/data-plane-ready and adds it to sandbox Pod placement after the required node-local components are Ready. The older services.netd.nodeSelector and services.netd.tolerations fields remain as compatibility aliases when the shared placement is unset.

Full Reference#

The reference below is generated from the Sandbox0Infra CRD schema produced by controller-gen, not manually maintained MDX.