Observability

Self-hosted Sandbox0 has three separate data paths. Configure each one for its own purpose.

Platform Telemetry#

Sandbox0 services write logs to container stdout and expose Prometheus-compatible metrics on their metrics ports. spec.observability can export Sandbox0-owned logs, metrics, and traces to your platform stack.

For cold-start diagnosis, collect manager_pod_lifecycle_stage_duration_seconds. Its stage label separates:

• scheduling
• the PodReadyToStartContainers sandbox and network boundary
• procd container start
• first PodIP observation
• Sandbox0 startup and readiness probe completion

The sandbox boundary is a Kubernetes lifecycle timestamp, not an individual CRI call duration. Collect kubelet_runtime_operations_duration_seconds{operation_type=~"run_podsandbox|create_container|start_container"} when you need node-level CRI timing.

Managed Collector#

Use managedCollector when Sandbox0 should install OpenTelemetry Collectors and export to your OTLP endpoint.

yaml
spec:
    observability:
        backend:
            type: external
            external:
                mode: managedCollector
                otlp:
                    endpoint: otel-gateway.example.com:4317
                    insecure: false
        resourceAttributes:
            deployment.environment: production
        collection:
            logs:
                enabled: true
            metrics:
                enabled: true
            traces:
                enabled: true

Existing Collector#

Use existingCollector when your platform already operates collectors. Sandbox0 does not install collectors in this mode; it injects standard OpenTelemetry trace environment variables into platform services.

yaml
spec:
    observability:
        backend:
            type: external
            external:
                mode: existingCollector
                otlp:
                    endpoint: http://otel-collector.observability.svc.cluster.local:4317
                    headersSecret:
                        name: otel-export-headers
                        key: headers
                    insecure: true

The legacy direct trace exporter remains supported and overrides trace endpoints derived from the export configuration:

yaml
spec:
    observability:
        traces:
            enabled: true
            exporter: otlp
            endpoint: http://otel-collector.observability.svc.cluster.local:4317
            insecure: true
            timeout: 10s
            sampleRate: "0.25"

The operator adds stable attributes such as service.name, sandbox0.region.id, sandbox0.cluster.id, k8s.namespace.name, k8s.pod.name, and k8s.node.name. Built-in values take precedence over user-provided values for the same keys.

When traces are not enabled through collection.traces or the legacy traces block, services use the noop trace exporter. Direct environment overrides still take precedence over operator defaults, including:

• OTEL_TRACES_EXPORTER
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
• OTEL_EXPORTER_OTLP_TRACES_HEADERS
• OTEL_RESOURCE_ATTRIBUTES

Official sample manifests:

• single-cluster/fullmode.yaml
• single-cluster/fullmode-gke-gvisor.yaml
• multi-cluster/control-plane.yaml
• multi-cluster/data-plane.yaml

Per-Sandbox Historical Data#

Configure ClickHouse once under spec.clickHouse. spec.sandboxObservability and spec.metering consume that region component; do not configure separate ClickHouse backends on individual services.

Historical queries return 503 unavailable when their backend is disabled or unavailable. They are separate from platform telemetry and from the metering usage ledger.

Builtin ClickHouse#

Use builtin ClickHouse when the infra-operator should install the region component.

yaml
spec:
    enterpriseLicense:
        secretRef:
            name: sandbox0-enterprise-license
            key: license.lic
    clickHouse:
        type: builtin
        builtin:
            image: clickhouse/clickhouse-server:24.8
            persistence:
                size: 50Gi
        databases:
            observability: sandbox0_observability
            metering: sandbox0_metering
        schemaMigration:
            enabled: true
    sandboxObservability:
        enabled: true
        backend: clickhouse
        audit:
            enabled: true
            deliveryMode: durable_async
            deliveryPersistence:
                size: 1Gi
        retention:
            auditDays: 90
            logDays: 7
            runtimeSampleDays: 30
        ingest:
            queueSize: 1024
            batchSize: 100
            flushInterval:
                duration: 1s
            requestTimeout:
                duration: 2s
            maxRetries: 3
            retryBackoff:
                duration: 100ms
    metering:
        enabled: true
        clickHouse:
            eventsTable: usage_events
            windowsTable: usage_windows
            watermarksTable: producer_watermarks
            sandboxStateTable: sandbox_projection_state
            storageStateTable: storage_projection_state

External ClickHouse#

Use an external backend when you already operate ClickHouse.

yaml
spec:
    enterpriseLicense:
        secretRef:
            name: sandbox0-enterprise-license
            key: license.lic
    clickHouse:
        type: external
        external:
            dsnSecret:
                name: sandbox0-clickhouse
                key: dsn
            connectTimeout:
                duration: 10s
        databases:
            observability: sandbox0_observability
            metering: sandbox0_metering
    sandboxObservability:
        enabled: true
        backend: clickhouse
        audit:
            enabled: true
            deliveryMode: durable_async
            deliveryPersistence:
                size: 1Gi
        retention:
            auditDays: 90
            logDays: 7
            runtimeSampleDays: 30
    metering:
        enabled: true

Audit Operations#

Cluster-gateway authorizes audit ingestion and the public event query and watch API. The enterprise license is mounted only into cluster-gateway.

The infra-operator also manages two independent key pairs:

• the active ctld process receives the network-producer private key
• cluster-gateway receives the corresponding producer public key
• cluster-gateway receives the independent audit-signing private key

ClickHouse table sandbox_audit_events is the canonical audit store. PostgreSQL is not a second audit store, and fsync-backed buffers are delivery state rather than query sources.

Delivery Mode#

spec.sandboxObservability.audit.deliveryMode controls admission for non-mutating Sandbox API requests, public exposure requests, and new ctld network flows.

State-changing Sandbox API requests always require canonical acknowledgement before execution and before returning their signed result. See Audit Delivery for failure behavior and caller-visible results.

Audit retention starts at cluster-gateway's ingested_at timestamp, when it creates or normalizes the canonical event. It does not start at the producer's occurred_at time or at ClickHouse acknowledgement. A delayed event can therefore have less than the configured retention period remaining when it becomes query-visible.

Persistence And Keys#

Audit currently requires spec.services.clusterGateway.replicas: 1. Infra-operator uses a ReadWriteOnce event-delivery PVC and the Recreate deployment strategy so a replacement gateway reattaches the same buffer before starting.

Production requirements:

• set audit.deliveryPersistence.storageClass to durable, encrypted storage that can reattach across nodes; omitting it uses the cluster default
• preserve the generated audit-signing Secret for the entire retention period
• retain previous public keys outside the cluster before destructive key replacement; inline verification uses only the active public key and reports older retained keys as signature_status=unavailable
• account for ctld network-flow buffers being node-local: they survive a same-node Pod restart, not permanent node loss
• alert on delivery records that have not reached ClickHouse; records are removed only after exact-event acknowledgement

Schema Compatibility#

Audit schema v2 uses sandbox_audit_events. Older sandbox_events rows lack trusted actor and signature data, so Sandbox0 retains them as legacy history rather than retroactively signing them.

At startup, an audit-enabled cluster-gateway validates the actual ClickHouse engine, partition key, sorting key, and required columns. It refuses audit operations against an old or incompatible custom table. Existing CRs that persisted the old default table value are routed to the canonical v2 table.

Monitoring#

The ctld network runtime exports:

• sandbox0_netd_audit_ingest_events_total
• sandbox0_netd_audit_ingest_batches_total

Alert on persist_failed, corrupt, retrying, replay_failed, and ack_failed results. Also monitor the ClickHouse service, cluster-gateway delivery PVC capacity and latency, and audit query errors.

External Controls#

Sandbox0 provides durable delivery, authenticated producer identity, tenant scoping, append-only application behavior, and tamper-evident row signatures. A production compliance design must also provide:

• highly available ClickHouse storage
• restricted ClickHouse mutation credentials
• encrypted backups and tested restores
• retention and legal-hold policy
• clock synchronization
• independent preservation of signing public keys or exported checkpoints

A per-row signature does not protect against an administrator who can rewrite the complete table and all of its backups.

Disable Or Retain Features#

Disabling audit stops new facts but does not delete existing rows. Set retention.auditDays to your required retention period before enabling audit in production. Legacy spec.sandboxObservability.type fields remain accepted for compatibility, but new installs should configure ClickHouse through spec.clickHouse.