Observability

Self-hosted Sandbox0 has three distinct data paths. Do not merge their storage or authority boundaries.

See Sandbox Observability for the public sandbox query APIs.

Platform Logs And Traces#

Direct systemd services write to the journal; Nomad service jobs write through the configured Nomad log sink. Collect both with your existing node agent. Include stable attributes for service, region, cluster, node ID, durable node UID, and boot ID.

Sandbox0 uses standard OpenTelemetry trace environment variables. Set them in /etc/sandbox0/<service>.env or the equivalent service-job environment:

bash
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otel.example.com:4317
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=Bearer%20<token>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,sandbox0.region.id=region-1,sandbox0.cluster.id=cluster-1
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.25

Keep secrets out of resource attributes. If no trace exporter is configured, services use the no-op path.

Prometheus Metrics#

Scrape each service's configured metrics port and both ctld HA endpoints. ctld slot A uses port 9192 and slot B uses 9193 in the supplied deployment assets; both remain available before primary election.

High-value claim metrics include:

• manager_runtime_slot_claim_end_to_end_duration_seconds;
• manager_runtime_slot_claim_phase_duration_seconds;
• HTTP request duration/error metrics at regional-gateway, scheduler, cluster-gateway, and manager; and
• the signed regional-gateway-to-procd SLO report produced by the acceptance tool.

Monitor the complete capacity conjunction rather than carrier count alone:

• ready resource-neutral slots by compatibility digest;
• ctld-reported free CPU and memory;
• active/retiring resource leases;
• connected authenticated node channels and heartbeat age;
• free NBD devices;
• terminal reconciliation age and retry count; and
• Nomad replacement-allocation lag.

ctld A/B#

Group HA alerts by node:

• ctld_ha_primary: alert when sum by (node) (ctld_ha_primary) != 1 persists;
• ctld_ha_role, ctld_ha_epoch, and ctld_ha_role_transitions_total: election/failover history;
• ctld_ha_synchronized and ctld_ha_standbys: takeover readiness;
• ctld_ha_state_duration_seconds: time in current role; and
• ctld_ha_lock_info: both peers on one node must report the same lock identity.

Runtime metric collection health is exposed through:

• ctld_runtime_metric_collection_duration_seconds;
• ctld_runtime_metric_collections_total;
• ctld_runtime_metric_collection_targets; and
• ctld_runtime_metric_samples_total.

Alert when a node has no synchronized standby, when boot identity changes without cleanup progress, when the primary cannot renew writers, or when lease cgroups remain occupied after termination was requested.

RootFS And Metering#

Manager exports:

• manager_rootfs_maintenance_runs_total;
• manager_rootfs_maintenance_duration_seconds;
• manager_rootfs_object_deletes_total;
• manager_rootfs_object_deletion_queue_depth;
• manager_rootfs_storage_bytes and manager_rootfs_storage_objects;
• manager_metering_outbox_pending_operations; and
• manager_metering_outbox_oldest_pending_age_seconds.

Also monitor importer leases, materializer backlog bytes, dirty-tail retirement reserve, conditional-publication failures, S3 request rate/status, PostgreSQL writer readiness, database pool saturation, and local branch-disk usage.

Per-Sandbox Historical Data#

Configure cluster-gateway directly with a ClickHouse backend:

yaml
sandbox_observability:
    backend: clickhouse
    audit_enabled: false
    clickhouse:
        dsn: ${SANDBOX0_OBSERVABILITY_CLICKHOUSE_DSN}
        database: sandbox0_observability
        events_table: sandbox_events
        logs_table: sandbox_logs
        runtime_samples_table: sandbox_runtime_samples
        retention_days: 30
        logs_retention_days: 7
        runtime_samples_retention_days: 30
        connect_timeout: 10s

Historical APIs return unavailable when the backend is disabled or unhealthy. They do not fall back to platform logs or PostgreSQL lifecycle tables.

Set skip_schema_migration: true only when an external migration process owns the exact expected ClickHouse schemas. Startup otherwise validates/creates the configured schema.

Metering Projection#

Enable metering on all producer/consumer services that participate in the region and configure the same ClickHouse table names:

yaml
metering:
    enabled: true
    clickhouse:
        dsn: ${SANDBOX0_METERING_CLICKHOUSE_DSN}
        database: sandbox0_metering
        events_table: usage_events
        windows_table: usage_windows
        watermarks_table: producer_watermarks
        sandbox_state_table: sandbox_projection_state
        storage_state_table: storage_projection_state
        connect_timeout: 10s

PostgreSQL metering projection state and projection_outbox are the producer persistence boundary. ClickHouse is rebuilt asynchronously from that outbox. A ClickHouse outage may increase projection lag, but it must not create a second usage truth or cause producers to invent independent counters.

Alert on outbox depth, oldest pending age, producer watermark gaps, duplicate or rejected operations, and ClickHouse insert latency/errors.

Signed Audit#

Signed centralized audit is a separately licensed per-sandbox history feature. When enabled, cluster-gateway requires:

• region_id and cluster_id;
• a dedicated network-producer public key;
• a distinct audit-signing private/public key pair;
• a durable sandbox_observability.audit_spool_dir; and
• a compatible ClickHouse events table.

ctld receives only the dedicated producer private key. It must not receive the audit-signing key. Preserve signing public keys for at least the full event retention period.

Delivery modes are:

State-changing sandbox API operations require canonical acknowledgement even when non-mutating operations use durable asynchronous delivery. Spool records are delivery state, not a second query store, and are removed only after exact event acknowledgement.

For durable_async, place the cluster-gateway spool on encrypted durable local storage and choose a failover design that does not discard an unacknowledged spool. Do not put two active writers on the same non-concurrent filesystem. Node-local ctld network spools survive process restart but not permanent disk loss; account for this explicitly in compliance design.

Monitor sandbox0_ctld_network_audit_ingest_events_total and sandbox0_ctld_network_audit_ingest_batches_total, especially persist_failed, corrupt, retrying, replay_failed, and ack_failed results.

Operational Dashboards#

At minimum, build dashboards for:

1. regional request rate, errors, and end-to-end claim latency;
2. node capacity, ready carriers, NBD devices, and resource leases;
3. ctld A/B election, synchronization, writers, and node channels;
4. RootFS import, attach, checkpoint, materializer backlog, S3 cost, and GC;
5. PostgreSQL writer health and transaction contention;
6. terminal cleanup age and unreleased capacity;
7. metering outbox and ClickHouse projection lag; and
8. audit spool and acknowledgement lag when audit is enabled.

Next Steps#