Observability

Self-hosted Sandbox0 has three distinct data paths. Do not merge their storage or authority boundaries.

See Sandbox Observability for the public sandbox query APIs.

Platform Logs And Traces#

Direct systemd services write to the journal; Nomad service jobs write through the configured Nomad log sink. Collect both with your existing node agent. Include stable attributes for service, region, cluster, node ID, durable node UID, and boot ID.

Sandbox0 uses standard OpenTelemetry trace environment variables. Set them in /etc/sandbox0/<service>.env or the equivalent service-job environment:

bash
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otel.example.com:4317
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=Bearer%20<token>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,sandbox0.region.id=region-1,sandbox0.cluster.id=cluster-1
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.25

Keep secrets out of resource attributes. If no trace exporter is configured, services use the no-op path.

Procd writes its ts log timestamps in UTC. This avoids reading the sandbox's local timezone files just to format the first log record on a demand-paged RootFS. It does not change the sandbox's TZ, application timezone, or other services' logging configuration.

Successful Claimed Nomad sandbox logs include claimPhases, an array of phase, durationMicros, and succeeded values from that claim's existing planner observations. ingress_to_planner includes routing and manager work before the planner; procd_probe includes token generation, readiness polling, and proof validation. Use these correlated phases to distinguish control-plane and readiness delays from node-side work. The planner's node_claim includes transport and response validation, while nodeClaimDurationMicros measures node-side processing. RootFS and runsc timings nest inside that node duration; do not add parent and child timings. These logs do not change public readiness timing, publication checks, or request deadlines.

Prometheus Metrics#

Scrape each service's configured metrics port and both ctld HA endpoints. ctld slot A uses port 9192 and slot B uses 9193 in the supplied deployment assets; both remain available before primary election.

High-value claim metrics include:

• manager_runtime_slot_claim_end_to_end_duration_seconds;
• manager_runtime_slot_claim_phase_duration_seconds;
• HTTP request duration/error metrics at regional-gateway, scheduler, cluster-gateway, and manager; and
• the canonical regional-gateway-to-procd SLO timing recorded by the acceptance tool.

Initial RootFS publication uses a per-sandbox row lock for idempotence and deletion fencing. Claims of different sandboxes share a read lock on the ready base artifact, so template reuse does not require an exclusive artifact queue. Treat database errors during this phase as failed claims, even if the remaining guests meet the latency gate; client retries must not hide those failures in a high-density acceptance report.

Monitor the complete capacity conjunction rather than carrier count alone:

• ready resource-neutral slots by compatibility digest;
• ctld-reported free CPU and memory;
• active/retiring resource leases;
• connected authenticated node channels and heartbeat age;
• free NBD devices;
• terminal reconciliation age and retry count; and
• Nomad replacement-allocation lag.

ctld A/B#

An unchanged carrier allocation set does not prove that guests survived a primary restart. The current primary owns the NBD server; its exit disconnects live RootFS devices and shuts down their mounted XFS filesystems. Planned updates require regional claim fencing and durable pause/drain before B/A replacement. Verify resumed sandboxes with actual user commands. After an unplanned loss, recovery uses the last committed RootFS generation.

Group HA alerts by node:

• ctld_ha_primary: alert when sum by (node) (ctld_ha_primary) != 1 persists;
• ctld_ha_role, ctld_ha_epoch, and ctld_ha_role_transitions_total: election/failover history;
• ctld_ha_synchronized and ctld_ha_standbys: takeover readiness;
• ctld_ha_state_duration_seconds: time in current role; and
• ctld_ha_lock_info: both peers on one node must report the same lock identity.

Runtime metric collection health is exposed through:

• ctld_runtime_metric_collection_duration_seconds;
• ctld_runtime_metric_collections_total;
• ctld_runtime_metric_collection_targets; and
• ctld_runtime_metric_samples_total.

Alert when a node has no synchronized standby, when boot identity changes without cleanup progress, when the primary cannot renew writers, or when lease cgroups remain occupied after termination was requested.

Distinguish retained, already-reclaimed external proof history from unresolved writers, physical artifacts, and unreleased capacity. Quiet history is excluded from periodic recovery until its local expiry deadline; a node returning after the regional proof-retention window can expire eligible exact-binding history without an online lookup. Record count alone is not evidence of a capacity leak. See Manager Node Authority for the expiry and physical-absence conditions.

An externally crash-fenced session with a retained branch is not quiet history. Its branch can hold the shared node retirement reserve and delay other sandboxes' cleanup. Periodic recovery checks regional terminal authority without waiting for the old consumer lease or attach grace; artifact deletion still requires exact regional authorization and local physical-absence proof. Normal unfenced consumers remain protected by their durable leases.

Node recovery and regional slot cleanup use the same crash operation for each writer incarnation. If node cleanup takes over an unfinished local crash, it finishes that operation before returning physical cleanup proof. The manager preserves any RootFS retirement proof already committed by the node and still requires allocation, network, mount, and resource-cgroup absence before releasing the lease. Repeated ownership conflicts are a recovery failure, not evidence that more node capacity is needed.

Monitor carrier replacement separately from guest exit and physical cleanup. Nomad can report a system-job stop evaluation complete with zero queued allocations before it has placed the replacement carrier. Terminal reconciliation requests another exact-allocation evaluation after observing terminal catalog state, even if the direct client artifacts are already absent. An unacknowledged request stays in the existing slot reconciliation queue; physical absence must not skip it. This does not restart an old carrier, change the job's desired count, admit a drained node, or replace the writer and local physical-absence proofs.

Distinguish task recovery failure from replacement delay. The driver persists normalized command/security configuration in version 2 task handles because Nomad does not retain its private raw driver configuration across client database persistence. An unsupported persisted driver handle version or persisted driver configuration is missing error rejects recovery; replacing it with default configuration or relaxing allocation/boot identity is unsafe. Plan upgrades to replace legacy carriers through normal lifecycle reconciliation. After a recovery rejection, verify new allocation readiness separately from terminal cleanup. Neither a successful driver recovery nor a refilled pool is evidence that cold claim plus the first real command meets the latency gate.

Track unresolved session age and actual cleanup progress alongside retry logs and regional verification errors, including HTTP 403. Bounded concurrency and backoff can reduce attempts without resolving a writer; a nil reconciliation error or a renewed consumer heartbeat is not recovery progress. Retry state resets after restart and is not an audit record. Neither retry suppression nor local proof deletion demonstrates a new terminal release or an SLO pass.

RootFS And Metering#

Manager exports:

• manager_rootfs_maintenance_runs_total;
• manager_rootfs_maintenance_duration_seconds;
• manager_rootfs_object_deletes_total;
• manager_rootfs_object_deletion_queue_depth;
• manager_rootfs_storage_bytes and manager_rootfs_storage_objects;
• manager_metering_outbox_pending_operations; and
• manager_metering_outbox_oldest_pending_age_seconds.

Also monitor importer leases, materializer backlog bytes, dirty-tail retirement reserve, conditional-publication failures, S3 request rate/status, PostgreSQL writer readiness, database pool saturation, and local branch-disk usage.

The node session owner's shared RootFS read cache admits at most eight distinct mapping/data source loads across its generation Readers, including coalesced bulk reads. Slots are held through checksum verification and cache admission; verified hits and callers sharing an existing flight do not consume another slot. Disabling cached payload storage does not disable this bound. Queued Readers rotate after each grant, so one generation's backlog does not monopolize the node. Admission and context-aware transport use the node I/O lifetime, not the short-lived claim request. This bounds source-load concurrency, not total process memory, all S3 HTTP requests, or wall-clock claim/first-command latency; production-width cold-node and cached-node acceptance remains necessary.

Within the default 128MiB RootFS read-cache budget, recently used decoded mapping pages have up to 16MiB of protection from bulk data eviction. For smaller caches, protection is at most one eighth of their budget. This is not another cache or prewarming: pages enter only after demand reads and validation, and oversized or displaced pages use the ordinary LRU space. Mixed-image data can still exceed the remaining cache capacity and cause remote reads on subsequent claims. Measure new sandboxes with retained node caches separately from empty node caches; neither case alone establishes a production latency guarantee.

The durable importer publishes Format2 RootFS generations, with compressed range addresses in authenticated mappings, decoded checksum verification and shared source admission. Format1 is no longer supported; retained Format1 data must have been migrated before upgrading. Already reclaimed external terminal proofs retain their exact historical descriptor bytes for the existing 48-hour replay window. Their binding and physical cleanup evidence are checked without opening an old-format RootFS; active sessions still require a supported format. Format activation does not establish a startup latency result: measure privileged lifecycle, populated size, mapping depth, and production-width claim plus first-command performance separately.

Incremental RootFS publication and batch materialization now rewrite only affected mapping paths, retaining untouched immutable child locators. Batch members share a bounded verified mapping cache and pack rewritten pages by dependency level. A small edit no longer requires expanding all base extents; compressed subviews also avoid downloading unchanged data merely to split a mapping. These are storage-path work bounds, not a claim latency or aggregate node memory guarantee. Full-image import and production-width populated RootFS tests remain separate acceptance requirements.

Full-image block publication streams mapping pages instead of retaining all nonzero range entries. The live mapping frontier contains at most one fanout of child locators per level; completed page payloads are released. Each sequential builder reuses one compression encoder and closes it at completion, and zero image ranges reuse the input buffer. The immutable-object inventory still grows with distinct published objects, and concurrent import/checkpoint jobs have their own bounded buffers. Monitor total allocation, peak RSS, staging disk, object inventory and job concurrency separately. These implementation bounds do not establish populated-image or production-density startup results.

RootFS object-size audits compare catalog payload bytes with content size before envelope encryption. The generic object-store Head/List sizes still describe physical stored bytes, including encryption overhead. Audits read bounded, fresh encryption-header metadata and do not scan whole object bodies or preload a tenant RootFS. A successful size check does not prove content integrity: normal reads still require frame authentication and expected payload checksums. Missing or unreadable metadata and size mismatches remain audit failures.

Per-Sandbox Historical Data#

Configure cluster-gateway directly with a ClickHouse backend:

yaml
sandbox_observability:
    backend: clickhouse
    audit_enabled: false
    clickhouse:
        dsn: ${SANDBOX0_OBSERVABILITY_CLICKHOUSE_DSN}
        database: sandbox0_observability
        events_table: sandbox_events
        logs_table: sandbox_logs
        runtime_samples_table: sandbox_runtime_samples
        retention_days: 30
        logs_retention_days: 7
        runtime_samples_retention_days: 30
        connect_timeout: 10s

Historical APIs return unavailable when the backend is disabled or unhealthy. They do not fall back to platform logs or PostgreSQL lifecycle tables.

Set skip_schema_migration: true only when an external migration process owns the exact expected ClickHouse schemas. Startup otherwise validates/creates the configured schema.

Metering Projection#

Enable metering on all producer/consumer services that participate in the region and configure the same ClickHouse table names:

yaml
metering:
    enabled: true
    clickhouse:
        dsn: ${SANDBOX0_METERING_CLICKHOUSE_DSN}
        database: sandbox0_metering
        events_table: usage_events
        windows_table: usage_windows
        watermarks_table: producer_watermarks
        sandbox_state_table: sandbox_projection_state
        storage_state_table: storage_projection_state
        connect_timeout: 10s

PostgreSQL metering projection state and projection_outbox are the producer persistence boundary. ClickHouse is rebuilt asynchronously from that outbox. A ClickHouse outage may increase projection lag, but it must not create a second usage truth or cause producers to invent independent counters.

Alert on outbox depth, oldest pending age, producer watermark gaps, duplicate or rejected operations, and ClickHouse insert latency/errors.

Usage-window paging first selects a bounded set of candidate primary keys. ReplacingMergeTree FINAL reads all versions of those keys before applying team, type, and cursor filters. Oversized candidate sets use the original canonical scan. A concurrent replacement fails the page if its canonical version was absent from the candidate snapshot; retries cannot silently advance the cursor past an unseen version. This supports historical version formats as well as current projections.

Signed Audit#

Signed centralized audit is a separately licensed per-sandbox history feature. When enabled, cluster-gateway requires:

• region_id and cluster_id;
• a dedicated network-producer public key;
• a distinct audit-signing private/public key pair;
• a durable sandbox_observability.audit_spool_dir; and
• a compatible ClickHouse events table.

ctld receives only the dedicated producer private key. It must not receive the audit-signing key. Preserve signing public keys for at least the full event retention period.

Delivery modes are:

State-changing sandbox API operations require canonical acknowledgement even when non-mutating operations use durable asynchronous delivery. Spool records are delivery state, not a second query store, and are removed only after exact event acknowledgement.

For durable_async, place the cluster-gateway spool on encrypted durable local storage and choose a failover design that does not discard an unacknowledged spool. Do not put two active writers on the same non-concurrent filesystem. Node-local ctld network spools survive process restart but not permanent disk loss; account for this explicitly in compliance design.

Monitor sandbox0_ctld_network_audit_ingest_events_total and sandbox0_ctld_network_audit_ingest_batches_total, especially persist_failed, corrupt, retrying, replay_failed, and ack_failed results.

Operational Dashboards#

At minimum, build dashboards for:

1. regional request rate, errors, and end-to-end claim latency;
2. node capacity, ready carriers, NBD devices, and resource leases;
3. ctld A/B election, synchronization, writers, and node channels;
4. RootFS import, attach, checkpoint, materializer backlog, S3 cost, and GC;
5. PostgreSQL writer health and transaction contention;
6. terminal cleanup age and unreleased capacity;
7. metering outbox and ClickHouse projection lag; and
8. audit spool and acknowledgement lag when audit is enabled.

Next Steps#