# Observability Self-hosted Sandbox0 has three distinct data paths. Do not merge their storage or authority boundaries. | Path | Purpose | Authority/destination | | --- | --- | --- | | Platform telemetry | Operate services and nodes | Logs, Prometheus metrics, OpenTelemetry traces | | Per-sandbox history | Query runtime samples, process logs, and signed audit facts | ClickHouse read model | | Metering | Usage truth, quota, showback, and export | PostgreSQL producer state/outbox projected to ClickHouse | See [Sandbox Observability](/docs/sandbox/observability) for the public sandbox query APIs. ## Platform Logs And Traces Direct systemd services write to the journal; Nomad service jobs write through the configured Nomad log sink. Collect both with your existing node agent. Include stable attributes for service, region, cluster, node ID, durable node UID, and boot ID. Sandbox0 uses standard OpenTelemetry trace environment variables. Set them in `/etc/sandbox0/.env` or the equivalent service-job environment: ```sh OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otel.example.com:4317 OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=Bearer%20 OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,sandbox0.region.id=region-1,sandbox0.cluster.id=cluster-1 OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.25 ``` Keep secrets out of resource attributes. If no trace exporter is configured, services use the no-op path. Procd writes its `ts` log timestamps in UTC. This avoids reading the sandbox's local timezone files just to format the first log record on a demand-paged RootFS. It does not change the sandbox's `TZ`, application timezone, or other services' logging configuration. Successful `Claimed Nomad sandbox` logs include `claimPhases`, an array of `phase`, `durationMicros`, and `succeeded` values from that claim's existing planner observations. `ingress_to_planner` includes routing and manager work before the planner; `procd_probe` includes token generation, readiness polling, and proof validation. Use these correlated phases to distinguish control-plane and readiness delays from node-side work. The planner's `node_claim` includes transport and response validation, while `nodeClaimDurationMicros` measures node-side processing. RootFS and runsc timings nest inside that node duration; do not add parent and child timings. These logs do not change public readiness timing, publication checks, or request deadlines. ## Prometheus Metrics Scrape each service's configured metrics port and both ctld HA endpoints. ctld slot A uses port `9192` and slot B uses `9193` in the supplied deployment assets; both remain available before primary election. High-value claim metrics include: - `manager_runtime_slot_claim_end_to_end_duration_seconds`; - `manager_runtime_slot_claim_phase_duration_seconds`; - HTTP request duration/error metrics at regional-gateway, scheduler, cluster-gateway, and manager; and - the canonical regional-gateway-to-procd SLO timing recorded by the acceptance tool. Initial RootFS publication uses a per-sandbox row lock for idempotence and deletion fencing. Claims of different sandboxes share a read lock on the ready base artifact, so template reuse does not require an exclusive artifact queue. Treat database errors during this phase as failed claims, even if the remaining guests meet the latency gate; client retries must not hide those failures in a high-density acceptance report. Monitor the complete capacity conjunction rather than carrier count alone: - ready resource-neutral slots by compatibility digest; - ctld-reported free CPU and memory; - active/retiring resource leases; - connected authenticated node channels and heartbeat age; - free NBD devices; - terminal reconciliation age and retry count; and - Nomad replacement-allocation lag. ### ctld A/B An unchanged carrier allocation set does not prove that guests survived a primary restart. The current primary owns the NBD server; its exit disconnects live RootFS devices and shuts down their mounted XFS filesystems. Planned updates require regional claim fencing and durable pause/drain before B/A replacement. Verify resumed sandboxes with actual user commands. After an unplanned loss, recovery uses the last committed RootFS generation. Group HA alerts by node: - `ctld_ha_primary`: alert when `sum by (node) (ctld_ha_primary) != 1` persists; - `ctld_ha_role`, `ctld_ha_epoch`, and `ctld_ha_role_transitions_total`: election/failover history; - `ctld_ha_synchronized` and `ctld_ha_standbys`: takeover readiness; - `ctld_ha_state_duration_seconds`: time in current role; and - `ctld_ha_lock_info`: both peers on one node must report the same lock identity. Runtime metric collection health is exposed through: - `ctld_runtime_metric_collection_duration_seconds`; - `ctld_runtime_metric_collections_total`; - `ctld_runtime_metric_collection_targets`; and - `ctld_runtime_metric_samples_total`. Alert when a node has no synchronized standby, when boot identity changes without cleanup progress, when the primary cannot renew writers, or when lease cgroups remain occupied after termination was requested. Distinguish retained, already-reclaimed external proof history from unresolved writers, physical artifacts, and unreleased capacity. Quiet history is excluded from periodic recovery until its local expiry deadline; a node returning after the regional proof-retention window can expire eligible exact-binding history without an online lookup. Record count alone is not evidence of a capacity leak. See [Manager Node Authority](/docs/self-hosted/configuration#manager-node-authority) for the expiry and physical-absence conditions. An externally crash-fenced session with a retained branch is not quiet history. Its branch can hold the shared node retirement reserve and delay other sandboxes' cleanup. Periodic recovery checks regional terminal authority without waiting for the old consumer lease or attach grace; artifact deletion still requires exact regional authorization and local physical-absence proof. Normal unfenced consumers remain protected by their durable leases. Node recovery and regional slot cleanup use the same crash operation for each writer incarnation. If node cleanup takes over an unfinished local crash, it finishes that operation before returning physical cleanup proof. The manager preserves any RootFS retirement proof already committed by the node and still requires allocation, network, mount, and resource-cgroup absence before releasing the lease. Repeated ownership conflicts are a recovery failure, not evidence that more node capacity is needed. Monitor carrier replacement separately from guest exit and physical cleanup. Nomad can report a system-job stop evaluation complete with zero queued allocations before it has placed the replacement carrier. Terminal reconciliation requests another exact-allocation evaluation after observing terminal catalog state, even if the direct client artifacts are already absent. An unacknowledged request stays in the existing slot reconciliation queue; physical absence must not skip it. This does not restart an old carrier, change the job's desired count, admit a drained node, or replace the writer and local physical-absence proofs. Distinguish task recovery failure from replacement delay. The driver persists normalized command/security configuration in version 2 task handles because Nomad does not retain its private raw driver configuration across client database persistence. An `unsupported persisted driver handle version` or `persisted driver configuration is missing` error rejects recovery; replacing it with default configuration or relaxing allocation/boot identity is unsafe. Plan upgrades to replace legacy carriers through normal lifecycle reconciliation. After a recovery rejection, verify new allocation readiness separately from terminal cleanup. Neither a successful driver recovery nor a refilled pool is evidence that cold claim plus the first real command meets the latency gate. Track unresolved session age and actual cleanup progress alongside retry logs and regional verification errors, including HTTP 403. Bounded concurrency and backoff can reduce attempts without resolving a writer; a nil reconciliation error or a renewed consumer heartbeat is not recovery progress. Retry state resets after restart and is not an audit record. Neither retry suppression nor local proof deletion demonstrates a new terminal release or an SLO pass. ### RootFS And Metering Manager exports: - `manager_rootfs_maintenance_runs_total`; - `manager_rootfs_maintenance_duration_seconds`; - `manager_rootfs_object_deletes_total`; - `manager_rootfs_object_deletion_queue_depth`; - `manager_rootfs_storage_bytes` and `manager_rootfs_storage_objects`; - `manager_metering_outbox_pending_operations`; and - `manager_metering_outbox_oldest_pending_age_seconds`. Also monitor importer leases, materializer backlog bytes, dirty-tail retirement reserve, conditional-publication failures, S3 request rate/status, PostgreSQL writer readiness, database pool saturation, and local branch-disk usage. The node session owner's shared RootFS read cache admits at most eight distinct mapping/data source loads across its generation Readers, including coalesced bulk reads. Slots are held through checksum verification and cache admission; verified hits and callers sharing an existing flight do not consume another slot. Disabling cached payload storage does not disable this bound. Queued Readers rotate after each grant, so one generation's backlog does not monopolize the node. Admission and context-aware transport use the node I/O lifetime, not the short-lived claim request. This bounds source-load concurrency, not total process memory, all S3 HTTP requests, or wall-clock claim/first-command latency; production-width cold-node and cached-node acceptance remains necessary. Within the default 128MiB RootFS read-cache budget, recently used decoded mapping pages have up to 16MiB of protection from bulk data eviction. For smaller caches, protection is at most one eighth of their budget. This is not another cache or prewarming: pages enter only after demand reads and validation, and oversized or displaced pages use the ordinary LRU space. Mixed-image data can still exceed the remaining cache capacity and cause remote reads on subsequent claims. Measure new sandboxes with retained node caches separately from empty node caches; neither case alone establishes a production latency guarantee. The durable importer publishes Format2 RootFS generations, with compressed range addresses in authenticated mappings, decoded checksum verification and shared source admission. Format1 is no longer supported; retained Format1 data must have been migrated before upgrading. Already reclaimed external terminal proofs retain their exact historical descriptor bytes for the existing 48-hour replay window. Their binding and physical cleanup evidence are checked without opening an old-format RootFS; active sessions still require a supported format. Format activation does not establish a startup latency result: measure privileged lifecycle, populated size, mapping depth, and production-width claim plus first-command performance separately. Incremental RootFS publication and batch materialization now rewrite only affected mapping paths, retaining untouched immutable child locators. Batch members share a bounded verified mapping cache and pack rewritten pages by dependency level. A small edit no longer requires expanding all base extents; compressed subviews also avoid downloading unchanged data merely to split a mapping. These are storage-path work bounds, not a claim latency or aggregate node memory guarantee. Full-image import and production-width populated RootFS tests remain separate acceptance requirements. Full-image block publication streams mapping pages instead of retaining all nonzero range entries. The live mapping frontier contains at most one fanout of child locators per level; completed page payloads are released. Each sequential builder reuses one compression encoder and closes it at completion, and zero image ranges reuse the input buffer. The immutable-object inventory still grows with distinct published objects, and concurrent import/checkpoint jobs have their own bounded buffers. Monitor total allocation, peak RSS, staging disk, object inventory and job concurrency separately. These implementation bounds do not establish populated-image or production-density startup results. RootFS object-size audits compare catalog payload bytes with content size before envelope encryption. The generic object-store `Head`/`List` sizes still describe physical stored bytes, including encryption overhead. Audits read bounded, fresh encryption-header metadata and do not scan whole object bodies or preload a tenant RootFS. A successful size check does not prove content integrity: normal reads still require frame authentication and expected payload checksums. Missing or unreadable metadata and size mismatches remain audit failures. ## Per-Sandbox Historical Data Configure cluster-gateway directly with a ClickHouse backend: ```yaml sandbox_observability: backend: clickhouse audit_enabled: false clickhouse: dsn: ${SANDBOX0_OBSERVABILITY_CLICKHOUSE_DSN} database: sandbox0_observability events_table: sandbox_events logs_table: sandbox_logs runtime_samples_table: sandbox_runtime_samples retention_days: 30 logs_retention_days: 7 runtime_samples_retention_days: 30 connect_timeout: 10s ``` Historical APIs return unavailable when the backend is disabled or unhealthy. They do not fall back to platform logs or PostgreSQL lifecycle tables. Set `skip_schema_migration: true` only when an external migration process owns the exact expected ClickHouse schemas. Startup otherwise validates/creates the configured schema. ## Metering Projection Enable metering on all producer/consumer services that participate in the region and configure the same ClickHouse table names: ```yaml metering: enabled: true clickhouse: dsn: ${SANDBOX0_METERING_CLICKHOUSE_DSN} database: sandbox0_metering events_table: usage_events windows_table: usage_windows watermarks_table: producer_watermarks sandbox_state_table: sandbox_projection_state storage_state_table: storage_projection_state connect_timeout: 10s ``` PostgreSQL `metering` projection state and `projection_outbox` are the producer persistence boundary. ClickHouse is rebuilt asynchronously from that outbox. A ClickHouse outage may increase projection lag, but it must not create a second usage truth or cause producers to invent independent counters. Alert on outbox depth, oldest pending age, producer watermark gaps, duplicate or rejected operations, and ClickHouse insert latency/errors. Usage-window paging first selects a bounded set of candidate primary keys. `ReplacingMergeTree FINAL` reads all versions of those keys before applying team, type, and cursor filters. Oversized candidate sets use the original canonical scan. A concurrent replacement fails the page if its canonical version was absent from the candidate snapshot; retries cannot silently advance the cursor past an unseen version. This supports historical version formats as well as current projections. ## Signed Audit Signed centralized audit is a separately licensed per-sandbox history feature. When enabled, cluster-gateway requires: - `region_id` and `cluster_id`; - a dedicated network-producer public key; - a distinct audit-signing private/public key pair; - a durable `sandbox_observability.audit_spool_dir`; and - a compatible ClickHouse events table. ctld receives only the dedicated producer private key. It must not receive the audit-signing key. Preserve signing public keys for at least the full event retention period. Delivery modes are: | Mode | Admission point | Tradeoff | | --- | --- | --- | | `durable_async` | fsync-backed local spool enqueue | Lower ClickHouse coupling; eventual query visibility | | `canonical_sync` | ClickHouse acknowledgement | Immediate query authority; added latency and availability coupling | State-changing sandbox API operations require canonical acknowledgement even when non-mutating operations use durable asynchronous delivery. Spool records are delivery state, not a second query store, and are removed only after exact event acknowledgement. For `durable_async`, place the cluster-gateway spool on encrypted durable local storage and choose a failover design that does not discard an unacknowledged spool. Do not put two active writers on the same non-concurrent filesystem. Node-local ctld network spools survive process restart but not permanent disk loss; account for this explicitly in compliance design. Monitor `sandbox0_ctld_network_audit_ingest_events_total` and `sandbox0_ctld_network_audit_ingest_batches_total`, especially `persist_failed`, `corrupt`, `retrying`, `replay_failed`, and `ack_failed` results. ## Operational Dashboards At minimum, build dashboards for: 1. regional request rate, errors, and end-to-end claim latency; 2. node capacity, ready carriers, NBD devices, and resource leases; 3. ctld A/B election, synchronization, writers, and node channels; 4. RootFS import, attach, checkpoint, materializer backlog, S3 cost, and GC; 5. PostgreSQL writer health and transaction contention; 6. terminal cleanup age and unreleased capacity; 7. metering outbox and ClickHouse projection lag; and 8. audit spool and acknowledgement lag when audit is enabled. ## Next Steps Query runtime metrics, logs, and audit events through the public API.