Observability

Sandbox0 provides runtime metrics, historical logs, and a signed audit ledger for each sandbox. These APIs read historical data without waking a paused sandbox.

Per-sandbox historical data is separate from platform telemetry and the metering usage ledger.

Runtime Metrics#

The metrics endpoint returns bounded, server-downsampled series ready for charts. The default query covers the last hour and requests CPU utilization, memory utilization, and network I/O. A query cannot span more than 30 days.

Query Parameters#

Metric Catalog#

Each series contains zero or more continuous segments:

• runtime resets and collection gaps start a new segment; clients must not connect lines across segments
• typed gaps explain missing intervals
• the response reports freshness, the effective step_seconds, and whether the result is partial
• missing or unsupported observations are omitted and never synthesized as zero

typescript
import { Client, SandboxRuntimeMetricName } from "sandbox0";

const client = new Client({ token: process.env.SANDBOX0_TOKEN! });
const metrics = await client.sandbox("sb_abc123").getMetrics({
    startTime: new Date(Date.now() - 60 * 60 * 1000),
    metrics: [
        SandboxRuntimeMetricName.SandboxCpuUtilization,
        SandboxRuntimeMetricName.SandboxMemoryWorkingSet,
        SandboxRuntimeMetricName.SandboxNetworkIo,
    ],
    maxPoints: 120,
});

Historical Logs#

Use this endpoint for retained logs. It is separate from the live output streams exposed by process and session APIs.

Audit Events#

The audit endpoint reads canonical signed facts from ClickHouse. It returns 403 feature_not_licensed when audit is not enabled or the cluster-gateway license does not contain sandbox_audit.

Query Parameters#

Event Model And Integrity#

List and watch responses recompute each returned row's digest and verify its signature. An exact event_id lookup also detects reused IDs:

• one canonical payload returns one row
• conflicting payload hashes return at most two variants and set event_id_conflict=true
• event_id_conflict does not replace signature_status, so callers can distinguish an ID collision from a signature failure

Pagination cursors and watch watermarks are response metadata. They are not repeated inside each signed event.

Audit Delivery#

Gateway operations produce an attempt before authorization or execution and a result after the downstream response or a failed admission decision.

The default durable_async path normally becomes query-visible after immediate delivery wakeups and the one-second flush/replay cycle. It is not a read-after-write guarantee.

Failure behavior is fail-closed at the applicable admission boundary:

A failed canonical mutation admission records a correlated result when result custody succeeds:

If terminal-result custody also fails, the response reports audit_result=unrecorded. Investigate the surviving attempt as an audit-completeness incident, not evidence that the operation ran.

Network Flow Events#

Network audit records the connection lifecycle:

• an allowed flow has one attempt and one result when the flow closes
• a long-lived keep-alive connection therefore has one allow attempt and one eventual flow result
• parsed HTTP or MCP operations may appear in bounded protocol details rather than as separate connections
• raw proxy and credential-resolution errors stay in local diagnostics because they may contain upstream or credential details

Watch Logs Or Events#

Set watch=true on the logs or events endpoint to receive application/x-ndjson in ingestion order.

Watch behavior:

• without cursor or start_time, the stream starts at request time and does not replay history
• end_time is not supported with watch=true
• runtime metrics use bounded time-series queries and do not expose a watch mode

Storage And Coverage#

ClickHouse is the only audit read source. Fsync-backed buffers retain pending events for retry, but buffered events are invisible to list and watch queries until ClickHouse acknowledges them. Logs and runtime samples use separate historical projections with independent retention periods.

Current producers are:

• cluster-gateway for authenticated Sandbox API access, including regional detail requests
• the ctld network runtime for network attempts and results
• manager for historical logs
• node-local ctld collectors for sandbox-wide CRI runtime samples, collected every 15 seconds with jitter

Historical queries return 503 when their backend is disabled or unavailable. The static metric catalog remains available so clients can build selectors without probing storage.

Coverage Limits#

• Metering windows remain separate usage-ledger records for quota, showback, and billing export.
• process.* and file.* facts prove authenticated gateway requests and their HTTP results. They do not observe every process effect, SSH/SFTP command, POSIX write, direct SandboxVolume operation, or manager-initiated reconciliation.
• Public exposure streams and WebSockets record a durable open attempt, but abrupt node or process loss can prevent the eventual close result.
• Cluster-gateway pending events require reattachable durable storage. Node-local network-flow buffers survive same-node Pod restarts, not permanent node loss.
• Per-row signatures detect modified returned facts. They do not prove that a privileged ClickHouse administrator never deleted a row.

Deployments that require completeness across node loss should preserve audit signing public keys and export signed facts or checkpoints to independently controlled immutable storage. See Self-hosted Observability for ClickHouse, retention, delivery persistence, and signing-key configuration.