How We Cut gVisor Metrics CPU by 72% at 424 Sandboxes

Observability should not be the busiest workload on an otherwise quiet sandbox node.

Sandbox0 keeps runtime metrics enabled for every sandbox. CPU, memory, network, process, and writable-rootfs data remain useful even when a sandbox is idle: they power historical charts, capacity decisions, debugging, and usage analysis. Our default collection cadence is 15 seconds.

At a few dozen sandboxes, the collection path looked unremarkable. At more than 400 gVisor sandboxes on an 8-vCPU worker, it became one of the largest consumers of the node.

The surprising part was not the amount of metric data. It was how gVisor obtained it.

The stock containerd shim implemented each stats request by launching:

text
runsc events --stats <container-id>

That runsc process then loaded container state, connected to the already-running gVisor sandbox, called an existing control RPC, serialized the result, and exited. Containerd repeated the sequence for every sandbox container. Recurring kubelet and Sandbox0 metric collection turned a convenient command boundary into sustained fork/exec work.

We changed the shim to call the same sandbox control RPC directly. The CRI contract, metric fields, and collection cadence stayed the same. In a matched production comparison at 424 ready sandboxes, average node CPU fell from 60.68% to 16.70%, while observed runsc events --stats executions fell from 11,254 to zero.

This post covers how we found the boundary, why our first attempt made steady CPU worse, how the direct path preserves gVisor's accounting behavior, and what we learned from carrying a small production fork.

The Metrics Path Was Standards-Based#

Sandbox0 does not use a private per-sandbox status protocol for runtime resource metrics. The node-local ctld service asks containerd for CRI PodSandboxStats, and containerd assembles the response from its pod cgroup, network collection, filesystem data, and task metrics.

For gVisor tasks, the path continued through the gVisor containerd shim:

One node-local bulk request therefore became many task metric requests. In upstream gVisor, Runsc.Stats implemented each one with exec.CommandContext and the runsc events --stats CLI.

The CLI was not computing the metrics in isolation. It eventually called containerManager.Event on the sandbox control socket. That RPC already returned the Sentry's per-container statistics and CPU usage information. The runsc container path then corrected CPU totals against the host cgroup before returning JSON to the shim.

This layering is reasonable for occasional administrative commands. A CLI gives operators one stable entry point, centralizes state loading, and keeps callers from depending on an internal RPC.

The cost model changes when the caller is a long-running shim, the command is issued continuously, and the node contains hundreds of sandboxes.

High Density Turned Process Startup into Steady Load#

The first controlled baseline used a fixed 8-vCPU, 64-GiB ACK worker with 443 ready gVisor sandboxes. With no claims in flight, node CPU averaged 72.80%. CPU p95 and p99 repeatedly reached 100%.

Process tracing showed large waves of short-lived runsc events --stats processes. Each process was individually valid and short-lived. Together, they created:

• process creation and executable startup work
• repeated runsc flag and configuration handling
• repeated state-file discovery and parsing
• bursts of scheduler activity
• extra context switches
• repeated setup before reaching the same sandbox control RPC

The obvious first hypothesis was a thundering herd. Containerd fanned a bulk request out across the node, so hundreds of stats subprocesses could overlap.

That hypothesis was only partly right.

Our First Fix Reduced Concurrency and Increased CPU#

Our first experiment moved the pacing boundary into ctld. Instead of issuing one unfiltered bulk CRI request, it discovered ready sandboxes, filtered unrelated entries, spread per-sandbox stats calls across the 15-second interval, and limited concurrency to four.

The change worked as a concurrency control:

Peak contention dropped. Individual calls completed much faster. Metric continuity also remained good: all 425 active sandboxes had 19 or 20 samples in the final five-minute window.

But the total amount of work did not fall. The stats exec rate was effectively unchanged, and the active ctld process now spent about three quarters of a CPU core continuously coordinating individual requests.

In a 180-second steady window, the paced version averaged 85.66% node CPU at 427 ready sandboxes. Every 30-second window remained between 83.53% and 88.46%. The earlier 443-sandbox bulk baseline had averaged 72.80%.

Those two densities were not a matched A/B benchmark, so we did not use them as the final performance result. They were still decisive diagnostic evidence: pacing changed when the work happened, but it did not eliminate the O(N) process startup work.

We closed the paced collector PR instead of adding more complexity around the wrong boundary. The investigation moved to the gVisor shim stats path.

Remove the Boundary, Keep the Contract#

The useful optimization point was inside the gVisor shim.

The shim was already a long-running process with the container ID, runsc root, request context, and permission to read the runtime state. It did not need a child process merely to reach the sandbox control socket.

The direct path does the following:

1. Validate the container ID before using it in a state-file pattern.
2. Find the existing runsc state file and take its advisory read lock.
3. Read the sandbox ID, process ID, and control socket from that state.
4. Resolve the host cgroup from the sandbox process.
5. Cache the control socket path and cgroup metadata for the sandbox lifetime.
6. Connect to the sandbox control socket.
7. Call containerManager.Event, the same RPC used by runsc events --stats.
8. Apply the same host-cgroup CPU correction as the existing runsc container path.
9. Return the existing runc.Stats shape to containerd.

The cache stores sandbox-lifetime metadata, not metric values. Every request still asks the running Sentry for current statistics. We also create a fresh control connection for the request instead of keeping a potentially stale RPC connection indefinitely.

If the control socket path exceeds the Unix socket address limit, the shim opens it with O_PATH and connects through /proc/self/fd. If the request context is canceled while the RPC is in flight, the shim closes the socket to unblock the call.

After any direct-path failure, the shim invalidates the cached metadata and falls back to the original CLI path. That fallback matters during rollout: an unexpected state layout, missing socket, cgroup error, or mixed runtime condition should cost one subprocess, not an entire missing collection cycle.

The implementation is in sandbox0-ai/gvisor PR #1. It changed three files: 215 lines in the shim implementation, 128 lines of focused tests, and the Bazel dependencies needed by the direct client.

Preserving CPU Accounting Was the Subtle Part#

Calling containerManager.Event directly is not sufficient by itself.

The Sentry reports per-container CPU usage, but gVisor's existing Container.Event path can use the host cgroup total for more accurate sandbox-level accounting. For a sandbox with multiple containers, it apportions the host cgroup usage according to the Sentry's per-container measurements.

The direct shim path mirrors that behavior:

• sum the Sentry usage for running containers
• select the requested container's share
• read total CPU usage from the host cgroup
• scale the selected usage proportionally
• fall back to Sentry usage when host cgroup usage is unavailable
• split cgroup usage evenly if the Sentry reports zero for every container

This is an important systems optimization rule: bypassing an expensive layer is safe only if you identify the semantics that layer was adding.

We wanted to remove process startup, not silently change the CPU numbers exposed through CRI.

Focused tests cover direct-provider selection, CLI fallback, state and socket discovery, and host-cgroup CPU apportionment. During the latest fork synchronization, we also issued 20 consecutive CRI stats requests while tracing for unexpected runsc events processes; all 20 used the direct path.

Matched Production Results#

After the controlled validation passed, we deployed the direct-stats build to the production ACK sandbox node pool and ran a matched comparison.

Both observation windows used:

• the same fixed 8-vCPU, 64-GiB worker
• 424 of 424 sandboxes ready
• 406 active and 18 idle sandboxes
• no pending or deleting sandboxes
• no Kubernetes metrics-server on the node
• Sandbox0 ctld runtime metrics enabled
• a 180-second measurement window

The direct-path process trace ran for 179.43 seconds and observed no recurring stats subprocesses or CLI fallback. The remaining forks, context switches, and CPU usage belong to the rest of the node workload; the optimization was never expected to make the host idle.

The strongest signals were the ones closest to the change:

• stats subprocesses fell to zero
• forks fell by 93.67%
• system CPU fell by 80.34%
• maximum load1 fell from 69.67 to 3.54

At the same time, the existing CRI stats path and metric fields remained available. We did not get the result by turning off observability, lengthening the collection interval, or dropping high-density sandboxes from the sample.

The full production comparison is recorded in the investigation result. These are production observations for one node shape and workload, not a claim that every gVisor deployment will see the same percentage.

Why This Matters for AI Sandbox Infrastructure#

Dense sandbox nodes behave differently from ordinary application nodes.

An agent platform may keep hundreds of small sandboxes ready so a new session can claim one without waiting for image pulls, runtime setup, or dependency initialization. Many of those sandboxes are intentionally quiet between tasks. The platform still needs fresh metrics for debugging, capacity, and user-facing observability.

That combination makes per-sandbox fixed overhead especially expensive.

On an 8-vCPU worker, spending most of the node on metric collection directly competes with:

• hot claim latency
• cold sandbox initialization
• active agent execution
• storage and network runtime work
• the headroom needed to absorb bursts

Removing the stats subprocess boundary returned CPU capacity without weakening the product contract. That is more valuable than optimizing a benchmark endpoint that production does not continuously exercise.

What We Learned#

Measure Work, Not Only Concurrency#

The paced collector made the system look healthier by several latency and concurrency measures. Maximum overlap dropped. Per-call p95 improved dramatically.

Steady CPU got worse because the total rate of expensive operations did not change.

When a system performs O(N) work on a cadence, concurrency limits answer "how many at once?" They do not answer "how much work per interval?" Both measurements matter.

Optimize at the Narrowest Redundant Boundary#

We could have invented a Sandbox0-specific metric endpoint, reduced the metric catalog, or maintained a complex client-side scheduler.

The smaller change was to preserve CRI, containerd, and the existing gVisor RPC, then remove the process that only connected those pieces.

This kept the optimization inside the component that owned the redundant boundary.

Compatibility Paths Make Runtime Changes Deployable#

The direct path is the default, but the CLI remains a fallback. Metadata is invalidated after failures, request cancellation is propagated, and the original response contract is preserved.

Those details are not where the headline performance comes from. They are what made the optimization safe enough to validate on production workers.

A Fork Needs an Exit Condition#

Sandbox0 published the initial production build as sandbox0-release-20260622.0-1, pinned the binaries and digests used by workers, and submitted the change upstream as google/gvisor PR #13711.

The upstream PR remains open. A gVisor contributor noted that a broader solution covering more runsc commands is in development and may eventually make this focused patch unnecessary. That is a good outcome.

Our current sandbox0-release-20260729.0-1 keeps the direct-stats change while synchronizing newer upstream work. The fork README treats runtime patches as an explicit inventory: when an official gVisor release retrieves shim stats without one subprocess per sandbox container and preserves the required accounting, this patch can be removed.

The goal is not to own a permanent gVisor fork. The goal is to make the production behavior correct today and make replacement criteria concrete.

The Small Boundary That Cost 11,254 Processes#

The final code change was not a new metrics system.

It was a shorter path through the system we already had:

text
before: shim -> subprocess -> control RPC
after:  shim -------------> control RPC

At low density, the extra process was easy to miss. At 424 sandboxes, it created 11,254 unnecessary stats subprocesses in one three-minute observation window.

Removing that boundary reduced average node CPU by 72.49%, kept runtime metrics enabled, and left the existing CRI contract intact.

That is the kind of optimization that matters in sandbox infrastructure: not doing less for each agent, but removing platform work the agent never needed.