01Documentation
Pause And Resume
Sandbox0 can pause an idle sandbox without deleting it. Pause checkpoints the writable root filesystem, releases the runtime pod, and keeps the sandbox identity and durable state. Resume creates a new runtime generation and restores the latest rootfs checkpoint.
Use this page when you need to:
- pause a sandbox explicitly
- resume a paused sandbox explicitly
- understand how
ttl,auto_resume, service routes, and SSH interact with pause state - inspect whether a sandbox is running or paused
Pause is a runtime lifecycle operation. It checkpoints the writable rootfs and deletes the runtime pod; it does not freeze cgroups or preserve running processes, memory, sockets, PID state, or live REPL sessions.
Self-hosted deployments must run ctld on sandbox nodes for checkpointed pause/resume. Pause saves the writable rootfs checkpoint and releases the runtime pod; it does not preserve process state.
What Persists#
| State | Across pause/resume | After sandbox delete or hard_ttl |
|---|---|---|
| Sandbox identity and configuration | Preserved | Deleted |
| Writable root filesystem | Latest checkpoint is restored | Deleted with the sandbox identity |
| Processes, memory, sockets, PID state, live REPL sessions | Not preserved | Deleted |
The rootfs checkpoint covers files written inside the sandbox filesystem. It is tied to the sandbox lifecycle. For named rootfs snapshots, restore, and fork operations, see Snapshot And Restore.
On nodes using the containerd overlayfs snapshotter, ctld checkpoints only the active snapshot upperdir and stores it as a standard OCI layer diff. Other snapshotters, or nodes where the overlayfs upperdir is not readable by ctld, fall back to containerd's built-in diff path.
Pause A Sandbox#
Pause a sandbox to release compute while keeping its identity, configuration, services, and latest writable rootfs checkpoint available for a later resume.
The pause request is asynchronous. Poll sandbox details until status is paused and paused is true before treating the checkpoint as resume-ready.
| Situation | Behavior |
|---|---|
| Pause request accepted | The API returns the current committed state while checkpoint upload and runtime pod deletion continue |
| Pause not yet committed | Sandbox details still show the previous committed state, usually running |
| File, context, service, or SSH access during a transition | The request uses the committed runtime generation or waits for the transaction, then continues after resume |
| Runtime access reaches an automatic TTL pause before commit | Sandbox0 cancels the automatic pause and continues on the existing runtime generation |
| Runtime access reaches an explicit pause | The explicit pause commits; access can then resume the sandbox when auto-resume is allowed |
pausing and resuming are not caller-visible states. A lifecycle transaction in progress is not a 409 Conflict; that status is reserved for actual policy or state conflicts such as disabled auto-resume, non-restartable routes, quota limits, deletion, or paused-only operations.
Runtime Failure Recovery#
Sandbox0 uses the durable pause flow to replace a failed runtime.
| Failure or condition | Recovery behavior |
|---|---|
procd exits unexpectedly, including OOM | Manager starts a non-cancelable recovery pause; ctld checkpoints the terminated containerd snapshot, then manager atomically commits the new rootfs head and paused state before deleting the old pod |
| Runtime liveness failure is transient | The existing runtime is not replaced |
| Runtime liveness fails continuously for 90 seconds | Manager starts the same durable pause flow and reconstructs the sandbox with a new runtime_generation |
| Runtime Pod disappears, including while manager is restarting | Manager confirms the absence through the Kubernetes API and reconstructs the same sandbox identity from its last committed rootfs head. A periodic anti-entropy scan covers missed Pod events |
| Kubernetes API is unavailable | Manager waits and retries; API unavailability is never treated as proof that a runtime is missing |
| Runtime cannot produce a checkpoint within the bounded attempt | Recovery keeps the last committed rootfs head and replaces the stuck runtime |
Explicit delete or hard_ttl occurs | Deletion takes precedence over crash recovery |
Recovery is independent of the sandbox auto_resume access policy. A supported runtime request waits for recovery to commit, then resumes when auto-resume is allowed.
Recovering the latest writes requires the terminated containerd snapshot to remain available on the same node. If the old pod or snapshot disappears first, Sandbox0 falls back to the last committed rootfs checkpoint and records degraded recovery. Files written after that checkpoint may be lost.
/api/v1/sandboxes/{id}/pause
go_, err := client.PauseSandbox(ctx, sandbox.ID) if err != nil { log.Fatal(err) } fmt.Println("Pause requested")
Resume A Sandbox#
Resume a sandbox that was paused manually or by TTL expiry.
| Situation | Restore behavior |
|---|---|
| Normal resume | Creates a runtime pod from the current template image and restores the latest committed rootfs checkpoint |
| Restore fails against the current template image | Retries once with the checkpoint's recorded base image digest; kubelet manages that image normally |
| Existing runtime pod is terminal | Waits for crash recovery to commit, then restores the recovered rootfs head |
| Terminated snapshot is unavailable | Restores the last committed checkpoint |
Persistent And Ephemeral Paths#
The writable rootfs persists across pause and resume except for runtime-owned ephemeral paths:
/tmpis recreated from the clean carrier or template state. Files written there do not survive pause and resume./var/tmpremains part of the persistent rootfs.- Other writable rootfs paths, such as
/rootand workspace directories, remain persistent.
This policy is applied while saving checkpoints, restoring checkpoint layers, and capturing incremental baselines. Sandbox0 also filters /tmp from older checkpoint layers during restore. The first restore of a large older checkpoint can still take time because the layer must be read once; the next pause writes a checkpoint without those /tmp contents.
Do not use /tmp for dependency caches, downloaded tools, or other state that must survive runtime replacement. Store that data under a persistent rootfs path.
/api/v1/sandboxes/{id}/resume
go_, err := client.ResumeSandbox(ctx, sandbox.ID) if err != nil { log.Fatal(err) } fmt.Println("Resume requested")
Inspect Pause State#
After requesting a pause or resume, fetch sandbox details to check the lifecycle status.
During asynchronous pause, paused remains false and status continues to reflect the cached runtime Pod state, usually running. It becomes true only after checkpoint upload has completed, the runtime pod has been released, and the sandbox is safe to resume from durable state.
During resume, callers may wait while Sandbox0 creates and initializes a new runtime generation. After the resume transaction commits, status is running, paused is false, and runtime_generation has advanced. If an automatic pause is canceled before commit, status remains running, paused remains false, and runtime_generation does not change.
gosb, err := client.GetSandbox(ctx, sandbox.ID) if err != nil { log.Fatal(err) } fmt.Printf("paused=%v ", sb.Paused) fmt.Printf("status=%s ", sb.Status) fmt.Printf("runtime_generation=%d ", sb.RuntimeGeneration)
TTL And Auto Resume#
ttl and hard_ttl interact with pause differently:
| Field | Behavior |
|---|---|
ttl | Runtime soft timeout. When it expires, Sandbox0 checkpoints the writable rootfs, pauses the sandbox, and releases runtime compute. |
hard_ttl | Sandbox hard timeout. When it expires, Sandbox0 deletes the sandbox identity and durable state, including rootfs checkpoints, even if the sandbox is already paused. |
auto_resume is the sandbox-level gate for whether inbound access can wake a paused sandbox:
- when
auto_resumeistrue, supported access paths can resume the sandbox automatically - when
auto_resumeisfalse, you must call resume explicitly
Common auto-resume entrypoints:
- sandbox runtime APIs such as files, contexts, and supervised sessions
- service routes configured with
resume: true; public service requests require sandboxauto_resume: true, routeresume: true, and a restartable service runtime (cmdorfunction) - SSH access through Sandbox0 SSH Gateway
For a paused sandbox, auto-resume creates a new runtime pod for the same sandbox identity and restores the saved rootfs checkpoint before the sandbox is used. Public service URLs remain stable until the sandbox is explicitly deleted or hard_ttl expires.
Supported runtime access distinguishes an accepted resume from a failed attempt:
503 unavailablewithsandbox is waking upmeans the accepted lifecycle transition has not committed yet; poll sandbox details before retrying the runtime request503 sandbox_resume_failedwithsandbox resume failedmeans that resume attempt ended unsuccessfully; clients may perform a bounded retry, but should not treat the failed attempt as still in progress
Running PIDs, memory, sockets, and client attachments do not survive pause. A supervised session keeps its logical identity and retained journal, then follows its runtime_recovery policy in the new runtime generation.
See Sandbox Services, Sandbox Functions, and SSH for the user-facing flows.
Typical Pattern#
For long-running workflows, a common pattern is:
- claim a sandbox with
ttlset - do active work through contexts and files
- let idle sandboxes pause automatically
- resume on the next user action, SSH session, or service route hit
- use
hard_ttlto cap the maximum lifetime
Next Steps#
Contexts
Run REPL and command contexts inside a sandbox and stream process output.
ContinueSupervised Sessions
Preserve logical process identity and replayable events across reconnects and runtime replacement.
ContinueFiles
Read, write, watch, and manage files inside a sandbox workspace.
ContinueSnapshot And Restore
Create named restore points and fork sandbox rootfs state.
Continue