Skip to documentation
API + guides

01Documentation

Pause And Resume

Sandbox0 can release runtime compute without deleting a sandbox. By default, pause publishes the writable block-COW RootFS head and retires the current runtime allocation. Resume claims a fresh resource-neutral carrier and starts a new runtime generation from the latest committed head.

ℹ

The default filesystem-only pause preserves files, identity, and durable configuration. It does not preserve processes, memory, sockets, PIDs, open file descriptors, or live REPL connections.

Experimental memory pause/resume explicitly opts into preserving supported process state with memory: true. It reuses the execution capture and restore machinery of system maintenance migration, but retains a checkpoint while no runtime is allocated. Explicit calls without memory: true, TTL pauses, and billing pauses keep their filesystem-only defaults. Supported automatic wake-up restores retained memory when available.

While paused, the Files API can read and inspect persistent files without resuming compute. Filesystem-only pauses also allow writes, directory creation, moves, and deletion through the same API and SDK methods. Memory-paused sandboxes allow offline reads only; file edits cannot invalidate or discard the retained process checkpoint.

Default Filesystem-Only Behavior#

StateAcross a successful pause/resumeAfter delete or hard_ttl
Sandbox ID and configurationPreservedDeleted
Writable RootFSLatest committed generation is restoredDeleted with the sandbox
Named RootFS snapshotsPreserved independently until deleted/expiredFollow snapshot retention
Processes, memory, sockets, and live sessionsNot preservedDeleted
runtime_idReplacedRemoved
runtime_generationIncremented on resumeRemoved

The writable RootFS is a block-COW filesystem. Ordinary RootFS paths such as /var/tmp and workspace directories persist after a successful checkpoint. /tmp is a runtime tmpfs and is recreated on pause/resume, as are virtual mounts such as /proc, /dev, and /sys. Template-defined ephemeralMounts are also excluded from filesystem-only pause/resume, RootFS snapshots, and filesystem-only forks. Compatible memory checkpoints can retain runtime tmpfs state as described below. Do not store durable data under any of these runtime-only mounts.

For named restore points, fork, and rebase, see Snapshot And Restore.

Explicit Runtime Restart#

POST /api/v1/sandboxes/{id}/restart records a durable request to end the current processes and start a fresh runtime from the externally published writable RootFS. This recovery entry point does not execute commands inside the sandbox. It discards process state, including a retained memory checkpoint. Sandbox0 never selects this operation as a fallback from memory resume. Applications may adopt their own automatic cold-recovery policy by submitting a separate, generation-fenced restart after confirming a terminal memory failure and physical cleanup. Such a policy loses retained processes and memory; only regionally committed files are available to the replacement. A timeout or an in-progress restore is not proof of failure and must not trigger a competing restart.

Send a stable Idempotency-Key header and a JSON body containing the exact observed expected_runtime_generation. For example, a source at generation 7 uses {"expected_runtime_generation": 7}. The key accepts 1–255 ASCII letters, digits, dots, underscores, colons or hyphens. Keep both values unchanged when retrying an uncertain submission. A changed source requires a new explicit request; retrying with a fresh generation can otherwise restart the wrong runtime.

The response is an operation receipt: 202 means accepted or still pending, while 200 on a repeated submission returns its terminal result. Acceptance is not completion. Poll GET /api/v1/sandboxes/{id}/restart-operations/{operation_id}. If the submission response was lost, recover the receipt with GET /api/v1/sandboxes/{id}/restart-operations?request_key={key}. Both reads require sandbox:read; submission requires sandbox:write and usage admission. Receipt reads remain available when the team is restricted from starting more compute. A 404 lookup means no matching receipt is visible yet; it does not prove a concurrent submission cannot commit. Retrying uses the same key and generation.

The operation moves through pausing, paused, and resuming as needed. Only succeeded proves a committed replacement, with result_runtime_generation. failed and superseded are terminal and cannot be replayed into another restart. Pending operations can include a timestamped retry observation with a fixed blocked_reason, such as retirement_pending, quota_exceeded or capacity_unavailable. An old-stage observation is not a current blocker. Timeouts do not cancel committed work; the manager resumes pending operations from PostgreSQL after restart. While retained, a successful receipt remains a historical result even if the sandbox later changes or is deleted. It does not assert terminal, execution, or application health.

External publication must succeed before replacement proceeds, and the old runtime must lose writer authority before the new one starts. A restart cannot recreate workload-deleted files or promise writes that were never committed. Use a named snapshot for an intentional file rollback.

Explicit Memory Pause And Resume#

This experimental mode is still undergoing failure and capacity validation. Use it only with a deployment that supports memory checkpoints. Both actions accept an optional JSON object:

json
{ "memory": true }

Omitting the body, using an empty object, or setting memory: false keeps the filesystem-only behavior. Choose memory on both pause and resume. An explicit memory resume requires a retained checkpoint matching the committed RootFS generation. Missing, incompatible, or failed memory restoration remains an error; it never starts filesystem-only replacement processes. A filesystem-only pause cannot later recover discarded process memory. Memory capture failures still report an error; pause never claims to have retained memory it did not save.

A successful memory resume waits for the regional commit, the node's durable adoption acknowledgement, and an access handshake for that exact restored runtime. Background reconciliation still completes interrupted delivery. If the request times out or is canceled after commit, retry memory resume for the same sandbox; the retry checks the committed runtime instead of restoring a second copy. Runtime access readiness does not guarantee that each application inside the sandbox has finished its own startup.

A successful memory pause retains supported guest processes, memory, PIDs, open files and runtime tmpfs state, together with the matching RootFS checkpoint. Resume restores that execution image into a fresh compatible carrier. The sandbox ID stays the same; runtime_id changes and runtime_generation advances. Existing SSH, streaming and external TCP connections are not guaranteed to survive. Reconnect clients after resume. A named RootFS snapshot still contains filesystem state only.

python
client.sandboxes.pause_and_wait( sandbox.id, memory=True, timeout_sec=360, ) client.sandboxes.resume_and_wait( sandbox.id, memory=True, timeout_sec=360, )
typescript
await client.sandboxes.pauseAndWait(sandbox.id, { memory: true, timeoutMs: 360_000, }); await client.sandboxes.resumeAndWait(sandbox.id, { memory: true, timeoutMs: 360_000, });
go
options := &sandbox0.SandboxLifecycleWaitOptions{ Memory: true, Timeout: 6 * time.Minute, } if _, err := client.PauseSandboxAndWait(ctx, sandbox.ID, options); err != nil { return err } if _, err := client.ResumeSandboxAndWait(ctx, sandbox.ID, options); err != nil { return err }

Memory checkpoint cost depends on the live working set, image transfer, storage bandwidth and restore compatibility. A warm carrier does not remove these costs; the normal claim latency target is not a memory restore guarantee. The example wait budget is a caller setting, not a completion-time guarantee. Capture uses a bounded regional upload reservation shared with migration. If that temporary budget is full, preparation retries until its deadline, then cancels without capture. An image larger than the supported reservation limit fails at admission. After pause commits and the source is physically terminal, the temporary budget is released while the retained image stays available to its owners. Neither case silently switches to filesystem-only pause. Before capture is authorized, expired preparation is canceled through the exact source protocol: API admission reopens and unused image staging is released. The existing process, writer and resource lease stay owned by the running sandbox. Once capture is authorized, this cancellation path is forbidden; an uncertain capture requires physical failure resolution. It never retries capture or reports successful memory retention. After exact source cleanup, the last committed RootFS remains available. If no retained image exists, an explicit memory resume returns an error. Starting fresh processes requires a separate filesystem-only resume or restart; it cannot recover the missing process state. The public status is failed even though paused remains true for filesystem recovery. Memory pause wait helpers raise SandboxLifecycleFailedError instead of treating that recovery state as a successful pause. Deletion records termination intent and waits for this cleanup or preparation cancellation before reclaiming the source's resources.

After restore execution is authorized, a failed target is stopped and physically reclaimed before its resource lease is released. The failed attempt consumes its runtime generation while retaining the paused checkpoint and committed disk head. The memory operation remains a memory operation after failure. Recovery retains the checkpoint and committed RootFS; starting fresh processes requires a separate explicit filesystem-only restart. Accepted scheduling intent survives manager restarts, and no replacement can start until the failed target, writer and resource lease are physically reclaimed. If restoring the parent of a running memory fork fails, retrying the fork reports that failure instead of recapturing the parent or creating another child. An accepted in-progress restore can remain starting; a physically resolved memory failure is reported as failed. Never interpret a paused recovery head as successful memory restoration or silently launch a cold replacement.

An HTTP timeout or client disconnection does not cancel an accepted durable operation. Inspect the sandbox and retry the same memory mode; never retry with memory: false as an error-recovery shortcut.

Memory capture must match the destination's runsc/platform/CPU and immutable runtime configuration. Incompatible runtime or workload changes return an error without starting filesystem-only replacement processes. Upgrading the platform procd alone does not discard otherwise compatible memory; restored processes keep their captured daemon. Template upgrades do not replace the sandbox's immutable committed RootFS head. If webhooks are configured, capture requires a durable webhook outbox; in-flight delivery and its acknowledgement drain before capture. Restored sessions retain their captured process identity rather than starting a new recovery attempt.

With auto_resume: true, supported inbound access restores a retained memory checkpoint when the paused sandbox has one. If no memory checkpoint is retained, automatic wake-up uses filesystem-only resume. A failed or incompatible retained memory restore remains an error and never switches to fresh processes. An explicit POST /resume without memory: true retains its filesystem-only default.

Conditional Memory Pause#

For a durable caller intent, submit POST /api/v1/sandboxes/{id}/pause with Idempotency-Key: pause-7-1 and {"memory": true, "expected_runtime_generation": 7}. The generation must be a positive safe integer. Keys accept 1–255 ASCII letters, digits, dots, underscores, colons or hyphens. Both fields and memory: true are required together.

The operation is bound to the authenticated team, user, sandbox, key and exact source generation. Preserve those coordinates on every retry, including after an uncertain reply or manager restart. The response includes operation_id and runtime_generation. A 202 with paused: false means checkpoint work remains; a 200 with paused: true confirms this operation's retained checkpoint. Repeat the same conditional request to observe completion. A generic paused status does not identify which request completed.

Another pending or retained pause, an aborted operation, or a changed runtime generation returns a conflict. The old request cannot pause a resumed successor. A new attempt after safe cancellation needs a new explicit key. Missing support returns an error without dropping the conditions or selecting disk-only pause.

The SDK entry points are sandboxes.pauseMemoryForOperation (TypeScript), sandboxes.pause_memory_for_operation (Python), and PauseSandboxMemoryForOperation (Go). These return acceptance or completion and do not automatically retry, refresh the generation, or poll generic sandbox state. The existing pause/resume helpers retain their previous behavior. Applications must separately qualify their credential and process restoration path.

Conditional Memory Resume#

Submit POST /api/v1/sandboxes/{id}/resume with a separate stable Idempotency-Key: resume-7-1 and all three body fields:

json
{ "memory": true, "expected_runtime_generation": 7, "checkpoint_id": "capture-operation-id" }

Use the actual operation_id from the completed conditional pause as checkpoint_id. It must identify the retained image owned by this paused sandbox. A never-run memory fork can have source generation zero. The source must have a safe integer successor; checkpoint IDs accept 1–256 visible ASCII characters and request keys use the same character set as conditional pause.

Persist the source generation, checkpoint ID, request key and authenticated identity before submission. Repeat those exact inputs after an uncertain reply; the accepted restore can continue in the background after a timeout. A successful 200 response includes operation_id for this caller's resume request, checkpoint_id, from_runtime_generation and runtime_generation. The result generation is exactly source plus one. The resume operation is distinct from the capture operation and the internal worker lifecycle ID. These coordinates prove the requested restore committed, not current application or storage health.

The request-to-lifecycle binding survives manager restart and checkpoint-reference consumption. A generic running runtime cannot satisfy it. A different request, checkpoint, source generation or later lifecycle conflicts. An aborted request cannot silently begin a new restore attempt. After safe terminal cleanup, a new explicit attempt requires a new key and the exact then-current paused generation and retained checkpoint. No path falls back to filesystem-only processes.

SDK methods are sandboxes.resumeMemoryForOperation (TypeScript), sandboxes.resume_memory_for_operation (Python), and ResumeSandboxMemoryForOperation (Go). They validate the conditional response and never retry automatically, refresh generation or poll generic running state. The legacy resume methods and automatic wake-up mode selection remain available.

Pause And Resume As RootFS Grows#

Long-running agents accumulate dependencies, code, and data in their writable RootFS. Sandbox0 separates that durable filesystem from runtime compute so later pause/resume operations do not need to copy the complete accumulated workspace.

Pause seals and publishes the writable branch's changes, reusing unchanged blocks from the committed generation. Resume attaches the latest committed generation and fetches stored blocks on demand as the runtime reads them, without downloading the entire RootFS before execution starts.

The design avoids work proportional to total RootFS bytes on every pause and resume. It does not guarantee fixed latency: unpublished writes, metadata, cache state, object-store access, and the runtime's read working set still affect the operation. Explicit memory checkpoint and restore additionally depend on the size of retained execution state.

Filesystem Pause Transaction#

POST

/api/v1/sandboxes/{id}/pause

bash
s0 sandbox pause sb_abc123

Pause is asynchronous:

  1. manager records planned retirement for the exact sandbox generation, allocation, slot, writer grant, and resource lease;
  2. manager requests an idempotent stop for the exact Nomad allocation;
  3. the elected ctld primary fences guest writes, syncs and seals the local block branch, and publishes immutable generation descriptors;
  4. PostgreSQL atomically advances the RootFS head and commits the sandbox as paused; and
  5. the plugin-independent terminal reconciler proves runsc, writer, mount, network state, allocation directory, and lease cgroup absent before returning node capacity.

The initial response can report starting/paused: false while this work continues. Poll sandbox details until status is paused and paused is true before treating the checkpoint as resume-ready.

A repeated pause request also reports paused: false until the previous runtime slot has completed physical cleanup. A committed RootFS head alone does not make the sandbox ready to resume.

After the pause intent is durably recorded, a temporary node-channel outage or completion deadline still returns 202 Accepted; the pause controller retries the same operation. This response does not confirm a saved checkpoint. Failure to record the initial intent still returns an error.

bash
s0 sandbox get sb_abc123

pausing is an internal lifecycle transaction, not a public status. Repeating the pause request is safe: retries recover the same durable operation instead of creating a second RootFS publication.

If S3 or PostgreSQL is unavailable after local seal, Sandbox0 keeps the exact retirement retryable and keeps its capacity fenced. It does not silently mark the sandbox paused or release a writer before the regional transaction commits.

Filesystem Resume Transaction#

POST

/api/v1/sandboxes/{id}/resume

bash
s0 sandbox resume sb_abc123

Resume:

  1. locks the paused sandbox and reserves the next runtime generation;
  2. resolves the same attested image/platform and latest committed RootFS head;
  3. atomically leases a compatible ready carrier and exact node CPU/memory;
  4. asks ctld to attach RootFS, prepare the lease cgroup, and apply network policy;
  5. starts stock runsc and procd; and
  6. commits running only after an authenticated first-command readiness proof.

Filesystem-only resume returns after the new command-ready binding commits. Memory resume reports its own restoration result; failures never change modes. The new runtime_id names a different physical allocation and runtime_generation increases by one.

A resume can fail as unavailable when there is no compatible carrier, node capacity, NBD device, node channel, or object-store access. The sandbox remains paused and its committed RootFS head is unchanged.

Runtime Failure Recovery#

A crash is not treated as a successful pause.

ConditionBehavior
Driver process disappearsRegional reconciliation remains authoritative; cleanup does not depend on the plugin returning.
runsc/procd or allocation fails without a planned retirementctld fences the old writer and the regional transaction crash-abandons its uncommitted branch. Resume uses the last committed RootFS head.
ctld primary failsThe synchronized standby is promoted and resumes the durable node operation.
Manager restartsAnother replica recovers the PostgreSQL lifecycle/terminal operation.
Node rebootsOnly the authenticated successor boot for the same durable node UID can reconcile the old incarnation, after proving old runtime state absent.
Nomad server record is purged earlyThe trusted exact client allocation directory and node absence proof remain required; server-record absence alone is insufficient.
ℹ

Writes after the last committed RootFS head can be lost after an unplanned runtime or node failure. Local dirty tail is never promoted without the exact writer and publication proof. Wait for a successful explicit pause or snapshot publication when a workflow needs a regionally durable point. Guest fsync confirms local WAL durability, not regional publication. Ordinary file writes alone do not prove a regional commit.

Capacity stays reserved until terminal proof is complete. This avoids running a replacement sandbox on CPU, memory, writer, mount, or network state that may still belong to the failed generation.

Inspect State#

Sandbox details expose:

  • status and paused: inspect both, because a physically resolved memory failure can report status: "failed" with paused: true;
  • empty runtime_id while paused;
  • the current runtime_generation;
  • soft and hard expiration timestamps; and
  • selected durable settings, including auto_resume, services, and resources. The details response is not a full copy of the original claim configuration.
python
sb = client.get_sandbox("sb_abc123") print("paused=", sb.paused) print("status=", sb.status) print("runtime_id=", sb.runtime_id) print("runtime_generation=", sb.runtime_generation)

During planned pause, the previous generation is not reusable once retirement begins. During resume, supported access may wait for the lifecycle transaction; callers never receive a mixed old/new runtime binding.

TTL And Auto Resume#

FieldBehavior
ttlSoft runtime timeout. Expiry requests the same planned pause transaction.
hard_ttlHard sandbox timeout. Expiry durably requests deletion of identity and RootFS state.

auto_resume controls whether supported inbound access may wake a paused sandbox. The wake-up mode follows the paused sandbox's committed state. A retained memory checkpoint selects memory restore; an incompatible or failed restore returns an error and keeps its checkpoint instead of starting fresh processes. When no memory checkpoint is retained, automatic wake-up uses filesystem-only resume and starts new processes from the committed RootFS. This mode selection happens before the restore attempt and is not a fallback after a restore failure. It does not disable platform-initiated crash fencing.

Common auto-resume entry points include sandbox runtime APIs, SSH, and restartable public routes configured with resume: true. A public route also needs sandbox auto_resume: true and a restartable cmd or function runtime. Public URLs stay stable across runtime generations.

Caller-visible auto-resume results include:

  • 503 unavailable with sandbox is waking up: an accepted resume has not yet committed;
  • 409 conflict: resume was rejected because another lifecycle operation is in progress or its retained state is incompatible;
  • 404 not_found: the sandbox was deleted, including between lookup and resume; and
  • 503 sandbox_resume_failed: that resume attempt ended unsuccessfully.

A rejected resume does not forward the original runtime request or retry it automatically. Refresh the sandbox state before deciding whether to retry a conflict; a deleted sandbox must not be resumed.

Lifecycle resume completion does not guarantee that a guest service or terminal is already accessible. Runtime access returns 503 runtime_unavailable when its transport is temporarily unavailable, and 409 runtime_generation_mismatch when the requested runtime identity is no longer active. Reconnect a terminal attachment with bounded backoff; do not blindly replay a mutating HTTP request whose result may have been lost. Guest service failures never authorize discarding a retained memory checkpoint or restarting its processes.

In filesystem-only mode, a supervised session preserves its logical identity and retained event journal, then follows its runtime_recovery policy in the new generation. Its old PID and sockets do not survive. During graceful runtime shutdown, procd receives the stop signal before its children so teardown does not turn recoverable sessions into normal exits. Sessions that had already exited or were explicitly stopped remain terminal; runtime_recovery: restart applies to sessions active at runtime replacement.

Sandbox0 Cloud Billing Pause#

Sandbox0 Cloud restricts new runtime use when prepaid credit is exhausted. Running sandboxes are automatically paused once outstanding usage and payment debt reach 10% of the team's most recent successful top-up amount. Without a successful top-up, any positive debt triggers the pause. Settlement and regional delivery are asynchronous, so this is a pause threshold rather than a precise limit on final charges.

Billing pause uses the default filesystem-only checkpoint. The sandbox ID and committed RootFS remain available; processes and memory do not survive. Once billing admission allows runtime use again, the sandbox can be resumed with a new runtime generation. hard_ttl continues to apply while a sandbox is paused.

Typical Pattern#

  1. claim a sandbox with a soft ttl;
  2. perform active work and create explicit snapshots at important milestones;
  3. refresh the soft deadline while work continues, then let it expire when idle;
  4. resume explicitly or through an allowed access path; and
  5. set hard_ttl when durable state must have a maximum lifetime.

Next Steps#

Gateway timeout budget#

Runtime API requests for contexts, sessions, file watching, and preview creation or renewal may synchronously wake a sandbox. Gateway upstream proxies allow at least 45 seconds for these requests, as they do for explicit create, resume, and fork requests. Cluster-gateway bounds its manager resume call to 45 seconds, including when the access will become a long-lived stream. An earlier caller deadline or cancellation still applies. Metadata requests retain the ordinary proxy timeout, and runtime operations after startup retain their normal procd timeout. The default 60-second gateway write timeout accommodates this startup budget; custom ingress or client timeouts must also allow time for resume.

Persistent Files API reads and mutations use a separate upstream budget of at least five minutes for storage attachment and publication. They do not automatically resume guest compute. Earlier caller deadlines still apply.