01Documentation
Pause And Resume
Sandbox0 can release runtime compute without deleting a sandbox. By default, pause publishes the writable block-COW RootFS head and retires the current runtime allocation. Resume claims a fresh resource-neutral carrier and starts a new runtime generation from the latest committed head.
The default filesystem-only pause preserves files, identity, and durable configuration. It does not preserve processes, memory, sockets, PIDs, open file descriptors, or live REPL connections.
Experimental memory pause/resume explicitly opts into preserving supported process
state with memory: true. It reuses the execution capture and restore machinery
of system maintenance migration,
but retains a checkpoint while no runtime is allocated. Explicit calls without
memory: true, TTL pauses, and billing pauses keep their filesystem-only defaults.
Supported automatic wake-up restores retained memory when available.
While paused, the Files API can read and inspect persistent files without resuming compute. Filesystem-only pauses also allow writes, directory creation, moves, and deletion through the same API and SDK methods. Memory-paused sandboxes allow offline reads only; file edits cannot invalidate or discard the retained process checkpoint.
Default Filesystem-Only Behavior#
| State | Across a successful pause/resume | After delete or hard_ttl |
|---|---|---|
| Sandbox ID and configuration | Preserved | Deleted |
| Writable RootFS | Latest committed generation is restored | Deleted with the sandbox |
| Named RootFS snapshots | Preserved independently until deleted/expired | Follow snapshot retention |
| Processes, memory, sockets, and live sessions | Not preserved | Deleted |
runtime_id | Replaced | Removed |
runtime_generation | Incremented on resume | Removed |
The writable RootFS is a block-COW filesystem. Ordinary RootFS paths such as
/var/tmp and workspace directories persist after a successful checkpoint.
/tmp is a runtime tmpfs and is recreated on pause/resume, as are
virtual mounts such as /proc, /dev, and /sys. Template-defined
ephemeralMounts are also excluded from filesystem-only pause/resume, RootFS
snapshots, and filesystem-only forks. Compatible memory checkpoints can retain
runtime tmpfs state as described below.
Do not store durable data under any of these runtime-only mounts.
For named restore points, fork, and rebase, see Snapshot And Restore.
Explicit Runtime Restart#
POST /api/v1/sandboxes/{id}/restart records a durable request to end the
current processes and start a fresh runtime from the externally published writable
RootFS. This recovery entry point does not execute commands inside the sandbox.
It discards process state, including a retained memory checkpoint. Sandbox0 never
selects this operation as a fallback from memory resume. Applications may adopt
their own automatic cold-recovery policy by submitting a separate, generation-fenced
restart after confirming a terminal memory failure and physical cleanup. Such a
policy loses retained processes and memory; only regionally committed files are
available to the replacement. A timeout or an in-progress restore is not proof of
failure and must not trigger a competing restart.
Send a stable Idempotency-Key header and a JSON body containing the exact
observed expected_runtime_generation. For example, a source at generation 7
uses {"expected_runtime_generation": 7}. The key accepts 1–255 ASCII letters,
digits, dots, underscores, colons or hyphens. Keep both values unchanged when
retrying an uncertain submission. A changed source requires a new explicit
request; retrying with a fresh generation can otherwise restart the wrong runtime.
The response is an operation receipt: 202 means accepted or still pending,
while 200 on a repeated submission returns its terminal result. Acceptance is
not completion. Poll GET /api/v1/sandboxes/{id}/restart-operations/{operation_id}.
If the submission response was lost, recover the receipt with
GET /api/v1/sandboxes/{id}/restart-operations?request_key={key}. Both reads require
sandbox:read; submission requires sandbox:write and usage admission. Receipt
reads remain available when the team is restricted from starting more compute.
A 404 lookup means no matching receipt is visible yet; it does not prove a
concurrent submission cannot commit. Retrying uses the same key and generation.
The operation moves through pausing, paused, and resuming as needed. Only
succeeded proves a committed replacement, with result_runtime_generation.
failed and superseded are terminal and cannot be replayed into another restart.
Pending operations can include a timestamped retry observation with a fixed
blocked_reason, such as retirement_pending, quota_exceeded or
capacity_unavailable. An old-stage observation is not a current blocker.
Timeouts do not cancel committed work; the manager resumes pending operations
from PostgreSQL after restart. While retained, a successful receipt remains a historical result
even if the sandbox later changes or is deleted. It does not assert terminal,
execution, or application health.
External publication must succeed before replacement proceeds, and the old runtime must lose writer authority before the new one starts. A restart cannot recreate workload-deleted files or promise writes that were never committed. Use a named snapshot for an intentional file rollback.
Explicit Memory Pause And Resume#
This experimental mode is still undergoing failure and capacity validation. Use it only with a deployment that supports memory checkpoints. Both actions accept an optional JSON object:
json{ "memory": true }
Omitting the body, using an empty object, or setting memory: false keeps the
filesystem-only behavior. Choose memory on both pause and resume. An explicit
memory resume requires a retained checkpoint matching the committed RootFS
generation. Missing, incompatible, or failed memory restoration remains an error;
it never starts filesystem-only replacement processes. A filesystem-only
pause cannot later recover discarded process memory. Memory capture failures
still report an error; pause never claims to have retained memory it did not save.
A successful memory resume waits for the regional commit, the node's durable adoption acknowledgement, and an access handshake for that exact restored runtime. Background reconciliation still completes interrupted delivery. If the request times out or is canceled after commit, retry memory resume for the same sandbox; the retry checks the committed runtime instead of restoring a second copy. Runtime access readiness does not guarantee that each application inside the sandbox has finished its own startup.
A successful memory pause retains supported guest processes, memory, PIDs,
open files and runtime tmpfs state, together with the matching RootFS checkpoint.
Resume restores that execution image into a fresh compatible carrier. The
sandbox ID stays the same; runtime_id changes and runtime_generation advances.
Existing SSH, streaming and external TCP connections are not guaranteed to
survive. Reconnect clients after resume. A named RootFS snapshot still contains
filesystem state only.
pythonclient.sandboxes.pause_and_wait( sandbox.id, memory=True, timeout_sec=360, ) client.sandboxes.resume_and_wait( sandbox.id, memory=True, timeout_sec=360, )
typescriptawait client.sandboxes.pauseAndWait(sandbox.id, { memory: true, timeoutMs: 360_000, }); await client.sandboxes.resumeAndWait(sandbox.id, { memory: true, timeoutMs: 360_000, });
gooptions := &sandbox0.SandboxLifecycleWaitOptions{ Memory: true, Timeout: 6 * time.Minute, } if _, err := client.PauseSandboxAndWait(ctx, sandbox.ID, options); err != nil { return err } if _, err := client.ResumeSandboxAndWait(ctx, sandbox.ID, options); err != nil { return err }
Memory checkpoint cost depends on the live working set, image transfer,
storage bandwidth and restore compatibility. A warm carrier does not remove
these costs; the normal claim latency target is not a memory restore guarantee.
The example wait budget is a caller setting, not a completion-time guarantee.
Capture uses a bounded regional upload reservation shared with migration. If
that temporary budget is full, preparation retries until its deadline, then
cancels without capture. An image larger than the supported reservation limit
fails at admission. After pause commits and the source is physically terminal, the
temporary budget is released while the retained image stays available to its
owners. Neither case silently switches to filesystem-only pause.
Before capture is authorized, expired preparation is canceled through the exact
source protocol: API admission reopens and unused image staging is released.
The existing process, writer and resource lease stay owned by the running sandbox.
Once capture is authorized, this cancellation path is forbidden; an uncertain
capture requires physical failure resolution. It never retries capture or reports
successful memory retention. After exact source cleanup, the last committed
RootFS remains available. If no retained image exists, an explicit memory resume
returns an error. Starting fresh processes requires a separate filesystem-only
resume or restart; it cannot recover the missing process state.
The public status is failed even though paused remains true for filesystem
recovery. Memory pause wait helpers raise SandboxLifecycleFailedError instead
of treating that recovery state as a successful pause.
Deletion records termination intent and waits for this cleanup or preparation
cancellation before reclaiming the source's resources.
After restore execution is authorized, a failed target is stopped and physically
reclaimed before its resource lease is released. The failed attempt consumes its
runtime generation while retaining the paused checkpoint and committed disk head.
The memory operation remains a memory operation after failure. Recovery retains
the checkpoint and committed RootFS; starting fresh processes requires a
separate explicit filesystem-only restart. Accepted scheduling intent survives
manager restarts, and no replacement can start until the failed target, writer
and resource lease are physically reclaimed. If restoring the parent of a running memory fork fails, retrying the fork reports that failure instead of
recapturing the parent or creating another child.
An accepted in-progress restore can remain starting; a physically resolved
memory failure is reported as failed. Never interpret a paused recovery head as
successful memory restoration or silently launch a cold replacement.
An HTTP timeout or client disconnection does not cancel an accepted durable
operation. Inspect the sandbox and retry the same memory mode; never retry with
memory: false as an error-recovery shortcut.
Memory capture must match the destination's runsc/platform/CPU and immutable runtime configuration. Incompatible runtime or workload changes return an error without starting filesystem-only replacement processes. Upgrading the platform procd alone does not discard otherwise compatible memory; restored processes keep their captured daemon. Template upgrades do not replace the sandbox's immutable committed RootFS head. If webhooks are configured, capture requires a durable webhook outbox; in-flight delivery and its acknowledgement drain before capture. Restored sessions retain their captured process identity rather than starting a new recovery attempt.
With auto_resume: true, supported inbound access restores a retained memory
checkpoint when the paused sandbox has one. If no memory checkpoint is retained,
automatic wake-up uses filesystem-only resume. A failed or incompatible retained
memory restore remains an error and never switches to fresh processes. An explicit
POST /resume without memory: true retains its filesystem-only default.
Conditional Memory Pause#
For a durable caller intent, submit POST /api/v1/sandboxes/{id}/pause with
Idempotency-Key: pause-7-1 and
{"memory": true, "expected_runtime_generation": 7}. The generation must be a
positive safe integer. Keys accept 1–255 ASCII letters, digits, dots, underscores,
colons or hyphens. Both fields and memory: true are required together.
The operation is bound to the authenticated team, user, sandbox, key and exact
source generation. Preserve those coordinates on every retry, including after
an uncertain reply or manager restart. The response includes operation_id and
runtime_generation. A 202 with paused: false means checkpoint work remains;
a 200 with paused: true confirms this operation's retained checkpoint.
Repeat the same conditional request to observe completion. A generic paused
status does not identify which request completed.
Another pending or retained pause, an aborted operation, or a changed runtime generation returns a conflict. The old request cannot pause a resumed successor. A new attempt after safe cancellation needs a new explicit key. Missing support returns an error without dropping the conditions or selecting disk-only pause.
The SDK entry points are sandboxes.pauseMemoryForOperation (TypeScript),
sandboxes.pause_memory_for_operation (Python), and
PauseSandboxMemoryForOperation (Go). These return acceptance or completion and
do not automatically retry, refresh the generation, or poll generic sandbox state.
The existing pause/resume helpers retain their previous behavior. Applications
must separately qualify their credential and process restoration path.
Conditional Memory Resume#
Submit POST /api/v1/sandboxes/{id}/resume with a separate stable
Idempotency-Key: resume-7-1 and all three body fields:
json{ "memory": true, "expected_runtime_generation": 7, "checkpoint_id": "capture-operation-id" }
Use the actual operation_id from the completed conditional pause as
checkpoint_id. It must identify the retained image owned by this paused
sandbox. A never-run memory fork can have source generation zero. The source
must have a safe integer successor; checkpoint IDs accept 1–256 visible ASCII
characters and request keys use the same character set as conditional pause.
Persist the source generation, checkpoint ID, request key and authenticated
identity before submission. Repeat those exact inputs after an uncertain reply;
the accepted restore can continue in the background after a timeout. A successful
200 response includes operation_id for this caller's resume request,
checkpoint_id, from_runtime_generation and runtime_generation. The result
generation is exactly source plus one. The resume operation is distinct from
the capture operation and the internal worker lifecycle ID. These coordinates
prove the requested restore committed, not current application or storage health.
The request-to-lifecycle binding survives manager restart and checkpoint-reference consumption. A generic running runtime cannot satisfy it. A different request, checkpoint, source generation or later lifecycle conflicts. An aborted request cannot silently begin a new restore attempt. After safe terminal cleanup, a new explicit attempt requires a new key and the exact then-current paused generation and retained checkpoint. No path falls back to filesystem-only processes.
SDK methods are sandboxes.resumeMemoryForOperation (TypeScript),
sandboxes.resume_memory_for_operation (Python), and
ResumeSandboxMemoryForOperation (Go). They validate the conditional response and
never retry automatically, refresh generation or poll generic running state.
The legacy resume methods and automatic wake-up mode selection remain available.
Pause And Resume As RootFS Grows#
Long-running agents accumulate dependencies, code, and data in their writable RootFS. Sandbox0 separates that durable filesystem from runtime compute so later pause/resume operations do not need to copy the complete accumulated workspace.
Pause seals and publishes the writable branch's changes, reusing unchanged blocks from the committed generation. Resume attaches the latest committed generation and fetches stored blocks on demand as the runtime reads them, without downloading the entire RootFS before execution starts.
The design avoids work proportional to total RootFS bytes on every pause and resume. It does not guarantee fixed latency: unpublished writes, metadata, cache state, object-store access, and the runtime's read working set still affect the operation. Explicit memory checkpoint and restore additionally depend on the size of retained execution state.
Filesystem Pause Transaction#
/api/v1/sandboxes/{id}/pause
bashs0 sandbox pause sb_abc123
Pause is asynchronous:
- manager records planned retirement for the exact sandbox generation, allocation, slot, writer grant, and resource lease;
- manager requests an idempotent stop for the exact Nomad allocation;
- the elected ctld primary fences guest writes, syncs and seals the local block branch, and publishes immutable generation descriptors;
- PostgreSQL atomically advances the RootFS head and commits the sandbox as paused; and
- the plugin-independent terminal reconciler proves runsc, writer, mount, network state, allocation directory, and lease cgroup absent before returning node capacity.
The initial response can report starting/paused: false while this work
continues. Poll sandbox details until status is paused and paused is
true before treating the checkpoint as resume-ready.
A repeated pause request also reports paused: false until the previous
runtime slot has completed physical cleanup. A committed RootFS head alone
does not make the sandbox ready to resume.
After the pause intent is durably recorded, a temporary node-channel outage
or completion deadline still returns 202 Accepted; the pause controller
retries the same operation. This response does not confirm a saved checkpoint.
Failure to record the initial intent still returns an error.
bashs0 sandbox get sb_abc123
pausing is an internal lifecycle transaction, not a public status. Repeating
the pause request is safe: retries recover the same durable operation instead
of creating a second RootFS publication.
If S3 or PostgreSQL is unavailable after local seal, Sandbox0 keeps the exact retirement retryable and keeps its capacity fenced. It does not silently mark the sandbox paused or release a writer before the regional transaction commits.
Filesystem Resume Transaction#
/api/v1/sandboxes/{id}/resume
bashs0 sandbox resume sb_abc123
Resume:
- locks the paused sandbox and reserves the next runtime generation;
- resolves the same attested image/platform and latest committed RootFS head;
- atomically leases a compatible ready carrier and exact node CPU/memory;
- asks ctld to attach RootFS, prepare the lease cgroup, and apply network policy;
- starts stock runsc and procd; and
- commits
runningonly after an authenticated first-command readiness proof.
Filesystem-only resume returns after the new command-ready binding commits.
Memory resume reports its own restoration result; failures never change modes.
The new
runtime_id names a different physical allocation and runtime_generation
increases by one.
A resume can fail as unavailable when there is no compatible carrier, node capacity, NBD device, node channel, or object-store access. The sandbox remains paused and its committed RootFS head is unchanged.
Runtime Failure Recovery#
A crash is not treated as a successful pause.
| Condition | Behavior |
|---|---|
| Driver process disappears | Regional reconciliation remains authoritative; cleanup does not depend on the plugin returning. |
| runsc/procd or allocation fails without a planned retirement | ctld fences the old writer and the regional transaction crash-abandons its uncommitted branch. Resume uses the last committed RootFS head. |
| ctld primary fails | The synchronized standby is promoted and resumes the durable node operation. |
| Manager restarts | Another replica recovers the PostgreSQL lifecycle/terminal operation. |
| Node reboots | Only the authenticated successor boot for the same durable node UID can reconcile the old incarnation, after proving old runtime state absent. |
| Nomad server record is purged early | The trusted exact client allocation directory and node absence proof remain required; server-record absence alone is insufficient. |
Writes after the last committed RootFS head can be lost after an unplanned
runtime or node failure. Local dirty tail is never promoted without the exact
writer and publication proof. Wait for a successful explicit pause or snapshot publication when a workflow
needs a regionally durable point. Guest fsync confirms local WAL
durability, not regional publication. Ordinary file writes alone do not prove
a regional commit.
Capacity stays reserved until terminal proof is complete. This avoids running a replacement sandbox on CPU, memory, writer, mount, or network state that may still belong to the failed generation.
Inspect State#
Sandbox details expose:
statusandpaused: inspect both, because a physically resolved memory failure can reportstatus: "failed"withpaused: true;- empty
runtime_idwhile paused; - the current
runtime_generation; - soft and hard expiration timestamps; and
- selected durable settings, including
auto_resume, services, and resources. The details response is not a full copy of the original claim configuration.
pythonsb = client.get_sandbox("sb_abc123") print("paused=", sb.paused) print("status=", sb.status) print("runtime_id=", sb.runtime_id) print("runtime_generation=", sb.runtime_generation)
During planned pause, the previous generation is not reusable once retirement begins. During resume, supported access may wait for the lifecycle transaction; callers never receive a mixed old/new runtime binding.
TTL And Auto Resume#
| Field | Behavior |
|---|---|
ttl | Soft runtime timeout. Expiry requests the same planned pause transaction. |
hard_ttl | Hard sandbox timeout. Expiry durably requests deletion of identity and RootFS state. |
auto_resume controls whether supported inbound access may wake a paused
sandbox. The wake-up mode follows the paused sandbox's committed state. A retained
memory checkpoint selects memory restore; an incompatible or failed restore
returns an error and keeps its checkpoint instead of starting fresh processes.
When no memory checkpoint is retained, automatic wake-up uses filesystem-only
resume and starts new processes from the committed RootFS. This mode selection
happens before the restore attempt and is not a fallback after a restore failure.
It does not disable platform-initiated crash fencing.
Common auto-resume entry points include sandbox runtime APIs, SSH, and
restartable public routes configured with resume: true. A public route also
needs sandbox auto_resume: true and a restartable cmd or function runtime.
Public URLs stay stable across runtime generations.
Caller-visible auto-resume results include:
503 unavailablewithsandbox is waking up: an accepted resume has not yet committed;409 conflict: resume was rejected because another lifecycle operation is in progress or its retained state is incompatible;404 not_found: the sandbox was deleted, including between lookup and resume; and503 sandbox_resume_failed: that resume attempt ended unsuccessfully.
A rejected resume does not forward the original runtime request or retry it automatically. Refresh the sandbox state before deciding whether to retry a conflict; a deleted sandbox must not be resumed.
Lifecycle resume completion does not guarantee that a guest service or terminal
is already accessible. Runtime access returns 503 runtime_unavailable when its
transport is temporarily unavailable, and 409 runtime_generation_mismatch
when the requested runtime identity is no longer active. Reconnect a terminal
attachment with bounded backoff; do not blindly replay a mutating HTTP request
whose result may have been lost. Guest service failures never authorize
discarding a retained memory checkpoint or restarting its processes.
In filesystem-only mode, a supervised session preserves its logical identity and retained event journal,
then follows its runtime_recovery policy in the new generation. Its old PID
and sockets do not survive.
During graceful runtime shutdown, procd receives the stop signal before its
children so teardown does not turn recoverable sessions into normal exits.
Sessions that had already exited or were explicitly stopped remain terminal;
runtime_recovery: restart applies to sessions active at runtime replacement.
Sandbox0 Cloud Billing Pause#
Sandbox0 Cloud restricts new runtime use when prepaid credit is exhausted. Running sandboxes are automatically paused once outstanding usage and payment debt reach 10% of the team's most recent successful top-up amount. Without a successful top-up, any positive debt triggers the pause. Settlement and regional delivery are asynchronous, so this is a pause threshold rather than a precise limit on final charges.
Billing pause uses the default filesystem-only checkpoint. The sandbox ID and
committed RootFS remain available; processes and memory do not survive. Once
billing admission allows runtime use again, the sandbox can be resumed with a
new runtime generation. hard_ttl continues to apply while a sandbox is paused.
Typical Pattern#
- claim a sandbox with a soft
ttl; - perform active work and create explicit snapshots at important milestones;
- refresh the soft deadline while work continues, then let it expire when idle;
- resume explicitly or through an allowed access path; and
- set
hard_ttlwhen durable state must have a maximum lifetime.
Next Steps#
Snapshot And Restore
Create named restore points, fork running or paused state, and rebase a paused RootFS.
ContinueSupervised Sessions
Preserve logical process identity and replayable events across reconnects.
ContinueGateway timeout budget#
Runtime API requests for contexts, sessions, file watching, and preview creation or renewal may synchronously wake a sandbox. Gateway upstream proxies allow at least 45 seconds for these requests, as they do for explicit create, resume, and fork requests. Cluster-gateway bounds its manager resume call to 45 seconds, including when the access will become a long-lived stream. An earlier caller deadline or cancellation still applies. Metadata requests retain the ordinary proxy timeout, and runtime operations after startup retain their normal procd timeout. The default 60-second gateway write timeout accommodates this startup budget; custom ingress or client timeouts must also allow time for resume.
Persistent Files API reads and mutations use a separate upstream budget of at least five minutes for storage attachment and publication. They do not automatically resume guest compute. Earlier caller deadlines still apply.