@volter/world-machine

The machine as a World service: a Firecracker microVM run as an ordinary process service, whose memory-and-disk snapshots are entries in the World's ledger. mark and branch --at machine@N take the machine back exactly as they take a twin back, with no change to the kernel or the runtime. It is the machine half of retake: an agent's world (machine memory and disk, and its twins' state) goes back to a mark while the agent's conversation, running outside the machine, continues.

The shape

| Part | Where | What | |---|---|---| | the ledger | the World, state service machine | one entry per checkpoint (snapshot:<id>: label, snapshot directory, memory/disk/store bytes, pause, created), written through world-core's applyTwinWrite; the head a branch restores is the last snapshot in its view (wholeLog) | | the machine host | Linux with KVM, or a Lima >= 2 VM with nestedVirtualization: true on a Mac | Firecracker in one slot (fixed tap, guest address and disk path), run in place: a restore replaces the slot's process, so the guest keeps its address and identity | | the store | a directory on the machine host on XFS (reflink) or btrfs | base-<hash>/ warm bases, snapshots/<id>/{vmstate,mem,disk.ext4}, slots/<slot>/; for the container backend also tree-<hash>/ (a filled tree, below) and slots/<slot>/ct-loaded (what the slot's machine was restored from) | | the guest agent | inside the guest, in every snapshot | steps the wall clock, resets the connections a restore left stale, writes /etc/world.env |

The service holds one persistent shell on the machine host (a new ssh per operation cost 75-125 ms on a loaded Mac), so each operation is one round trip.

• Checkpoint flushes the disk's dirty pages, pauses the guest, writes the pages dirtied since the last snapshot or load (a Firecracker Diff snapshot) and the device state to /dev/shm, reflink-clones the disk, resumes it, then materializes one full memory file for the entry (a reflink copy of the parent's with the diff applied by snapshot-editor edit-memory rebase), so a restore always loads one file.
• Restore loads this branch's head (or the warm base when the ledger has none) with mem_backend File, steps the guest's wall clock to the World's clock (TWIN_WORLD_CLOCK_FILE when set, else the World host's now), resets every connection the restore left stale, and rebuilds the guest's route out.
• Nothing runs until asked. The service answers at once and boots nothing (state: idle). The first GET /machine or POST /machine/checkpoint brings the machine up (the ledger's head, else the warm base); POST /machine/restore loads the head itself. A branch's services start before branch forks their ledgers, so a retake is branch --at followed by POST /machine/restore, and the new branch's machine is loaded once, at its mark (it used to boot its head at start and be replaced by the restore a second later).
• The warm base is the declared guest image cold-booted once, provisioned by the declared script and snapshotted whole, cached in the store under a hash of everything that shapes it. A World's machine never cold-boots (20-60 s measured).
• Copy-on-write is a precondition: the store must reflink (cp --reflink=always). There is no fallback to a full copy; a whole-image copy dominated every pause it was part of (1-13 s measured).
• Egress: the guest has no gateway and forwarding from its tap is dropped. Its only way out is one forwarder per twin on the tap's host address, at a stable guest-visible port per vendor, pointed at this World's *_TWIN_URL at every restore and start (so a checkout re-points it). The guest reads the stable URLs from /etc/world.env (STRIPE_TWIN_URL=http://172.16.0.1:29772), with the rest of the World's live env (VOLTER_WORLD_ENV_FILE: its fake credentials and app values; its proxy as a forwarder at a stable port, which the twins' own addresses bypass; a certificate file it names (its CA and trust bundle) copied to /etc/volter-world-<name>, all written at each start and restore; what only its host can use, a directory there or a loopback URL of its other listeners, is left out and named in the file), and NODE_OPTIONS requiring world-core's redirect, written with the files it loads to /opt/volter-world at each start and restore too: a real SDK calling its vendor's host reaches the twin, as under volter world run. Every login shell exports the file (/etc/profile.d/volter-world.sh). A World variable whose value is empty is left unset in the machine and named in the file. world init --repo writes a repository's .env.example keys that way, and set, even to nothing, they hid the app's own .env.local (Next.js and dotenv never override a set variable). A deliberately empty value therefore reads as unset in the machine, as it does on the host: init writes no empty placeholders, and up leaves an empty World value unset.

The container backend (backend: "container", not yet run)

On a host where a microVM would be nested (a Mac, Firecracker inside a Lima VM), first-touch guest memory faults through two hypervisors, so a real project's install ran 20–80× slower than one level up. The container backend (container.ts, ct.sh) runs the machine as the host's own processes in a crun container instead, marked by CRIU:

| | firecracker | container | |---|---|---| | the machine | a microVM on KVM | a crun container in a network namespace holding the same guest address | | its tree | a reflinked disk.ext4 | one reflinked ext4 image holding the container's root (so NFS handles on an unchanged file stay valid across a restore) | | a mark | a memory diff, the disk reflinked | the container paused, the image reflinked, a full CRIU dump (arm64 has no soft-dirty tracking), resumed; the dump's pages then composed against the parent's (compose.py) | | a restore | the snapshot loaded, the clock stepped, stale sockets reset by the guest agent | the image reflinked back, CRIU restore, stale sockets reset from the host (ss -K in the namespace) | | the clock | stepped to the World's | the host kernel's, which is now; a frozen World clock is not followed | | the World's env, certificates, redirect | written by the guest agent | written into the tree |

Its config adds:

• backend;
• image (the OCI image the tree is made from) and imageSize (default 40G);
• binds (host directories bound read-only into the container);
• export (a path in the tree the host's NFS server v3 serves to one client, unexported and re-exported around every swap of the image, with a fixed fsid);
• publishPorts ({"ranges": [[3000, 3999]], "except": [...]}, none by default): the machine's listening ports in those ranges, published on the host's loopback at the same port as the machine's own localhost (publish.py).

The app in the machine registers http://localhost:<port> with a vendor, and a twin on the host delivered its webhooks to the host's own loopback, where nothing listened. A Lima host also forwards those ports on to the Mac.

A port is published only when:

• it is allowed: the machine runs code the user did not write, and a port it held first would be taken from a host service that binds it later;
• the machine listens on it on a wildcard or loopback address, and on no address of its own (how its doors bind);
• the host does not listen on it already.

Each connection enters the machine's namespace to its loopback. setup adds skip-in-flight to CRIU's default config (/etc/criu/default.conf, which crun's CRIU reads). A connection the kernel had accepted but the paused process had not made a mark fail (measured under requests to a published port); skipped, its client sees a reset and retries. With it, 8 of 8 marks succeeded under steady requests. setup installs crun, CRIU and podman and checks CRIU. base fills the tree, runs the provisioning once in the container, starts world-machine-init (which runs each /etc/world-machine/services.d/*.sh to completion: a service script starts its daemon and returns once it listens) and marks it.

Measured on the pieces, not on this code (Volter Harness container/retake/README.md):

• a next dev and Postgres froze 1.28 s for a CRIU dump and restored in place in 1.6 s, warm;
• consecutive marks share 71% of their pages, and zstd takes a mark 6×;
• a twin replaced across a restore cost one in-flight request.

The container's processes are told io_uring does not exist (seccomp, ENOSYS): CRIU cannot dump one, and Node's libuv uses it where the kernel offers it. Both loops (the store's image and the tree's) use direct I/O: through the page cache every page was cached once per layer, and a large dump waited on writeback. A store setup --make-store creates is btrfs with compress-force=zstd:1 where the host has btrfs (an XFS image made before is kept). After each mark, compose.py rewrites its CRIU page files so that pages identical to the parent's are reflinked from it and only new ones are data. The files stay CRIU's own, byte for byte, and a next dev mark of 1.3 GB kept 40–131 MB (mean 89 MB). Every mark dumps: whether nothing ran since the parent cannot be told exactly (a freeze and thaw retry the processes' interrupted syscalls, as a small real wake-up does, and no counter on a VZ guest tells the two apart: it has no PMU to count user instructions; Volter Harness container/retake/README.md, "a turn in which nothing ran").

The container base's identity is the image by its id on the host (a tag rebuilt or pulled anew is a new image), its size, the provisioning, the host functions that make it and its dump (wm_ct_base_fill with the init it installs, wm_ct_provision, wm_ct_start, wm_ct_bundle, wm_ct_net, wm_ct_mark, as bash prints them, so a comment is no input), the binds and the slot's addresses. A change to any other host function keeps the base (it hashed all of ct.sh, and each such change rebuilt it: 49 s measured). With no image id on the host (podman failing, the image removed) there is no base name: a machine with a head restores it anyway, one without reads error saying so, and a collection stops before deleting anything.

The filled tree (tree-<hash>: the image's tree after the fill, keyed by the image id, the size, the fill and the guest's address) is kept in the store. A base rebuilt for anything but those starts from a reflink of it: 1.5 s where filling from the image took 12.6 s (a warm podman export of the 1.4 GB, 37,000-file image alone was 8.2 s). base pulls an image from a registry in the background first, since the base is named by its id; the fill never pulls, so a registry tag is refreshed only by a pull on the host.

POST /machine/prepare {"position": N} (container backend) is a retake's first half, run by the machine of the World being left while the World branches. It restores the head a branch forked at machine@N will have, pauses it at once with its way out closed, and marks the slot as handed over to that snapshot. When the World stops that machine's service, the container is left as it is. The branch's machine, asked to restore, finds its own head waiting there paused and adopts it: its egress is pointed at the branch's twins, the World's files are written, stale connections are reset, and the processes resume, with no second restore. Anything else (no handover, another snapshot, a running container) takes the whole restore as before. With "live": true nothing is restored or paused: the running machine itself is handed over, and N must be this ledger's head (a mark just taken). A partial retake uses it (the twins go back, the machine stays): its way out is closed and only its connections through the World's forwarders are reset, so its processes and the connections into it (the harness's) run on; its calls to a twin are refused until the branch adopts it (about a second). A failure opens the way out again. Measured through supercode's restore_world:

| | machine | whole retake | |---|---|---| | before | 0.57–0.81 s | 2.3–2.6 s | | adopted, no dev server | 0.14–0.19 s | 1.17–1.25 s | | adopted, next dev (1.3 GB) in the machine | 0.32 s | 2.4 s |

The branch itself grew under the concurrent restore (2.0 s against 1.0–1.7 s), which is why the dev-server case gained less.

Two services operate the one slot during a retake, the machine being left and the branch's. So on the container backend every host operation takes the slot's lock (slot.lock, flock, up to 120 s), except the store-wide gc. The checks behind the handover are made inside the same locked operation as the swap:

• a prepare refuses a slot that is not its machine's;
• a prepare that fails once the swap began leaves the machine stopped with its way out closed, never the mark running into the World being left;
• a failed adopt falls through to the whole restore.

Background processes the host starts (forwarders, the publisher, the composer, the base fill) do not inherit the lock's descriptor.

Run end to end on 2026-09-25 (Volter Harness container/retake/README.md, "the container machine's first run"). A mark paused the machine for 0.2–0.6 s, and a whole retake took 2.4–3.0 s, of which the machine took 0.85 s. Files, process memory and a twin came back exactly, and a Mac's NFS handles stayed valid. With an export, the host's NFS server stops its threads for a restore's swap, so a hard-mounted client waits instead of getting an error. That server is the host's, so every export on it waits.

Declaring it

The machine config (paths are on the machine host; ~/ is that host's home; provision is beside the config on the World's host):

{
  "host": { "kind": "lima", "instance": "tm", "limaHome": "~/.lima-tm" },
  "store": "~/.cache/volter-world-machine",
  "kernel": "~/fc/vmlinux",
  "rootfs": "~/fc/rootfs.ext4",
  "sshKey": "~/fc/id_rsa",
  "provision": "provision.sh",
  "vcpus": 2,
  "memMib": 1024
}

Optional: firecracker and snapshotEditor (default <store>/bin/…, installed by setup), slot (default), tap (wm0), hostIp (172.16.0.1), guestIp (172.16.0.2), egressPorts ({"STRIPE": 21000} to fix a vendor's port) and room ({"paths": [...], "reserveBytes": 2147483648}). A mark is refused with 507 unless the filesystems under the store have room for it: the store, the file backing it when it is a loop image, and each of paths. Room means the last mark's growth, at least 1 GiB, plus reserveBytes. On a Mac, name the home the Lima VM mounts: its free space is the Mac's, beneath the VM's own sparse disk. When the lowest layer fills, a write fails as an I/O error in every layer above it, which can corrupt the store; a refused mark only goes untaken. Not covered: the warm base's own mark, and what the machine's processes write between marks. On Linux host is {"kind":"linux"} (the default). The rootfs must authorize sshKey's public half for root and have python3 (the guest agent).

The service, declared after the twins it routes to (a service sees the *_TWIN_URLs of the services started before it):

{
  "id": "machine", "type": "process",
  "execution": { "process": { "command": "bun", "args": ["node_modules/@volter/world-machine/src/cli.ts", "serve", "--config", "machine.json"] }, "controlPlane": true },
  "endpoint": { "port": "auto" },
  "bindings": { "injectEnv": "MACHINE_URL" }
}

Commands

Every command returns in well under 30 s; the one-time base build is a sequence of short steps.

| Command | Does | |---|---| | world-machine setup --config <file> [--make-store <size>] | on the machine host: the /dev/kvm udev rule (udev resets the device's mode), socat, the tap, Firecracker v1.17.0 and snapshot-editor when absent, and with --make-store, where the store cannot reflink already, a loop-mounted XFS image at the store (in fstab). Idempotent | | world-machine base --config <file> | advances the warm base one step (copy the rootfs, boot, install the agent and start the provisioning script, then the Full snapshot; for the container backend: pull a registry image, fill the tree or reflink the filled one, provision, mark) and says what it waits on; exit 3 until it prints "ready": true. It uses the slot, so a running machine is stopped | | world-machine gc --config <file> | deletes the snapshots, warm bases and filled trees no registered World's branch names, then commits and trims the store's filesystem; prints what it kept, what it deleted and the bytes freed (the store's df, before and after) | | world-machine serve --config <file> | the service; the World adds --port and --root. It registers its World in the store at once and collects after the machine's first bring-up (or 10 s after start when nothing brings it up), in the background: a first GET waited 1.5 s behind a collection deleting two bases |

Doors

Every door needs the World's door key: authorization: Bearer <key>, the 64 hex digits serve writes once to .world-machine-token beside the configuration it is given (--config), mode 0600. That file is on the machine's host only (never under the tree the machine runs on or a bind of it), and a caller reads it there: supercode-retake from the World's app folder, where its machine.json is. Loopback is not a boundary here, since the machine's processes reach this host's loopback through the World's egress proxy, which forwards local destinations. Every door also refuses (403) a request carrying Origin, naming a Host other than this listener's loopback address, or POSTing a body that is not application/json. The port is loopback, but a VM's loopback is forwarded to its host (Lima), where a web page could otherwise send a cross-origin POST, or read an answer through a rebound DNS name.

| Door | Answers | |---|---| | GET /machine | { state, head: {id,label}\|null, loaded, guest: {ip, sshPort, sshKey, user}, egress: {<VENDOR>: url}, message? }; state is running, displaced (another World's machine took the slot; a restore takes it back), no-base or error (for the container backend also when its image has no id on the host), or idle before anything asked for it (a GET brings it up, and its answer carries broughtUp with the restore's facts when it did) | | POST /machine/checkpoint {label, keep?} | { id, pauseMs, memBytes, diskBytes, storeBytes, memWriteMs, diskCloneMs, materializeMs }; appends one ledger entry | | POST /machine/restore | { id, label, from, loadMs, restoreMs, clockMs, clockErrorMs, clockSteps, staleReset, resetMs, … , totalMs } | | POST /machine/present {positions} | [{ position, id, present }]: whether each ledger position's snapshot is still in the store (complete, not .pending) | | GET /machine/clock | { errorMs, guestRttMs, hostOffsetMs, hostRttMs }: the guest's wall clock against the World's, now | | POST /machine/compare {position, path?} | container backend, read only: what a restore to that position would change, { files: {back, gone, changed, counts, skipped, partial}, processes: {restarted, stopped, then, now} }, from the mark's image mounted read-only (path defaults to the exported project; regular files and symlinks, bounded at 30,000 files and 15 s) | | POST /machine/run {argv, user?, timeoutMs?} | container backend: a command run in the running machine (crun exec), as user (a name the machine's passwd resolves) or its root, bounded by timeoutMs (60 s, at most 300 s) with the machine's own timeout, which ends what it started there; { code, stdout, stderr, timedOut? }, each output's last 16,000 bytes. A caller composing a service's restore (stop it, files, start it) uses it; an error starting refused: ran nothing | | POST /machine/files {position, paths} | container backend: those paths (absolute in the machine, or relative to the exported project) back as they were at the position, the machine running on, { paths: [{path, done, changes, sample}] }; done is restored (from the mark's image: a folder as it was, what was made in it since removed; a file by its content, never its time; owners, modes and times), removed (the mark did not have it) or absent. Every path is checked before any is touched (an error starting refused: changed nothing); with noLinks, a path holding a symlink anywhere under it, in the mark or now, is refused (a data directory whose tablespace or log lives elsewhere); the live tree is written component by component without following a symlink, and a path whose folders are a symlink in either tree is refused. Hard links, ACLs and extended attributes are not carried |

memBytes is the dirty memory written, storeBytes the store's df growth (read after a syncfs: XFS frees an unlinked file's blocks in the background) since the last restore or checkpoint, and diskBytes that growth less memBytes (the disk's copy-on-write blocks).

A retake

At a mark: POST /machine/checkpoint, then the World's positions (volter world log --json, the highest position per service). To go back: volter world branch <name> --at machine@N,stripe@M, then POST /machine/restore on the new branch's machine (its URL is MACHINE_URL in the World's env). With a frozen World clock, carry the parent's instant with volter world clock set first.

The clock

A restored guest wakes at its snapshot's wall time. The restore moves it through the guest agent: the machine host's clock is measured against the World host's (the shortest of five round trips over the persistent shell); after one warm-up request (the first request after a load waits on the guest waking, its pages faulted in from the snapshot file), three probes measure the guest's error against the target and the agent moves its clock by that difference against its own time (shift), so the step does not depend on when it arrives; probe and shift repeat (at most four times) until the guest reads within 3 ms. With a frozen World clock the guest is moved to the frozen instant and runs on from it, so GET /machine/clock then reads the time since the restore.

Stale connections

Every established TCP connection between the guest and the host on its link is stale after a restore: its far end belongs to the host as it was at the snapshot. A server listening in the guest (a harness's executor, sshd) comes back holding the mark's client, which is gone, and refuses or waits on a new one that resumes the same session; a client in the guest waits out its own timeout. Once the clock is moved, the guest agent resets all of them except its own connection (the kernel's socket destroy over sock_diag netlink, what ss -K does, in the agent's own process: the guest kernel needs CONFIG_INET_DIAG_DESTROY), so each peer sees a reset at once and reconnects. The restore's answer counts them (staleReset, resetMs).

Retention

A snapshot is kept while any existing branch's machine ledger names it, through the branch's whole view (the parent's history up to the fork included); a purged branch's own snapshots go. Every snapshot directory is self-contained (its memory and disk are reflink copies, never references into another directory), so deleting one never breaks another. The store is shared by every World whose machine runs on the host, so each serve registers its World (<store>/roots.json: the World's .volter/worlds directory) and a collection reads every registered World's branches, dropping registrations whose directory is gone. It deletes snapshot directories no ledger names, except one a checkpoint is still writing (.pending until the ledger entry is written) and one a running slot's machine was loaded from (slots/<slot>/loaded, read only while that slot's Firecracker runs; a container slot's ct-loaded, written before a restore reads it and removed when its machine's service stops); and warm bases no kept snapshot was built on, except this configuration's base, a loaded one and one being built. On the container backend a snapshot never reads its base, so there only a branch's head keeps the base it was taken on, and a filled tree stays while it is this configuration's or a kept base (or base build) was made from it. Then the store is committed and fstrim gives the space back to the host (a trim before btrfs committed the deletes gave nothing back; on a Mac the Lima VM's root discards online, so the space reaches the Mac's sparse disk file at the root's next commit). A ledger it cannot read stops it before anything is deleted. A checkpoint that failed half-way leaves its .pending directory behind, which a collection keeps.

retain (a number, optional) bounds one World's own marks: a collection run by that World's serve keeps each machine ledger's head snapshot, and of the World's other snapshots the newest retain by the host time in their id; the rest go even though a ledger names them, and each is listed in <store>/expired.list. Other Worlds' snapshots are left to their own serve. A snapshot another registration's ledger also names is kept, so a mark shared with a branch still registered is not counted against retain. With retain set, serve collects at start and after every tenth checkpoint. A snapshot being restored (slots/<slot>/restoring, written before its presence is checked and removed once the restore returns) is kept like a loaded one. A restore or prepare of a snapshot that is gone is refused before anything stops, naming whether it was let go past retain or is missing; POST /machine/present tells a caller beforehand which positions can still be restored.

thin (true or false, optional, needs retain) thins with age instead of cutting at retain: past the newest retain, one snapshot of each calendar hour stays for a day and one of each UTC day for a month (src/thin.ts). The one kept is each bucket's oldest, so a newer mark never displaces it: a snapshot that stayed stays until its tier ends, and an hour's keeper goes only as it crosses into the day tier, unless it is its day's oldest. A checkpoint taken with keep (POST /machine/checkpoint {label, keep: true}: a mark the agent pinned) is never let go, the newest 20 such; they count among the newest retain like any other.

Not done

• One slot per machine host: two Worlds' machines share it (the later restore wins; the other reads displaced).
• diskBytes is derived from df, not from the disk's own changed extents.

Measured (2026-09-24): on a loaded box, so readings, not results

Surface: this package's own doors (POST /machine/checkpoint, POST /machine/restore, GET /machine/clock) and the product CLI (volter world branch --at, world log --json) on a World of the Stripe twin and this service. Apple M3 Pro, macOS 26.4; Lima 2.2.0 VM (vz, nested virtualization, 4 CPU / 6 GiB, Ubuntu, kernel 7.0); Firecracker v1.17.0; CI guest kernel 6.18 and Ubuntu 24.04 rootfs (2 vCPU / 1 GiB, 2 GiB ext4 disk); the store a 12 GiB loop-mounted XFS image. The Mac was not quiet during any reading: load average 33-90 and 80,000-240,000 page compressions per 2 s (vm_stat), from other work on the box. Every timing below is therefore a reading under that load, not a result; the quiet-box numbers the acceptance asks for were not taken.

| What | Target | Read (loaded) | |---|---|---| | checkpoint pause | < 200 ms | 61-134 ms over six checkpoints (memory write 23-68 ms, disk clone 20-56 ms); 202-604 ms before the diff went to /dev/shm | | after the pause | | materialize 95-186 ms | | storage per checkpoint | ≈ changed disk + dirty memory | 11-13 MB of dirty memory per turn-sized checkpoint, 1.7-2.2 MB after 5-10 s idle; disk 140-200 KiB (store growth after syncfs; filefrag counted 284 KiB of changed disk blocks in one) | | restore load | < 100 ms | snapshot/load 17-69 ms; the whole in-place replacement (process, disk clone, load) 67-250 ms | | restore, answered | | 0.72-1.6 s; the guest's first reply after the load (memory faulted in from the file, through nested virtualization, on a compressing host) takes 0.2-1.7 s of it | | guest clock after restore | within ~50 ms | -1.0 to -2.9 ms (World host to machine host round trip 4-14 ms bounds the offset to half of it) | | a whole retake (world branch --at + restore) | < 1 s | 6.3 s: branch 4.9 s (the machine's start in it 1.1 s), restore 1.4 s; 8.5-10 s before the restore was trimmed |

A dev server in the guest (pid 364) came back from every restore with the same pid and startup nonce and its in-memory notes as of the mark (['1', '2', 'b1'] after notes b2, b3 were written and a retake went to b1), and /work/proof held the mark's rows. The World branch held the Stripe customers up to the mark. The guest reached the twin only through its forwarder (200); an outside address did not answer and the guest has no default route.

The lab's guest clock came back a constant 6 s short. ssh guest date -s @$(date) reads the time before the ssh login, so the guest lands behind by the login's duration: here the same command left the guest 1,534 ms behind with a 1.5 s login under load, where this service's step had read -1.4 ms a moment before. (That the lab's 6 s was its login time is inferred from this, not measured there.)