@vzn/vx-reapi
v0.0.659
Published
Bazel Remote Execution API plugin for vx — remote cache (ActionCache + CAS) against NativeLink, BuildBuddy, Buildbarn or bazel-remote.
Maintainers
Readme
@vzn/vx-reapi
Website: https://vznjs.github.io/vx/ · this plugin: https://vznjs.github.io/vx/plugins/vx-reapi/ · source: https://github.com/vznjs/vx
A vx remote cache backed by any server speaking Bazel's Remote Execution API — NativeLink, BuildBuddy, Buildbarn, bazel-remote: mature server implementations, none of which we had to write, because a REAPI server is deliberately dumb.
bun add -d @vzn/vx @vzn/vx-reapi// .vx/workspace.ts
import { defineWorkspace } from '@vzn/vx/config'
import { reapi } from '@vzn/vx-reapi'
export default defineWorkspace({
// Reads try the local cache first, then the remote; a remote hit is copied to local.
plugins: [reapi({ endpoint: 'grpcs://cache.example.com:443' })],
})Off CI a run reads the remote and does not write it unless its policy names
the remote. The execution record below follows the same policy: a run that
does not read the remote replays no record, and one that does not write it
records none under the task's key. Give the write credential only to the default branch's CI and a
read-only one to laptops and pull request jobs: the server's token is the
boundary (docs/security.md § Cache poisoning). A server that refuses a
save's UpdateActionResult (PERMISSION_DENIED, UNIMPLEMENTED) is taken
at its word for the run: later saves fail the same way without uploading.
ReapiRemoteCache is the layer class behind reapi(), for a workspace that
composes cache layers by hand. The package exports reapi, ReapiPluginOptions,
ReapiRemoteCache and its ReapiOptions; the wire client, the Merkle
encoders and the executor are internal. ReapiOptions is the connection
as the plugin resolves it: the same fields, with the PEM text itself
(tlsCaPem, tlsClientCertPem, tlsClientKeyPem) in place of the files,
and onWarn for a degraded-but-recovered call; reapi() refuses those four,
reading files and warning through vx. With no endpoint configured (or a blank
one) the plugin declines and costs nothing, so it is
safe to leave declared. An endpoint that is not host[:port], with an
optional grpc(s):// or http(s):// scheme (a trailing / is dropped), or a gRPC resolver target
(unix:, unix-abstract:, dns:, ipv4:, ipv6:), is refused at startup
with a line naming the setting. So is a TLS endpoint at an IP address:
Bun's TLS refuses an IP as the server name, so name the server by host
name, or use grpc:// for plaintext. The plugin reads no environment
variable: every setting is an option, and a value from CI is named in
.vx/workspace.ts (endpoint: process.env.REAPI_ENDPOINT). execute:
true turns on remote execution (off by default: a plugin must not move
where a build runs merely by being configured for caching). execute is
a boolean: a string (process.env.X) is refused. instanceName, like
the endpoint, is trimmed, and a blank one is unset. headers
adds gRPC metadata to every call: a hosted server's API key goes there
(headers: { 'x-buildbuddy-api-key': process.env.BB_KEY! }); a name
gRPC metadata cannot carry (a space, a -bin suffix) is refused at startup,
and a header or platform value that is no string (an unset variable) as
reapi() is called.
toolName and toolVersion (default vx, 0.0.0) fill each call's
RequestMetadata.tool_details, and correlatedInvocationsId groups several
runs as one build in a server's UI.
TLS is on unless asked off, as in Bazel: a bare host:port, a grpcs://
or https:// endpoint, and a dns:/ipv4:/ipv6: target all use it.
Plaintext takes a grpc:// or http:// endpoint or tls: false; a unix:
or unix-abstract: socket is plaintext (tls: true turns it on). It uses the system roots unless told otherwise. A server behind a private
CA takes tlsCertificate, a PEM file of that CA; one that asks for
mutual TLS takes tlsClientCertificate and tlsClientKey — Bazel's --tls_certificate, --tls_client_certificate and
--tls_client_key. Any of them turns TLS on unless tls: false is set; a file that cannot be read
is refused at startup, naming the setting.
How a vx cache key becomes a REAPI entry
A CAS digest is the sha256 of the content, so it cannot be derived from a
vx cache key before the bytes exist — has(key) could never answer. The
ActionCache supplies the missing indirection:
| vx | REAPI |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cache key | synthetic action digest — sha256("vx-reapi-v1\0" + key) |
| artifact (tar.zst) | one CAS blob, referenced by the ActionResult's output_files |
| task duration | stdout_raw on the ActionResult |
| cache miss | GetActionResult → NOT_FOUND |
| cache hit | GetActionResult asking for stdout and the artifact inline: a server that honours it (bazel-remote, up to ~1 MiB) answers a small hit in one round trip |
| cache save | an artifact up to 256 KiB is read once and goes in one BatchUpdateBlobs, unprobed; a larger one is probed with FindMissingBlobs and streamed only if missing |
| stored key | kept: the save writes no UpdateActionResult, unless core says the stored bytes did not read back (replace); then the blob is sent unprobed and the entry rewritten |
The vx-reapi-v1 prefix does two jobs: it keeps vx keys out of the address
space of real Bazel action digests on a shared server, and it makes a future
change to this mapping miss cleanly rather than read bytes written under
different rules.
Servers may normalise an inline stdout_raw into a CAS blob and hand back a
stdout_digest instead (bazel-remote does). The read path accepts either.
Bun and chunk size
chunkBytes defaults to 65535 and is not a throughput knob. Bun's
node:http2 client hangs — it does not error — when a request carries more
than one message and any single message exceeds a ceiling that the server's
flow-control behaviour decides. Go's gRPC servers grow their window
dynamically (a WINDOW_UPDATE then a SETTINGS raise) and Bun mishandles the
tail of that sequence; a node:http2 server, which does not do it, accepts
4 MB writes happily.
See Bun #30342 and
#26915, largely fixed by
#31584 — which is why the ceiling
rose from ~64 KB on Bun 1.3.x to ~216 KB on 1.4.0 rather than the hang going
away. Hence Bun >= 1.4.4 is required (core's own minimum, the package's
engines.bun), and the plugin refuses to start on anything older with a
named error: the alternative is a wedged upload with nothing for a user to
act on.
The default is the one size with no peer-dependence — 65535, the RFC 7540
default initial window every peer must honour with no WINDOW_UPDATE at
all (SAFE_CHUNK_BYTES). 128 KB was the default until it stalled a 1 MiB
write against bazel-remote in 2 of 12 fresh runs (Bun 1.4.2), each costing
the call's 30 s deadline; 65535 stalled in none, and costs ~51% on a 32 MiB
upload (390 → 590 ms on loopback). A larger chunkBytes is still accepted:
reapi({ endpoint: 'grpcs://cache.example.com:443', chunkBytes: 128 * 1024 })The stall is a RACE, not a boundary. So above the safe size the client
downgrades adaptively — a deadline on a multi-message write retries once at
SAFE_CHUNK_BYTES with a warning, turning a lost coin-flip into a logged
retry instead of a failed task. The deadline counts in either spelling: the
client's own DEADLINE_EXCEEDED, or the CANCELLED a grpc-go server such as
bazel-remote sends when the call's grpc-timeout runs out first (a
CANCELLED before the deadline is the server's own and is not retried). The full probe matrix is in
packages/vx/docs/design/plugin-executor-reapi-2026-08.md §14.
Repeat runs skip the worker
Every successful remote execution writes an execution
record under the task's vx key (vx-reapi-exec-v2), listing its outputs by digest plus
its stdout and stderr (each inline up to 64 KiB, a CAS blob past it), written
while the outputs come down. A later run whose vx cache missed but whose key
already has a record skips the Merkle build, the upload pass and Execute
entirely: the outputs are already in the CAS, and both streams replay from
the record, as a vx hit replays them. A run core does not save
(--no-cache, a read-only policy, a tainted upstream), or whose policy does
not write the remote (a run off CI, --cache-policy=remote:r), records under a key
only this run's executor knows instead: its dependants in this run graft
from it, and no later run replays it as the task's hit. A v1 record kept no stderr and is
never read: its key executes once more and writes a v2 one. A record
with a non-zero exit code (vx writes none, but the action cache is the
server's) is no record either: the task executes, and a dependant whose
upstream's outputs exist only as such a record is refused. --force
bypasses it, as does a policy that does not read the remote
(--cache-policy=remote:).
The input tree is read after the key was taken, so each file's bytes are
held to the git blob id the key hashed for it. A file edited, removed or
replaced by a directory
in between (an edit mid-run under vx watch) still runs, but the
execution is not recorded under a key that no longer describes it; the
record once replayed the edited outputs on every machine after the file
was restored. The input root holds exactly the files the key hashed: core
keys no package file, so a project's package.json is in it only as a
listed input (@vzn/vx-lockfile lists it on every cached task).
A file over 1 MiB is hashed as it streams past and held by path, not its
bytes: an action's inputs cost a buffer, not their size, while it waits on
the upload. The upload reads such a file again and refuses it if its bytes
no longer carry the digest (the task fails, saying which file changed), so
a rewrite mid-run is never stored under the old digest
(tests/big-inputs.test.ts).
This matters most under --cache-download=none, where deferral leaves no
local cache entry behind, so vx's own probe misses on every later run
and the record is what makes the second run cheap. Records are checked
against the CAS first (FindMissingBlobs, the outputs and any stream blob) — the action cache and the
CAS evict independently, so a record that outlived its blobs falls
through to a real execution rather than "succeeding" with nothing.
The replay itself is a cache read too: a transport failure reading the
record, a stream or any output warns and executes, after removing
whatever the half-done replay created (core cleaned the outputs once,
before the replay, and the real run's result must not inherit them).
The probe sees a Tree by its own digest, not the blobs inside it, so
one of those gone is caught by the replay: under any capture shape it
fails the replay, and the task executes. (A fresh result under a
whole-tree capture still only warns, since the capture holds more
than the outputs.)
An UPSTREAM's evicted blobs get the opposite answer, because there is
nothing to fall through to. When a dependency's outputs live only in
the CAS — vx grafts them by reference precisely because no local copy
exists — and those blobs are gone, the action cannot be built with the
inputs its key names. vx fails the task and names the upstream rather
than shipping the action without them: a command that tolerates the
absence exits 0, and that successful-but-wrong result would be cached
under a key asserting those bytes were present. Which upstream bytes a
command actually reads is unknowable — that is what dependsOn
declares — so the refusal is the only sound reading, and it matches
what core does when a deferred producer cannot be materialised.
Re-run the upstream (--force) to repopulate the store. An upstream with
no record at all and its outputs nowhere on this machine (its outputs
listed in the local index but gone, or a deferred result no record could
hold) is refused the same way.
Bringing a task's OWN outputs back gets the same treatment. Core's
contract is that once an executor returns, the declared outputs are on
disk, because the ordinary save path then tars whatever it finds — so
an output blob that cannot be fetched is a hole that would be cached
under a key that stands for a complete build. Under a literal capture the
worker returns only what output_paths named, so every returned file
is a declared output and an unfetchable one fails the task. The
exception is a glob whose FIRST segment is a wildcard (*.js): it has
no REAPI spelling, so it is sent as '' — whole-working-directory
capture — and inputs and undeclared siblings come back too. Those
cannot be told apart from real outputs, so a missing blob there only
warns, unless a declared glob names it: that is a hole in a declared
output and fails the task under either shape. Inline output bytes are
checked against their digest like fetched ones; a mismatch is fetched.
A capture cut at a wildcard holds more than the outputs: src/*.gen.js
is sent as src, and the worker returns the sources beside the
generated files. Only what a declared glob names (or what sits under a
directory it names) is written back; the rest stays as it is on disk.
Writing all of it once put the worker's copy of the sources over the
user's, so an edit made while the action ran was lost. A directory a
literal glob names is written whole. Where a write really lands, after
the result's own links are placed, must be declared too: a declared link
to .git/hooks and a declared directory under it wrote a hook.
The existence probe confirms the artifact, not just the entry
has() — what vx run --dry uses to predict hit vs miss —
reads the ActionCache entry AND checks the artifact blob is still in the
CAS. The second call is not redundant, because servers disagree: measured
against both, bazel-remote validates an ActionResult's referenced blobs
and hides a dangling entry, while NativeLink serves it. Without the
check, a plan would report cache hit (remote) for a task that then
executes for real. It costs one extra round trip and only for a PREDICTED
HIT — a miss still answers in a single call.
Downloads are verified
Every blob read — ByteStream and batch alike, compressed or not — is
re-hashed with the negotiated digest function and length-checked against
the digest it was requested under. Bytes that don't match are refused with
a named integrity error instead of being written into the local
content-addressed store: a corrupt or poisoned remote degrades to a miss
(the cache invariant), never to wrong bytes under a trusted name. Uploads
were always server-verified; this is the mirror on the read side, the same
check Bazel's client performs. A streamed read (the cache artifact) is
hashed as its bytes pass and errors at its end on a mismatch with core's
CorruptArtifactError, so the hit is a miss and the rebuild's save
replaces the entry and its blob: one corrupt entry is one rebuild, not
one on every run. The size is held as the bytes arrive:
a body past its digest's size is refused at the byte that passes it, a
zstd reply is decoded no further than that size, and a batch entry for a
digest not asked for is dropped.
A verified blob still lands where the server's ActionResult says, so its
paths are held to the workspace: an output path or Tree name that climbs
out (.., absolute), a link whose target leaves the workspace (read as
the OS follows it, through the links the result placed), a
directory that resolves out through a link, and a file or link that
lands, through one, where no declared glob names are refused before anything
is written through them; a link standing at an output file is replaced, never written
through; a Tree file's setuid and setgid bits are dropped. A Tree may name
one subtree many times; a result whose Trees together expand to more paths
than the file system has free inodes is refused before anything is written,
as is a Tree holding a path past PATH_MAX (4096 bytes; 1024 on macOS).
A glob's matches are recorded each as a Tree of its own, a subtree named
under many names encoded once; a capture past the free inodes or
PATH_MAX is recorded whole, unsplit.
Artifacts stream
An artifact up to 256 KiB is read whole and sent in one batch, with no
probe first. Up to the batch limit (about 4 MiB) it is read whole once,
probed with FindMissingBlobs, and those bytes sent. Past that the cache
layer never holds an artifact whole: put digests the file-backed Blob
vx hands it in one pass over its stream, asks FindMissingBlobs, and
uploads from a second pass, read chunkBytes at a time as the ByteStream
write drains, identity-encoded (the artifact is zstd already). get returns the ByteStream read as a
Response, each message taken from the call as vx writes the previous one
to disk.
Deadlines: a wedged server degrades, it does not hang
Every call carries a deadline, because the killer case is not a server that is
DOWN — that is an instant UNAVAILABLE the cache layer degrades to a miss —
but one that accepts TCP and never answers. Without a deadline no error ever
happens and the first probe hangs the whole run.
There are TWO deadlines, and the split matters:
| option | covers | default |
| --------------- | ------------------------------------------------------------------------------------- | ---------------------------- |
| metaTimeoutMs | Capabilities, GetActionResult, UpdateActionResult, FindMissingBlobs, QueryWriteStatus | min(callTimeoutMs, 15 000) |
| callTimeoutMs | ByteStream transfers, Batch{Read,Update}Blobs, Split/SpliceBlob | 30 000 |
A control-plane message is small and bounded: a healthy server answers in
single-digit milliseconds. A bulk transfer is size-proportional and
legitimately slow — capturing a node_modules tree is what pushes real
deployments to raise callTimeoutMs into the minutes. With one knob for both,
buying headroom for that upload also buys every metadata probe the same
minutes before it can degrade, which is the opposite of what the deadline is
for.
Each deadline, executeTimeoutMs and queueTimeoutMs among them, must be
a positive number of ms up to 2147483647 (a longer timer fires at once);
anything else is refused at the reapi() call.
That is not hypothetical. A NativeLink instance degraded into a state where it answered every ActionCache MISS in 3 ms and every HIT never — idle CPU, nothing in its logs, cleared by a restart with identical on-disk data. With a single 180 s deadline, every task burned three minutes on a lookup before failing. Now the probe gives up in 15 s and the run re-executes.
What the run prints is one line per kind of failure:
[vx] plugin '@vzn/vx-reapi': probe <key> at <endpoint> failed: 4 DEADLINE_EXCEEDED: ….
Core's layered cache names the request, the vx key and the server (a
gRPC status carries none of them), and a later request that fails with
the same status is counted, not repeated: the run ends with
[vx] plugin '@vzn/vx-reapi': N more requests failed the same way: ….
Execution streams are deliberately NOT bounded by either: queueing behind a
busy worker pool is legitimate. A wedged server still cannot reach Execute,
because the deadline-bounded Capabilities call runs first. The wait for a
worker is bounded by queueTimeoutMs when set: no EXECUTING within it, the
Execute stream is closed, the operation is cancelled with
Operations.CancelOperation (one attempt on metaTimeoutMs; a server
that refuses it, or lacks the service, may still run the action, and its
result lands in the action cache; that refusal is a warning) and the task is given back to vx, which
runs it here and says so once:
[vx] <task>: plugin '@vzn/vx-reapi' (executor 'vx/reapi') gave it back: no worker started the action within queueTimeoutMs (…ms); its operation was cancelled — running it here.
A task placed exec.remote: 'only' fails with that reason instead. Unset,
the task's own exec.timeout bounds the queue as it bounds the run (the
task fails as timed out); with neither, the wait is unbounded.
A stream that drops with a transient status, or ends cleanly before its
operation is done, re-attaches with WaitExecution rather than running the
action again. Only attaches that deliver no operation spend the budget:
the fourth such in a row fails the task. A stream that delivered one (a
proxy cutting a long, progressing action) starts the count over, so a
long action survives any number of cuts. The wait before each re-attach
grows per cut in a row (100, 400, 1600, then 5000 ms) and starts over once
a stream advances the stage or stays open 10 s. If
WaitExecution answers NOT_FOUND — the server lost the operation, as a
restart does — nothing is left running to re-attach to, and the action is
executed again on the same budget. Past the budget the task fails and the
operation is cancelled, so the action does not run on unwatched.
An Execute answered FAILED_PRECONDITION (inputs the server lacks, evicted
after FindMissingBlobs or never stored) uploads them again — the blobs its
PreconditionFailure names MISSING, else every input, without asking
FindMissingBlobs — and executes once more; a second FAILED_PRECONDITION
fails the task.
Once a worker reports EXECUTING, the task's exec.timeout (or
executeTimeoutMs, 10 minutes when neither is set) bounds the wait, and
the bound holds across a re-attach: firing during the backoff between a
dropped stream and its WaitExecution, it ends the task there rather than
being lost. It is also what ends a server that answers EXECUTING and hangs
up on every attach. Past it the operation is cancelled too (a refused
cancel is a warning).
The run stopping (Ctrl-C, an embedder's abort) or the task's exec.timeout
closes the stream and cancels the operation with CancelOperation, and an
action not yet submitted is not sent. A record replay, an upstream record
read or the upload under way stops there too, and writes nothing after: what
a replay already created is taken back, so a retry starts clean. The task then ends as a killed local
one does (exit 143, SIGTERM): past exec.timeout it is failed (timed
out) with its key in vx last.
A ByteStream Read, like every unary call, retries UNAVAILABLE,
RESOURCE_EXHAUSTED and INTERNAL three times (100, 400 and 1600 ms) before it
counts as failed. A streamed read (the artifact of a remote hit) spends
the same budget across the whole blob: a cut after bytes have reached the
reader re-opens the Read at read_offset = what the reader has, and the
digest is checked over the joined bytes as before. INTERNAL is on the list because it is how the gRPC client
reports a call cut in transit: the RST_STREAM(INTERNAL_ERROR) a proxy sends
when the server behind it goes away, or a stream that ends with no gRPC
status. The same three statuses are what re-attach a dropped execution
stream.
A server that stays down pays that backoff once: after a call spends its retries on UNAVAILABLE, the cache path's calls (unary, Read, Write) give up at their first UNAVAILABLE until the server answers again, so a refused port costs a five-task run 2.6 s, not 24.
A status the server answers with and no retry heals (PERMISSION_DENIED on Execute, a refused upload) fails the task with a line naming the task and the status, never as a vx "internal error".
A failed RECORD read is never a failed task. The execution-record lookup is a shortcut past the worker, so a transport error there means "no usable record" and the task executes normally, with a warning naming why. The UPSTREAM record reads are the deliberate exception — those decide whether a dependency's bytes exist at all, and carrying on past a failure there is how an action runs without its inputs and caches the result. A fresh execution's stdout or stderr that cannot be read, or whose blob the CAS no longer holds, fails the task naming the digest, and no record is written: those are the task's result, not a shortcut.
Tests
bun test runs the unit suite anywhere. The round-trip suites need real
servers, which vx run test.live --filter @vzn/vx-reapi brings with Docker:
services.cache (bazel-remote on :19092) and services.exec (NativeLink on
:51051) run in the foreground until the run ends. CI runs it in the Linux
DTE run, where they start on the agent that runs test.live.
Without an endpoint those tests skip; test.live sets VX_REQUIRE_REAPI=1 and
VX_REQUIRE_REAPI_EXEC=1, which turn an absent endpoint into a failure so the
suites cannot silently vanish.
Protocol coverage
All 14 RPCs across the five services, not a working subset:
| Service | RPCs |
| --------------------------- | ---------------------------------------------------------------------------------------------- |
| Execution | Execute, WaitExecution |
| ActionCache | GetActionResult, UpdateActionResult |
| ContentAddressableStorage | FindMissingBlobs, BatchUpdateBlobs, BatchReadBlobs, GetTree, SplitBlob, SpliceBlob |
| Capabilities | GetCapabilities |
| ByteStream | Read, Write, QueryWriteStatus |
Protocol features in use, not just reachable:
- Digest negotiation — SHA256 by default (the universal baseline; the Merkle encoders must hash with the SAME function as every upload, so auto-upgrading would mix functions inside one action). The plugin offers no other.
- zstd compression —
compressed-blobs/zstd/…resource names on ByteStream andcompressor: ZSTDon batch updates, enabled only whensupported_compressorssays so. RequestMetadatain the well-known binary header (tool name/version, action id, correlated invocations id) — how a server groups an action's dozens of CAS/AC calls into one build in its UI.- Inline stdout/stderr on
ExecuteRequest, sparing two CAS round trips per finished action. A failed execution status that carries a partial result (a worker past its timeout) still prints what the command wrote. - Execution stages —
QUEUED/EXECUTING/COMPLETEDdecoded fromExecuteOperationMetadata, so a queued action is distinguishable from a hung one. Action.platform(v2.2) alongsideCommand.platformfor older servers.NodeProperties—unix_modeandmtimeon tree nodes; a mode (andis_executable) lands as its permission bits minus the process umask.- Output directories via the
Treeblob anOutputDirectory.tree_digestaddresses, plus output symlinks (a v2.0 server'soutput_file_symlinks/output_directory_symlinkswhen it sends only those). A tree's small files are fetched together across its directories (BatchReadBlobs, 64 MiB at a time), not one call per directory; the result's own output files, 64 MiB at a time too. - Upload minimality —
FindMissingBlobsfirst (split so no request passes a 4 MiB message), then batched blobs while they fit the server's budget and ByteStream beyond it.
Remote execution
Off by default. Remote execution changes where a user's build runs, which is not something a plugin should switch on merely by being configured for caching:
reapi({
endpoint: 'grpcs://grpc.example.com:443',
execute: true,
platform: { 'container-image': 'docker://alpine:3.20', OSFamily: 'Linux' },
capacity: 64, // concurrent remote tasks; becomes the scheduler's pool
})A cache-only server (bazel-remote advertises exec_enabled: false) makes the
plugin decline the executor with a warning rather than submit work that
will never be answered. Only cacheable tasks are eligible — a task with no
cache: true has no described inputs, so a worker would run it against an
empty input root. A cacheable task's root holds its declared input files
and its upstreams' declared outputs. An upstream with no cache runs on
this machine and leaves no record, so its outputs ship from this disk; a
file it writes outside its declared outputs never reaches the worker.
Verified end-to-end against a live NativeLink scheduler + worker: input tree
uploaded, QUEUED → EXECUTING → COMPLETED streamed, stdout returned inline,
declared outputs materialised byte-correct, and the worker attributed. Every
hand-rolled encoder AND decoder is pinned byte-for-byte against protobufjs
over the same vendored protos — the decoder tests exist because a wrong field
number parses garbage without ever erroring (tests/encoding.test.ts).
One environmental note for NativeLink specifically: its official image is
distroless, so a worker inside it has no /bin/sh and cannot run any vx task.
tests/helpers/nativelink.md has the three-command busybox rehost.
node_modules: install as an action
REAPI workers are stateless, and vx deliberately treats node_modules as
ambient environment rather than a cache input — so a remote task cannot see
the packages a build needs. The answer is the design doc's §7.4 recipe,
exec.remote: 'only':
install: {
exec: { command: 'bun install --frozen-lockfile', remote: 'only' },
inputs: { files: ['package.json', 'bun.lock'] },
outputs: { files: ['node_modules/**'] },
cache: true,
},
build: {
dependsOn: ['install'],
exec: { command: 'tsc -p .' },
inputs: { files: ['src/**'] }, outputs: { files: ['dist/**'] }, cache: true,
},What actually happens, all verified live against a NativeLink scheduler +
worker (tests/vx-run-e2e.test.ts):
installexecutes on a worker — so platform binaries build for the worker's platform, not the laptop's — and runs once per lockfile change, ever: repeats are satisfied from an execution record the plugin keeps under the task's vx cache key.- Its outputs never land on the submitter's disk: not materialised, not
restored, and the local
node_modulesa dev installed is never cleaned. - A dependent task's input tree grafts the install outputs by reference
(per-file digests from the execution record; whole directories as
re-canonicalised REAPI
Trees), so the bytes flow worker→CAS→worker and never transit the submitter. The graft applies ONLY to outputs that exist nowhere locally: when an upstream's outputs are materialised on this machine, local disk is truth — two machines racing a nondeterministic miss can leave the artifact store and the execution record holding results of different executions under one pure-input key, and a worker fed the record would see bytes this machine's own tasks do not. - An upstream that writes inside
install's directory (aprisma generateintonode_modules/.prisma) is merged into it: the worker sees both. A DECLARED input file under a grafted directory is replaced by the graft, with a warning. - With no remote executor declared,
installis a local no-op and dependents use whatever the machine has ambient — a laptop run behaves exactly as it did before the field existed.
The execution record lives under sha256("vx-reapi-exec-v2\0" + key) — a
second AC namespace beside the artifact mapping, listing outputs file-by-file
with workspace-relative paths.
What the worker's environment contains
An action's Command carries exactly two of vx's three environment lists,
sorted by name — the proto requires that, so equivalent Commands hash alike:
exec.env.define— literalname: valuepairs from the task config. They read the same on every machine, so they are safe to put into the action identity, and they are already in the vx cache key.inputs.env— the values this machine resolved for those names, for each name the task's local child would get too (it is also inexec.env.passThrough). They are in the vx cache key by definition, so a change to one already produces a different action. A name unset here, or one the config only tracks, is left out of theCommand, so the worker sees what a local run would: shipping a tracked-only value ran the worker on something a local run never saw, under the same key (item 1092).
A define wins over an inputs.env entry of the same name: it is the more
explicit statement of intent.
Colour is the one exception: the action gets the FORCE_COLOR and
NO_COLOR the local child sees, this machine's or vx's own
FORCE_COLOR=1 (set unless the task sees a FORCE_COLOR or a non-empty
NO_COLOR), so a remote entry holds the same bytes a local one does.
Nothing else crosses. In particular exec.env.passThrough does not reach a
remote worker, and neither does the essential allowlist (PATH, HOME,
TMPDIR, …) — those are the submitting machine's resolved environment, and
the worker has its own. Two reasons, and both matter:
- Host values in the
Commandwould enter the action digest, so a laptop and a CI runner would never share a remote entry — the same reason the vx key excludes them. - A
Commandblob lives in the CAS.passThroughis where secrets go, and a token written to a shared content-addressed store is readable by anyone who can name its digest.
So a task that needs a value on a worker must define it (config literal) or
list it in both inputs.env and exec.env.passThrough (host value,
keyed, and seen by a local run the same way). A task whose command reads
a passThrough secret is one to keep local with exec: { remote: false }.
