@vzn/vx-reapi

A vx remote cache backed by any server speaking Bazel's Remote Execution API — NativeLink, BuildBuddy, Buildbarn, bazel-remote: mature server implementations, none of which we had to write, because a REAPI server is deliberately dumb.

npm install -D @vzn/vx @vzn/vx-reapi   # or: pnpm add -D -w · yarn add -D (-W on Yarn 1) · bun add -d

// vx.workspace.ts
import { defineWorkspace } from '@vzn/vx/config'
import { reapi } from '@vzn/vx-reapi'

export default defineWorkspace({
  // Reads try the local cache first, then the remote; a remote hit is copied to local.
  plugins: [reapi({ endpoint: 'grpcs://cache.example.com:443' })],
})

ReapiRemoteCache is the layer class behind reapi(), for a workspace that composes cache layers by hand. The package exports reapi, ReapiPluginOptions, ReapiRemoteCache and its ReapiOptions; the wire client, the Merkle encoders and the executor are internal. ReapiOptions is the connection as the plugin resolves it: the same fields, with the PEM text itself (tlsCaPem, tlsClientCertPem, tlsClientKeyPem) in place of the files, and onWarn for a degraded-but-recovered call; reapi() refuses those four, reading files and warning through vx. With no endpoint configured (or a blank one) the plugin declines and costs nothing, so it is safe to leave declared. An endpoint that is not host[:port], with an optional grpc(s):// or http(s):// scheme, or a gRPC resolver target (unix:, unix-abstract:, dns:, ipv4:, ipv6:), is refused at startup with a line naming the setting. VX_REAPI_ENDPOINT / VX_REAPI_INSTANCE configure it from the environment, and VX_REAPI_EXECUTE=1 turns on remote execution the way execute: true does (off by default: a plugin must not move where a build runs merely by being configured for caching). execute is a boolean: a string (process.env.X) is refused. instanceName is the option form of VX_REAPI_INSTANCE, and headers adds gRPC metadata to every call: a hosted server's API key goes there (headers: { 'x-buildbuddy-api-key': process.env.BB_KEY! }). toolName and toolVersion (default vx, 0.0.0) fill each call's RequestMetadata.tool_details, and correlatedInvocationsId groups several runs as one build in a server's UI.

TLS is on for a grpcs:// or https:// endpoint, or with any PEM below; a bare host:port is plaintext, unless tls: true turns it on (tls: false turns it off). It uses the system roots unless told otherwise. A server behind a private CA takes tlsCertificate (or VX_REAPI_TLS_CERTIFICATE), a PEM file of that CA; one that asks for mutual TLS takes tlsClientCertificate and tlsClientKey (VX_REAPI_TLS_CLIENT_CERTIFICATE / VX_REAPI_TLS_CLIENT_KEY) — Bazel's --tls_certificate, --tls_client_certificate and --tls_client_key. Any of them turns TLS on unless tls: false is set; a file that cannot be read is refused at startup, naming the setting.

How a vx cache key becomes a REAPI entry

A CAS digest is the sha256 of the content, so it cannot be derived from a vx cache key before the bytes exist — has(key) could never answer. The ActionCache supplies the missing indirection:

| vx | REAPI | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | cache key | synthetic action digest — sha256("vx-reapi-v1\0" + key) | | artifact (tar.zst) | one CAS blob, referenced by the ActionResult's output_files | | task duration | stdout_raw on the ActionResult | | cache miss | GetActionResult → NOT_FOUND | | cache hit | GetActionResult asking for stdout and the artifact inline: a server that honours it (bazel-remote, up to ~1 MiB) answers a small hit in one round trip | | cache save | an artifact up to 256 KiB is read once and goes in one BatchUpdateBlobs, unprobed; a larger one is probed with FindMissingBlobs and streamed only if missing |

The vx-reapi-v1 prefix does two jobs: it keeps vx keys out of the address space of real Bazel action digests on a shared server, and it makes a future change to this mapping miss cleanly rather than read bytes written under different rules.

Servers may normalise an inline stdout_raw into a CAS blob and hand back a stdout_digest instead (bazel-remote does). The read path accepts either.

Bun and chunk size

chunkBytes defaults to 65535 and is not a throughput knob. Bun's node:http2 client hangs — it does not error — when a request carries more than one message and any single message exceeds a ceiling that the server's flow-control behaviour decides. Go's gRPC servers grow their window dynamically (a WINDOW_UPDATE then a SETTINGS raise) and Bun mishandles the tail of that sequence; a node:http2 server, which does not do it, accepts 4 MB writes happily.

See Bun #30342 and #26915, largely fixed by #31584 — which is why the ceiling rose from ~64 KB on Bun 1.3.x to ~216 KB on 1.4.0 rather than the hang going away. Hence Bun >= 1.4 is required, and the plugin refuses to start on anything older with a named error: the alternative is a wedged upload with nothing for a user to act on.

The default is the one size with no peer-dependence — 65535, the RFC 7540 default initial window every peer must honour with no WINDOW_UPDATE at all (SAFE_CHUNK_BYTES). 128 KB was the default until it stalled a 1 MiB write against bazel-remote in 2 of 12 fresh runs (Bun 1.4.2), each costing the call's 30 s deadline; 65535 stalled in none, and costs ~51% on a 32 MiB upload (390 → 590 ms on loopback). A larger chunkBytes is still accepted:

reapi({ endpoint: '…', chunkBytes: 128 * 1024 })

The stall is a RACE, not a boundary. So above the safe size the client downgrades adaptively — a deadline on a multi-message write retries once at SAFE_CHUNK_BYTES with a warning, turning a lost coin-flip into a logged retry instead of a failed task. The deadline counts in either spelling: the client's own DEADLINE_EXCEEDED, or the CANCELLED a grpc-go server such as bazel-remote sends when the call's grpc-timeout runs out first (a CANCELLED before the deadline is the server's own and is not retried). The full probe matrix is in packages/vx/docs/design/plugin-executor-reapi-2026-08.md §14.

Repeat runs skip the worker

Every successful remote execution writes an execution record under the task's vx key (vx-reapi-exec-v1), listing its outputs by digest plus its stdout (inline up to 64 KiB, a CAS blob past it), written while the outputs come down. A later run whose vx cache missed but whose key already has a record skips the Merkle build, the upload pass and Execute entirely: the outputs are already in the CAS, and stdout replays from the record. --force bypasses it.

The input tree is read after the key was taken, so each file's bytes are held to the git blob id the key folded for it. A file edited or removed in between (an edit mid-run under vx watch) still runs, but the execution is not recorded under a key that no longer describes it; the record once replayed the edited outputs on every machine after the file was restored. The project's own package.json, which the key folds whether or not a glob lists it, is always in the input root.

This matters most under --download=none, where deferral leaves no local cache entry behind, so vx's own probe misses on every later run and the record is what makes the second run cheap. Records are checked against the CAS first (FindMissingBlobs) — the action cache and the CAS evict independently, so a record that outlived its blobs falls through to a real execution rather than "succeeding" with nothing. The replay itself is a cache read too: a transport failure reading the record, its stdout or any output warns and executes, after removing whatever the half-done replay created (core cleaned the outputs once, before the replay, and the real run's result must not inherit them). The probe sees a Tree by its own digest, not the blobs inside it, so one of those gone is caught by the replay: under any capture shape it fails the replay, and the task executes. (A fresh result under a whole-tree capture still only warns, since the capture holds more than the outputs.)

An UPSTREAM's evicted blobs get the opposite answer, because there is nothing to fall through to. When a dependency's outputs live only in the CAS — vx grafts them by reference precisely because no local copy exists — and those blobs are gone, the action cannot be built with the inputs its key claims. vx fails the task and names the upstream rather than shipping the action without them: a command that tolerates the absence exits 0, and that successful-but-wrong result would be cached under a key asserting those bytes were present. Which upstream bytes a command actually reads is unknowable — that is what dependsOn declares — so the refusal is the only sound reading, and it matches what core does when a deferred producer cannot be materialised. Re-run the upstream (--force) to repopulate the store.

Bringing a task's OWN outputs back gets the same treatment. Core's contract is that once an executor returns, the declared outputs are on disk, because the ordinary save path then tars whatever it finds — so an output blob that cannot be fetched is a hole that would be cached under a key claiming a complete build. Under a literal capture the worker returns only what output_paths named, so every returned file is a declared output and an unfetchable one fails the task. The exception is a glob whose FIRST segment is a wildcard (*.js): it has no REAPI spelling, so it is sent as '' — whole-working-directory capture — and inputs and undeclared siblings come back too. Those cannot be told apart from real outputs, so a missing blob there only warns, unless a declared glob names it: that is a hole in a declared output and fails the task under either shape. Inline output bytes are checked against their digest like fetched ones; a mismatch is fetched.

A capture cut at a wildcard holds more than the outputs: src/*.gen.js is sent as src, and the worker returns the sources beside the generated files. Only what a declared glob names (or what sits under a directory it names) is written back; the rest stays as it is on disk. Writing all of it once put the worker's copy of the sources over the user's, so an edit made while the action ran was lost. A directory a literal glob names is written whole.

The existence probe confirms the artifact, not just the entry

has() — what vx run --dry and --graph use to predict hit vs miss — reads the ActionCache entry AND checks the artifact blob is still in the CAS. The second call is not redundant, because servers disagree: measured against both, bazel-remote validates an ActionResult's referenced blobs and hides a dangling entry, while NativeLink serves it. Without the check, a plan would report cache hit (remote) for a task that then executes for real. It costs one extra round trip and only for a PREDICTED HIT — a miss still answers in a single call.

Downloads are verified

Every blob read — ByteStream and batch alike, compressed or not — is re-hashed with the negotiated digest function and length-checked against the digest it was requested under. Bytes that don't match are refused with a named integrity error instead of being written into the local content-addressed store: a corrupt or poisoned remote degrades to a miss (the cache invariant), never to wrong bytes under a trusted name. Uploads were always server-verified; this is the mirror on the read side, the same check Bazel's client performs. A streamed read (the cache artifact) is hashed as its bytes pass and errors at its end on a mismatch, so vx's ingest fails and the hit is a miss. The size is held as the bytes arrive: a body past its digest's size is refused at the byte that passes it, a zstd reply is decoded no further than that size, and a batch entry for a digest not asked for is dropped.

A verified blob still lands where the server's ActionResult says, so its paths are held to the workspace: an output path or Tree name that climbs out (.., absolute), a link whose target leaves the workspace (read as the OS follows it, through the links the result placed), and a directory that resolves out through a link are refused before anything is written through them; a link standing at an output file is replaced, never written through; a Tree file's setuid and setgid bits are dropped.

Artifacts stream

An artifact up to 256 KiB is read whole and sent in one batch, with no probe first. Up to the batch limit (about 4 MiB) it is read whole once, probed with FindMissingBlobs, and those bytes sent. Past that the cache layer never holds an artifact whole: put digests the file-backed Blob vx hands it in one pass over its stream, asks FindMissingBlobs, and uploads from a second pass, read chunkBytes at a time as the ByteStream write drains, identity-encoded (the artifact is zstd already). get returns the ByteStream read as a Response, each message taken from the call as vx writes the previous one to disk.

Deadlines: a wedged server degrades, it does not hang

Every call carries a deadline, because the killer case is not a server that is DOWN — that is an instant UNAVAILABLE the cache layer degrades to a miss — but one that accepts TCP and never answers. Without a deadline no error ever happens and the first probe hangs the whole run.

There are TWO deadlines, and the split matters:

| option | covers | default | | --------------- | ------------------------------------------------------------------------------------- | ---------------------------- | | metaTimeoutMs | Capabilities, GetActionResult, UpdateActionResult, FindMissingBlobs, QueryWriteStatus | min(callTimeoutMs, 15 000) | | callTimeoutMs | ByteStream transfers, Batch{Read,Update}Blobs, Split/SpliceBlob | 30 000 |

A control-plane message is small and bounded: a healthy server answers in single-digit milliseconds. A bulk transfer is size-proportional and legitimately slow — capturing a node_modules tree is what pushes real deployments to raise callTimeoutMs into the minutes. With one knob for both, buying headroom for that upload also buys every metadata probe the same minutes before it can degrade, which is the opposite of what the deadline is for.

Each deadline must be a positive number of ms; anything else is refused when the plugin starts. executeTimeoutMs and queueTimeoutMs are held to the same rule when the executor starts, so with execute off they are not checked.

That is not hypothetical. A NativeLink instance degraded into a state where it answered every ActionCache MISS in 3 ms and every HIT never — idle CPU, nothing in its logs, cleared by a restart with identical on-disk data. With a single 180 s deadline, every task burned three minutes on a lookup before failing. Now the probe gives up in 15 s and the run re-executes.

What the run prints is one line per kind of failure: vx/reapi: probe <key> at <endpoint> failed: 4 DEADLINE_EXCEEDED: …. Core's layered cache names the request, the vx key and the server (a gRPC status carries none of them), and a later request that fails with the same status is counted, not repeated: the run ends with vx/reapi: N more requests failed the same way: ….

Execution streams are deliberately NOT bounded by either: queueing behind a busy worker pool is legitimate. A wedged server still cannot reach Execute, because the deadline-bounded Capabilities call runs first. The wait for a worker is bounded by queueTimeoutMs when set: no EXECUTING within it, the Execute stream is closed, the operation is cancelled with Operations.CancelOperation (one attempt on metaTimeoutMs; a server that refuses it, or lacks the service, may still run the action, and its result lands in the action cache) and the task is given back to vx, which runs it here and says so once: [vx] <task>: vx/reapi: no worker started the action within queueTimeoutMs (…ms); its operation was cancelled — running it here. A task placed exec.remote: 'only' fails with that reason instead. Unset, the task's own exec.timeout bounds the queue as it bounds the run (the task fails as timed out); with neither, the wait is unbounded. A stream that drops with a transient status, or ends cleanly before its operation is done, re-attaches with WaitExecution (three times, backing off 100, 400 and 1600 ms) rather than running the action again. If WaitExecution answers NOT_FOUND — the server lost the operation, as a restart does — nothing is left running to re-attach to, and the action is executed again on the same budget. Once a worker reports EXECUTING, the task's exec.timeout (or executeTimeoutMs) bounds the wait, and the bound holds across a re-attach: firing during the backoff between a dropped stream and its WaitExecution, it ends the task there rather than being lost. The run stopping (Ctrl-C, an embedder's abort) cancels the operation stream the same way, and an action not yet submitted is not sent.

A ByteStream Read, like every unary call, retries UNAVAILABLE, RESOURCE_EXHAUSTED and INTERNAL three times (100, 400 and 1600 ms) before it counts as failed. A streamed read (the artifact of a remote hit) spends the same budget across the whole blob: a cut after bytes have reached the reader re-opens the Read at read_offset = what the reader has, and the digest is checked over the joined bytes as before. INTERNAL is on the list because it is how the gRPC client reports a call cut in transit: the RST_STREAM(INTERNAL_ERROR) a proxy sends when the server behind it goes away, or a stream that ends with no gRPC status. The same three statuses are what re-attach a dropped execution stream.

A server that stays down pays that backoff once: after a call spends its retries on UNAVAILABLE, the cache path's calls (unary, Read, Write) give up at their first UNAVAILABLE until the server answers again, so a refused port costs a five-task run 2.6 s, not 24.

A status the server answers with and no retry heals (PERMISSION_DENIED on Execute, a refused upload) fails the task with a line naming the task and the status, never as a vx "internal error".

A failed READ is never a failed task. The execution-record lookup is a shortcut past the worker, so a transport error there means "no usable record" and the task executes normally, with a warning naming why. The UPSTREAM record reads are the deliberate exception — those decide whether a dependency's bytes exist at all, and carrying on past a failure there is how an action runs without its inputs and caches the result.

Tests

bun test runs the unit suite anywhere. The round-trip suite needs a real server:

docker run -d -p 19092:9092 buchgr/bazel-remote-cache:latest \
  --dir /data --max_size 1 --grpc_address 0.0.0.0:9092 --http_address 0.0.0.0:8080

VX_REAPI_TEST_ENDPOINT=127.0.0.1:19092 bun test

Without an endpoint those tests skip; CI sets VX_REQUIRE_REAPI=1, which turns an absent endpoint into a failure so the suite cannot silently vanish. The remote-execution suites need an execution server in VX_REAPI_EXEC_ENDPOINT; CI sets it with VX_REQUIRE_REAPI_EXEC=1.

Protocol coverage

All 14 RPCs across the five services, not a working subset:

| Service | RPCs | | --------------------------- | ---------------------------------------------------------------------------------------------- | | Execution | Execute, WaitExecution | | ActionCache | GetActionResult, UpdateActionResult | | ContentAddressableStorage | FindMissingBlobs, BatchUpdateBlobs, BatchReadBlobs, GetTree, SplitBlob, SpliceBlob | | Capabilities | GetCapabilities | | ByteStream | Read, Write, QueryWriteStatus |

Protocol features in use, not just reachable:

• Digest negotiation — SHA256 by default (the universal baseline; the Merkle encoders must hash with the SAME function as every upload, so auto-upgrading would mix functions inside one action). The plugin offers no other.
• zstd compression — compressed-blobs/zstd/… resource names on ByteStream and compressor: ZSTD on batch updates, enabled only when supported_compressors says so.
• RequestMetadata in the well-known binary header (tool name/version, action id, correlated invocations id) — how a server groups an action's dozens of CAS/AC calls into one build in its UI.
• Inline stdout/stderr on ExecuteRequest, sparing two CAS round trips per finished action. A failed execution status that carries a partial result (a worker past its timeout) still prints what the command wrote.
• Execution stages — QUEUED / EXECUTING / COMPLETED decoded from ExecuteOperationMetadata, so a queued action is distinguishable from a hung one.
• Action.platform (v2.2) alongside Command.platform for older servers.
• NodeProperties — unix_mode and mtime on tree nodes.
• Output directories via the Tree blob an OutputDirectory.tree_digest addresses, plus output symlinks (a v2.0 server's output_file_symlinks / output_directory_symlinks when it sends only those). A tree's small files are fetched together across its directories (BatchReadBlobs, 64 MiB at a time), not one call per directory.
• Upload minimality — FindMissingBlobs first (split so no request passes a 4 MiB message), then batched blobs while they fit the server's budget and ByteStream beyond it.

Remote execution

Off by default. Remote execution changes where a user's build runs, which is not something a plugin should switch on merely by being configured for caching:

reapi({
  endpoint: 'grpcs://grpc.example.com:443',
  execute: true,
  platform: { 'container-image': 'docker://alpine:3.20', OSFamily: 'Linux' },
  capacity: 64, // concurrent remote tasks; becomes the scheduler's pool
})

A cache-only server (bazel-remote advertises exec_enabled: false) makes the plugin decline the executor with a warning rather than submit work that will never be answered. Only cacheable tasks are eligible — a task with no cache block has no described inputs, so a worker would run it against an empty input root. A cacheable task's root holds its declared input files and its project's package.json.

Verified end-to-end against a live NativeLink scheduler + worker: input tree uploaded, QUEUED → EXECUTING → COMPLETED streamed, stdout returned inline, declared outputs materialised byte-correct, and the worker attributed. Every hand-rolled encoder AND decoder is pinned byte-for-byte against protobufjs over the same vendored protos — the decoder tests exist because a wrong field number parses garbage without ever erroring (tests/encoding.test.ts).

One environmental note for NativeLink specifically: its official image is distroless, so a worker inside it has no /bin/sh and cannot run any vx task. tests/helpers/nativelink.md has the three-command busybox rehost.

node_modules: install as an action

REAPI workers are stateless, and vx deliberately treats node_modules as ambient environment rather than a cache input — so a remote task cannot see the packages a build needs. The answer is the design doc's §7.4 recipe, exec.remote: 'only':

install: {
  exec: { command: 'pnpm install --frozen-lockfile', remote: 'only' },
  cache: {
    inputs: { files: ['package.json', 'pnpm-lock.yaml'] },
    outputs: { files: ['node_modules/**'] },
  },
},
build: {
  dependsOn: ['install'],
  exec: { command: 'tsc -p .' },
  cache: { inputs: { files: ['src/**'] }, outputs: { files: ['dist/**'] } },
},

What actually happens, all verified live against a NativeLink scheduler + worker (tests/vx-run-e2e.test.ts):

• install executes on a worker — so platform binaries build for the worker's platform, not the laptop's — and runs once per lockfile change, ever: repeats are satisfied from an execution record the plugin keeps under the task's vx cache key.
• Its outputs never land on the submitter's disk: not materialised, not restored, and the local node_modules a dev installed is never cleaned.
• A dependent task's input tree grafts the install outputs by reference (per-file digests from the execution record; whole directories as re-canonicalised REAPI Trees), so the bytes flow worker→CAS→worker and never transit the submitter. The graft applies ONLY to outputs that exist nowhere locally: when an upstream's outputs are materialised on this machine, local disk is truth — two machines racing a nondeterministic miss can leave the artifact store and the execution record holding results of different executions under one pure-input key, and a worker fed the record would see bytes this machine's own tasks do not.
• With no remote executor declared, install is a local no-op and dependents use whatever the machine has ambient — a laptop run behaves exactly as it did before the field existed.

The execution record lives under sha256("vx-reapi-exec-v1\0" + key) — a second AC namespace beside the artifact mapping, listing outputs file-by-file with workspace-relative paths.

What the worker's environment contains

An action's Command carries exactly two of vx's three environment lists, sorted by name — the proto requires that, so equivalent Commands hash alike:

• exec.env.define — literal name: value pairs from the task config. They read the same on every machine, so they are safe to put into the action identity, and they are already in the vx cache key.
• cache.inputs.env — the values this machine resolved for those names, for each name the task's local child would get too (it is also in exec.env.passThrough). They are in the vx cache key by definition, so a change to one already produces a different action. A name unset here, or one the config only tracks, is left out of the Command, so the worker sees what a local run would: shipping a tracked-only value ran the worker on something a local run never saw, under the same key (item 1092).

A define wins over an inputs.env entry of the same name: it is the more explicit statement of intent.

Nothing else crosses. In particular exec.env.passThrough does not reach a remote worker, and neither does the essential allowlist (PATH, HOME, TMPDIR, …) — those are the submitting machine's resolved environment, and the worker has its own. Two reasons, and both matter:

• Host values in the Command would enter the action digest, so a laptop and a CI runner would never share a remote entry — the same reason the vx key excludes them.
• A Command blob lives in the CAS. passThrough is where secrets go, and a token written to a shared content-addressed store is readable by anyone who can name its digest.

So a task that needs a value on a worker must define it (config literal) or list it in both cache.inputs.env and exec.env.passThrough (host value, keyed, and seen by a local run the same way). A task whose command reads a passThrough secret is one to keep local with exec: { remote: false }.