Flume

A disciplined harness for AI-derivation pipelines. One tick = one phase × one agent invocation = one commit (or zero). The harness enforces output shape, capability scoping, validation gates, and baton mechanics; prompts say what the agent produces, not what discipline to remember.

What it is

Flume turns a prose multi-stage AI pipeline into a typed declarative chain. Each Phase is a typed declaration — prompt template, writable paths, concurrency mode, gates, handoff rules. The dispatcher runs one tick at a time, validates the resulting commit, and signals the next phase to wake.

Each tick is stateless. The agent invocation reads committed artifacts from disk; there is no in-memory continuity. If a commit fails a gate, the harness reverts and the phase re-runs against the same input plus the validation error.

Posture

• Disk is truth. Every tick is a fresh agent invocation reading committed artifacts. No sessions, no in-memory continuity.
• Stateless ticks. One tick = one phase × one agent invocation = one commit (or zero).
• Harness enforces, prompts state. Validation gates, capability scoping, output shape, and baton mechanics live in the harness. Prompts say what the agent produces, not what discipline to remember.
• Conservative derivation. A derived layer never invents intent absent from its source.
• Structured handoff. Inter-phase contracts are JSON schemas, not prose conventions. Markdown is reserved for documentation and prose surfaces.

Quickstart

Prerequisites: Node 22 or newer, and git 2.36 or newer — the engine reads git worktree list --porcelain -z, which git grew in 2.36; on an older git, worktree reclamation degrades loudly and nothing else does.

One npm package ships two things. The engine (@dtmd/flume) is mechanism: ticks, worktrees, gates, verdicts, pickability. The harness package (@dtmd/flume/harness) is flume's own opinion about how to run it — plan slices, a build phase, their prompts and discipline, the entry extension, the judges, the records. Adopting the harness is one command:

npx flume-harness init

It writes, into the current repository:

• .flume/declaration.ts — the one file your environment decides: where a per cite may point, what each phase may write, the test runner the judge drives, which plan slices run. A skeleton; its placeholders are yours to fill in.
• .flume/chain.ts — the hop the engine loads, applying the package's factory to that declaration. The engine refuses a load without it, and nothing in it is yours to tune.
• .flume/package.json — "type": "module", and nothing else: the module scope the two files above load in. flume is ESM-only, and a chain loaded as CommonJS stops resolving it on node 22. Your own repository's manifest is untouched — a CommonJS repo adopts and its chain still loads.
• .flume/plan/pending/.gitkeep — the empty queue directory, one JSON file per entry once a plan slice writes them. Nothing else creates it, and a plan slice refuses over an absent queue. The placeholder is what keeps the directory in the tree once every entry has shipped — git holds no empty directory, and the engine reads only *.json directly under it, so nothing reads the placeholder as work.
• .flume/plan/state/plan-derive.json and .flume/plan/state/plan-sweep.json — the plan cursors, stamped at the commit you adopted on, so your first plan wave is about what lands next rather than about every line of spec written before flume was here. Written only where the repository has a commit for a cursor to stand at; adopt before your first one and init says so and writes neither.
• .flume/PROTOCOL.md — the project-side conventions no declaration encodes.
• the runtime .gitignore lines, and @dtmd/flume in your package.json.

It never rewrites what it wrote: adoption is once (a second init refuses over a state root that is already there), and upgrading is one version bump plus the release's migration note.

Install the dependency with whichever package manager this repo uses, edit the declaration's placeholders, then:

npx flume tick      # one phase × one agent invocation
npx flume status    # baton state
npx flume loop      # tick until hibernation

A consumer never copies a prompt, a slice, or a judge out of the package — what it wants to change, it declares.

The engine-level path: a chain you write

Take this one when flume's workflow is not the workflow you want: different phases, different discipline, or just the plugin surface with nothing layered over it. No harness package involved.

npm install --save-dev @dtmd/flume

A chain is a plugin loaded into the engine, not a consumer of it: the engine calls your chain's factory with its own API, so the chain never imports an engine value and never resolves an engine copy of its own. One engine per process, whichever binary you invoked — there is no second copy for instanceof or module state to split across. Your only engine import is import type, which is erased at runtime, so the declared dependency exists purely for types.

Drop a .flume/chain.ts into your repo:

// .flume/chain.ts
import type { Chain, ChainFactory, Phase } from "@dtmd/flume";

const factory: ChainFactory = (flume) => {
  const echo: Phase = { name: "echo", description: "Hello-world tick: write a note to disk.", promptPath: "prompts/echo.md", concurrency: "singleton", writablePaths: ["notes/**"], gates: [], handoff: () => [] };
  const chain: Chain = { phases: [echo], humanOnly: [] };
  return { chain };
};

export default factory;

Gates, agents, and schema helpers arrive on that flume parameter (flume.tscGate, flume.claudeCode, flume.pendingGate, …) — see docs/CHAIN-AUTHORING.md.

Author .flume/prompts/echo.md — the prompt template (Markdown plus {{KEY}} placeholders from promptArgs) — and the same three verbs above drive it.

For a readable single-phase starter with each field on its own line, see examples/minimal-chain.ts. For multi-phase pipelines (plan-inbox → plan-derive → build), copy examples/cascade-chain.ts instead.

The chain

A Chain is an ordered list of Phases. Each Phase declares:

• prompt — a template rendered against TickContext (committed files, baton state, pending entries).
• writablePaths — globs the agent's commit must fit inside. Anything outside reverts the commit.
• concurrency — "singleton" (one tick per phase per cycle) or "fanout" (multiple parallel ticks across disjoint pending entries).
• gates — typecheck, tests, lint, custom — each declared afterCommit (per-tick) or afterMerge (post-fanout).
• handoff — which phase(s) wake after a successful commit.

The pending entries themselves are typed. A plan-style phase writes .flume/plan/pending/<tag>.json, one file per entry; the schema (PendingEntry, PendingList) is the contract between plan and build. Built-in gates (tscGate, vitestGate, eslintGate, writablePathsGate) cover the common cases; custom gates are plain functions returning GateResult.

Chain residency

One chain per .flume. The chain lives at <configDir>/chain.ts, and the two roots move independently: FLUME_DIR relocates the mutable state root, FLUME_CONFIG_DIR relocates the chain and prompts dir. Relocating state never changes which chain governs the tick.

Per-checkout variation is already served, twice over:

• A checkout runs on whatever branch the operator is on. Edit .flume/chain.ts there and the variation lives and dies with that branch — linked worktrees give concurrent divergence, since each resolves its own checkout's chain.
• A chain is code. The factory is handed the resolved roots (api.paths) before it returns a chain, so one repo-resident chain can shape itself around where this run's state lives.

Concurrency

Phases declared concurrency: "fanout" partition their pending entries by file overlap (partitionByFileOverlap). Disjoint entries spawn parallel worktrees; each worktree runs its agent invocation and afterCommit gates in isolation. As each entry finishes, the dispatcher cherry-picks its span onto a candidate — a tree of its own at the current trunk tip — runs the afterMerge gates there, and fast-forwards the trunk to the candidate once they are green. A cherry-pick conflict keeps that entry in pending; an afterMerge failure discards that one candidate, so the trunk never names the commit and the siblings it already fast-forwarded to stay shipped.

Worktree branches are named flume/<checkout>/<entry-slug>, and the engine mints no level beneath the worktree base. Two efforts are two checkouts, each with its own state root and so its own base, so identical entry tags in two efforts already address two directories — but linked checkouts share one ref namespace and one git common dir, which directories do not separate. The <checkout> segment does: a name the checkout owns, stable for its life and distinct between any two checkouts of one repository, so two efforts at one phase mint two branches instead of colliding on one. The per-entry claim a build tick stakes carries the same segment, so a sibling checkout's claim on a tag of the same spelling never hides this checkout's entry.

Worktrees are the only isolation primitive the harness ships. Docker / sandbox layers are deferred until a chain needs a boundary a worktree cannot draw.

Where state lives

Everything is on disk under .flume/, and the root has two owners — the runtime, and whichever chain ticks against it.

Harness-managed state — every name here is one the runtime spells itself (STATE_ROOT_NAMES and the accessors in src/paths.ts):

• .flume/awake/<phase> — baton flag files. Presence = phase is awake.
• .flume/held/<phase> — hold markers, the operator's end of the baton: flume hold <phase> writes one and clears the phase's flag, flume wake removes it. A hold outranks a handoff's wake, so a handoff never removes one.
• .flume/plan/pending/<tag>.json — structured handoff between plan and build, one file per entry so two producers and a ship merge without a conflict. This is the default location; a chain moves the directory with Chain.pendingDir.
• .flume/prior-attempts/<keyspace>/ — one record per reverted attempt, written beside the baton so it outlives the worktree that produced it, and scoped by keyspace (entry/, phase/) so a tag and a phase name that slugify alike never share a file.
• .flume/rendered-prompts/ — each invocation's fully rendered prompt, persisted before the agent runs, and kept while a verdict the history log still holds names it. The directory is bounded by that history's own retention and by no second setting, and a prompt inside a live run's window is never trimmed — it is an agent whose row has not landed yet.
• .flume/worktrees/<slug>/ — one worktree per tick: the entry's slug under fanout, the phase's own under singleton. The base dir is relocatable — by the operator via FLUME_WORKTREES_DIR, by the chain via Chain.worktreesBase (below).
• .flume/loop.pid — cross-process loop lock, present while a flume loop runs against this state root (below).
• .flume/stop — graceful-stop flag: flume stop writes it, a live loop ends on it and flume loop refuses to start over it.
• .flume/tick-verdict/<phase>.json, .flume/tick-verdicts.jsonl — each phase's last verdict and the bounded history behind them (readTickVerdicts). One file per phase because a flume loop run holds one child per awake phase at once, and the supervisor reads each child's facts back under the phase it named that child with.
• .flume/invocations/<phase>.jsonl — the usage rows the phase's running tick has appended, one per agent that has returned. Written as each agent returns rather than when the tick ends, so a tick that dies after paying for an agent keeps that agent's row; the verdict above composes its invocations[] from this file and the next tick of that phase clears it.

Everything else under the state root is its chain's — placed by the chain rather than the runtime, and the chain's alone to move. This repo's own chain writes:

• .flume/plan/state/ — the plan state the harness package keeps across ticks, one file per writing slice: the derive cursor in the derive slice's file, the sweep cursor and the sweep's continuation signal in the sweep slice's, the per-lane drained-run stamps in the inbox slice's. Each is a typed artifact read through the package's own accessor, and each plan phase's fence admits its own file alone, so two slices stamping in one wave merge as disjoint files.
• .flume/plan/questions/ — the forks plan parks for a human, one file per open question: present while the question is open, deleted when it is answered, so any session can add one without conflicting with another.
• .flume/inbox/ — transient findings queue, one file per finding, drained by plan.
• .flume/sessions/<timestamp>.jsonl — captured agent NDJSON (opt-in via withSessionCapture), rooted at api.paths.flumeDir so it tears down with the rest of the dock.

Ticks read these on entry and write them on commit. Across ticks, the disk is the only carrier of state.

Relocating state: FLUME_DIR / FLUME_CONFIG_DIR

The two halves of .flume/ relocate independently via env vars:

• FLUME_DIR moves the mutable state — the baton (awake/), pending (plan/), worktrees (worktrees/), prior-attempt records (prior-attempts/), and each invocation's rendered prompt (rendered-prompts/). A chain's own per-run artifacts ride along only because the chain roots them at api.paths.flumeDir — this repo's session logs (sessions/) are chain-placed, not runtime-owned.
• FLUME_CONFIG_DIR moves the chain + prompts — chain.ts and the prompt files it references.

Both default to <repoRoot>/.flume. A set-but-relative value resolves against the cwd. They cross the loop→tick process boundary by inheritance: the supervisor's flume tick children run with no env: override, so they see the same resolved values.

This buys an attach-work-detach posture: point FLUME_DIR at a tmpdir outside the working tree, run the loop, and tear the whole footprint down with a single rm — no state bleeds into <repoRoot>/.flume.

export FLUME_DIR="$(mktemp -d)/flume-dock"
flume loop                  # baton, pending, worktrees, chain artifacts under FLUME_DIR
rm -rf "$(dirname "$FLUME_DIR")"   # one rm removes the whole dock

One checkout still resolves one state root: a dock pointed at a tmpdir while <repoRoot>/.flume already holds runtime state of its own — a baton, a worktree base, a verdict log — is refused where the roots resolve, naming both. Two efforts at once are two checkouts (git worktree add), not two roots under one. A bay holding only your chain is a config dir, not a second root, so the posture above composes against an adopted repo as written.

A relocated dock is expected to live outside the repo (e.g. a tmpdir), so .gitignore needs no change: the default <repoRoot>/.flume stays ignored as today, and an out-of-tree dock is invisible to git by construction. The one-rm guarantee holds only if every per-run artifact your chain writes also lives under FLUME_DIR — see docs/CHAIN-AUTHORING.md for the chain-author requirement.

Relocating worktrees: FLUME_WORKTREES_DIR, Chain.worktreesBase

Every tick provisions a worktree — a fanout entry's, or a singleton phase's own (spec/worktrees.md, "Singleton runs in a worktree") — and both default to <flumeDir>/worktrees — inside the state root, so they move with FLUME_DIR and are covered by the one-rm teardown.

Two things move that base, in this order: FLUME_WORKTREES_DIR when set (a relative value resolves against the cwd), else the chain's own Chain.worktreesBase when it declares one, else the default. The env var is the operator's, on a host whose chain file they may not own, so it wins.

Chain.worktreesBase is a function of the roots the runtime resolved — (paths) => string, returning an absolute path, called once when the chain loads (see docs/CHAIN-AUTHORING.md). A function rather than a path because chain.ts is committed and placement is per host; declaring one is how a chain relocates its worktrees without exporting an environment variable at module scope, before flume's own module has loaded.

The relocation exists for one specific hazard: an agent whose working directory contains the root checkout's path as a prefix (the default <repoRoot>/.flume/worktrees/<entry> does) can derive the root from its own cwd and operate there instead of in its worktree — a stray write the writable-paths guard never sees, because it lands outside the worktree being diffed. Pointing FLUME_WORKTREES_DIR at a directory outside every repo-path prefix (e.g. a sibling tmpdir) removes the vector. If you relocate worktrees outside FLUME_DIR, they leave the one-rm footprint — they are ephemeral (created and removed per wave), but a crashed run can strand one there.

One loop per state root

flume loop takes <flumeDir>/loop.pid by exclusive-create — the same stake the tip claim below is taken with, so two starts racing cannot both take it — and writes its pid there. A second loop started against the same state root is refused (exit 1, naming the holder's pid) while the recorded pid is alive — two supervisors racing one baton would corrupt plan/build state. A stale pidfile left by a dead process is reclaimed automatically, and the lock is dropped on normal exit, SIGINT, SIGTERM, and a hangup (SIGHUP, or SIGBREAK on win32), so no manual cleanup is ever required.

The lock lives under flumeDir, not the repo: the state root is the resource that races, and a dock relocated via FLUME_DIR carries its lock with it — two loops against different docks over the same repo are allowed.

One flume writer per tip

loop.pid guards the state root; it says nothing about the tip (the ref HEAD resolves to) the state root's ticks commit onto — two runs with separate state roots, or a bare tick racing a loop, can still write to the same ref. flume loop closes that gap with an advisory per-ref claim: it claims the tip at start and releases it at exit, exclusive-create at <git-common-dir>/flume/tip-claims/<ref path> (e.g. .git/flume/tip-claims/refs/heads/main). The common dir resolves identically from every linked worktree, so a claim taken in one is visible from all of them. A second loop against the same ref refuses (exit 1), naming the holder's pid; a stale claim (holder process dead) is reclaimed silently, the same stake as the state-root lock above. A bare flume tick takes the same claim around its single tick, so two of them on one ref refuse rather than interleave; a loop-spawned tick child runs under its supervisor's claim and takes none of its own. Both verbs refuse outright (exit 1) on a detached HEAD, since the claim keys on a named ref.

It is advisory, not exclusive against every possible writer — a signal plus a fact when the signal is bypassed. flume status reports the current tip's claim, and the state root that claim was taken for (tip claimed by pid N for <state root>, or stale), alongside supervisor liveness.

Every run that ends says how it ended in <flumeDir>/run-end.json — the reason (hibernation, the stop flag, the tick budget, the abort threshold, a signal, or one of the walls the supervisor fail-fasts on), the signal or the child exit code that identified it, and the instant — and flume status prints it. A death no handler runs on (SIGKILL, TerminateProcess) records nothing, and status reports that from the stale loop.pid the dead run left rather than from the previous run's record.

Trunk contract: HEAD is truth

Commits land on the checked-out branch of the working tree the loop runs in. Every tick's agent commits on a private flume/** branch in its own worktree, and the span cherry-picks back onto HEAD — a singleton phase's exactly as a fanout wave's (spec/worktrees.md, "Singleton runs in a worktree"). The runtime never switches branches — there is no trunk configuration to point it elsewhere. Checkout is a human act: whatever branch is checked out when the loop starts is the branch the run ships to.

An operator committing mid-tick is ordinary history, not interference. Before each cherry-pick the dispatcher asks one question — does another process hold a live claim on this tip? — never whether the tip still matches a sha it recorded earlier. No live foreign claim means whatever moved the ref was not an engine, so the span picks onto whatever tip is current and the foreign commit sits under it, exactly as if it had landed between ticks; git's own conflict detection arbitrates content (a conflicting pick aborts, and that entry stays pending for a retry against the tip that now carries the foreign commit), and the chain's afterMerge gates own the merged tree. The pending-ledger commit absorbs the same way, which is what keeps a tick from ending with the queue behind the tree it describes.

Refusal is reserved for what absorption cannot make safe. A live claim held by another process is a second engine interleaving merges onto one ref: the wave makes no commit for every entry it has not already merged, those entries stay pending, and the tick reports a tip-moved fact. On the agent's side the check is ancestry rather than equality — a flume/** branch has exactly one legitimate writer, so an agent that commits twice has produced a completed tick, and only a recorded base that is no longer an ancestor of the observed HEAD (something rewrote it out from under the agent) refuses, naming both shas. That split is what the claim above buys beyond a bypassable signal: an engine instance always holds one and an operator never does, so it is how the dispatcher tells legal history from a second writer without inferring anything from the commits themselves.

Concurrent efforts: one checkout each

One loop per tip. Singleton ticks, fanout cherry-picks, and merge-gate reverts all mutate the working tree's HEAD; two loops writing to one ref race it. The engine partitions no state below the checkout — it mints no per-effort namespace, seeds no second root, and offers no selector that retargets one — because a partition under one checkout buys separate files and no separate execution: the tip claim serializes two efforts sharing a tip whatever their files are called.

So a repository running several efforts at once gives each one a checkout of its own, via git worktree, and each resolves its own .flume:

git worktree add -b docs-refresh-wip ../flume-docs-refresh
cd ../flume-docs-refresh
flume loop --max 20
flume status    # awake phases + pending count, for this checkout

Contention on git's shared .git/worktrees metadata is accepted: a race fails one git command → one tick, the entry stays pending, and the stateless-tick loop retries. Overlapping writablePaths across concurrent checkouts is operator responsibility.

The history an effort produced — the commits it caused, on whatever branch it ran on — stays exactly where it landed; integrating or discarding it is an ordinary git operation, the operator's to run.

Where state itself needs to live elsewhere, FLUME_DIR and FLUME_CONFIG_DIR relocate the two roots independently, and nothing else does. Full per-command contracts — steps, refusals, exit codes — in docs/CLI.md.

Status

v0.x — stable enough to depend on for a project that lives ≥3 months without rework. The four core types (Phase, Chain, Gate, and the pending entry schema) carry the compatibility line; the current version is the one package.json and the top of CHANGELOG.md name.

Pre-1.0 ships minor-version breaking changes when the public API surface needs to shift; patch versions never break. Each break lands with a ### Breaking entry in CHANGELOG.md. Stabilization at 1.0 follows enough usage signal to commit under semver.

Pointers

• docs/LAYERS.md — the three layers and their borders: who owns what, how each is extended from outside, where a finding goes.
• docs/INTENT.md — design rationale: the spine, what stays prose, what becomes JSON, non-goals.
• docs/MIGRATING-0.23.md — upgrade note for a chain on 0.22.x: trunk moving only by fast-forward to a judged candidate, the consumer owning its test scope (Lane.excludes gone), GateContext.assignment, every tree provisioned, gate-child deadlines, and five smaller breaks.
• docs/MIGRATING-0.22.md — upgrade note for a chain on 0.21.x: priority off the entry core (strip it before upgrading), the queue as a forest with a computed order, ticks loading only the repository's Claude Code settings, and six smaller breaks.
• docs/MIGRATING-0.21.md — upgrade note for a chain on 0.20.x: three breaks at the render and loader seams — renderPrompt off the root's value surface, taking a template's bytes, and chainLoader returning LoadedChain — and the behavior changes a long-running wave brings.
• docs/MIGRATING-0.20.md — upgrade note for a chain on 0.19.x: one break, the legacy queue and plan state pages off the plan fence, and two behavior changes, an inbox record counting once committed and a CI lane reading only the tip's own run.
• docs/MIGRATING-0.19.md — upgrade note for a chain on 0.18.x: two shared files become two directories. The pending queue becomes one JSON file per entry, which moves a chain field, a gate-context field, three parse exports, every fence glob and prompt span naming the queue, and the queue you already have on disk; the harness package's plan state becomes one file per plan slice, which moves the accessors, the fence helper, one prompt arg, and the page you already have on disk.
• docs/MIGRATING-0.18.md — upgrade note for a chain on 0.17.x: one API break, the suspect-flake marker off the gate-revert record, with the gate's own blamesSpan: false in its place.
• docs/MIGRATING-0.17.md — upgrade note for a chain on 0.16.x: no API change, one on-disk one. The loop lock and the tip claim each state the holder's pid on the first line and the instant it took the guard on the second, so a mixed-version fleet against one state root is the step to plan for.
• docs/MIGRATING-0.16.md — upgrade note for a chain on 0.15.0: five renames and engine-taken defaults, plus the opt-in port onto the harness package. Each section opens with the grep that says whether it applies to your chain.
• docs/MIGRATING-0.15.md — the note behind it, for a chain on 0.14.0: four breaks, led by the voluntary-bail → clean-exit rename a typecheck does not catch, plus the engine facts 0.15.0 now reports so your chain can stop rebuilding them.
• docs/MIGRATING-0.14.md — and the note behind that, for a chain on 0.13.0: one break (the queue path moves onto the chain) and the hand-carried blocks 0.14.0 lets you delete. Earlier notes — 0.11, 0.12, 0.13 — are linked from the release they cut in CHANGELOG.md.
• docs/MIGRATING-0.10.md — upgrade checklist for any chain on 0.2.x, 0.6.x, or 0.9.0 moving onto @dtmd/[email protected]. Routing table up front; work only the sections your starting version needs.
• examples/minimal-chain.ts — single-phase starter.
• examples/cascade-chain.ts — multi-phase plan-inbox → plan-derive → build pipeline.

License

MIT. See LICENSE.