@ai-dossier/sched
v0.21.0
Published
Deterministic scheduler core for dossier batch cycles — queue, slots, persistent state machine, dispatch engine with completion verification and stall/escalation ladder, PR watching with script-based teardown, and batch failure recovery (attribution, bise
Maintainers
Readme
@ai-dossier/sched
Deterministic scheduler core for dossier batch cycles — queue, worker slots, typed state
machines, crash-safe persistence, the dispatch engine (#464: spawning agent
processes, verifying their completion against ground truth, mechanizing the
stall/escalation ladder), and since #468 the PR watcher + tail work (parked-PR
watching, script-based teardown, cheap-tier report dispatch — retiring the fleet
pattern of re-dispatching a full-cycle run for the tail), and since #472 batch failure
recovery (attribution, bisect, one bounded fix, eviction, dissolve). The scheduler itself never
invokes an LLM — it spawns the agent process the operator configured and reconciles
the durable record (ai-dossier runstate / gh / git) that the spawned run leaves
behind.
Design: RFC-0001 Batch Cycles (rfcs/0001-batch-cycles.md)
§B/C.1/D — the §D state machines are frozen verbatim into the types below.
This package is the deterministic replacement for fleet-cycle's LLM-prose supervision,
whose named failure — slots sitting idle after a subagent finished — is a scheduling bug
this state machine makes impossible to forget.
CLI surface
Consumed through the monorepo CLI (@ai-dossier/cli ≥ 0.19.0):
ai-dossier sched enqueue --issues 101,105..109 --deps 100 --tier strong # flags
ai-dossier sched enqueue --from-manifest batch-prep.json # batch-prep output
ai-dossier sched start # the dispatch engine: spawn, verify, escalate, watch parked PRs (Ctrl-C stops it)
ai-dossier sched start --once # a single reconcile+refill tick (cron-style)
ai-dossier sched status # queue (+pr/cleanup), parked PRs, slots, batches, blocked/failed
ai-dossier sched pause # stop NEW assignments; live units keep running
ai-dossier sched resume
ai-dossier sched abandon --issue 42 --reason "operator abort"
ai-dossier sched abandon --batch b1 # dissolve; members requeue as full-cycle
ai-dossier sched stats --issues 4..9 # per-issue tokens/cost from ~/.dossier/runs.jsonl (#524)
ai-dossier sched stats --batch b1 --project owner-repo # batch member/tail/report/fix costs from raw dispatch logs (#564)Every subcommand except stats (without --batch) takes --project <slug> (default:
owner-repo of the current directory, falling back to the repo basename — fleet-cycle's
convention) and --json. stats without --batch reads ~/.dossier/runs.jsonl, a
single global file, not the per-project state — it takes --json and --issues only;
see "runs.jsonl telemetry" below for the resulting cross-repo caveat (the same issue
number in two repos sums together). stats --batch <id> instead takes --project like
every other subcommand and reads that project's ~/.dossier/sched/<project>/runs/
directory directly, reconstructing costs from the raw dispatch logs rather than
runs.jsonl (#564) — see "Batch members (#564)" below.
Since #507, enqueue additionally reads each candidate issue's live GitHub labels (one
gh issue view --json labels call per issue, resolved against the current directory's repo
unless --repo <owner/name> is passed) and lands an issue carrying decision-pending /
needs-clarification / epic / decomposed as blocked (reason: label:<name>) instead
of queued — without spending a slot on an agent that would only rediscover the same block.
A failed gh lookup fails open: the issue enqueues normally, with a warning and a
label-check-failed journal event.
The dispatch engine (#464)
sched start runs a tick loop (default 60s, --interval or reconcile_interval_ms)
where every mechanical supervision decision is code, not remembered prose:
- Dispatch (AC1) — a runnable unit is spawned as a detached agent process
(
claude -p --output-format stream-json --verbose --model <tier model>by default —jsonbuffers the whole session into a single write at exit, which left a 0-byte log for any dispatch killed before a clean exit (#524); opencode fallback; command/prompt/tier-models configurable), prompt on stdin, output appended toruns/<unit>.log. Each tier may fully override the command/model/prompt independently viadispatch.tiers.<tier>(#527) — a MIXED agent-CLI ladder, not just a different model on the same CLI: a unit can start onopencodeformechanical/midand be rescued onclaudeatstrong.dispatch.tiersis additive over the top-levelcommand/tier_models/promptshorthand — any field a tier leaves unset falls back to the shorthand, so an existing config with notiersresolves exactly as before. The opencode fallback runsopencode run --auto …(#506) — a git worktree is anexternal_directoryto opencode, whose default"ask"policy a headless session can only auto-reject, killing the agent mid-phase;--autoapproves any request not explicitly denied. pid, phase, role, and last-progress are persisted instate.json. Agents are unref'd: they survive a sched crash (restart reconciles by pid). The default full-cycle and fix prompts (DEFAULT_PROMPT_TEMPLATE,DEFAULT_FIX_PROMPT_TEMPLATE) appendNO_BACKGROUND_EXIT_INSTRUCTION(#497) — a headless session ends the instant the model stops responding, so an agent that starts a long build/test command and reports "waiting for it to finish" abandons the run with the subprocess still going; the instruction tells it to run such commands in the foreground and wait, or poll until they finish.DEFAULT_REPORT_PROMPT_TEMPLATEis excluded — it never spawns a long command. The prompt instruction alone was not enough (#591): agents kept arming theMonitortool to wait on a background command and ending their turn anyway, which the engine can only see as an unverified exit. Everyclaude-family command template (top-levelcommandand each tier's owncommandTemplate, #527) gets--disallowedTools Monitorappended automatically — setdispatch.disallowed_tools: []inconfig.jsonto opt out, or list your own tools to deny instead of the default["Monitor"]. Matched on the binary's basename, so an absolute or wrapper path (/usr/local/bin/claude) still gets it; never applied to a non-claudecommand or one that already carries the flag itself, so anopencodetier is unaffected. - Completion verification (AC2) — an agent exiting is never proof of completion.
On exit, the unit completes only when ground truth confirms it: the issue's latest
runstate milestone is
report done, or GitHub says the issue is closed — except a report-agent slot (role: 'report'), whose issue is already closed at merge: the closed signal is suppressed and only areport donemilestone completes it (#500). An unverified exit rides the recovery ladder like a stall. Areport donemilestone must also postdate the slot's ownspawned_at(±60s clock-skew tolerance, #575) — a re-enqueued issue's PREVIOUS run's report milestone is ignored (journaledstale-milestone-ignored) rather than instantly completing a freshly-spawned agent on its first reconcile tick; a legacy slot with nospawned_atdegrades to the old, unfenced check. Batch members get the same fence on their own completion signal (isMemberComplete,phase=review status=done mode=slot). - Reconciliation tick (AC3) — every tick detects externally-advanced state (someone
finished the work outside sched → complete, kill the leftover agent, reclaim the
slot), orphaned pids after a restart (dead pid on a running slot → exit rail →
verify), and progress (a new milestone
at=or a new pushed commit — the branch from the setup milestone watched viagit ls-remote). - Stall/escalation ladder (AC4) — no new milestone AND no new pushed commit for
stall_timeout_ms(default 30 min) → kill the agent and redispatch the same unit one tier stronger (mechanical → mid → strong; the resume rails carry work forward). The redispatch reads the NEXT tier's own resolved command (#527) —resolveDispatchpre-resolves every tier once per tick, so a mixed-CLI ladder rescues on a different agent CLI, not just a different--modelflag on the same one. Cap 2 escalations — or a stall at the strongest tier — fails the unit and blocks its TRANSITIVE dependents (dep-failed:<issue>). The timeout is phase-aware (#495): theimplementphase alone can run 1-3h on a large monorepo with zero intermediate milestone or pushed commit, so it gets a longer built-in default (90 min,DEFAULT_PHASE_STALL_TIMEOUT_MS) than every other phase's 30-min default — selected by the phase now IN FLIGHT (the last milestone'snext=, not the last completed phase; before any milestone posts it falls back to the slot's own phase). A built-in phase default is a FLOOR against the globalstall_timeout_ms— raising the global never silently shortensimplement's allowance. Override any phase viadispatch.phase_stall_timeout_ms: { "<phase>": <ms> }inconfig.json(validated against the known phase vocabulary — an unrecognized key is a config error, not a silent no-op); an explicit override always wins verbatim, even below the built-in default. A phase not listed keeps its built-in default (floored by the global) or falls back to the globalstall_timeout_msoutright. - Immediate refill (AC5) — a slot freed by a terminal state is refilled in the SAME
tick; a runnable unit never waits while a slot is idle (pinned by a regression test).
Refill was always synchronous; what previously had no journal trace was the release
itself — see
slot-releasedbelow (#525). - Journal (AC6) — every event (assigned, spawned, exit-detected, external-advance,
progress, stalled, redispatched, fence-written, fence-failed, unit-failed,
dependents-blocked, slot-released, suspect-dispatch, dispatch-unhealthy,
run-log-recorded, run-log-no-usage, run-log-skipped, run-log-failed, engine-stale,
engine-auto-upgrade-attempted, engine-auto-upgrade-failed, stale-milestone-ignored, …) is
appended to
events.jsonl;sched statusshows the live phase per unit, plus each slot'sgenandfencedstate (#504).engine-stale/engine-auto-upgrade-attempted/engine-auto-upgrade-failed(#537) are journaled OUTSIDE the engine —sched start's CLI-side staleness check appends them directly, not throughtick(). The label events (label-blocked/label-check-failed/label-cleared) come from BOTH sides:sched enqueueappends the first two at enqueue time before dispatch (#507), and since #544 the engine appends all three from its own per-tick label re-check.slot-released(#525) marks the exact tick a held slot reachesidleon a per-issue dispatch terminal path — verified completion, external-advance, a direct failure, a blocked dependent's release, or a detached-ship park — carrying the freedslotid and a closedreason(verify-complete/external-advance/unit-failed/report-failed/dependents-blocked/parked, the exportedSlotReleaseReasonunion), journaled right after that path's own cause event so an occupancy report reads release time directly instead of inferring it from the nextassignedon that slot. Not yet journaled bysched abandonor by batch-slot release, which walk a slot toidlethrough their own copies of the same edge table (tracked as a follow-up). - Dispatch-health pause (#505) — an unverified exit within
SUSPECT_DISPATCH_WINDOW_MS(60s) of a slot's last progress issuspect-dispatch: real work rarely produces zero milestones that fast, but an operator-billing quota/auth wall (a Claude Code weekly limit, a provider credit cap, …) that rejects the agent's very first request does, every time.DISPATCH_UNHEALTHY_THRESHOLD(2) consecutive suspect-dispatches from DIFFERENT units — the cross-unit correlation that tells a wall apart from one unit's own flakiness — auto-pauses new assignments (state.paused = true, journaleddispatch-unhealthy) exactly likesched pause: already-live slots keep running, only new assignments stop, including report-agent dispatch. The per-unit stall/escalation ladder above is unchanged — this only stops MORE units from being dispatched into a known-bad wall. A healthy dispatch outcome (verified completion or park) resets the streak; the pause itself clears only viasched resume(never automatically — an operator's explicit "I've addressed this," not a heuristic that could re-dispatch into a wall that hasn't actually cleared), which also clears the streak sosched status's warning doesn't linger against a wall the operator already acted on.
Config schema moves to 1.4.0 (#527): dispatch gains tiers — a per-tier
{ command?, model?, prompt? } spawn spec. command/tier_models/prompt remain valid
as the shorthand and are the fallback for any field a tiers entry leaves unset, so a
1.3.0 config with no tiers at all resolves identically to before — there is no on-disk
migration, only resolution-time fallback in resolveDispatch.
Config schema moves to 1.6.0 (#562): dispatch gains suite_command — an explicit argv
override for the aggregate batch-suite command, the middle tier of the new resolution
order (cap run test.full manifest → dispatch.suite_command → a repo-detected safe
default that never forwards extra flags through an unrecognized wrapper script). Also new:
the blocked BatchStatus and batch-blocked journal event — an unreadable suite report
blocks the batch (worktree and every member commit preserved) instead of dissolving it,
distinct from dissolving's "a red suite named no offender." blocked is a new
persisted-field value, not a new field, so no state.json schema-version bump or
backfill is needed on load — but it is NOT downgrade-safe: an older build's
BATCH_STATUSES (derived from its own BATCH_TRANSITIONS) will reject a state.json
containing a blocked batch with "unknown batch status", bricking that project's whole
SchedStore.load(). Resume or abandon any blocked batch before downgrading past this
version.
Config schema moves to 1.7.0 (#563): a new top-level dissolve_policy key —
{ fraction, min_evictions_before_dissolve }, both required when the key is present —
overrides the batch dissolve threshold (default { fraction: 1/3,
min_evictions_before_dissolve: 1 }, RFC-0001 §F.8's ⅓ with no additional floor). An
absent key falls back wholesale to the default; an invalid one degrades the WHOLE config
file to built-in defaults, same as every other config field (loadConfig's
degrade-to-defaults contract). This release also adds the partial dissolveBatch
strategy and the batch-preserved journal event (see Batch failure recovery above) — both
are behavioral, not persisted-shape changes, so they carry no schema-version bump of their
own.
Config schema moves to 1.8.0 (#565): a new top-level default_batch_priority key
(integer) — the BatchEntry.priority a batch gets when created with no explicit
batch_priority (see Unit priority below). Absent → DEFAULT_BATCH_PRIORITY (10); an
invalid value degrades the whole config file to built-in defaults, same contract as every
other field.
Two engine-safety policies were explicit product decisions on #464:
- Pid identity is hybrid-verified (decision 1, option C). Every spawn records the
child's
/proc/<pid>/statstart-time and persists it instate.json(pid_start);kill/isAliverefuse a pid whose current start-time no longer matches — a reused pid is never signalled, across engine restarts too. Platforms without/proc(macOS/Windows) and legacy pids without a recorded start-time stay best-effort. - Unreachable ground truth pauses decisions (decision 2, option A). A FAILED
milestone poll (
undefined) is distinct from a verifiably-empty trail (null): while a poll is unreachable (gh auth expired,ai-dossiermissing from a cron PATH, network down), stall and verify-fail decisions pause for that unit — an outage can never kill a healthy agent or fail a unit as "unverified". An agent that exits during an outage holds inverifyinguntil truth returns. Each pause is journaled asground-truth-unreachable. - A completion milestone is fenced to its own dispatch (#575).
isVerifiedComplete(issues) andisMemberComplete(batch members) both accept the current dispatch'sSlotEntry.spawned_atand reject areport done/review done mode=slotmilestone that predates it (±60s clock-skew tolerance) — a re-enqueued issue or a member re-added to a fresh batch run must not read as instantly complete against a PREVIOUS run's milestone. The rejection is journaled asstale-milestone-ignored;spawned_at=null(a legacy slot) degrades to the old, unfenced check.
This applies to issue:<n> unit dispatch (dispatchAssignments). batch:<id> units run
through a separate pass with its own claim/reconcile logic — see
Batch dispatch (#523) below.
Zombie-run fencing (#504)
The ladder redispatches the SAME run, so a takeover inherits the run id and its milestone
trail. In the #472 race that turned out to be a hole: enterRecovery kills the pid it
knows about, but an agent it cannot see or signal — throttled, cwd outside the worktree —
survives, and nothing on the trail tells that agent it was replaced. Both runs implemented
the same issue, and both kept posting milestones on one trail. The doctrine, one step past
"an agent exiting is not proof of merge": no visible process is not proof of death.
A generation now fences the trail:
- Before the takeover is spawned, the engine calls
ai-dossier runstate fence, which posts astatus=supersededmilestone carryinggen=<n>andtakeover=<label>. Written first, on purpose, so it survives the takeover dying too. - The takeover is told its generation in its prompt and passes
--gen <n>to everyrunstate post. The CLI refuses any post below the trail's fenced generation, so the superseded agent cannot extend the trail even though it never checks — and an agent running an older dossier implicitly sits at generation 0, fenced out the moment generation 1 exists. ai-dossier runstate check --issue <n> --run <id> --gen <g>exits3when the caller has been superseded: the checkpoint a workflow runs before implement, review, and ship.- A takeover that posts NOTHING is watched on the shorter of
fence_takeover_timeout_ms(default 15 min) and the phase's own stall allowance — the fence window can only ever bring recovery forward, never delay it — so a takeover that dies at birth re-enters the ladder in minutes and the next fence supersedes it in turn. The first progress signal disarms the short window.ESCALATION_CAPstill bounds the whole ladder. - Report agents ride the same rail: a fenced report slot is told its generation too, or
its
report donemilestone would be refused and it would recover to the cap on a PR that already merged. - The read side is hardened, because a milestone is an issue comment: only comments from
an account with write access count as a fence, a forged
takeover=label is dropped rather than echoed into an agent's prompt, and the engine refuses to fence a run id that does not belong to the issue it is working on (that would journal success while the real zombie stayed free to write).
Fencing is defense-in-depth, not a precondition: if the fence cannot be written (no run id
on the trail yet, gh unreachable, no fencer configured) the redispatch proceeds unfenced
and journals fence-failed. Stranding a stalled unit forever would be the worse failure —
but the unprotected redispatch is never silent.
Batch failure recovery (#472)
What happens when a batch's aggregate suite goes red, or its PR will not merge (RFC-0001 §F.2/F.8/F.9).
Wired into sched start since #523 — the validating → attributing → fixing/evicting
rail below is called directly from batch-dispatch.ts's runValidate/evictOffender
(a red AGGREGATE suite, after every member individually went green). A member that never
went green in the first place (its own gate failed) evicts through a separate, simpler
rail that never touches this module — see
Batch dispatch (#523). These modules remain independently tested
against real scratch repos.
validating → attributing → fixing (ONE bounded attempt) → validating
→ evicting (revert the member's commits) → validating
evictions > max(ceil(N × fraction), min_evictions_before_dissolve), or a
revert conflict → dissolving → members requeued (`dissolve_policy`, #563)
same threshold crossed, but the survivors' re-run suite came back green
→ reviewing (batch preserved; only the evicted
members requeue — strategy=partial, #563)
suite report unreadable, after the fallback retry when one applied
→ blocked → validating (nothing requeued/reverted; #562)
awaiting-merge (CONFLICTING | auto-merge-blocked)
→ rebasing → re-validating → shipping
→ (2nd occurrence) dissolving into two half-batches- Attribution (AC1) —
attributeByOverlapmaps each failing test to a member by focused-test match, then by changed-path overlap. Exactly one candidate attributes; more than one is AMBIGUOUS and none is UNATTRIBUTED — neither is ever guessed. When the caller supplies aBisectSpec, both go torunAttributionBisect: a realgit bisect runover the branch'sgood..badrange executing ONLY the failing tests, whose first-bad commit is mapped to a member through the(#N)subject trailer on the branch's issue-boundary commits (every unresolved test is then attributed to that member). A first-bad commit with no trailer, or an abbreviated sha matching two commits, reportsunattributablerather than blaming a neighbour. The bisect refuses to run at all unless the test command actually discriminates — it must fail atbadAND pass atgood, so a missing runner cannot silently convict the earliest member — and it always resets the checkout to where it found it. Without aBisectSpec, overlap is the whole verdict and unresolved tests stay unattributed. - One bounded fix attempt (AC2) —
beginFixAttemptreturns the mid-tier command and prompt for the CALLER to spawn (sched never invokes an LLM) and records the attempt. A second call for the same member returnsnull: the next step is eviction, so a batch cannot burn its budget on one broken member. - Eviction (AC2) —
evictMembersreverts the member's commits newest-first across members (an eviction group reverts together), requeues it as full-cycle withfailure_evidenceattached (batch, reason, failing tests, attribution method, reverted commits), re-runs the suite and checks the dissolve trigger. A conflicting revert is aborted so the worktree is clean and the batch dissolves — the reverts that already landed ride along on the abandoned branch, which is why it is abandoned rather than reused. An eviction group that reaches an already-shipped member dissolves instead of reverting merged work. Crossing the dissolve trigger no longer always dissolves (#563): if the re-run suite came back green for the survivors — and every evicted member's commits were actually found and reverted — the batch is PRESERVED instead: trimmed to its survivors and carried straight toreviewing, only the evicted members requeue. A red or unreadable re-run, or an evicted member whose commits were never found on the branch, still dissolves in full. - Dissolve (AC3) —
dissolveBatchmarks the batchdissolvedand requeues every UNSHIPPED member:full(each as its own full-cycle run),halved(one or two freshforminghalf-batches — a single remaining member yields one — entries retagged, eviction groups inherited where they survive the split), orpartial(#563 — never marks the batchdissolvedat all: drops the evicted members frombatch.membersand itseviction_groups/ranges, transitionsvalidating → reviewing, and requeues nothing itself, since the caller's eviction loop already did; falls through tofullif that would leave zero survivors, so an empty batch never ships). Shipped and terminal members keep their outcome; nothing green is discarded, and no git runs — the batch branch is simply left behind unmerged, since sched deletes nothing. Every dissolve decision — all three strategies — journals its policy inputs (N=,evictions=,threshold=), so it is explainable without re-deriving the formula. - PR conflict (AC4) —
handlePrConflictrebases the batch branch, re-runs the suite and re-ships ONCE. A second occurrence, a conflicting rebase, a failed fetch, an unusablebase_branch, a checkout that is not on the batch branch, or a red suite after a clean rebase dissolves into two half-batches. - Milestones (AC5) — every eviction and dissolve posts a
batch-validate/batch-shipmilestone to the batch ANCHOR issue viaai-dossier runstate post, with the reason, the evicted/requeued/preserved members and the attribution method (a successful re-ship postsbatch-ship awaiting-merge); each per-member outcome is journaled and kept in the batch'sevictions(the classifier feedback signal). A batch with noanchoror norun_idcannot post — the CLI requires both — so the milestone it could not post is journaled in full instead of vanishing.
Twelve journal events carry the detail: suite-failed, attributed, fix-dispatched,
fix-resolved, member-evicted, revert-conflict, batch-rebased, batch-dissolved,
batch-preserved (#563 — the dissolve threshold was crossed but the survivors' re-run
suite came back green, so the batch ships them instead of dissolving), batch-blocked
(#562 — the suite report was unreadable), batch-split and milestone-post-failed, plus
git-failed for any git command that returned non-zero (the injected ExecFn collapses
every git failure into null, so the command that produced one is always recorded).
Schema 1.3.0 carries the new state: BatchEntry gains anchor, branch, run_id,
eviction_groups, evictions, fix_attempts and rebase_attempts; QueueEntry gains
failure_evidence. 1.2.0 states migrate on load.
Schema 1.4.0: SlotEntry gains role ('cycle' | 'report') — set when the slot is
assigned and never resynced from polled milestones the way phase is, so a report
agent's completion-suppression signal survives phase drifting back to the issue's
pre-report milestone mid-run (#500). 1.3.0 states migrate on load: role is inferred
from the unit's queue entry (shipped + pr + cleanup — the same guard that assigns
a report slot in the first place) with the persisted phase as a fallback when no
matching entry exists. A backfilled role is a best-effort inference, not a guarantee —
see validateState in state.ts for the exact rule.
Schema 1.6.0: SlotEntry gains gen (number — the runstate generation the slot's agent
owns, 0 for a first dispatch) and fenced_at (ISO string or null — set when a takeover is
fenced in, cleared by its first progress signal; #504 above). 1.5.0 states migrate on
load: nothing was fenced before fencing existed, so 0/null is the exact backfill, not
a guess. Both reset with the slot on release (CLEARED_SLOT_FIELDS).
Schema 1.5.0: SchedState gains consecutive_suspect_dispatches (number) and
last_suspect_dispatch_unit (string or null) — the dispatch-health pause's cross-unit
suspect-dispatch streak (#505 above). The two fields are a single fact and must agree
(0 ⇔ null); validateState rejects a state where they disagree. 1.4.0 states migrate
on load: no suspect dispatches were ever tracked under them, so 0/null is the exact
backfill, not a guess.
Schema 1.8.0: SlotEntry gains spawned_at (ISO string or null — when the CURRENTLY
held unit was (re)spawned, distinct from last_progress_at, which later progress
signals overwrite) and log_offset_at_spawn (number or null — the dispatch log's byte
size at that same instant). Both feed runs.jsonl per-dispatch telemetry (#524): the
log is per-UNIT and opened in append mode, so a redispatch's output lands after the
prior dispatch's in the same file — log_offset_at_spawn is what lets the engine read
only the current dispatch's own slice rather than concatenating (claude) or
double-counting (opencode) a prior one. 1.6.0 states migrate on load, backfilling both
to null — an in-flight dispatch's start time and log position are unknown, not zero —
as do 1.7.0 states (#523 took 1.7.0 for the batch fields; these two landed after it).
Both reset with the slot on release (CLEARED_SLOT_FIELDS).
Schema 1.9.0: SchedState gains last_label_poll_at (ISO string or null — when the
engine last re-read hard-block labels, #544 below). 1.8.0 and earlier states migrate on
load, backfilling null: no label re-check ever ran under them, so the first tick after
the upgrade polls immediately rather than waiting out a throttle window it has no
evidence for — the exact backfill, not a guess.
Schema 1.10.0 (#565): QueueEntry gains priority (integer, default 0) and BatchEntry
gains priority (integer, default DEFAULT_BATCH_PRIORITY = 10) — see Unit priority
below. 1.9.0 and earlier states migrate on load, backfilling absent OR explicit null to
those same defaults: nothing before this field existed was ever weighted differently, so
the backfill is exact, not a guess.
Unit priority (#565)
readiness.ts's runnableUnits ranks EVERY candidate — issues and batches together — by
priority desc, then readiness age (updated_at) asc, then a numeric tiebreak (an
issue's own number, or a batch's anchor issue) asc, via the exported compareByPriority
comparator and its entryRank/batchRank helpers. A batch's default priority
(DEFAULT_BATCH_PRIORITY, 10, or the configured default_batch_priority) outranks a
full-cycle entry's default (0) so a ready batch is offered a free slot before a
same-readiness issue competing for it — closing the gap
docs/reports/batch-pilot-2-execution.md §13.4 found (an operator manually deferring
full-cycle entries by hand so a batch could claim its slot).
Applying that ordering to who actually DISPATCHES took more than sorting: engine.ts's
dispatchAssignments (the issue-only dispatch pass) now runs computeAssignments as a
READ-ONLY dry run over both kinds each tick, discards the returned state, and applies
only the issue winners — reserving a free slot for a higher-priority ready batch instead
of handing it to a same-tick issue. batch-dispatch.ts's own ready-batch claim loop
(which never goes through computeAssignments/runnableUnits itself) later that same
tick claims the capacity the reservation left free, sorted by the same comparator when
more than one batch is ready. The reservation is gated on the batch pass actually being
configured (batchExec/runBatchSuite) — without that gate, a ready batch with
nothing able to claim it would withdraw capacity from issues forever instead of one tick.
sched enqueue --priority <n> sets a full-cycle entry's own priority; with --mode slot
it instead sets the BATCH's priority — a batch-level fact like anchor/run_id,
agreement-checked on a later join (an incremental --more-members-expected call, or a
manifest split across --from-manifest calls, must never silently re-point it). sched
reprioritize --issue <n>|--batch <id> --priority <n> adjusts a queued unit's weight in
place — no abandon/re-enqueue round trip, which would also reset every other field
enqueueEntries does not accept as a re-supply — and deliberately does not bump
updated_at (the readiness-age tiebreak), journaling a reprioritized event with the
previous value instead. sched status's Queue and Batches tables both show a priority
column; a slot-mode member's own priority is never read by the scheduler (only the BATCH
row governs assignment), so the Queue table renders - for it rather than a number that
looks load-bearing but is not. A batch dissolve (recovery.ts's dissolveBatch) carries
the parent batch's priority forward onto both split halves.
Batch dispatch (#523)
batch-dispatch.ts's runBatchTick — called from tick() after the issue-level pass,
only when batchExec/runBatchSuite are both configured on EngineDeps — drives every
batch:<id> unit through:
ready → executing(member i/N) ⟲ → validating → reviewing → shipping
→ awaiting-merge → merged → deployed → reported → done
failure rails: executing → dissolving (a member self-reports blocked)
validating → attributing → (fixing | evicting) → validating → dissolving
validating → blocked (suite report unreadable, #562) → validating
executing → blocked (gate-inconclusive:<cap>, #583) → executing
(`sched resume --batch <id>` re-runs the gate; nothing requeued/reverted)- One shared worktree/branch per batch, claimed once by a deterministic (no LLM)
batch-setupstep, namedbatch/<id>-<date>, plus a freshai-dossier runstate mintagainst the anchor issue. Tries a pool claim first (npx worktree-pool claim— already warm by construction,BatchEntry.pool_claimed); on the coldgit branch/push/worktree addpath, batch-setup warms the worktree itself before returning (#561) —cap run worktree.preparewhen the repo's manifest declares it, else package-manager-detected install/build (@ai-dossier/worktree-pool's command resolution — respects the repo's.worktree-pool.jsonproject_subdir/warm_commandseven in repos that never use the pool for anything else). - Members run serially, one fresh
slot-cycleagent at a time, in the shared worktree. A member's completion signal isphase=review status=done mode=sloton its OWN issue (slot-cycleposts no phase of its own pastreview— ship is batch-owned); its commit range on the batch branch is recomputed (git log) after every member and kept onBatchEntry.rangesfor eviction. An incremental gate (ai-dossier cap run typecheck.run/test.focused, when the repo has a manifest) runs after each member before advancing — a second, independent check that the member's self-reported "done" is real. Three-way outcome policy (#583):task-failedevicts the member (same rail as a self-reported block);automation-broken/capability-unavailable— the gate itself couldn't reach a verdict — block the batch instead of silently proceeding (gate-inconclusive:<cap>,member_gates/blocked_reasononBatchEntry, surfaced insched status);sched resume --batch <id>re-runs the gate later to resolve the block once the capability is fixed. - The batch's single slot is claimed FRESH for each live step (a member, the tail agent, the report agent, a bounded fix agent) — never held across a wait. The aggregate suite itself runs with NO slot claimed at all (deterministic engine work, not an LLM step).
- Two failure rails. A member that never went green evicts directly (nothing to attribute — see the #472 section above for what "directly" skips). A red AGGREGATE suite (every member individually green, but integration-level conflict) routes through the #472 attribution/fix/evict library.
- The tail, after the last member: the aggregate suite runs deterministically; green
spawns ONE bounded strong-tier agent that runs
review-issueaggregate mode thenship-issuebatch mode (rebase-merge, aCloseslist) and parks the PR exactly like a detached full-cycle run; the engine's own PR watcher (a batch-granularity mirror of the per-issue one) accepts the merge and dispatches a cheap mechanical-tier agent forreport-issue's batch variant. - Scope cuts, recorded rather than discovered later: no
git bisectstage for an ambiguous aggregate failure (an unattributable red suite dissolves instead); no per-phase stall/escalation ladder for batch sub-agents (a dead-without-verification agent is treated as blocked, not redispatched stronger).
Schema 1.7.0: BatchEntry gains worktree (absolute path of the shared batch worktree,
null until batch-setup lands), ranges (MemberRange[] — each member's commit range,
recomputed after every member completes) and pr (the batch PR parked on auto-merge,
persisted so a restart mid-watch still knows what to poll). 1.6.0 states migrate on load:
no batch was ever dispatched under them, so null/[]/null is the exact backfill, not
a guess. BatchEntry also gains pool_claimed (#561) — pre-#561 states carry no such
key at all, and backfill it to false on load (no batch was ever pool-claimed before
batch-setup had pool integration). Config schema moves to 1.3.0: dispatch gains
member_prompt, batch_tail_prompt and batch_report_prompt (the three new agent
prompt templates).
Schema 1.11.0 (#583): BatchEntry gains member_gates (most recent incremental-gate
result per member, keyed by issue number as a string — {capability, outcome,
output_tail, at}) and blocked_reason (why the batch is blocked — persisted so
sched status can show it; previously blockBatch only journaled/posted the reason,
never stored it on the entry, so this also retroactively covers the #562 case). 1.10.0
states migrate on load: no gate has ever produced a non-ok verdict, and no batch has
ever been blocked, under them, so {}/null is the exact backfill, not a guess.
New journal events: batch-setup-done, batch-setup-failed, member-advanced,
batch-warmup-done, batch-warmup-failed (#561 — the cold-path warm step only; a pool
claim emits neither). gate-inconclusive (#583 — the incremental gate came back
automation-broken/capability-unavailable rather than a definite ok/task-failed;
sits alongside batch-blocked as the per-member analogue of the aggregate suite's
"block, don't dissolve" precedent). Member/tail/report/fix-agent spawn, progress,
completion and park events reuse the existing unit-generic names (assigned/spawned/unit-failed/
external-advance/pr-parked/merge-accepted/report-dispatched/teardown-done/
teardown-failed) with unit = batch:<id>.
API surface
import {
SchedStore, // persistence: load/save/withLock per project dir
enqueueEntries, // validated queue appends (cycles, dupes, mode/batch rules)
parseManifest, // batch-prep JSON → EnqueueInput[]
computeAssignments, // pure: fill idle slots with runnable units, bounded by max_slots
runnableUnits, // pure: which units may run right now (dep-gated), in assignment
// order — priority desc → readiness age → issue/anchor (#565)
compareByPriority, // the priority/age/tiebreak comparator runnableUnits sorts with —
// also used directly by batch-dispatch.ts's ready-batch claim loop
entryRank, batchRank, // PriorityRank of a QueueEntry / BatchEntry (#565)
reprioritizeIssue, // sched reprioritize --issue: adjust priority in place, no
reprioritizeBatch, // abandon/re-enqueue round trip; refuses a terminal unit
tick, // one engine cycle: reconcile + verify + refill + spawn,
// and since #468: park-watch, teardown, report dispatch
runLoop, // the sched start loop (tick, sleep, repeat)
type TickResult, // what one tick did (spawned/parked/merge-accepted/stale-reconciled/
// dependents-unblocked/report-dispatched/teardown/completed/
// redispatched/failed/blocked, and since #544
// label-cleared/label-blocked/label-check-failed)
// — since #523 also carries
// `batch:<id>` unit ids (issue numbers for `blocked`)
type EngineDeps, // inject everything the engine touches (store/journal/spawn/ground
// truth/clock/repoDir/teardownExec/fencer/batchExec/runBatchSuite/
// runBatchCapability — #523)
createSpawnDeps, // real detached-spawn process I/O
createExecGroundTruth, // runstate/gh/git ground truth via subprocesses (injectable exec);
// since #468 also gh pr view PR state + setup info from comments
resolveDispatch, // config → resolved command/prompt/report-prompt/tier-models/timers/
// per-tier spawn specs (tiers — #527)
buildTierCommand, // resolved dispatch + tier + issue → argv, using that tier's OWN
// command/model (#527) — what the mixed-CLI ladder spawns with
resolveTierSpawn, // resolved dispatch + tier + issue → { cmd, model } together (#527) —
// the single call every spawn site uses so a journal entry can
// never disagree with what was actually spawned
journalCmdModelFields, // { cmd, model } → spawned/redispatched/fix-dispatched journal fields
stallTimeoutForPhase, // the stall allowance for the phase now in flight (#495 per-phase
// map → global, hardened against a prototype-name phase)
stallTimeoutForSlot, // #504: that allowance, shortened to fenceTakeoverTimeoutMs while
// a takeover has posted nothing (Math.min — never longer)
takeoverInstruction, // the TAKEOVER prompt suffix appended for gen > 0
SUPERSESSION_CHECKPOINT_INSTRUCTION, // the check-before-implement/review/ship clause
// every dispatch prompt carries
createExecRunFencer, // default fencer: shells `ai-dossier runstate fence --json`
parseFenceGeneration, // fence stdout → the installed generation (null = unfenced)
type RunFencer, // inject the takeover-record writer: (issue, run, phase, takeover)
type FenceOutcome, // {ok, gen} | {ok: false, reason} — a failure carries its cause
FENCE_TIMEOUT_MS, // fence subprocess timeout (60 s — two gh round trips)
DEFAULT_PHASE_STALL_TIMEOUT_MS, // built-in per-phase stall allowances (implement: 90 min)
buildReportPrompt, // report-agent prompt ({issue}/{pr}/{cleanup}/{gen} substituted)
reportTierFor, // report (re)dispatch tier after N escalations
isParkedMilestone, // ship-phase awaiting-merge + pr= → the park signal
prOfMilestone, // a milestone's pr= key as a positive integer
parsePrViewJson, // gh pr view --json → PR truth (mergedAt/mergeable/blocked label)
parseSetupInfo, // gh issue view --json comments → teardown inputs
runTeardown, // #468 script teardown for a merged unit (pool return / worktree remove)
isSafeWorktree, // worktree-path containment check (CWE-22)
TEARDOWN_TIMEOUT_MS, // teardown subprocess timeout (120 s)
attributeByOverlap, // #472 pure stage-1 attribution: failing tests → members
parseVitestJson, // vitest --reporter=json → failing tests
isReadableVitestReport, // #562: a parseable { testResults: [...] } document exists —
// distinct from "zero failures"
parseBoundaryCommits, // git log → issue-boundary commits via the (#N) trailer
memberRanges, // boundary commits → each member's commit list
runAttributionBisect, // stage-2: real git bisect over the failing tests only
beginAttribution, // validating → attributing (overlap, then bisect if needed)
beginFixAttempt, // the ONE bounded mid-tier fix dispatch instruction
resolveFixAttempt, // record its outcome, back to validating
evictMembers, // revert + requeue with evidence + suite re-run + dissolve check
checkDissolveTrigger, // pure: evicted > max(ceil(N × fraction), min floor) — dissolve_policy, #563
dissolveBatch, // full | halved | partial (#563); preserves everything green
blockBatch, // #562: unreadable suite report → blocked; no requeue, no revert
type BlockOptions, // { reason, milestonePhase? } for blockBatch
handlePrConflict, // rebase + re-ship once, then dissolve into halves
createExecMilestonePoster, // batch milestones via `ai-dossier runstate post`
expandEvictionGroups, // members that must revert together (§E.4 eviction groups)
requeueMember, // the one requeue path abandon/evict/dissolve all take
isPreservedMember, // the single definition of "already green"
createBatch, // the single BatchEntry constructor
type RecoveryDeps, // inject exec/repoDir/journal/milestone-poster/suite-runner/clock
type SuiteRunner, // re-runs the aggregate suite after a revert or rebase
type BatchMilestonePoster, // batch-milestone sink (createExecMilestonePoster is default)
Journal, // append-only events.jsonl
appendJsonl, // the shared mkdir+append+swallow JSONL write
transitionIssue, transitionBatch, transitionSlot, // typed §D transitions
TRANSITIONS, // the transition tables themselves (for previews)
buildStatusReport, // machine-readable status incl. blocked/failed sets
validateState, // strict persisted-state validation (1.0.0-1.9.0 files migrate)
DEFAULT_ISSUE_PRIORITY, DEFAULT_BATCH_PRIORITY, // priority defaults (0 / 10, #565)
IllegalTransitionError, EnqueueError, CorruptStateError, LockTimeoutError,
SchedNotFoundError,
EngineTooOldError, // state schema newer than installed engine — not corruption (#537)
// #524: per-dispatch runs.jsonl telemetry (see "runs.jsonl telemetry" below)
buildSchedRunLogEntry, // AgentRunUsage-sourced RunLogEntry for one completed dispatch
appendSchedRunLog, // JSONL append to ~/.dossier/runs.jsonl, gated by schedTelemetry (not cli's auditLog)
readDispatchLog, // read a unit's dispatch log, optionally from a byte offset
schedRunsLogPath, // ~/.dossier/runs.jsonl (re-export of @ai-dossier/core's runsLogPath)
schedTelemetryEnabled, // false when the operator set schedTelemetry:false in ~/.dossier/config.json
usageParserFor, // claude/opencode usage-parser selection by spawned binary
type SchedRunLogInput, // buildSchedRunLogEntry's input shape
dispatchLogPath, // <runsDir>/<unit>.log — shared by spawn (offset) and record (read)
fileSizeOrZero, // byte size of the dispatch log at spawn time, or 0
// #564: reconstruct a batch's dispatch costs from raw per-unit logs on
// disk, for batches with no runs.jsonl coverage (pre-#564, or torn down)
listBatchDispatchLogs, // every raw dispatch log found for a batch id, parsed from its filename
buildBatchRunLogEntries, // ...to RunLogEntry rows, same shape a live dispatch produces
type BatchLogEntry, // one parsed log entry (member/tail/report/fix)
runBatchTick, // #523: one batch reconcile+refill pass; called by tick() after
// the issue pass — loads/saves state itself, holds no lock
// across the call
type BatchDispatchDeps, // inject store/journal/groundTruth/spawnDeps/exec/runSuite/
// runCapability(optional, returns CapabilityGateResult)/fsExists(optional)
type BatchTickResult, // spawned/completed/parked/mergeAccepted/failed (batch:<id> ids)
// + blocked (issue numbers, dissolve-requeued)
type CapOutcome, // ok | task-failed | automation-broken | capability-unavailable
type CapabilityGateResult, // {outcome: CapOutcome, outputTail?, reason?} — runCapability's return shape (#583)
resumeBlockedGate, // #583: sched resume --batch <id> — re-run the gate that blocked a batch
buildMemberPrompt, buildBatchTailPrompt, buildBatchReportPrompt, // #523 prompt builders
DEFAULT_MEMBER_PROMPT_TEMPLATE, DEFAULT_BATCH_TAIL_PROMPT_TEMPLATE,
DEFAULT_BATCH_REPORT_PROMPT_TEMPLATE,
isMemberComplete, isMemberBlocked, // member milestone predicates (mode=slot gated)
isBatchTailParked, // batch-ship awaiting-merge + pr= — the batch park signal
isBatchPhaseDone, // <phase> done on the anchor (batch-review/batch-report)
batchOfUnit, // batch:<id> → <id>; null for issue units or malformed ids
} from '@ai-dossier/sched';All state functions are pure (state in, new state out — the worktree-pool pattern);
SchedStore is the only state-I/O boundary and every mutation runs under its lock. The
engine polls ground truth OUTSIDE the lock and mutates state under it, so a slow gh
call never blocks other sched commands. Almost all process I/O is injectable — the
tests spawn fake agents and stub ground truth; no LLM calls anywhere. The one exception
is runs.jsonl telemetry (#524): appendSchedRunLog/readDispatchLog read/write the
real filesystem directly rather than going through an injected dependency, with only
EngineDeps.homeDir (a path override, test-only) as a seam — see "runs.jsonl
telemetry" below.
State layout
~/.dossier/sched/<project>/
├── state.json # hot operational truth — atomic tmp+fsync+rename writes;
├── config.json # durable intent: max_slots, stall_timeout_ms, reconcile_interval_ms,
│ # pr_poll_interval_ms, dispatch (incl. report_prompt,
│ # phase_stall_timeout_ms, fence_takeover_timeout_ms, tiers — #527,
│ # suite_command — #562, disallowed_tools — #591), auto_upgrade — #537,
│ # dissolve_policy — #563
├── events.jsonl # append-only event journal (the operator's flight recorder)
├── runs/ # per-unit agent output logs (issue-<n>.log)
└── .sched-lock/ # cross-process directory mutex (pid; stolen from dead holders)Sched also writes OUTSIDE this per-project tree: one runs.jsonl entry per completed
dispatch goes to ~/.dossier/runs.jsonl (#524) — the same global, cross-project file
cli's ai-dossier run already appends to, read by ai-dossier sched stats and
ai-dossier history alike. See "runs.jsonl telemetry" below.
runs.jsonl telemetry (#524)
packages/sched/src/run-log.ts closes a gap where scheduler-dispatched agents never
appeared in ~/.dossier/runs.jsonl — per-issue cost could not be baselined. One entry
is appended per completed dispatch (recordDispatchRunLog in engine.ts, called from
every place a dispatch's exit is first detected: the dead-pid rail, the
external-advance rail, a stall-timeout kill, and a dependents-blocked kill), sourced
from the agent's modelUsage map — never blended with the top-level usage block,
the fix for a ~43% fabricated-saving discrepancy the two blocks were found to produce.
recordDispatchRunLog/recordMemberRunLog also return the last tool the dispatch called
(parseLastToolUse, @ai-dossier/core, #591), when the log yielded one — the exit itself
attributes to a concrete cause (e.g. Monitor) without opening the transcript. It rides the
non-terminal verify-incomplete event on every unverified exit and, once the escalation
ladder is exhausted, the terminal unit-failed (agent-exited-unverified /
unverified-exit-at-strongest-tier) as last_tool; a stall-timeout kill carries it too. Only
the dead-pid detection rail and the stall kill record a fresh log slice in the same tick —
a slot already exited/verifying when reconciled again has none to attribute.
The dispatch log (runs/<unit>.log) is per-UNIT and opened in append mode
(createSpawnDeps), so a redispatched unit's second agent writes its output AFTER the
first's, in the SAME file — SlotEntry.log_offset_at_spawn (schema 1.8.0, stamped
right before every spawn) is what lets recordDispatchRunLog read only the current
dispatch's own slice, so a redispatch's entry is never corrupted by concatenation
(claude) or double-counted (opencode) against a prior dispatch's output.
Every dispatch log opens with a {"type":"sched-dispatch","ts":…,"cmd":[…]} preamble
line written at spawn, followed by a {"type":"sched-dispatch","event":"spawned","pid":…}
marker once the child exists — so a log is never 0 bytes for a unit that ran, each
dispatch's slice is self-describing, and a slice can be joined to its events.jsonl
record by pid rather than by timestamp alone. Every @ai-dossier/core usage parser
skips that type.
Why the default dispatch command streams. --output-format json buffers the entire
session and writes ONE object at process exit. The batch pilot's six 0-byte logs are
exactly the six units advanced by ground truth (external-advance) and killed while
still alive — the one-shot write never happened. --output-format stream-json --verbose
fills the log per turn, and parseAgentUsage sums per-turn assistant usage when a
dispatch was killed before its final result event, so an interrupted run still reports
real tokens instead of null.
Opt-out. Writing is gated by schedTelemetry in ~/.dossier/config.json (default
on), read directly here because sched cannot depend on cli. This is deliberately
NOT the CLI's auditLog, which scopes ai-dossier run's own entries: honouring it would
silently leave an opted-out operator with zero scheduler cost visibility, and ignoring it
would just as silently widen a flag whose documented scope is the audit log. A skipped
write is journaled run-log-skipped reason=telemetry-disabled, so a missing entry is
never indistinguishable from a lost one.
Reading the journal when a row is blank. An entry whose token fields are all null is
journaled run-log-no-usage with a reason — log-unreadable, log-empty, or
no-usage-events — so a row of dashes in sched stats can be explained without
re-deriving it. A successful append is journaled run-log-recorded; a failed one,
run-log-failed with the target file. Dispatches ended by sched abandon release the
slot without recording, so they are not costed. (finalizeRunLogEntry in run-log.ts
is the single implementation of this journal-then-append tail, shared by
recordDispatchRunLog here and batch dispatch's recordMemberRunLog below — #564.)
Batch members (#564). batch-dispatch.ts spawns members/tail/report/fix agents
directly (deps.spawnDeps.spawn()), bypassing recordDispatchRunLog above entirely —
runs.jsonl had zero coverage for batches even after #524/#531 shipped the per-issue
capture. recordMemberRunLog (batch-dispatch.ts) closes that gap for MEMBER
dispatches, attributed to the same issue:<n> unit scheme ordinary dispatches use, so a
member's cost shows up in the default sched stats view with no new read-side logic.
Tail/report/fix agents still never write to runs.jsonl (wiring that in needs each of
their spawn functions to stamp SlotEntry.spawned_at first, same as the original
member bug); sched stats --batch <id> (packages/sched/src/batch-stats.ts) instead
recovers their cost — and any historical batch's, predating #564 or already torn down —
by reading the raw dispatch logs on disk directly, the same recovery a human previously
did by hand (docs/reports/batch-pilot-2-execution.md §13). Tokens/cost/model reproduce
exactly; Duration/Tier are always - for a --batch-reconstructed row (a raw log
carries neither the dispatch's spawn time nor its tier) — a structural limit of
after-the-fact recovery, not a missing-data bug.
- Crash safety: a process killed between writes leaves the previous complete state,
never a partial file; restart resumes identically (proved by
restart.test.ts) — running slots with dead pids are re-detected and verified, slots leftassignedby a crash between assign and spawn are spawned, and a dispatched entry no slot holds is requeued. - Corrupt state is loud:
load()throwsCorruptStateErrornaming the file — never a silent queue reset.state.jsonis deletable and rebuildable from GitHub, which remains the system of record. Exception: astate.jsonwritten by a newer schema than the installed engine is not corruption —load()throws the more specificEngineTooOldError(#537), pointing at an engine upgrade rather than at deleting real queue data. - Schema: state/config files from #460 (schema 1.0.0), #464 (1.1.0), #468 (1.2.0),
#472 (1.3.0), #500 (1.4.0), #505 (1.5.0), #504 (1.6.0), #523 (1.7.0) and #524 (1.8.0)
load and migrate to 1.9.0 automatically (slot
branch/last_head/pid_start, slotrole(inferred from the unit's queue entry, with the persistedphaseas a fallback — #500), entrypr/cleanup/failure_evidence, batchanchor/branch/run_id/eviction_groups/evictions/fix_attempts/rebase_attempts, state-levellast_pr_poll_atbackfill to null, state-levelconsecutive_suspect_dispatches/last_suspect_dispatch_unitbackfill to0/null— #505, slotgen/fenced_atbackfill to0/null— #504, and slotspawned_at/log_offset_at_spawnbackfill tonull/null— #524, and state-levellast_label_poll_atbackfill tonull— #544). max_slotsbounds live units (assigned | running | recovering); dependency edges gate readiness — an issue with an unmerged dependency, and a batch behind an unmerged batch, are never runnable.- Pause stops new assignments only (including report-agent dispatch — #505); abandon
routes through the typed failure rails (
evicted → requeued{full}for batch members — nothing green is discarded). A pause can be manual (sched pause) or automatic (dispatch-health, #505 above);sched resumeclears both the flag and the dispatch-health streak.
Hard-block labels are re-read every tick (#544)
#507's enqueue pre-screen resolves an issue's GitHub labels in the CLI and lands the
entry as blocked reason=label:<name>. That screen runs once, at enqueue time — so
before #544 a decision the human resolved never reached the queue: the entry stayed
blocked forever and sched status kept printing a stale reason. The engine now re-reads
the labels itself, on both sides of the same check:
- A
label:<name>-blocked entry whose label is gone returns toqueued(label-cleared), and normal dependency gating takes it from there — a free slot can pick it up in the same tick. - A dispatchable entry that GAINED a hard-block label moves to
blocked(label-blocked) before the dispatch pass, so a fresh human hand-off is never dispatched over. An already-dispatchedunit is left alone: a late label must not abandon a live agent's work. - A blocked entry whose label CHANGED gets its reason refreshed in place.
- An unreachable read (
ghdown, auth expired) journalslabel-check-failedand decides NOTHING.issueLabelsreturnsundefinedfor a failed read and[]for a verifiably unlabelled issue — flattening the two would dispatch over a live hand-off whenever GitHub is flaky.
The watch set is every label-blocked entry plus the runnable issue units a dispatch
could actually place this tick — the latter capped at max_slots, so the per-tick gh
cost tracks the SLOT count rather than the backlog, and skipped entirely while the
scheduler is paused. A tick with work re-reads every tick; a tick with nothing else to
do (no live slot, nothing runnable) re-reads at most every label_poll_interval_ms
(default 10 min), from the persisted last_label_poll_at — so an idle fleet parked on
human decisions stays cheap. The timestamp advances only when a read actually returned
something, so sched status never claims a check that a gh outage prevented.
HARD_BLOCK_LABELS lives in labels.ts and is re-exported by
cli/src/hard-block-labels.ts, so the enqueue screen (#507), the classify screen (#538)
and this one cannot drift apart.
The max_slots cap is exact while nothing is blocked (freeCapacity <= max_slots, and
blocking nothing preserves candidate order, so every unit dispatched below was read). On
a tick that DID block something, units outside the read window are deferred for one tick
rather than dispatched on information nobody gathered — which costs nothing in the common
case, so #525 AC5's same-tick refill is untouched whenever no label moved.
Scope note. Per-issue dispatch only. Batch members (mode: 'slot') and batch anchors
are not re-screened — runnableUnits filters to mode: 'full', and runBatchTick has
its own claim path — so a hard-block label landing on a batch member mid-wave is not
caught here; enqueue still refuses to enqueue an already-labelled issue as a batch
member. Nor does the screen cover a unit that becomes dispatchable INSIDE the tick's lock
(its dependency shipped this very tick, or requeueOrphanedDispatches returned it to the
queue) and is dispatched before the next snapshot reads it: closing that would mean
deferring every in-lock arrival by a tick, which is exactly the guarantee #525 exists to
provide. Both are narrower than the gap this section closes — before #544 nothing was
re-screened at all — but neither is closed by it.
The PR watcher + tail work (#468)
Dispatched runs park their PR on auto-merge (detached ship mode — the default
prompt instructs it) and exit. The engine owns everything after the park:
- Park detection (AC1) — an agent exit whose latest milestone is the ship
phase's
awaiting-merge(withpr=) is a VERIFIED park, not an unverified exit: the entry moves toparked, the slot is released (a waiting unit consumes zero slots), and the watcher takes over. - PR watching (AC1) — parked PRs are polled every
pr_poll_interval_ms(default 150 s — "every 2–3 min", persistedlast_pr_poll_atso a restart honors the cadence; checked on each reconcile tick when due, so areconcile_interval_mslonger than the interval slows the effective cadence) viagh pr view --json state,mergedAt,mergeable,labels. A merge is accepted only when state is MERGED andmergedAtis non-null and the issue is closed — never inferred from an agent exit. An unreachable poll pauses the watcher (decision 2, option A). - Failure states (AC3) —
CONFLICTING, closed-unmerged, or theauto-merge-blockedlabel fail the unit with the reason and block its TRANSITIVE dependents. The engine never merges anything itself. - Gating on MERGE, not park (AC4) —
parkedis not a satisfied status: dependents stay blocked until the merge lands (parked → shipped). - Teardown as a script (AC2) — on merge, the run's setup milestone
(recovered once from the issue's comments — collaborator-authored only, and
the worktree path must pass a containment check before any destructive
subprocess) chooses the script: pool-claimed worktrees run
worktree-pool return --path <wt> --json(the pool's own self-check is the verification); cold worktrees rungit worktree remove --force <wt>with a path-gone check. Both are verify-first idempotent; a failed step recordscleanup=failed-<step>on the entry and in the journal — degradation, never unit failure. - Report dispatch (AC2) — once teardown is recorded (when a slot is
free — a waiting report consumes zero slots), a mechanical-tier report
agent is spawned with the report-phase prompt (
dispatch.report_prompt;{issue}/{pr}/{cleanup}substituted — the cleanup status rides into the report). The slot recordsrole: 'report'at assignment, fixed for the assignment and never resynced byphase-updated(#500) — the issue is already closed at merge, so for a report-role slot the closed signal is suppressed and only areport donemilestone completes the unit. A report that stalls climbs the same ladder (mechanical → mid → strong, cap 2), and at the cap the unit completes (done, reasonreport-escalation-cap) with areport-failedjournal event — the work is merged, so dependents are never re-blocked. The full-cycle tail-run pattern (re-dispatching a whole run for teardown+report) is retired. - Stale-failure reconcile (#501) — a
failed reason=auto-merge-blockedentry stops being watched the instant it leavesparked, but an operator can manually clear the watcher's block and re-queue the same PR outside the engine entirely.pollParkedPrsalso polls these stale-failed entries on the same cadence — no second poll pass, though each watched entry still costs its owngh pr view/gh issue view— for up to 7 days after the failure (STALE_RECONCILE_WINDOW_MS); past that an abandoned failure is left alone rather than polled forever. Once the PR showsMERGEDwithmergedAtset and the issue closed (the same three-part gate as AC1 above), the entry flipsfailed → shipped, re-enters the normal teardown → report-dispatch path (items 5–6 above), and unblocks whatever
