ctxbench

A controlled experiment that measures whether a context layer changes what an AI coding agent achieves — run against the author's own product, and reported where it loses.

Read the chart with the intervals in mind — it does not draw them. Treatment sits at or above control in every bar, and none of those gaps is statistically distinguishable from zero at this N: the aggregate 95% CI is [−13.6, +13.6] percentage points, wider than any bar-to-bar difference shown. Single run, one repeat, indicative not conclusive. The bars are the data; the results table is the finding.

git clone https://github.com/br9704/ctxbench && cd ctxbench && npm i
npm run report          # regenerates every number below from committed raw artifacts

The headline: no detectable effect on completion rate, in any category. Aggregate paired delta +4.5pp, 95% CI [−13.6, +13.6], McNemar p = 1.000 — 3 of 22 tasks discriminated at all, against a pre-registered floor of 6. Then the control arm was re-run against itself with an identical flag set and disagreed with itself on 3 of 24 tasks — as large as or larger than the 1 task separating the arms. The noise floor is at least as large as the signal.

What it does

Every vendor selling a context layer for AI coding agents — a memory MCP, a shared-context service, a CLAUDE.md convention — claims it makes the agent better. Almost none of them have measured it. ctxbench is the measurement: two arms, everything else held constant. The agent alone, and the same agent plus one context provider, on the same tasks, at the same pinned model, graded only by whether the repository's own test suite goes green.

The property the whole design turns on is that the only difference between the two arms is one file. Both arms invoke the same binary with the same flags; only the --mcp-config path differs. There is a unit test asserting the two argv arrays differ in exactly one element, and a runtime assert on every single rollout checking the agent's own system/init event reports exactly the expected MCP servers (all connected), zero leaked skills, and the pinned model. A rollout that fails the assert is discarded and logged — never silently retried into the data.

It is a harness, not a study. Provider is a small interface — five required methods and one optional — and the control arm is a provider too (none), so the control path runs the same code as the treatment path and a harness asymmetry cannot masquerade as an effect. Three reference implementations ship: none, aethereum (hosted MCP), and file (a static CLAUDE.md). All three pass the same parameterised conformance suite, which is the artifact you run against your own context layer. The core loads providers by dynamic import() only, with a test asserting on the source that no static provider import exists — so the "runs with zero Aethereum code installed" promise is machine-enforced rather than intended.

It was built to evaluate Aethereum, which its author also builds. That is a conflict of interest, so the design is arranged to make a favourable result hard to manufacture. The analysis was pre-registered in METHODOLOGY.md before the harness existed — arms, metrics, statistical method, the minimum detectable effect, and the exact interpretation wording, all fixed before any result was seen. Every shared-context fixture is mechanically diffed against its own defect patch for symbol leakage. The README's results table and its "where it does not help" section are generated by npm run report, not typed, so the honest section cannot drift into optimism by being edited.

Tasks are 24 hand-authored defects injected into pinned commits of two real TypeScript repositories. Not SWE-bench: memorisation is impossible when the defect never existed in any training corpus, and SWE-bench Verified is 92% single-file anyway, which starves the categories this experiment cares about. Each task must prove itself against an oracle gate — broken in exactly the declared places, fully fixed by reversing its defect patch — or it is deleted rather than patched around.

Architecture

flowchart TD
    CLI["cli.ts"] --> BATCH["run.ts — batch<br/>randomised interleave, resumable, cost guard"]
    BATCH --> RUNNER["runner.ts — one rollout"]

    RUNNER -->|"prepare(task)"| SANDBOX["Sandbox seam<br/>LocalSandbox · DockerSandbox"]
    RUNNER -->|"configureTask(task)"| PROVIDER["Provider seam<br/>none · aethereum · file"]

    SANDBOX -->|"pristine workspace at the<br/>pinned commit, defect folded in"| AGENT
    PROVIDER -->|"AgentConfig — the ONLY<br/>difference between arms"| AGENT["agent.ts<br/>claude -p --output-format stream-json"]

    AGENT -->|"system/init"| PURITY{"purity assert<br/>MCP servers · skills · pinned model"}
    PURITY -->|"mismatch"| DISCARD["invalid_environment<br/>excluded from BOTH arms, logged"]
    PURITY -->|"ok"| GRADE["grade.ts<br/>fail_to_pass AND pass_to_pass"]

    GRADE --> ART["results/RUN/TASK/ARM/REPEAT/<br/>trajectory.jsonl · run.json · report.json"]
    ART --> STATS["stats.ts<br/>paired deltas · McNemar · BCa bootstrap"]
    STATS --> REPORT["report.ts --inject README.md"]
    REPORT --> ART

Two design decisions carry most of the weight. A provider may only influence the agent through AgentConfig, plus the context files it declares — never the prompt, the sandbox or the grading. Declared files are diffed against git status and the rollout aborts if any other tracked file changed, so a provider owns its context and cannot touch the code under test. That restriction is what keeps the single-difference property true no matter who writes the provider. And Sandbox is a seam rather than a hard Docker dependency: Docker was absent on the build machine, so LocalSandbox (clone at pinned commit → npm ci from the committed lockfile → record the exact toolchain into every artifact) is the v1 default, and DockerSandbox ships implemented but auto-selects only when a daemon answers. That trade is disclosed, not hidden: the host toolchain becomes a variable, mitigated by pinning and by both arms of a task always running on the same machine in the same session — a shared confound rather than a differential one.

The full decision log, with the evidence behind each one, is in masterplan.md (amendments A1–A4).

How it was built

The plan was wrong about its own founding assumption, and probing beat reading. Before any code, twelve findings were measured first-hand rather than taken from docs. Three of them killed planned approaches outright: --bare — the flag the whole purity design assumed — pins the agent to exactly [Bash, Edit, Read] and cannot be widened, which would have floored completion in both arms; SWE-smith is Ubuntu-and-Python only; and PR-mining was impossible because the candidate repos have zero merge commits between them. The biggest one came later: the project was planned around an MCP schema tax of roughly 9,800 tokens (the planning assumption; a byte-count estimate over the server's own tools/list independently predicts ≈9,400, METHODOLOGY §3.2), and the measured figure — identical no-op prompt, three replicates identical to the token — is 945 tokens (+3.1%). Claude Code 2.1.232 registers all 30 MCP tools but does not put their schemas in context. That overturned the premise the experiment was designed to control for, and it is stated as an overturned premise rather than quietly corrected.

Three bugs would each have published a false result. A grading bug on macOS (/var versus /private/var) meant no test file ever matched its entry, so every rollout in both arms graded unresolved — a perfect, plausible, entirely false null. It was caught only because a trajectory was read by hand and the agent had visibly fixed the bug and said so. A design flaw left each defect as an uncommitted change, so the agent's first orienting command — git diff — printed the exact inverse of the fix; ten of twelve pilot rollouts solved, four of them in under 30 seconds. And a reporting bug selected run directories lexicographically, printing superseded numbers under a correct-looking heading. The oracle gate found four more before any run: a snapshot test embedding a relative timestamp that would have manufactured a difference out of wall-clock drift, a stale pin, a red baseline, and real fixture leakage — a contract line that spelled out, verbatim, the fix for one of the tasks. The clean-clone gate then found live room bearer tokens committed under a dot-directory that the analysis code skipped; they were deleted and purged from git history.

The most valuable result was an accident. The third arm's batch re-ran the control arm with an identical flag set on the identical task set — an unplanned partial replicate, and exactly what a single-repeat design is supposed to be unable to see. It disagreed with itself on 3 of 24 tasks (2 discounting one harness timeout), against a between-arm difference of 1 task net. It also dismantled the one efficiency story that looked like a finding: three tasks that took 1128s, 551s and 1354s in the first control run took 82s, 97s and 68s in the second. Publishing "the context layer makes the agent 68% faster" would have been arithmetically true and substantively false. analyseContinuous now reports medians beside means and labels a tail-driven effect as such, with three unit tests pinned against the real shapes.

The receipts — every sprint's as-shipped delta, every deferral with its reason, every decision with its why — are in masterplan.md. It is a build log, not a feature list.

Results

Results — full-001

| arm | n | solved | pass@1 | mean tokens | mean output | median wall | p95 wall | mean turns | |---|---|---|---|---|---|---|---|---| | control | 24 | 21 | 87.5% | 382,546 | 3,208 | 39.2s | 950.0s | 10.6 | | treatment | 22 | 21 | 95.5% | 369,365 | 2,245 | 47.0s | 87.5s | 10.1 |

Paired analysis, by category

| category | pairs | control | treatment | delta | 95% CI (BCa) | McNemar p | verdict | |---|---|---|---|---|---|---|---| | cross-module | 6 | 5/6 | 5/6 | +0.0pp | [-66.7pp, +33.3pp] | 1.000 (b=1, c=1) | no detectable effect at this N | | multi-file | 7 | 6/7 | 7/7 | +14.3pp | [+0.0pp, +42.9pp] | 1.000 (b=1, c=0) | no detectable effect at this N | | multi-turn | 4 | 4/4 | 4/4 | +0.0pp | [+0.0pp, +0.0pp] | 1.000 (b=0, c=0) | no detectable effect at this N | | single-file | 5 | 5/5 | 5/5 | +0.0pp | [+0.0pp, +0.0pp] | 1.000 (b=0, c=0) | no detectable effect at this N | | ALL | 22 | 20/22 | 21/22 | +4.5pp | [-13.6pp, +13.6pp] | 1.000 (b=2, c=1) | no detectable effect at this N |

Efficiency, paired per task

| metric | pairs | control mean | treatment mean | mean delta | 95% CI (BCa) | control median | treatment median | median delta | verdict | |---|---|---|---|---|---|---|---|---|---| | total tokens | 22 | 385,834 | 369,365 | -16,469 (-4.3%) | [-166,908, 46,077] | 310,070 | 338,844 | +37,695 | no detectable difference at this N | | output tokens | 22 | 3,279 | 2,245 | -1,034 (-31.5%) | [-2,745, -188.0] | 2,168 | 2,169 | -7.0 | tail effect only — typical task unchanged | | turns | 22 | 10.5 | 10.1 | -0.3 (-3.0%) | [-3.6, 1.1] | 9.0 | 10.0 | +1.0 | no detectable difference at this N | | wall-clock (s) | 22 | 192.6 | 60.7 | -131.9 (-68.5%) | [-315.8, -43.0] | 39.2 | 47.0 | +2.7 | tail effect only — typical task unchanged |

Read the medians before the means. For output tokens and wall-clock (s) the interval excludes zero on the MEAN, while the median difference is either negligible or points the other way. That is the signature of an effect living entirely in the tail: a handful of very long control runs move the mean, and the typical task is unchanged. Quoting the mean alone would be true of the arithmetic and false about the experience.

Net of the measured schema tax. The treatment arm is handed 945 tokens of tool definitions before turn one (measured, three identical replicates). Subtracting that, the mean total-token difference attributable to behaviour is -17,414 tokens. The tax is ~3% of a rollout's input side, so it explains almost none of any gap either way.

Latency is the weakest of these: it was measured on a machine that also ran build work (METHODOLOGY confound 13). A latency difference alone is not claimed as an effect.

Every figure above is regenerated from the committed artifacts by npm run report. Deltas are treatment minus control, in percentage points. The CI is the primary output; the p-value is a footnote (METHODOLOGY §4.2 and §4.4).

Modelled cost estimate: $11.28 USD at list prices. The experiment ran on a Claude subscription, so this is not money spent.

Where it does not help

No detectable effect on completion rate in any of the measured groupings, including the aggregate.

| category | pairs | control | treatment | delta | 95% CI | verdict | |---|---|---|---|---|---|---| | cross-module | 6 | 5/6 | 5/6 | 0.0pp | [-66.7, 33.3] | no detectable effect at this N | | multi-file | 7 | 6/7 | 7/7 | 14.3pp | [0.0, 42.9] | no detectable effect at this N | | multi-turn | 4 | 4/4 | 4/4 | 0.0pp | [0.0, 0.0] | no detectable effect at this N | | single-file | 5 | 5/5 | 5/5 | 0.0pp | [0.0, 0.0] | no detectable effect at this N | | ALL | 22 | 20/22 | 21/22 | 4.5pp | [-13.6, 13.6] | no detectable effect at this N |

In 2 of these groupings, both arms solved every task. That is not evidence that the arms are equivalent — it is a measurement with no resolution left. A benchmark at ceiling reporting "no effect" is reporting its own insensitivity, and the honest conclusion is that these tasks are too easy to discriminate between the arms.

The single-file category was pre-registered as an expected null before any run: a fix confined to one small file is exactly where shared project context should not help. It did not help there. It is reported first because that is what was promised, not because the result happened to be convenient.

A third arm: does a plain CLAUDE.md do the same thing?

providers/file writes the same fixture content as the Aethereum arm receives, rendered as markdown into a CLAUDE.md at the repo root. Same information, different delivery mechanism — which replicates the setup of the AGENTS.md study (arXiv:2601.20404) inside this harness.

| arm | n | solved | pass@1 | mean tokens | mean output | median wall | p95 wall | mean turns | |---|---|---|---|---|---|---|---|---| | control | 24 | 24 | 100.0% | 424,480 | 3,570 | 68.3s | 126.1s | 11.3 | | treatment | 24 | 24 | 100.0% | 376,184 | 2,931 | 58.0s | 95.6s | 10.3 |

Completion saturated: 24/24 in both arms, every category, no detectable effect. No efficiency metric survived either — the one interval that excluded zero (output tokens) is tail-driven and flagged as such. Full tables: npm run report:file-arm.

One finding from building it is worth more than the result. --setting-sources "" — the flag that makes the control arm clean — also suppresses CLAUDE.md and AGENTS.md discovery entirely. Measured three ways: with "" the agent answers UNKNOWN about a codename sitting in CLAUDE.md; with project it answers correctly; --disable-slash-commands is irrelevant. So this arm runs with --setting-sources project and differs from control in two flags rather than one, which is stated here rather than buried. Its purity assert is unchanged and still passes: zero MCP servers, zero skills.

A useful side effect: because the main experiment ran with --setting-sources "", gitpulse's own checked-in 103-line CLAUDE.md was not ambient context for either arm. The control arm really was clean — verified, not assumed.

The number that reframes every other number here

The third arm's batch re-ran the control arm with an identical flag set on the identical task set — an unplanned partial replicate, and the one thing a single-repeat design is supposed to be unable to see. ("Identical flag set", not byte-identical: the recorded agentFlags arrays differ in the ordering of --setting-sources "" and in the run-id inside the --mcp-config path. Both arms carry --setting-sources "", which is the flag that matters, and the pinned toolchain, commit and lockfile hashes are byte-equal.)

$ npx tsx scripts/compare-runs.ts full-001 file-001 --arm control

| | results/full-001 | results/file-001 | |---|---|---| | tasks compared | 24 | 24 | | resolved | 21/24 | 24/24 | | median wall-clock | 40s | 69s |

3 of 24 tasks (12.5%) flipped with no change to configuration, task, model or prompt.

| task | results/full-001 | results/file-001 | wall-clock | |---|---|---|---| | cc-context-danger-band | unresolved | resolved | 1128s → 82s | | cc-limit-unit-contract | unresolved | resolved | 551s → 97s | | cc-threshold-colours | unresolved (timeout) | resolved | 1354s → 68s |

One of those three was a harness timeout, not agent nondeterminism. cc-threshold-colours hit the 20-minute per-rollout ceiling in full-001 and was SIGTERM'd at 1354s (outcome: timeout, terminalReason: aborted_tools); on the replicate it finished in 68s. It is counted as a fail, as pre-registered. Stated plainly because it is one third of the evidence in this section: the like-for-like self-disagreement excluding it is 2 of 24, which is still at least as large as the 1-task difference between the arms. The other two flips are genuine same-config disagreements.

The noise floor is at least as large as the signal. The same arm, changing nothing, disagreed with itself on 3 tasks (2 excluding the timeout). The measured difference between the two arms in the main experiment was 1 task net (+4.5pp).

Like for like, the comparison is closer than "3 versus 1" makes it sound: the between-arm analysis was also 3 discordant tasks (b=2, c=1) that netted to 1. Three discordant within a single arm against three discordant between arms is the honest framing, and it still says the same thing — a between-arm result this size is comfortably inside the range you get from running the same arm twice.

It also dismantles the one efficiency story that looked like a finding. The wall-clock "effect" in the main run was carried by four very long control rollouts — and three of those same tasks, re-run identically, finished in 82s, 97s and 68s instead of 1128s, 551s and 1354s. Those excursions were noise, not a property of the control arm.

This is what "single-run — indicative, not conclusive" means in practice, and it is why the pre-registered wording was "no detectable effect at this N" rather than a percentage-point claim.

What was measured

| | | |---|---| | Agent | Claude Code 2.1.232, headless | | Model | claude-sonnet-5 (pinned; drift recorded per rollout) | | Tasks | 24 hand-authored defects in two real TypeScript repos, at pinned commits | | Categories | 6 single-file · 8 multi-file · 6 cross-module · 4 multi-turn | | Arms | control (no context) · treatment (Aethereum hosted MCP) · file (static CLAUDE.md) | | Rollouts | 118 across four batches; 46/48 and 48/48 on the two full runs, 4 symmetric exclusions | | Repeats | 1 — single-run, indicative, not conclusive | | Grading | test execution only; no LLM-as-judge | | Modelled spend | $30.61 USD across all batches, list prices, not money spent (subscription auth) |

Both arms invoke the same binary with the same flags:

claude -p "<task prompt>" --output-format stream-json --verbose \
       --model claude-sonnet-5 --max-turns 60 --permission-mode bypassPermissions \
       --strict-mcp-config --mcp-config <ARM FILE> \
       --setting-sources "" --disable-slash-commands --no-session-persistence

Only --mcp-config differs.

Other things this repository found out

The MCP schema tax is ~945 tokens, not ~10,000. Measured on an identical no-op prompt — three replicates, identical to the token — handing the agent 30 MCP tools costs 945 tokens (+3.1%). All 30 tools are registered and listed, but Claude Code 2.1.232 does not put their schemas in context; ~31 tokens per tool is a name and a one-line description. Any token gap between the arms beyond ~945 is therefore behavioural, not the price of holding the tools. Reproduce with npx tsx scripts/measure-overhead.ts.

A benchmark can hand the agent the answer without anyone noticing. The first pilot applied each defect as an uncommitted change, so the agent's first orienting command — git diff — printed the exact inverse of the fix. Ten of twelve rollouts resolved — four of them in under 30 seconds, though two still took over four minutes, so speed alone was not what gave it away. Defects are now folded into the pinned commit. Full write-up: results/pilot/NOTES.md.

Verification

$ npm test
 Test Files  16 passed | 1 skipped (17)
      Tests  210 passed | 1 skipped (211)
   Duration  4.11s

| gate | result | how to re-run | |---|---|---| | unit + conformance tests | 210 passed, 1 skipped (live test, needs credentials) | npm test | | types | clean | npm run typecheck | | lint | clean | npm run lint | | oracle gate — every task broken exactly where declared, fixed by reversing its defect | 24/24 | npm run validate-tasks | | fixture leakage — no defect symbol appears in any shared-context fixture | 0 leaks / 24 tasks | npx vitest run scripts/leakage.test.ts | | clean-clone — fresh clone runs the harness end to end | 12/12 | bash scripts/clean-clone-test.sh | | provider conformance — all three providers, one suite | 24 cases | npx vitest run src/provider.spec.ts |

CI runs lint, typecheck and tests on every push and pull request (.github/workflows/ci.yml). Experiment rollouts are deliberately not run in CI: they need a subscription token, and a benchmark that silently re-runs itself on every push is a benchmark you cannot audit.

Usage

git clone https://github.com/br9704/ctxbench && cd ctxbench && npm i

npm run bench -- tasks          # the task set and its category split
npm run bench -- providers      # available context providers
npm run report                  # regenerate every published number from results/

$ npm run bench -- tasks

cc-branch-and-limit-defaults       multi-file     4 f2p
cc-context-danger-band             multi-file     3 f2p
cc-control-strip-chokepoint        cross-module   1 f2p
...
24 tasks
  cross-module   6
  multi-file     8
  multi-turn     4
  single-file    6

Running the experiment itself needs a logged-in Claude Code. The none and file arms need nothing else; the aethereum arm additionally mints a room per task.

npm run bench -- run --control none --treatment file --dry-run
npm run bench -- run --control none --treatment file

| command | what it does | |---|---| | run | run the experiment — --tasks, --control, --treatment, --repeats, --dry-run | | tasks | list the task set with categories and fail_to_pass counts | | providers | list available context providers | | init-provider <name> | scaffold your own provider against the interface |

Validate the task set at any time — every task must prove it is broken in exactly the declared places and fully fixed by reversing its defect patch, or it is deleted:

npm run validate-tasks

Add your own context layer

Implement one interface, pass one conformance suite: docs/ADDING-A-PROVIDER.md. The file provider is the worked example — a real arm in these results, not a toy.

npm run bench -- init-provider mylayer
npx vitest run src/provider.spec.ts

What is committed

tasks/<id>/     task.yaml · defect.patch · fixture.json
results/<run>/  per rollout: trajectory.jsonl · run.json · report.json  + summary.json
METHODOLOGY.md  the pre-registration, and every deviation from it
masterplan.md   the build log, including what went wrong

Every published number is regenerable from those artifacts. If a figure cannot be regenerated, it does not belong in this README — including the ones that flatter the product under test.

Two levels of enforcement, and the difference matters. The results tables and the "where it does not help" section sit between <!-- RESULTS:BEGIN/END --> and <!-- NULL:BEGIN/END --> markers and are overwritten mechanically by npm run report; they cannot be hand-edited into optimism because the next run would revert them. Every other table here — the third-arm comparison, the noise-floor tables, "What was measured", the verification gates — is hand-written from the same artifacts and verified against them, but not marker-enforced. Both are regenerable; only the first is self-defending. npm run report:file-arm prints the third-arm numbers for checking but does not inject them.

Limitations

1. Only 3 of 22 tasks discriminated between the arms. On the paired task set the arms ran 90.9% control against 95.5% treatment, and two of four categories saturated completely. (The arm-summary table above reads 87.5% / 95.5% because it counts all admissible rollouts, including the two control-arm tasks whose treatment partner was excluded; the pre-registered primary analysis is the paired one, and it is +4.5pp, not the 8.0pp the unpaired figures imply. Quote the paired numbers.) With three discordant pairs, no possible split could have reached significance — the pre-registered arithmetic (METHODOLOGY §4.2) puts the floor at six. That is a limitation of the task set, not a finding about context layers: a benchmark near ceiling reporting "no effect" is largely reporting its own insensitivity.

2. The one task that went against treatment was an infrastructure failure that was not excluded. cc-registry-prototype-lookup is the sole discordant pair favouring control, and it is what makes McNemar b=2, c=1 rather than b=2, c=0. Its treatment rollout died at turn 5 after 64s with outcome: agent_error, terminalReason: api_error. METHODOLOGY §4.5 pre-registers that infra failures are excluded symmetrically, and the two fetch failed rollouts were; this one was not, because it was classified as an agent-side error rather than a harness one. That is a judgement call, applied inconsistently, and it is disclosed here rather than left for a reviewer to find. It cuts against the product under test — excluding it would have moved the aggregate delta up, not down — but the deviation is real either way.

3. Run-to-run noise is at least as large as the measured effect. The control arm re-run against itself flipped 3 of 24 tasks — 2 of 24 if you discount the one that was a harness timeout — against a between-arm difference of 1 task net (3 discordant either way). This is the most important caveat here.

4. The efficiency "effects" are tail effects, and the tail turned out to be noise. Means favoured treatment on wall-clock (−68%) and output tokens (−31%) with intervals excluding zero; the medians said otherwise (+2.7s, −7 tokens). Four long control rollouts carried the means, and re-running three of them identically produced 82s, 97s and 68s.

5. Solo agent. Aethereum's actual value proposition is multi-agent, cross-machine coordination. A single-agent benchmark measures only the read-shared-context half, and that is the most likely reason for a null result here.

6. The treatment is not purely informational. The provider's initialize block instructs the agent ("Coordinate before you build, not after"), so the treatment arm is "context layer + behavioural prompt". This design cannot separate them.

7. N = 24, single repeat. pass@3 and pass^3 are not reported — they need repeats the locked budget could not buy, and they are exactly what would answer the reliability question this result raises.

8. The benchmark author wrote the tasks and the fixtures. Bounded by the oracle gate and a mechanical leakage check, not eliminated.

9. LocalSandbox is not a container. Docker was unavailable; DockerSandbox ships implemented but unexercised, and is labelled as such.

10. Cost figures are modelled, not spent. The runs executed on a Claude subscription, so total_cost_usd is a client-side list-price estimate and no per-run charge was incurred.

11. One agent, one model. Claude Code 2.1.232 with claude-sonnet-5. The Agent seam exists so a second CLI can be added; it was not built in v1.

The full list — thirteen confounds — is in METHODOLOGY.md §5, written before any result existed.

Status

Sprints 0–8 are closed with every acceptance gate passed: the Provider and Sandbox seams, the Agent seam with its runtime purity assert, 24 oracle-validated tasks, four completed batches, 210 tests, and a clean-clone gate at 12/12. The full experiment, the third arm, the analysis, the pre-registration and the write-up are all done and committed.

The owner gates batched at the end (masterplan Sprint 9, amendment A4/D7) cleared on 2026-08-15: the modelled spend was signed off, the 24-task set is LOCKED, and publication was approved. This repository is public and tagged v1.0.0. The companion write-up is drafted at docs/POST-DRAFT.md and has not been posted anywhere yet.

ctxbench is not yet on npm — provenance attestation needs to run from CI rather than a laptop. There is no npm badge above until it is actually published, because a badge that 404s is worse than no badge. Everything in this repository works from a clone without it: npm run bench -- <cmd>.

Post-v1 backlog: a second agent CLI through the Agent seam, repeats for pass@k / pass^k, and a rolling task refresh as a contamination hedge.

Licence · Author

MIT — see LICENSE. Cite via CITATION.cff.

Built by Bruno Jaamaa · brunojaamaa.dev · @br9704