Trust, but verify — one claim at a time.

A pi extension with one job: reproduce the quantitative results of an academic paper, and tell you honestly which ones survive contact with reality.

Install · Usage · How it works

A paper lands on your desk claiming 94.2% accuracy. There's no code — or there's a repo that's missing the training script, hardcodes a path to someone's laptop, and references a config file that isn't in the commit. You need to know: does the number hold? Today that means days of archaeology — guessing at hyperparameters, reconstructing the data pipeline from a figure caption, and slowly losing track of which of the paper's ten claims you've actually checked.

pi-repro does that archaeology for you. It treats the paper as the source of truth and the code as a lead to follow, not a thing to trust. It reconstructs what's missing, runs what it can, and refuses to fudge the difference between what the paper said and what your machine did.

You hand it a paper (link, PDF, arXiv id, or DOI) and — optionally — a reference GitHub repo. It reads the paper, pulls out every number you care about as a tracked claim, audits the shipped code for the gaps it always has, plans the reproduction, runs each experiment, and files a report on what replicated, what didn't, and what flat-out refused to run.

It follows the paper's method and reports whatever falls out, even when that's an inconvenient truth. It will never nudge a result toward the published value by quietly leaving the method behind — because the gap between "the paper says 94.2%" and "I got 94.2% on my machine" is where science actually happens.

Why you'd want this

• Researchers — before you build on someone's result, confirm the foundation holds. Reproduce the baseline you're about to beat so your delta is real and not a difference in setup.
• ML engineers — vet a flashy SOTA claim against your own hardware and data pipeline before you bet a sprint on integrating it.
• Open-source contributors — reproduce a paper's results to back a from-scratch implementation, file an issue when the published code doesn't match the paper, or hand maintainers an auditable report instead of a vague "I couldn't get your numbers."
• Peer reviewers & area chairs — turn "the authors claim X" into "I ran it and got X (or didn't)", with a paper trail you can attach.

Install

pi install npm:pi-repro
# or, from a local checkout:
pi install file:/path/to/pi-repro

Usage

In a project where you want to do the work:

reproduce https://arxiv.org/abs/XXXX.XXXXX  (optionally: repo https://github.com/...)

The repro-create skill drives everything from there: ingest, gap analysis, plan, run, report. All state lands in a single .repro/ folder at the project root, so the work survives restarts and context resets — pick up exactly where you left off, and hand the folder to anyone who wants to check your work.

How it works

ingest paper ─► extract claims (claims.json) ─► gap-analyze repo (gap.md) ─► plan (plan.md)
      │
      ▼
 for each claim:  run_reproduction ─► compare reproduced vs reported ─► log_result (status)
      │            └─ optional bounded per-claim debug loop if blocked/mismatch
      ▼
 report.md   (per claim: ✓ reproduced / ~ partial / ✗ mismatch / ⛔ blocked / · pending)

The whole thing is verification, not optimization. When a claim doesn't reproduce, that's a finding — not a bug to paper over.

When gap-analysis finds that the method already ships as a complete, installable package, pi-repro stops and asks you: use the existing package (validates the result fast) or reimplement from scratch (a stronger test of whether the paper is reproducible from its description). Your choice is recorded in config.json.

Tools

| Tool | What it does | |------|--------------| | init_reproduction | Scaffold the .repro/ session (once). Idempotent. | | register_claim | Record/update one reproducible quantitative claim from the paper. | | run_reproduction | Run a command (bash -lc) and capture output. Doesn't judge. | | log_result | Record a reproduced value and classify it vs the reported value. | | reproduction_status | Show all claims, reported vs reproduced, and totals. |

Skills

| Skill | Role | |-------|------| | repro-create | Orchestrator / spine — the entry point. | | repro-ingest | Paper → paper.md + registered claims. | | repro-gap-analysis | Reference repo → gap.md (implemented vs missing). | | repro-report | Claims + logs → report.md. |

The .repro/ session

All state lives in one folder at the root of the project being reproduced — human readable, version-controllable, and the single source of truth:

| File | Purpose | |------|---------| | config.json | name, paper source, repo, language, virtualenv, implementation mode, tolerance, loop budget | | paper.md | extracted summary, method, datasets, hyperparams, compute | | claims.json | structured claims — the source of truth for status | | gap.md | implemented vs missing/partial analysis of the reference repo | | plan.md | environment, data acquisition, per-claim approach and ordering | | log.jsonl | append-only run/result log | | report.md | the final reproduction report | | env/setup.sh | reproducible environment setup | | sources/ | downloaded paper artifacts / clone notes |

How claims are judged

log_result compares the reproduced value to the reported value using a relative tolerance (default 5%, configurable per-claim or per-session):

• within tolerance → reproduced (✓)
• beating the paper in the better direction → reproduced (✓)
• within 3× tolerance → partial (~)
• further off → mismatch (✗)
• couldn't run → blocked (⛔)

No grade inflation: a number only counts as reproduced when the method that produced it is the paper's method.

Configuration

• PI_REPRO_SHORTCUT — key for the fullscreen dashboard (default ctrl+r; set to none to disable).

Development

npm install
npm run typecheck   # tsc --noEmit
npm test            # node --experimental-strip-types --test tests/*.test.mjs

npm test requires a Node 22+ build with TypeScript stripping support (the same requirement pi has for loading .ts extensions). On a Node built without it the tests can be run by transpiling the modules first.

License

MIT