@tduyng/pi-autoresearch

An experiment loop turned on with one command: measure, read the number, keep or discard. The tools handle git, scope and the noise floor.

An optimization loop run by hand fails in three predictable ways. The benchmark gets edited until it shows a win. A regression gets committed because the number moved. The log is re-read whole every iteration, at more cost than the change.

Nothing at cold start. The extension adds one slash command, and the two tools are declared to the model only while /autoresearch runs. The extension removes them on off, clear, or when max_runs is reached.

Keep or discard stays with the model

No rule can tell whether 1 percent came from the diff, whether the code is now simpler, or whether a clean run was jitter. The tools measure and the model decides. A rule that decided would be wrong in exactly the interesting cases.

The measurement is crash-safe on disk: run_experiment writes a pending measurement before anything else, so a restart can still log it and a crash mid-iteration loses nothing.

The benchmark cannot move under the comparison. The script is hashed, editing it voids the comparison, and /autoresearch rebaseline is the deliberate way to accept a new baseline.

A keep cannot escape its scope. scope globs in the frontmatter bound the files a keep may commit, so an experiment cannot quietly rewrite a file it was not measuring.

The loop

edit -> run_experiment -> read the output -> log_experiment -> repeat

run_experiment snapshots the working tree, runs .auto/measure.sh, parses METRIC name=value lines, runs the optional checks, and returns the output, the metrics, the changed files, the noise advice and any scope violations. It does not touch git history.

log_experiment takes status: keep | discard | crash | checks_failed and the model's reasoning in asi. A keep commits the files the run touched with a conventional subject built from the description. Every other status restores those files.

| Mechanism | Effect | | --- | --- | | Full benchmark output | Phase timings and errors reach the model, so the next hypothesis comes from data | | Noise floor from median absolute deviation | Separates jitter from a real win, without overriding the model | | Pending measurement | One run between the tools, crash-safe, so a restart can still log it | | Scope globs | A keep cannot touch files outside the declared scope | | Benchmark hash | Editing the benchmark to move the number voids the comparison | | Worktree lock and dirty baseline | One loop per worktree, and files another session left dirty are never committed | | Finalize in temporary worktrees | Kept runs become branches without moving HEAD under another session | | Compaction digest from disk | Prompt, recent runs, dead ends and ideas survive a reset with no model call | | Conventional commits | type(scope): summary, at most 72 characters, one Metric: trailer |

The project files

.auto/prompt.md carries the frontmatter and the objective:

---
name: vitest startup
metric: total_ms
unit: ms
direction: lower
command: bash .auto/measure.sh
scope: [vitest.config.ts, test/**]
max_runs: 40
min_gain: 0.5
---

What is optimized, the constraints, and what a good change looks like.

min_gain is the smallest relative improvement the advice calls real, in percent. It never blocks a keep.

| File | Purpose | | --- | --- | | .auto/prompt.md | Frontmatter config, the objective, what has been tried | | .auto/measure.sh | The benchmark, printing METRIC name=value | | .auto/log.jsonl | Append-only record | | .auto/checks.sh | Optional correctness checks, run after a passing benchmark | | .auto/ideas.md | Optional backlog | | .auto/pending.json | The run waiting for a decision | | .auto/lock.json | The session that owns this worktree |

The first run adds .auto/ to .git/info/exclude, including from a linked worktree, so none of it lands in a commit.

Commands

| Command | Effect | | --- | --- | | /autoresearch <goal> | Start a session, or resume the one in .auto/. On the trunk branch the first call creates a loop worktree and stops there, so the loop never runs in the checkout you are editing; run the command again inside that directory. | | /autoresearch status | Print the digest | | /autoresearch off | Stop and remove the tools | | /autoresearch rebaseline | Accept a changed benchmark script as a new baseline | | /autoresearch finalize | Split kept runs into independent branches from the merge base | | /autoresearch clear | Delete the log and stop |

finalize groups kept runs by file overlap and creates one branch per group in temporary worktrees. It never checks the current worktree out to another branch, and it verifies the union of the branches against the original before it reports.

Worktrees and other sessions

One loop owns a worktree at a time: /autoresearch takes .auto/lock.json and refuses a second session before it runs anything. Files that were already dirty when the loop started are recorded in the lock and never committed or reverted.

For a second loop, use a separate git worktree, which gets its own .auto/, branch and log. The first run on trunk creates autoresearch/<name>-<timestamp> in a worktree at ~/worktrees/<repo>/autoresearch-<name>-<timestamp> and leaves the shared checkout on trunk. An existing feature branch is used as it is.

Compaction and restarts

While the loop is active, session_before_compact replaces pi's model summary with a deterministic digest built from .auto/: the prompt body, recent runs, kept runs, dead ends, ideas and any pending measurement. No model call, and no history lost to a summary that forgot the metric.

After a restart of the same session the loop re-arms from the session's activation entry and sends one short resume message. A fresh session never auto-activates.

Install

pi install npm:@tduyng/pi-autoresearch

Local checkout: pi install /path/to/dotfiles/pi/agent/extensions/pi-autoresearch

It needs a git repository, since a keep commits, the lock is a file, and finalize uses worktrees. The measure script runs through bash -c.

Run it in an interactive session or through RPC. pi -p exits as soon as the slash command returns, before the loop's first turn, so a print run cannot drive it.

Limits

• A run has a default timeout of 600 seconds, and the measure script's own timeout is the one that applies.
• Auto-resume gives up after 200 turns, and an idle resume after 40.
• Output is trimmed to the last 40 lines, with 128 KB retained and a 20 MB cap on the temp log.
• finalize reports a conflict instead of resolving one. Splitting the kept runs is a bookkeeping job, not a merge.