@mittalsuraj18/opencode-auto-research

An OpenCode plugin that implements an automated benchmark-driven optimization loop — a lightweight, OpenCode-native take on the autoresearch pattern popularized by Andrej Karpathy.

What is Autoresearch?

The autoresearch pattern — introduced by karpathy/autoresearch (83K+ stars) — is a simple but powerful idea: give an AI agent a measurable goal and let it experiment autonomously. The agent modifies code, runs a benchmark, checks if the result improved, keeps or discards the change, and repeats. You wake up in the morning to a log of experiments and (hopefully) a better codebase.

This plugin brings that pattern directly into OpenCode as a first-class plugin — no external scripts, no manual orchestration. The loop runs inside your existing OpenCode session with built-in git isolation, auto-compaction, and scope enforcement.

How it differs from other tools

| Tool | Approach | Key Difference | |------|----------|----------------| | This plugin | OpenCode-native plugin | Runs inside OpenCode session; auto-compaction; git branch isolation; scope enforcement | | karpathy/autoresearch | Standalone script + program.md | Single-file optimization (ML training); agent reads markdown instructions | | ratchet | CLI orchestrator | Generates agent prompts; handles git/benchmark externally; multi-armed bandit strategy selection | | auto-optimize | Claude Code skill | Structured reasoning pipeline (Opus planner); noise-floor validation; disassembly analysis | | darwin-derby | CLI orchestrator | Swarm mode; git-push-based proposals; evaluation hidden from agents | | Artificial General Research | Claude Code skill | Fresh context per iteration; Metric+Guard+Rework pattern; stuck detection protocol | | VeRO | Python evaluation harness | Optimizes LLM-based agent code; subprocess-isolated evaluations | | Maleick/AutoResearch | OpenCode + Hermes plugin | Subagent-first; multi-runtime (OpenCode + Hermes); recursive self-improvement; 15+ slash commands |

Overview

This plugin enables OpenCode agents to systematically optimize code performance through:

1. Benchmark harness (autoresearch.sh) — measures target metrics
2. Experiment loop — modify code, run benchmark, evaluate, keep or discard
3. Auto-compaction — context is compacted after every iteration to prevent overflow
4. Git integration — commits on keep, resets on discard, dedicated branches
5. Scope enforcement — restrict which files the agent can and cannot modify
6. Confidence scoring — MAD-based statistical confidence in improvements

Installation

From npm (recommended)

npm install @mittalsuraj18/opencode-auto-research

Add to your opencode.json:

{
  "plugin": ["@mittalsuraj18/opencode-auto-research"]
}

Restart OpenCode. The /autoresearch command and all four tools are now available.

From local files

Place the built plugin in .opencode/plugins/ or ~/.config/opencode/plugins/. See OpenCode plugin docs for details.

Quick Start

Option 1: The /autoresearch command (easiest)

Just type in OpenCode:

/autoresearch optimize compile time

What happens

1. If an experiment is already active → resumes it with the new goal
2. If no experiment is active → starts a new one:
   • Creates autoresearch.sh if missing
   • Calls init_experiment with an appropriate benchmark name and metric
   • Runs the baseline with run_experiment
   • Logs the baseline with log_experiment keep
   • The auto-iteration loop begins

Resume behavior

/autoresearch

Without a goal, it continues the existing experiment. With a goal, it updates the experiment's goal and continues.

No manual opencode.json configuration needed — the command is registered automatically for autocomplete.

Option 2: Direct tool usage

1. Create autoresearch.sh in your project root that prints metrics:

   #!/bin/bash
   # Run your benchmark here...
   METRIC compile_time_ms=1200
   METRIC bundle_size_bytes=45000
   ASI hypothesis=reduced_loop_iterations
   ASI next_action_hint=try_unrolling_factor_4

2. Call init_experiment with your benchmark name and metric

3. Call run_experiment to run the baseline

4. Call log_experiment with status: keep to establish the baseline

5. The agent will auto-iterate, optimizing the metric

Tools

init_experiment

Initialize a new autoresearch experiment session.

| Parameter | Type | Description | |-----------|------|-------------| | name | string | Name of the experiment | | goal | string? | What to optimize | | primary_metric | string | Main metric to track | | metric_unit | string? | Unit (ms, bytes, etc.) | | direction | "lower" | "higher" | Better direction | | scope_paths | string[]? | Files agent may modify | | off_limits | string[]? | Files agent must NOT modify | | max_iterations | number? | Max experiments per segment | | new_segment | boolean? | Start fresh segment |

Auto-behaviors:

• Creates autoresearch.md if missing
• Creates autoresearch/* branch if not on one
• Auto-commits harness as baseline on autoresearch branch

run_experiment

Run the benchmark harness (bash autoresearch.sh).

| Parameter | Type | Description | |-----------|------|-------------| | timeout_seconds | number? | Max runtime (default: 600) |

Output:

• Parsed metrics from METRIC name=value lines
• ASI (Agent State Info) from ASI key=value lines
• Truncated raw output (4KB / 10 lines max)
• Full log saved to ~/.opencode-autoresearch/<project>/runs/<id>/benchmark.log

log_experiment

Log the result and update experiment state.

| Parameter | Type | Description | |-----------|------|-------------| | metric | number | Primary metric value | | status | "keep" | "discard" | "crash" | "checks_failed" | Whether to keep changes | | description | string | What this run tested | | metrics | Record<string, number>? | Additional metrics | | asi | Record<string, unknown>? | Agent state info | | justification | string? | Why this status |

Auto-behaviors:

• keep on autoresearch branch → commits changes
• discard/crash on autoresearch branch → git reset --hard HEAD + git clean -fd
• discard/crash on other branch → only reverts run-modified files
• Detects scope deviations (modified files outside scope_paths or in off_limits)
• Updates autoresearch.md
• Checks max_iterations; disables mode if reached

update_notes

Update experiment notes.

| Parameter | Type | Description | |-----------|------|-------------| | body | string? | Replace entire notes | | append_idea | string? | Append bullet to ideas |

Benchmark Harness Format

Your autoresearch.sh must print metrics in this format:

#!/bin/bash
# Run your benchmark here...
METRIC compile_time_ms=1200
METRIC bundle_size_bytes=45000
ASI hypothesis=reduced_loop_iterations
ASI next_action_hint=try_unrolling_factor_4

• METRIC <name>=<value> — one per line, numeric values only
• ASI <key>=<value> — optional, any string value (hypothesis, next_action_hint, rollback_reason, etc.)
• Exit code 0 = success, non-zero = failure

How the Loop Works

┌─────────────────────────────────────────────────────────┐
│  AUTORESEARCH LOOP                                      │
│                                                          │
│  1. init_experiment → create branch, set goal & metric    │
│  2. run_experiment → execute autoresearch.sh              │
│  3. Agent analyzes results                               │
│  4. log_experiment (keep/discard/crash)                   │
│     ├─ keep    → commit changes, update best             │
│     ├─ discard → git reset --hard HEAD + clean           │
│     └─ crash   → git reset --hard HEAD + clean           │
│  5. Auto-compact session context                         │
│  6. Continue from step 2                                 │
└─────────────────────────────────────────────────────────┘

Each iteration is fully automated. The agent modifies code within the configured scope, runs the benchmark, evaluates the result, and decides whether to keep or discard. After logging, the session is compacted to prevent context overflow, and the loop continues indefinitely until max iterations are reached or the user interrupts.

Git Workflow

1. init_experiment creates branch: autoresearch/<goal>-<YYYYMMDD>
2. run_experiment runs benchmark, records modified files
3. log_experiment keep commits changes with formatted message
4. log_experiment discard resets worktree to HEAD
5. At any point: git log shows the experiment history

This mirrors the git workflow from karpathy/autoresearch's program.md, but is handled automatically by the plugin rather than requiring the agent to manually manage git commands.

Auto-Compaction

After every log_experiment, the plugin automatically triggers a session compaction via client.summarize(). This:

• Summarizes the conversation history
• Preserves experiment context (goal, baseline, best result)
• Injects a synthetic "continue" message via experimental.compaction.autocontinue
• Keeps the agent loop running indefinitely without context overflow

Without auto-compaction, each iteration adds context until the model's context window fills up and the loop degrades. This plugin solves that by compacting after every iteration while preserving the essential experiment state through experimental.session.compacting hooks.

Scope Enforcement

The plugin tracks which files the agent modifies during each experiment:

• scope_paths — restrict modifications to only these paths
• off_limits — explicitly forbid modifications to these paths
• Deviation detection — log_experiment flags any modifications outside the declared scope

This prevents the agent from accidentally modifying critical files (e.g., test fixtures, config files, lock files) during its optimization loop.

Confidence Scoring

The plugin uses a Median Absolute Deviation (MAD)-based confidence score to evaluate whether an improvement is statistically meaningful:

• Low confidence → the improvement may be within measurement noise
• High confidence → the improvement is likely real

This helps the agent make informed decisions about whether to keep aggressive changes or revert to the baseline.

Storage

• SQLite: ~/.opencode-autoresearch/<encoded-project-path>.db
  • sessions table: experiment configuration
  • runs table: benchmark results
• Logs: ~/.opencode-autoresearch/<project>/runs/<id>/benchmark.log
• Markdown: ./autoresearch.md in project root

Configuration

No configuration required. The plugin auto-detects:

• Current git branch
• Project directory
• Available models (for compaction)

Source Layout

| File / Dir | Role | |------------|------| | src/index.ts | Plugin entry point. Registers 4 tools, the /autoresearch command, system-prompt injection, and compaction hooks. | | src/types.ts | Central type definitions (ExperimentState, AutoresearchRuntime, etc.). | | src/state.ts | Runtime state helpers (createRuntimeStore, buildExperimentState). | | src/storage.ts | SQLite persistence layer. | | src/git.ts | Git branch detection, commit/reset helpers. | | src/helpers.ts | Shared formatting and parsing utilities. | | src/tools/init-experiment.ts | init_experiment tool — creates branch, session, baseline. | | src/tools/run-experiment.ts | run_experiment tool — executes autoresearch.sh, parses METRIC/ASI lines. | | src/tools/log-experiment.ts | log_experiment tool — commits on keep, resets on discard/crash, updates autoresearch.md. | | src/tools/update-notes.ts | update_notes tool — appends to experiment notes. | | src/prompts/system.md | Template for injected system prompt when autoresearch mode is active. | | src/prompts/setup.md | Template used during experiment setup. |

Features

• Automated experiment loop with keep/discard decisions
• Git branch isolation (autoresearch/*) — experiment safely without touching main
• Auto-commit on keep / auto-reset on discard — clean state after every iteration
• Scope deviation detection — restrict what the agent can modify
• Confidence scoring (MAD-based) — distinguish real improvements from noise
• Max iteration enforcement — prevents runaway loops
• Auto-compaction after every iteration — unlimited iterations without context overflow
• Secondary metric tracking — monitor additional metrics alongside the primary one
• ASI (Agent State Info) logging — pass hypotheses and hints between iterations
• Persistent experiment notes — update_notes persists across compactions
• autoresearch.md auto-generation and updates — experiment log in your repo
• /autoresearch slash command — start or resume experiments with one command
• OpenCode-native — no external scripts, runs inside your existing session

Comparison with the Original Autoresearch Pattern

Karpathy's autoresearch introduced a simple loop: modify code → run benchmark → keep or revert → repeat. The agent reads a program.md file with instructions and manages git manually. This plugin builds on that foundation with several OpenCode-native improvements:

| Aspect | karpathy/autoresearch | This Plugin | |--------|----------------------|-------------| | Runtime | Standalone (any agent) | OpenCode plugin | | Git management | Manual by agent | Automatic (plugin handles commit/reset) | | Context management | Agent-dependent (often degrades) | Auto-compaction with state preservation | | Metric parsing | Agent reads raw output | Structured METRIC/ASI protocol | | Scope control | Single-file convention | Explicit scope_paths / off_limits | | Confidence | None (manual threshold) | MAD-based statistical confidence | | Session persistence | Git log only | SQLite + git + markdown | | Branch isolation | Manual by agent | Automatic (autoresearch/* branch) | | Command interface | Prompt-based | /autoresearch slash command + 4 tools |

Limitations

• No TUI dashboard widget (OpenCode server plugins cannot render UI)
• No synthetic auto-resume messages (mitigated by strong system prompt + compaction auto-continue)
• No custom tool renderers (standard OpenCode tool output)
• No multi-agent swarm mode (see darwin-derby for swarm experiments)
• No multi-armed bandit strategy selection (see ratchet for bandit-based approaches)
• No built-in noise-floor validation (see auto-optimize for variance checks)

License

MIT