refute

An adversarial verification panel. Give it a claim, a PR diff, or a finding, and it spawns N independent skeptic agents — each with a different lens, each told to refute it. Only what the panel can't break gets through.

It's the opposite of an agreeable assistant. Most LLM tooling is built to help, complete, and agree — exactly the wrong prior for verification. refute inverts it: every agent's only job is to find the one concrete reason you're wrong.

npx @dr_inker/refute "Our backtest uses each day's closing price to decide that day's trade, so it has no lookahead bias."

✗ REFUTED  Our backtest uses each day's closing price to decide that…  (4/5 refuted · score 0.78)
  ↳ kill shot: Deciding a trade from the same bar's close is classic lookahead: at decision time
    (before the close) that price is unknown, so live fills can never match the backtest.
  • Correctness   [refuted 0.90] the close isn't available at the moment the trade is decided…
  • Evidence      [refuted 0.80] no out-of-sample or walk-forward result is offered…
  • Edge cases    [refuted 0.70] gaps/halts make the same-bar close especially unrealizable…
  • Security      [holds 0.40] not a security concern
  • Consistency   [refuted 0.75] "no lookahead" contradicts using same-bar information…

refute exits 1 when something is refuted and 2 when the panel can't run (e.g. no API key) — so it fails your CI loudly instead of passing silently.

Why a panel of skeptics

A single reviewer asked "is this right?" tends to be agreeable, and a confident-but-wrong LLM answer is the hardest kind to catch. A panel fixes three things at once:

• Independence — each refuter runs blind to the others, so one missed angle doesn't sink the verdict.
• Diversity — different lenses (correctness, evidence, edge-cases, security, consistency, feasibility) catch different failure modes.
• A material-flaw bar — a refuter only refutes on an actually fatal or major flaw; "incomplete", "could be more precise", or an unmentioned caveat doesn't count. This calibration is what stops a panel of skeptics from refuting sound claims (see the eval below).

Does the panel actually work? (an honest eval)

npm run eval runs a labeled dataset through three arms: a single neutral reviewer, the panel (N=5 skeptics, one model), and a hetero panel (same lenses, refuters round-robined across two models). False claims should be refuted; sound ones should survive.

Dataset v2 (2026-07-09): 87 claims — 17 easy (the v1 set) + 70 hard. The hard claims were adversarially authored to fool a careful single reviewer (subtly-false expert-sounding statements, plus suspicious-sounding-but-TRUE bait for over-flaggers), then blind-verified: an independent model re-derived every label without seeing it, and claims with any disagreement or contestability were dropped. Arms measured on gemini-2.5-flash (single + panel) and gemini-2.5-flash + claude-haiku-4-5 (hetero).

| metric | single | panel (N=5) | hetero (N=5, 2 models) | |---|---|---|---| | recall, hard subset — flawed claims caught | 95.7% | 97.9% | 100% | | false-alarm, hard subset — sound claims wrongly flagged | 8.7% | 34.8% | 30.4% | | accuracy, hard subset | 94.3% | 87.1% | 90.0% | | recall (all 87) | 96.6% | 98.3% | 98.3% | | accuracy (all 87) | 94.3% | 85.1% | 88.5% | | LLM calls / prompt tokens | 87 / 9k | 434 / 221k | 424 / 224k | | median latency per claim | 4.7s | 16.6s | 15.0s |

What this honestly shows:

• The panel's recall edge is real. On the hard set the single reviewer waved through 2 of 47 flawed claims; the hetero panel caught all 47, including ones the single reviewer got wrong (e.g. "an American put on a non-dividend stock is never optimally exercised early"). If a missed flaw is expensive, the panel earns its cost.
• Model diversity beats the self-ensemble on every metric. Same recall, better precision, false alarms 41% → 31% (all-claims). Lens diversity is not model diversity — five copies of one model share its blind spots. If you run a panel, run --models.
• Over-flagging is still the panel's weakness. Skeptics wrongly refute ~30% of the suspicious-sounding-but-true bait claims (vs ~9% for the single reviewer) — note this true-set is adversarially suspicious, so production false-alarm rates should be lower. The single reviewer still wins on raw accuracy (94% vs 89–87%).
• Net: the panel is recall-maximizing — a pre-filter that flags for human review, with auditability (per-lens verdicts + a named kill-shot) as its second edge. It is not (yet) a drop-in replacement for one strong reviewer where false alarms are costly, which is why the CI action defaults to comment-only. At ~25× the tokens and ~3× the latency, you are buying recall and an audit trail, not accuracy.

History: the v1 eval (17 easy claims) saturated — the single reviewer scored 100% across the board, so the panel could only lose. Building it still earned its keep: the first panel version refuted every sound claim (100% false alarms), which drove the calibration that now holds it at ~30%: severity gating (a vote only counts for a material flaw), a fatal-override (one decisive flaw wins; scattered nitpicks need a majority), a strict rubric ("incomplete ≠ wrong"), and a quorum floor. Next lever: per-lens calibration and letting off-topic lenses abstain.

How it works

flowchart LR
  T["Target<br/>claim · PR diff · finding"] --> P((Panel))
  P --> R1["Refuter<br/>Correctness"]
  P --> R2["Refuter<br/>Evidence"]
  P --> R3["Refuter<br/>Edge cases"]
  P --> R4["Refuter<br/>Security"]
  P --> R5["Refuter<br/>Consistency"]
  R1 --> A["Aggregate<br/>majority + strongest kill-shot"]
  R2 --> A
  R3 --> A
  R4 --> A
  R5 --> A
  A --> V{{"refuted · survived · inconclusive"}}

Each lens becomes one refuter (an independent LLM call) that returns a structured verdict { severity, refuted, confidence, reasoning, killShot }, where only a fatal/major severity counts as a refutation. Aggregation is pure and deterministic: a claim is refuted when one lens lands a confident fatal flaw, or a (configurable) majority of lenses land a material flaw; the reported kill shot is the highest-severity objection. A refuter that errors is excluded from the vote rather than tanking it — but a verdict only stands if a quorum (a majority of the dispatched panel) returned valid votes; below quorum the result is inconclusive, so a mostly-errored panel can never quietly pass or fail your build on one refuter's say-so.

Install & use

npm install -g @dr_inker/refute      # or: npx @dr_inker/refute …
export OPENROUTER_API_KEY=sk-or-...   # https://openrouter.ai/keys

# a claim
refute "Mean reversion is the dominant, reliable edge on 5-minute Bitcoin."

# a pull request / patch (use - for stdin)
git diff origin/main...HEAD | refute --pr -

# a findings file (JSON array/object, or one claim per line)
refute --findings audit-findings.json

# tune the panel
refute --lenses security,edgecases "the upload endpoint validates file type by extension" -m anthropic/claude-sonnet-4.6
refute -n 6 --threshold 2 "…"        # all 6 lenses, fail on any 2 refutations (n caps at 6 — duplicate lenses would double-count)

# a heterogeneous panel — refuters round-robin across models. Lens diversity is not model
# diversity: one model's blind spots are correlated across all its refuters; mixing models
# is the defense (and output then names which model said what).
refute --models anthropic/claude-sonnet-4.6,google/gemini-2.5-pro,deepseek/deepseek-chat "…"
refute "…" --json                     # machine-readable output

Lenses

| id | attacks | |----|---------| | correctness | a concrete case/step where it's factually wrong | | evidence | unsupported, cherry-picked, or non-reproducible claims | | edgecases | boundaries, empty/null/overflow, races, ordering | | security | injection, auth bypass, data exposure, unvalidated input | | consistency | contradictions, hidden assumptions, over-generalization | | feasibility | performance, cost, latency, operational reality |

As a library

import { runPanel, claimTarget, pickLenses, OpenRouterClient } from '@dr_inker/refute';

const client = new OpenRouterClient(); // reads OPENROUTER_API_KEY
const result = await runPanel(client, claimTarget('All prime numbers are odd.'), {
  lenses: pickLenses(5),
  model: 'anthropic/claude-sonnet-4.6',
  skepticPrior: true,
  concurrency: 5,
});
console.log(result.outcome, result.killShot); // 'refuted', '2 is an even prime.'

The LLMClient interface is one method (complete), so the panel is provider-agnostic and trivially mockable — the test suite runs the full engine against a deterministic MockClient, no network.

In CI (GitHub Action)

Drop in .github/workflows/refute.yml (panel runs on every PR diff and comments the kill-shots), or use the composite action:

- uses: Dr-Inker/[email protected]
  with:
    openrouter-api-key: ${{ secrets.OPENROUTER_API_KEY }}
    refuters: '5'
    # fail-on-refuted: 'true'   # opt-in — see below

Non-blocking by default. On the current eval the panel over-flags (~30% false-alarm rate on adversarially suspicious-sounding sound claims), so it ships as a commenting reviewer: kill-shots land on the PR, the check stays green. Opt into blocking (fail-on-refuted: 'true' / REFUTE_BLOCKING in the workflow) once you trust its precision on your repo — a gate that cries wolf gets uninstalled.

Fork PRs don't receive secrets: the action skips with a visible warning rather than failing every external contribution.

Diffs are pre-filtered before they reach the panel: lockfiles/generated files (package-lock.json, yarn.lock, *.min.js, node_modules/, …) are dropped, and the diff is truncated on file boundaries to --max-diff-bytes (default 200 kB) with an explicit marker so the panel knows what it didn't see — omitted files are NOT reviewed.

Limitations (read before trusting it)

• Prompt injection. A PR diff or findings file is untrusted input to an LLM. The refuter prompt instructs the panel to treat embedded instructions as data (and as evidence of a flaw), but prompt-level guards are best-effort, not a security boundary — a determined author can still steer a verdict. Don't use the panel as your only gate on attacker-controlled input.
• Calibration. The measured false-alarm rate is ~30% on adversarially-baited sound claims (single reviewer: ~9%); treat "refuted" as "worth a human look", not as ground truth.
• Shared blind spots. All refuters run on the same model by default, so they share its failure modes; lens diversity is not model diversity. Use --models to run a heterogeneous panel.

Providers

Default is OpenRouter, but OpenRouterClient speaks plain OpenAI-compatible /chat/completions, so it also drives Google Gemini (baseUrl: https://generativelanguage.googleapis.com/v1beta/openai, model: gemini-2.5-flash), Anthropic's compat endpoint (baseUrl: https://api.anthropic.com/v1), local models (Ollama/LM Studio), or any OpenAI-style gateway — as a library pass { apiKey, baseUrl }, on the CLI set REFUTE_API_KEY + REFUTE_BASE_URL. Override the default model anywhere with REFUTE_MODEL. Providers that reject response_format: json_object (Anthropic's compat layer does) degrade gracefully: the client retries once without it and remembers; the verdict parser tolerates prose-wrapped JSON anyway. The client also keeps a cumulative usage tally ({ calls, promptTokens, completionTokens }) so you can see what a panel costs.

Exit codes

| code | meaning | |------|---------| | 0 | everything survived | | 1 | something was refuted | | 2 | inconclusive — the panel couldn't produce a verdict (bad key, usage error, network, below quorum) |

Pass --no-fail to always exit 0.

Design principles

• Refute, don't please. A vague "looks fine" is a failed review.
• Honest by construction. No fabricated confidence; errors surface as errors; CI fails loudly, never silently.
• Pure core. Aggregation, parsing, and target handling are deterministic and unit-tested.

Development

npm install
npm test          # 62 unit tests, no network
npm run build
npm run eval      # live 3-arm eval (needs GEMINI_API_KEY / ANTHROPIC_API_KEY / OPENROUTER_API_KEY); see eval/run-eval.ts header

License

MIT © Sam (Dr. Inker)