poke-crucible

Run masses of headless Gen 3 OU battles to rank Teams by how often they win. Both sides of every Battle are driven by one fixed, competent heuristic Pilot, so a result measures the Team, not the player. Three capabilities, each a single seam built on the last:

• evaluate (Slice 1) — rank Teams by Strength against a fixed Gauntlet, or round-robin.
• critique (Slice 2) — an LLM Critic explains one Team's weaknesses and matchup holes.
• design (Slice 3) — the Designer evolves stronger Teams, with the LLM proposing mutations and naming the Archetypes it discovers.

Vocabulary lives in CONTEXT.md; the specs are the PRDs under docs/; decisions are the ADRs.

Install & build

npm install
npm run build      # compiles TypeScript to dist/

Requires Node ≥ 20. The stack is in-process TypeScript on the @pkmn/* toolkit (@pkmn/sim, @pkmn/sets, @pkmn/data) — no server, no network, no Python (ADR 0001).

CLI

# Absolute Strength vs the fixed Gauntlet
node dist/cli.js eval --mode gauntlet team.txt

# Relative Strength: rank several of your own Teams round-robin
node dist/cli.js eval --mode round-robin teamA.txt teamB.txt teamC.txt

# Pipe a single Team paste in on stdin
cat team.txt | node dist/cli.js eval

Options: --mode <gauntlet|round-robin> (default gauntlet), --battles <N> (default 100), --seed <string>, --gauntlet-version <v>, --out <file> (default poke-crucible-results.json). Each team file is standard Showdown export/paste text; multiple Teams in one file are separated by === [gen3ou] Name === header lines.

The CLI prints a ranked Strength table (with a 95% confidence interval and a per-matchup breakdown) and writes the full structured result to --out.

Programmatic API

The base seam is evaluate — the Critic and Designer build on it, not on the CLI:

import { evaluate } from 'poke-crucible';

const result = await evaluate({
  teams: showdownPasteText,              // string or array of pastes/{name,paste}
  opposition: { mode: 'gauntlet' },      // or { mode: 'round-robin' }
  battlesPerPairing: 100,
  seed: 'my-seed',
});
// result: ranked Strength rows (per-matchup win-rates tagged with Archetype),
// the seed, the Gauntlet version, and any illegal-Team validation errors.

Illegal Teams never throw past the seam — they come back as result.errors, named, so a bad Team can't silently poison a run.

How it works

• Pilot (ADR 0002): one shared Gen 3 heuristic, identical on both sides, built in competence layers — secure a KO → pivot out of a losing matchup → recover → lay Spikes / spin → spread Toxic/burn/paralysis → set up → max damage. Every Strength number is relative to this Pilot.
• Determinism: a master seed threads through @pkmn/sim's PRNG; each pairing's Battles derive sub-seeds deterministically and sides are swapped 50/50, so a fixed seed reproduces identical results.
• Gauntlet (ADR 0003): a fixed, versioned roster of Archetype-labeled Gen 3 OU Teams. Every result records the version, so 62% Strength means the same thing next week. Two versions ship: g1 (default; hand-authored) and g2 (real Smogon ADV OU sample teams, fetched verbatim, --gauntlet-version g2). g1 stays the default baseline; swapping it is a deliberate rebaseline.
• Parallelism: --concurrency <N> (or POKE_CONCURRENCY) runs Battles across N worker threads. All Battles of a run are dispatched together so the pool stays saturated; results are byte-identical to serial (each Battle keeps its own seed). Default is 1 (inline). It pays off on big, expensive runs — the Designer, large evaluations — where it gives roughly a 2× speedup on 4 workers; for small/fast runs the thread overhead can make it a wash, hence the default.

Gauntlet provenance (a deliberate deviation)

The design envisioned sourcing the Gauntlet from Smogon's sample teams + usage stats via fetch-and-parse. That is deferred: the g1 roster is hand-authored from canonical Smogon Gen 3 OU sets (see src/gauntlet/teams-g1.ts). Because the Gauntlet is versioned and every result records its version, re-sourcing later is a g2 bump, not a rewrite — nothing silently drifts.

Critic (LLM analysis — Slice 2)

Turn an evaluation into a readable, grounded critique of one Team:

node dist/cli.js critique team.txt
node dist/cli.js critique --model claude-sonnet-5 --effort low --battles 100 team.txt

The Critic runs the evaluation, then asks Claude (via the Claude Agent SDK) to explain the Team's weaknesses, the Archetypes that exploit them, the mechanism, and concrete legal fixes — writing a structured Critique JSON. It is told the Pilot under-rates stall (ADR 0004), so it caveats defense-dependent verdicts rather than presenting them as ground truth.

Auth (individual use): the Critic runs on your Claude subscription, not a pay-per-token key. Log in with Claude Code (or set CLAUDE_CODE_OAUTH_TOKEN), and make sure ANTHROPIC_API_KEY is unset — it shadows the subscription OAuth token and reverts to per-token billing. The model is called with all tools disabled and low reasoning effort; nothing runs on your filesystem.

Programmatically the seam is critique(input, model) / critiqueTeam(paste, model, opts); the model is any CritiqueModel (use StubCritiqueModel in tests, AgentSdkCritiqueModel live). See docs/prd-slice-2-critic.md.

Designer (discovery — Slice 3)

Evolve stronger Teams automatically. A population of legal Teams is scored each generation by a composite fitness (Gauntlet Strength + peer round-robin + a worst-matchup robustness term), bred by mutation and crossover, with the LLM proposing variants — driven by the Critic's findings — and naming the Archetypes it discovers.

# Evolve from a starting Team (omit the file to search from scratch)
node dist/cli.js design --population 8 --generations 4 team.txt

# Random operators only — no LLM, no subscription call
node dist/cli.js design --no-llm --population 8 --generations 4

Options: --population, --generations, --battles (per fitness pairing), --model / --effort (Designer LLM), --no-llm, --seed, --gauntlet-version, --out. Every candidate is validated legal Gen 3 OU; the run is deterministic under its seed and reports each Team's lineage, Archetype name, fitness components, plus the budget used and total Battles run. To bound compute it caches unchanged Teams, screens cheaply before spending full battles on contenders, and honours a hard battle cap.

Programmatically the seam is design(input, model, options); the model is any DesignerModel (StubDesignerModel for tests, AgentSdkDesignerModel live), and an optional critic lets the Critic drive the mutation proposals. Auth is the same subscription path as the Critic. See docs/prd-slice-3-designer.md.

Fitness is deliberately not raw win-rate (ADR 0004) — it rewards well-rounded Teams over glass cannons and dampens the Pilot's offense lean (see the limitation below for how far that goes).

Testing (the trust gate)

npm run typecheck
npm test

The suite asserts external behavior only, through the evaluate() seam (never the Pilot's individual move choices): determinism, sanity ordering (a strong Team out-Strengths a weak one), archetype non-bias (a solid stall Team stays competitive in a head-to-head ladder of the eight archetypes and can beat offense — the live check that the Pilot isn't biased against defense), legality, and stable win-rates. Every Gauntlet Team is asserted legal Gen 3 OU.

Known limitation: the Pilot's offense lean

The shared greedy Pilot systematically under-rates stall — offense plays near-optimally under greedy heuristics while stall needs long-term planning. In an archetype-vs-archetype ladder the best stall Team lands mid-lower (≈38% Strength, 6th of 8), not mid-pack.

This was probed exhaustively (ADR 0004): three Pilot improvements (anticipatory switching, tempo tweaks, a positional-eval Pilot) all failed to lift stall; real-sim look-ahead is infeasible (@pkmn/sim state can't be resumed after deserialization); and the Designer's composite fitness only partly corrects the lean — stall's worst matchups are genuinely ~0% under the Pilot, so a robustness term can't recover them.

The lean is therefore a known, bounded, documented property of the greedy-Pilot design, not a bug — every Strength number is explicitly relative to this Pilot (ADR 0002). Closing it needs a stronger Pilot (a shallow search or ML policy — deferred) or a non-Pilot signal of a Team's merit; both are future work. The Critic caveats defense-dependent verdicts and the Designer's fitness avoids rewarding glass cannons, so the slices degrade gracefully rather than silently trusting the lean.