swarmkit-eval

Centralized evaluation infrastructure for agent harnesses — one matrix engine that runs (task × arm × model × seed) evals, scores them against ground truth (never agent-asserted state), and reports accuracy–cost Pareto curves with paired confidence intervals.

It owns the parts a rigorous eval needs and nobody else gives you: the per-cell workspace lifecycle, ground-truth grading, and the statistics (paired cluster-bootstrap CIs, pass^k, Pareto). Everything domain-specific — the tasks, the system-under-test, the grader — plugs in behind small seams.

Status: 0.0.x, incubating. The core loop, stats, backends, store, reporting, and the multi-agent / retrieval paths are built and tested; the API may still shift. See the design doc.

Install

npm install swarmkit-eval

Zero required runtime dependencies. SDK-backed cloud backends load lazily — install only what you use:

npm install e2b           # for the E2B microVM backend
npm install @wandb/sdk    # for the optional Weights & Biases run-tracking exporter
# modal: driven via a Python bridge (no npm dep)

The EC2 backend uses the local aws and ssh CLIs, so it adds no Node dependency.

Requires Node 20+ (ESM only).

Quick start

A complete, zero-token run: the built-in smoke benchmark (3 arms × 3 seeds) driven by the deterministic MockAdapter on the local in-process backend, graded against sealed checkpoints, with a paired-CI report.

import {
  runEval, buildReport, renderMarkdownReport,
  InProcessBackend, LocalResultStore, MockAdapter,
  smokeBenchmark, smokeMockSpec, SMOKE_ARMS,
  type EvalConfig,
} from "swarmkit-eval";

const config: EvalConfig = {
  runId: "demo",
  configVersion: "v0",
  benchmark: "smoke",
  arms: SMOKE_ARMS,                // stock | notes | graph
  models: [{ name: "mock" }],
  seeds: [1, 2, 3],
  backend: "in-process",
  concurrency: { cells: 4, modelConnections: 1 },
  output: { dir: ".eval-runs", trace: false },
};

const results = await runEval(config, {
  benchmark: smokeBenchmark,
  backend: new InProcessBackend(),
  store: new LocalResultStore(".eval-runs"),
  adapter: new MockAdapter(smokeMockSpec),
});

const report = buildReport(results, config, { baselineArmId: "stock" });
console.log(renderMarkdownReport(report));

Runs are content-addressed in the store, so re-running resumes — only cells whose prompt/scaffold/model/configVersion changed recompute.

The model: one engine, N configurations

The matrix is the cross product tasks × arms × models × seeds; each point is a cell. All the diversity lives in a handful of orthogonal seams — an eval is a configuration of these, not a fork:

| Seam | Question it answers | Examples | |---|---|---| | Arm | What knob am I varying? (the IV) | stock / notes / graph; a topology; a search config; a per-arm workspace file overlay (scaffold.files — vary the SUT's own bytes) | | ExecutionAdapter | What's under test? | native-cli (claude), hal, marble, mab, a retrieval SUT, mock | | ExecutionBackend | Where does it run? | in-process, docker, e2b, modal, ec2 | | Grader | How is it scored? | checkpoint, self, retrieval, tac (TheAgentCompany eval.py), swe (SWE-bench eval script), Harbor trial rewards | | EvalUnit | Independent or ordered? | cell (stateless) / stream (learning curve) | | BenchmarkAdapter | Where do tasks come from? | your load() + grader choice + optional ResourceSpecs; external harnesses such as SWE-Atlas/Harbor |

Anti-reward-hacking is structural: the adapter receives a PublicCell (gold stripped by an allowlist); only the grader sees the sealed task. status: success | failure | env_error is first-class — infra failures are excluded from aggregates and retried, never counted as task failures.

Execution modes

BenchmarkAdapter.execution selects the control flow:

• native — the core drives a single agent in an isolated workspace, graded on artifacts. The default.
• marble — multi-agent: N agents over a shared workspace + sidecar services across phases, with coordination KPIs (R / O / c / E_c / A_e). See adapters/marble.
• hal — wraps the HAL meta-harness for academic benchmarks (AppWorld / SWE-bench / GAIA / USACO). See adapters/hal/README.md.
• mab — the MARBLE / MultiAgentBench topology sweep. See adapters/mab/README.md.
• tau — τ²-bench: multi-turn tool-agent customer-service domains (airline / retail / telecom); the arm is a system-prompt addendum. See adapters/tau/README.md.
• mirrorcode — Epoch's MirrorCode code-reimplementation benchmark via Inspect; MirrorCode owns Docker compose setup + scoring while swarmkit owns matrix/store/stats. See adapters/mirrorcode/README.md.
• harbor — Harbor-owned benchmark jobs, currently used by SWE-Atlas QA / Test Writing / Refactoring: Harbor owns sandbox setup, agent execution, verifier upload, and reward calculation while swarmkit owns matrix identity, cache/resume, and report aggregation. See sweAtlasBenchmark, runSweAtlasHarborFlow, and the SWE-Atlas adapter guide. A no-token E2B smoke is available with cd src/eval && SWE_ATLAS_TASK_LIMIT=3 npm run smoke:swe-atlas after sourcing the needed cloud credentials. GPT-5-class verifier compatibility, E2B account-safe resource caps, configurable E2B sandbox lifetimes, Modal diagnostic kwargs, and redacted Harbor log tails are handled at the adapter boundary.
• env — multi-turn interactive environments: the agent (a Policy) acts in a stateful world over maxSteps turns (ALFWorld / τ-bench-style), scored on the episode.
• retrieval — a non-agent search/memory SUT scored by IR metrics (recall/precision/MRR/nDCG) against qrels, with an expensive corpus index modeled as a ResourceSpec (built once per scope, shared across cells). minimem is the reference client.
• memory-QA — ingest-then-QA benchmark support for LoCoMo and LongMemEval: shared loaders, normalized conversation/question/evidence types, store-neutral retrieval scoring, fusion sweeps, word-overlap metrics, Mem0-style judge prompt helpers, and paired QA comparison utilities. See adapters/memory-qa.

What you get for free

| Layer | Capabilities | |---|---| | Backends | in-process (local) · docker (orphan-reaping teardown) · e2b (Firecracker microVM, pause/resume, pre-baked claude template) · modal · ec2 (AWS CLI + SSH for heavyweight hosts) — all behind one port; grading is backend-agnostic. See backends/README.md. | | Stats | Paired cluster-bootstrap Δ-vs-baseline CIs (the methodologically correct unit — by task, not by (task,seed)), MDE, pass^k, Pareto frontier, IR metrics, coordination KPIs, per-arm envErrorRate + differential-dropout guard. | | Store / resume | Content-addressed (cellKey, contentHash) cache, resumes across runs; chained-prefix hashing for learning streams; resource-hash folding. | | Report | Markdown + HTML + JSON, provenance (git SHA), per-arm metric CIs, paired comparisons, accuracy–cost Pareto. Optional exporters push the report to external dashboards (Weights & Biases run tracking). | | Routing | A LiteLLM-gateway config layer with authoritative spend-log cost + no-silent-model-swap guarantees. See models/README.md. | | Trajectories | Opt-in (output.captureSession) per-cell transcript capture in the sessionlog format (prompt / assistant turns / tool calls) + the graded outcome — an RL-ready (transcript, reward) pair, on any backend. See consolidation.md. | | Shared resources | ResourceSpec — expensive cross-cell resources (a corpus index, a service stack) built lazily once per scope, memoized + refcounted, with a regression gate. |

On the roadmap (typed but not yet implemented): the judge grader (autoevals), a flag CLI, a remote result store, and the second (model-connection) concurrency gate.

Exporters (optional)

The report on disk (report.{md,html,json} from writeReport) is the canonical artifact. Exporters are optional side-channels that also push a finished report to an external dashboard. They are never the engine, grader, sweep, or trace store — purely observability — and they are best-effort: a failed export returns { ok: false, error } and never fails the eval.

Weights & Biases — run tracking only. Logs the run config, per-arm summary scalars (success / S_partial with CIs, cost, tokens, latency, custom metrics), and result tables to a W&B run. Off by default; runs only when configured (WANDB_PROJECT set; the SDK reads WANDB_API_KEY itself). Install the optional @wandb/sdk peer dep.

import { buildReport, writeReport, exportToWandb, wandbOptsFromEnv } from "swarmkit-eval";

const report = buildReport(results, config, { baselineArmId: "stock" });
const { json } = await writeReport(`${config.output.dir}/${config.runId}`, report);

const w = wandbOptsFromEnv();          // null unless WANDB_PROJECT is set → a clean no-op
if (w) {
  const res = await exportToWandb(report, { ...w, reportJsonPath: json });
  if (res.ok) console.log("W&B run:", res.url);
}

buildWandbPayload(report) is pure (no I/O / SDK) — it returns exactly what gets logged — and you can inject your own WandbSink (e.g. a Python-wandb bridge) via { sink }. Full details, the env-var table, SDK-version notes, and the deliberate non-goals (no Sweeps, no Weave) are in report/exporters/README.md.

Authoring a benchmark

import type { BenchmarkAdapter } from "swarmkit-eval";

export const myBench: BenchmarkAdapter = {
  id: "my-bench",
  execution: "native",
  grader: { kind: "checkpoint" },          // score task.checkpoints against the finished workspace
  async load() {
    return [{
      id: "t1",
      benchmark: "my-bench",
      prompt: "…instruction for the agent…",
      checkpoints: [{ id: "done", weight: 1, check: { type: "fileExists", path: "out.txt" } }], // SEALED
    }];
  },
};

Pair it with an ExecutionAdapter (the SUT) and a backend, then call runEval. For a model-backed agent, use the NativeCliAdapter (headless claude -p) or SandboxedCliAdapter (claude inside the sandbox).

Layout

core/        matrix · orchestrator · sanitize (sealed boundary) · store · resources · contentHash · types/
backends/    in-process · docker · e2b · modal · ec2  (+ e2b-template/)
adapters/    mock · native-cli · sandboxed-cli · marble/ · hal/ · mab/ · tau/ · mirrorcode/ · env/ · swe/ · tac/ · memory-qa/ · sessionlog
grade/       checkpoint · self · retrieval · tac · swe · registry · gate/
stats/       paired CIs · pass^k · Pareto · retrieval (IR) · coordination
models/      LiteLLM gateway config + spend-log cost
report/      markdown / html / json / Pareto / provenance  (+ exporters/: optional W&B run tracking)

The full rationale, decisions log, and methodology invariants live in swarmkit-eval-design.md.