@coder/aegis

Aegis runs controlled A/B experiments on black-box coding agents. Each trial executes in an isolated sandbox behind a capture proxy that records the full model-traffic transcript; artifact bundles are graded offline and reported with paired statistics — so you can measure whether a skill, prompt, model, or harness change actually helps.

One install ships both surfaces:

• The aegis CLI — a self-contained Go binary (embedded TypeScript eval compiler and web UI) that scaffolds, validates, runs, grades, and reports experiments.
• The @coder/aegis library — the TypeScript eval-authoring facade with bundled type declarations, so EVAL.ts files get IntelliSense and can be executed directly with node.

Installation

npm install -g @coder/aegis
aegis --help

Compiling EVAL.ts eval layouts requires Node.js >= 24 on PATH.

Quick start

# Scaffold an eval workspace with a runnable starter layout.
aegis init my-evals && cd my-evals

# Check that the starter compiles and is well-formed.
aegis validate evals/example-hello

# Configure the model upstream once (credentials go to the OS keyring) …
aegis setup

# … then smoke-run one live trial end to end.
aegis run evals/example-hello --provider exec --smoke --confirm-budget --allow-unsafe-host-graders

The starter drives the real claude-code harness through the exec provider, so the smoke run additionally needs the claude CLI on PATH (verify with claude --version) and an Anthropic credential — either via aegis setup above or an ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN environment variable.

Evals are authored in TypeScript — one EVAL.ts per experiment declaring the hypothesis, the agent under test, variants, graders, and a spend budget:

import { defineExperiment } from "@coder/aegis";

export default defineExperiment("my-experiment", (evals) => {
  // hypothesis, subject, variants, graders, budget …
});

Agent skills

The package ships its authoring guides as plain-markdown agent skills: read them from node_modules/@coder/aegis/skills/ or via aegis skills list and aegis skills get <name> (start with eval-authoring).

Learn more

• Repository and full documentation
• Eval authoring guide
• Issue tracker