@mutagent/evaluator

A generic, subject-agnostic AI-agent auditor — a reviewer, never the executor. It verifies that a skill or agent did what it should, repeatably, operator-out-of-loop, and emits a 4-tab master-audit report.

The evaluator ships zero subject-specific logic. Everything skill-specific lives in a generated subject profile under subjects/<name>/. The mutagent-diagnostics profile (132 criteria · 70 components) is simply the first one; the same agent profiles and audits any other subject the same way.

Install

pnpx @mutagent/evaluator --help

The three modes

| Mode | Command | What | |------|---------|------| | A — Audit Run | pnpx @mutagent/evaluator audit-run <runId> --subject <name> | Loads subjects/<name>/ + a run-bundle → the 4-tab report + a two-track scorecard. | | B — Profile Subject | pnpx @mutagent/evaluator profile-subject <def> <traces> --name <name> | GENERATES subjects/<name>/{eval-matrix,behavior-tree,methodology-review}.yaml + an interesting-dataset shortlist. This is how a matrix is born — generated, not embedded. | | C — Methodology Review | pnpx @mutagent/evaluator methodology-review <runId> --subject <name> | Process self-feedback — is the methodology the right/efficient choice + how to rearrange (advisory). | | coordinator | pnpx @mutagent/evaluator variance-check <bundleA> <bundleB> | Compare 2 variants → delta + 15-dim determinism trend. Executor ≠ reviewer. |

How it works

Two-track rollup (never merged)

• Track-1 GATE — binary, severity-gated. Component PASS iff 0 CRIT/HIGH fail; run PASS iff all components pass.
• Track-2 TREND — the 15-dimension determinism variance score, separate.

Deterministic vs pinned judge

The split is a pure function of each criterion's checkMethod:

• Deterministic (no model): deterministic-script · typebox-schema · gate.
• Pinned judge (model id + temperature=0, recorded; output masked): trace-cross-ref · trajectory-diff — reads the transcript against the subject's behavior-tree.yaml.

Byte-identity

The audit is itself deterministic. runId / timestamps / absolute paths are masked (scripts/mask.ts, a versioned set) so two audits on one bundle produce a byte-identical scorecard.

Package layout

mutagent-evaluator/
├─ .claude/skills/mutagent-evaluator/
│  ├─ SKILL.md                            # operational contract (eval-dev engine + *audit surface; standalone OR borrowed by skill-builder SIMULATE)
│  ├─ assets/agents/
│  │  ├─ evaluator.md                     # unified v2 cell — discover / judge-trajectory / judge-criterion modes (critique-before-verdict, C-PIN)
│  │  ├─ dataset-builder.md               # *build-dataset GENERATOR (EV-046) — tuples → NL queries → realism filter
│  │  └─ audit-executor.md                # v1 *audit surface executor (re-homed from the former .claude/agents/)
│  └─ .meta/design-principles.md          # the Constitution — block-organized Design Principles
├─ references/
│  ├─ error-analysis.md                   # *discover — 7-step error analysis → emergent criteria
│  ├─ write-judge-prompt.md               # *build-evals — 4-component judge, critique-before-verdict, binary
│  ├─ validate-evaluator.md               # *validate — TPR/TNR · Rogan-Gladen · bootstrap CI
│  └─ methodology.md                      # v1 determinism-control reference (under *audit; principles.md redirects → .meta)
├─ schemas/                               # the 4 shape contracts (YAML)
├─ lenses/                                # generic pinned-judge prompts
├─ workflows/                             # audit · data-leak · variance (self-contained)
├─ scripts/                               # the generic engine (TypeScript)
├─ subjects/mutagent-diagnostics/         # the FIRST subject profile (data, not code)
└─ assets/brand/                          # self-contained Mutagent brand (theme + wordmark)

Self-containment

The evaluator takes IN everything it needs: the ported audit + variance workflows, the distilled methodology, and the Mutagent brand asset all live inside this package. It references no SOURCE file under mutagent-diagnostics/ (or any other mutagent-system/* package) — the source stays sealed. It does, however, ACCEPT .mutagent/diagnostics/{runId}/ as a runtime INPUT bundle directory (the run-bundle data it audits) — that is data fed in at audit time, not a source reference.

License

Apache-2.0. See LICENSE + NOTICE.