Curiocity

An evals/testing harness that drives interactive coding-agent CLIs (Claude Code, Codex CLI) through a prompt over a real PTY, reads each CLI's native on-disk transcript as the source of truth, auto-answers the agent's genuine questions via LLM, then scores every run with deterministic checks + an LLM judge and gates CI on the aggregate.

• Curiocity (orchestrator) discovers cases, builds the agent × case × repeat matrix, forks one worker per cell, aggregates, reports, and picks the exit code.
• Curion (worker) owns exactly one trial in an isolated workspace and returns one verdict + metrics.

The full design lives in architecture.md. This README is how to run it. The demo/ folder is the living example.

Prerequisites

• Node ≥ 20.
• A C/C++ build toolchain — node-pty compiles a native addon at install (Xcode Command Line Tools on macOS, build-essential on Linux). Without it, install fails. CI images need build tools.
• The agent CLIs installed and authenticated by you. Curiocity never manages agent auth. Install claude and/or codex, sign in once, done.
• An unsandboxed process. A sandboxed harness blocks the agent from writing under $HOME, so no transcript ever appears. validate/run preflight agent-home writeability and fail fast. Run outside any sandbox.
• LLM API keys for the harness roles (question-answering + judging). See Models, effort & cost.

Install & run

Published to npm, so no clone needed:

npx -y curiocity@latest validate --source ./cases
npx -y curiocity@latest run --source ./cases

Local dev from this folder:

npm install           # builds node-pty
npm run dev -- validate --source demo/cases    # tsx, no build step
npm run build && node dist/cli.js run --source demo/cases

Quickstart (demo)

The demo runs two cases against Claude Code and Codex on the cheap model tier. It needs an authenticated claude/codex and an Anthropic key (see below).

# 1. Dry-run: see the resolved matrix + config, run nothing.
node dist/cli.js run --source demo/cases --config demo/curiocity.demo.json --dry-run

# 2. Discovery + preflight only: valid cases, skip reasons, P10 check.
node dist/cli.js validate --source demo/cases

# 3. Run the suite (evaluate + cost + stats ON by default).
node dist/cli.js run --source demo/cases --config demo/curiocity.demo.json

# 4. Re-score a stored run with new stats/thresholds — never re-runs agents.
node dist/cli.js report ./curiocity-results/run-<timestamp>

Commands

curiocity run --source <dir> [options]        # suite mode: evaluate/cost/stats ON
curiocity run --prompt <file|text> [options]  # inline mode: OFF by default (--evaluate to enable)
curiocity report <resultsDir> [--reporter json,markdown] [--config <file>]
curiocity validate --source <dir>             # discovery dry-run + P10 preflight

run options (all override config; every one usable in CI):

| Option | Meaning | |---|---| | --agent <id> (repeatable) | limit agents | | --case <glob> (repeatable) | limit cases by name | | --repeats <n> | override case repeats | | --concurrency <n> | pool size (1 = serial debugging) | | --timeout <sec> | per-trial wall-clock cap | | --evaluate / --no-evaluate | toggle evaluation (D9 default per mode) | | --collect-cost / --no-collect-cost | toggle cost collection | | --only-evaluator <id> / --skip-evaluator <id> | narrow the eval pipeline | | --mirror | stream PTY output live | | --keep-workspace | keep all workspaces (not just failed ones) | | --dry-run | print resolved matrix + config, run nothing | | --prompt/--qna/--eval/--src | inline-case inputs | | --fast-model / --workhorse-model / --judge-model <provider/model> | tier overrides | | --agent-model <id>=<model> / --agent-effort <id>=<effort> (repeatable) | per-agent model/effort | | --out <dir>, --config <file> | paths |

Authoring a case

--source <folder>: each immediate subfolder is one case. A subfolder is runnable only when all five files are present, else it is skipped with a logged reason (validate lists both).

| File | Role | |---|---| | prompt.md | The task, passed to the agent as its launch argument. | | config.json | Agents, timeout, repeats, evaluators, provisioning, setup/teardown. | | qna.md | Policy for answering the agent's genuine questions — approvals, denials, and a hard "if unsure, abort". Consumed only by the answering LLM. | | evaluation.md | Prose rubric for the LLM judge, passed verbatim. Deterministic checks do NOT live here. | | src.zip | Source archive unzipped into the workspace before launch. "From scratch" tasks ship a minimal/empty zip. |

Inline case: --prompt <file|text> (+ optional --qna, --eval, --src, --agent) builds the same case in memory. Missing pieces get neutral defaults; evaluation is off unless --evaluate + --eval.

Evaluators

Declared as evaluators entries in config.json; use names a built-in. gate:true makes a failure fail the trial; weight sets its share of the scored mean.

An evaluator that throws (e.g. the judge's model key returns insufficient_quota, or an external command exits non-zero) is an infra failure, not a low score: its record is flagged error:true, the combiner verdict is discarded, and the whole trial gets status eval-error — excluded from score statistics like the other error statuses and surfaced as exit code 3.

• file-exists — globs that must / must-not exist in the final workspace.

• command — run a build/test/lint string via shell; assert the exit code.

• trajectory-check — assert tool_call events matched a pattern (the "did our plugin actually run" gate). toolPattern is one regex or a per-agent map.

• hook-transcript-check — proves a Claude Code plugin's own hooks fired, distinct from Curiocity's always-present capture hooks. Version-agnostic: derives the expected hook events dynamically from whatever the live, checked-in plugins/core-claude/hooks/hooks.json declares — today just SessionStart, so the check verifies SessionStart fired the plugin's own command; if the plugin later declares PreToolUse/PostToolUse/etc., the check adapts automatically, no doc or code change needed. It then scans the run's raw transcript for system entries whose subtype ends _hook_summary and confirms each expected event both fired and ran the plugin's own command (matched by .js script basename, or a hookSpecificOutput token for printf-style SessionStart hooks). Params: pluginManifest, ignoreEvents (default ["PostCompact"]), requireCommands (default true); emits hook_events_{declared,checked,fired,plugin_matched} metrics. The event→subtype mapping itself is empirically confirmed only for Stop → stop_hook_summary; other event names fall back to the same snake_case + _hook_summary pattern — which is why it fails loud, rather than silently passing, if the transcript has zero recognizable *_hook_summary entries.

• llm-judge — judge model scores 0–100 from evaluation.md + distilled trajectory + produced artifacts (artifacts globs, size-capped) + QnA log. The result also carries confidenceLevel (0–100, self-reported — the judge's own estimate of how solid its verdict is, required in its output schema) and perplexityLevel (0–100, measured from token logprobs over the generated output when the provider exposes them; absent otherwise, e.g. Anthropic models — warned once per model per run, never an error).

• external — run any program that reads a JSON object (file paths, not blobs) on stdin and prints this object on stdout:

  {"values": [{"name": "<metric>", "value": <0-100>, "confidenceLevel": <0-100>, "perplexityLevel": <0-100>}]}

  Per value entry, name and value are required; confidenceLevel and perplexityLevel are optional. Every number is 0–100. Each value is recorded as a named metric on the trial and rolled up per metric name. scoreMetric/passThreshold optionally turn one metric into the pass/score; otherwise the metrics are informational. A non-zero exit, invalid JSON, a timeout, or any out-of-range number makes the evaluator throw — an infra error flagged error:true that turns the trial into eval-error (exit code 3), not a clean gate-aware fail.

Example (from demo/cases/healthcheck/config.json):

{
  "agents": ["claude-code", "codex"],
  "timeoutSec": 600,
  "repeats": 1,
  "evaluators": [
    { "use": "file-exists", "must": ["HEALTHCHECK.md", "**/HealthController.java"], "gate": true },
    { "use": "llm-judge", "rubric": "evaluation.md", "artifacts": ["**/*.md"], "weight": 1.0 },
    { "use": "external", "command": "node", "args": ["count-changed-files.mjs"], "timeoutSec": 30 }
  ],
  "combiner": "gated-mean"
}

The default combiner gated-mean: every gate:true result must pass (else the trial fails with score capped), then the verdict score is the weighted mean of scored results vs passThreshold (default 60).

Configuration

Precedence, lowest to highest: built-in defaults < top-level config (--config, default ./curiocity.config.json) < case config.json < CLI flags. Provisioning merges by name (same name overrides, new adds). Setup/teardown arrays concatenate (top-level first).

Top-level config sketch:

{
  "codingagents": { "claude-code": { /* profile */ }, "codex": { /* profile */ } },
  "models":  { "fast": "anthropic/claude-haiku-4-5", "workhorse": "anthropic/claude-sonnet-5" },
  "pricing": {                                      // optional; enables $ in cost-rollup
    "anthropic/claude-haiku-4-5":  { "inputPer1M": 1.0, "outputPer1M": 5.0 },
    "anthropic/claude-sonnet-5": { "inputPer1M": 3.0, "outputPer1M": 15.0 }
  },
  "provision": { "mcps": [], "plugins": [] },
  "setup": [], "teardown": [],
  "gate": { "minScore": 60, "minPassRate": 0.8, "maxStddev": 10 },
  "concurrency": 4,
  "out": "./curiocity-results"
}

Plugins. A provision.plugins entry is { "name": "...", "path": "<dir-or-.zip>" }; a relative path resolves against the case directory (a plugin without a path is a config error). Only claude-code provisions plugins today — rendered as a session-scoped claude --plugin-dir <path> flag per plugin, so ~/.claude is never touched; codex doesn't support plugins yet. MCPs are unchanged (the claude-code adapter still renders them into a workspace .mcp.json; codex renders MCPs its own way, via -c overrides).

Setup/teardown and the command evaluator run user-authored shell lines (shell:true) — they take shell syntax by design and are trusted at the case-authoring level. The external evaluator invokes a program with an explicit argv (no shell). No agent output is ever interpolated into a shell command.

Models, effort & cost

Three harness roles map to "provider/model" strings: fast (high-frequency classification), workhorse (question replies), judge (defaults to workhorse). Providers: anthropic, openai (add one = dependency + a line in llm/providers.ts).

Canonical current Anthropic model ids (use these exactly; don't invent variants): Haiku 4.5 = claude-haiku-4-5, Sonnet 5 = claude-sonnet-5, Opus 4.8 = claude-opus-4-8. Curiocity's harness-role strings take an anthropic/ provider prefix (e.g. anthropic/claude-sonnet-5); the agent's own agentModel field takes the bare id (e.g. claude-sonnet-5).

The agent's own model/effort is a separate dimension from the harness roles: set agentModel/agentEffort on the profile, per-case, or via --agent-model/--agent-effort.

Cost policy — track and warn, never abort. Aborting mid-suite contaminates benchmark results.

• Cheap tier is the default for the bulk of testing: Claude Code on Sonnet at low reasoning effort (supports auto permission mode; Haiku does not), Codex on gpt-5.4-mini. Reserve smarter/full-effort models for final validation.
• Token counts are always reported. Dollar amounts come only from the config pricing map; unpriced models report tokens-only with a warning.
• An optional budgetUsd over-budget warns once and keeps going.

Keys. Resolved once at startup, held in memory, shipped to workers over IPC, masked in logs, never written to disk. Per provider, precedence is CURIOCITY_<PROVIDER>_KEY then the provider-standard var (e.g. ANTHROPIC_API_KEY), checked first in the environment, then in a .env file in the current working directory. A provider with no key is fine unless a role actually needs it.

Env forwarded to the agent. Curiocity forwards the child/agent environment on a deny-list basis: the full parent env passes through (so toolchains like JAVA_HOME/Maven work as they would in a normal shell), and only vars matching ANTHROPIC_*/OPENAI_* (case-insensitive) are stripped — except exactly two provider keys, ANTHROPIC_API_KEY and OPENAI_API_KEY, which are forwarded straight through to the agent CLI (via filterAgentEnv), irrespective of adapter or envRemove config: in CI there's no interactive OAuth session, so the harness must forward the key for the agent to authenticate, and the same key also feeds the harness's own judge/QnA LLM calls. The exemption is exact-match only, so a near-miss name like MY_ANTHROPIC_API_KEY doesn't match the strip prefix and simply passes through like any other env var. NOTE: other secrets already in the environment (AWS_*, GITHUB_TOKEN, …) are intentionally passed through too — a deliberate trade-off so the harness behaves like a normal shell/CI step, not a secret vault.

Base URLs. Optional provider base URLs use the same source precedence and are validated as http:// or https:// before any trial starts. Per provider, precedence is CURIOCITY_<PROVIDER>_BASE_URL, then <PROVIDER>_BASE_URL, then CURIOCITY_BASE_URL. CURIOCITY_BASE_URL intentionally fans out to every provider for multi-provider gateways like Bifrost; use provider-specific vars when providers need different origins or route prefixes.

Stats

Computed per (case × agent) group and suite-wide, from stored trial JSON (so report applies new stats retroactively):

• score-stats — min/max/mean/median/stddev of scores (repeats > 1).
• pass-rate — passed / (passed + failed); error-status trials excluded.
• stability — stable-pass / flaky / stable-fail per group.
• cost-rollup — tokens (+$ if priced) itemized per model × per source (agent vs harness fast/workhorse/judge). Only the $ total is additive across models.
• time-rollup — measured decomposition: total wall + per-phase walls, plus agentPureMs (agent execution, excluding all harness reaction) vs harnessReactMs (LLM time per model vs overhead), and turn metrics (turnsTotal, questionTurns, interruptions).

Exit codes

| Code | Meaning | |---|---| | 0 | all groups pass all gates; no error-status trials | | 1 | gate failure (score / pass-rate / flakiness) | | 2 | config error or total infra failure (invalid config, no runnable trials, preflight failed) | | 3 | partial infra failure: some trials ended in an error status, but every gate on completed trials passes |

Gate failure takes precedence over partial-infra (both → 1). The gate is evaluated per (case × agent) group: minScore vs mean, minPassRate vs pass-rate, maxStddev vs stddev (repeats > 1 only). Any violating group fails the suite.

Results

Each run writes a timestamped dir under --out:

run-<timestamp>/
  suite.json   suite.md            # aggregate: config snapshot, matrix, stats, gate
  trials/<case>/<agent>/<repeat>/
    trial.json trajectory.jsonl raw-transcript.jsonl screen.log workspace.diff

report <dir> recomputes stats + reporters + gate from the stored TrialResults. It never re-runs agents, evaluators, or the judge.

Development

npm run dev        # tsx CLI, no build
npm run build      # tsup → dist/
npm run lint       # tsc --noEmit (strict)
npm test           # vitest: unit + integration
npm run smoke      # integration only: full fork+PTY loop against the mock agent, zero tokens

Contract tests against your locally installed CLIs: npm run contract:claude, npm run contract:codex. Live E2E is manual/nightly and needs authenticated CLIs.

E2E suite (tests/e2e-tests/): two live cases — coding-vanilla (plain Claude Code, control) and coding-rosetta (Claude Code + the live, checked-in plugins/core-claude plugin, referenced via a relative ../../../plugins/core-claude path and loaded via --plugin-dir) — both run the same health-endpoint coding task through the full gated lifecycle; coding-rosetta additionally gates on hook-transcript-check. Each case's src.zip is the maintainer-provided fixture archive copied verbatim from test-library/: coding-vanilla uses spring-boot-react-mysql-not-initialized.zip, coding-rosetta uses spring-boot-react-mysql-initialized.zip (a Rosetta-pre-initialized workspace) — neither archive is reprocessed. Both run against the base config tests/e2e-tests/curiocity.e2e.json. tests/e2e-tests/build-src-zip.mjs is a manual-only utility for deriving a fresh fixture locally; it is not used by the workflow or any automated path. The manual .github/workflows/e2e-testing.yml workflow (workflow_dispatch) verifies the live plugin is present, stages the fixtures, runs Curiocity, and uploads all evidence.

Known npm audit findings

npm audit reports 5 vulnerabilities (1 critical, 1 high, 3 moderate), all in the vitest → vite → esbuild / @vitest/mocker / vite-node chain:

• All are dev-only (test runner). They never ship — the published package is dist/ only (see files in package.json).
• All describe the vite/vitest dev server or UI server, which this project never runs; CI uses vitest run.
• The only fix is a major vitest bump (vitest@4), which risks the test gates. Deferred to a dedicated upgrade rather than folded into a release. Re-audit after any vitest upgrade.