@accelerate-data/promptfoo-eval-harness

Shared Promptfoo + OpenCode eval harness. Owns model/tier policy, provider wiring, package discovery, Promptfoo state export, and artifact guards. Consumers own eval YAML, prompts, fixtures, and assertions.

Providers

All providers route through the single Promptfoo bridge file://scripts/framework/_node_bridge.js. Set provider_kind in the provider config: block to select a provider.

The shipped tier config (config/eval-tiers.toml) defaults every tier to openhands_sdk. Provider selection is centralized there: the framework overrides any package-local providers block with the tier mapping, so switch providers by editing the tier in your own eval-tiers.toml, not a package.

| provider_kind | Status | Model alias examples | Notes | | --- | --- | --- | --- | | opencode_cli | stable | opencode-mock, opencode-anthropic | In-process; requires opencode binary on PATH. | | openhands_sdk | stable | mock/openhands-mock, openhands/anthropic-claude-3-5-sonnet | Subprocess via uv run --with openhands-sdk==1.22.1; supports multi-turn. | | claude_agent_sdk | stable | claude-sonnet-4-6, claude-opus-4-7, claude-haiku-4-5 (aliases: opus, sonnet, haiku) | Subprocess via uv run --with claude-agent-sdk==0.2.85; async lifecycle; multi-turn via stateful ClaudeSDKClient; default tools Read/Write/Edit/Glob/Grep (Bash & web gated via permissions.allow_shell / allow_web). | | opencode_sdk | stable | anthropic/claude-sonnet-4-6, openai/gpt-4o (any model the OpenCode server accepts) | In-proc Node provider via @opencode-ai/sdk; boots an ephemeral 127.0.0.1:0 OpenCode server per run, dispatches turns through one client.session.prompt per case; agents limited to {build, plan, general}; requires Node ≥ 20. | | codex_sdk | stable | gpt-4o, gpt-4.1 (any model the Codex SDK accepts) | In-proc Node provider via @openai/codex-sdk (CJS); reserves a per-session HOME and a per-case git init-ed workspace before invoking Codex.startThread; multi-turn via a single Thread; defaults sandbox_mode=workspace-write and model_reasoning_effort=medium (overridable via extra); requires Node ≥ 20. |

Plugin Glue Providers

Plugin glue providers wrap or augment the bridge-routed kinds above with runtime metadata, auto-reply gates, plugin discovery, and run telemetry. They are referenced from tier scenarios via the framework:// URL scheme and consume the new optional [runtime] fields documented in docs/setup.md.

| Wrapper | Wraps | Reads (optional [runtime] fields) | | --- | --- | --- | | framework://codex-sdk-provider.js | codex_sdk (via bridge) | bootstrap_prompt, agent_id, agent_entrypoint_file, opencode_config, model, project_dir | | framework://claude-agent-sdk-provider.js | claude_agent_sdk_node (Node-side, NOT the Python claude_agent_sdk bridge) | bootstrap_prompt, auto_reply_text, max_auto_replies, idle_turn_stop, plugin_subdirs, agent_id, agent_entrypoint_file, opencode_config, model, empty_output_retries | | framework://opencode-cli-plugin-provider.js | opencode_cli (sibling of framework://opencode-cli-provider.js) | bootstrap_prompt, opencode_plugin_link_path, capture_on_failure, write_run_metadata, load_local_env, opencode_parser_module, opencode_runner_command, agent_id, agent_entrypoint_file |

The base framework://opencode-cli-provider.js (the locked §7.4 contract) is byte-identical and stays the default for non-plugin scenarios; the plugin sibling composes the base without subclassing it. The Claude wrapper introduces a second transport (claude_agent_sdk_node) alongside the existing Python-bridge claude_agent_sdk kind — see docs/design.md for when to pick which.

Scenarios

Framework-owned mock-mode scenarios under tests/harness-scenarios/packages/. All run without live API keys and are exercised by the nightly CI workflow on main.

| Scenario | provider_kind | Description | | --- | --- | --- | | minimal-smoke | opencode_cli | Single-turn smoke test that exercises the bridge end-to-end. | | opencode-cli-compatibility | opencode_cli | Layer 4 regression locking all five §7.4 opencode_cli behaviors; 3 cases. | | openhands-mock-multi-turn | openhands_sdk | 3-turn conversation via mock SDK; validates NDJSON IPC multi-turn path. | | claude-mock-multi-turn | claude_agent_sdk | Multi-turn through the Python claude_agent_sdk kind with deterministic mock responses. | | codex-sdk-mock-multi-turn | codex_sdk | Multi-turn through the in-proc codex_sdk kind. | | opencode-sdk-mock-multi-turn | opencode_sdk | Multi-turn through the in-proc opencode_sdk kind (ephemeral server). |

See Setup Guide → Scenarios for the vars.turns multi-turn pattern and consumer-package authoring rules.

Quick Start

Bootstrap a new repo:

npx --package @accelerate-data/promptfoo-eval-harness eval-harness-init

This scaffolds tests/evals/ with package.json, opencode.json, config/eval-tiers.toml, and a harness-smoke package, installs dependencies, and adds a Dependabot entry to .github/dependabot.yml so the repo receives PRs when a new version is released.

Smoke-test the install immediately (no live API key needed):

OPENCODE_MOCK_MODE=1 npx ad-evals run tests/harness-scenarios/packages/minimal-smoke

Verify the full install:

cd tests/evals
npm test                     # contract tests
npm run doctor               # print resolved paths
npm run eval:harness-smoke   # one live execution
npm run eval:smoke           # smoke across all packages

Dependencies are installed automatically on the first ad-evals run and re-installed whenever package-lock.json changes.

Migration from v0

v0 tier configs (no provider_kind field) are accepted at runtime but deprecated. Run the in-place migration to avoid a v1.1.0 breaking change:

npx --package @accelerate-data/promptfoo-eval-harness eval-harness-init --upgrade --migrate-from-v0

Full guide: docs/migration-v0-to-v1.md

Usage

# Run the smoke filter across all packages
npm run eval:smoke

# Run all tests in all packages
npm run eval:regression

# Run one package
ad-evals run packages/my-feature/promptfooconfig.json

# Open the Promptfoo UI
npm run view

# Print resolved state paths
npm run doctor

Documentation

| Doc | What it covers | | --- | --- | | Setup Guide | Bootstrap, verify, write a package, run evals, wire CI; also covers scenarios, provider key matrix, parallelism, multi-turn evals, LLM judge & custom assertions, workspace isolation & git ops — give this to a coding agent | | Design | Framework architecture and ownership boundary | | Changelog | Release notes per version |

What the Framework Owns

• CLI entrypoint (ad-evals)
• Bootstrap (eval-harness-init)
• Path resolution across worktrees
• Promptfoo and OpenCode environment export
• Package discovery rules
• Provider wiring
• Resolved config materialization
• Artifact cleanup guard
• Default tier → agent mapping

What Your Repo Owns

• opencode.json agent definitions (model, steps, permissions)
• Package configs under packages/<name>/
• Prompts, fixtures, vars
• Domain assertions
• Scenario inventory and per-package documentation

Contributing

See CONTRIBUTING.md.