Pathgrade

Evaluate AI coding agents with Vitest or Jest. Write evals as normal .eval.ts files, run Claude Code, Codex, Cursor, or trusted-internal OpenCode fixtures in isolated workspaces, and score the result with deterministic checks, rubric judges, and tool-usage assertions.

Why Pathgrade?

• Write evals in plain TypeScript with Vitest or Jest
• Run each trial in an isolated workspace and HOME directory
• Seed trials from fixtures, real skills, or mocked MCP servers
• Score both final artifacts and the workflow that produced them
• Debug long conversations with preserved workspaces and run snapshots
• Use the same evals locally and in CI

Quick Start

Prerequisites: Node.js 20.11+, Vitest 4+ or Jest 30+, and at least one configured agent runtime. Claude uses the bundled @anthropic-ai/claude-agent-sdk binary by default; Codex requires the codex CLI; Cursor requires the cursor-agent CLI. OpenCode requires the exact Wix-registry v1.18.14 macOS ARM64, Linux ARM64, or Linux x64 executable.

yarn add -D @wix/pathgrade

For Jest projects, install Jest too:

yarn add -D @wix/pathgrade jest

Authentication

By default, Pathgrade tries to reuse the agent CLI's native auth before falling back to explicit environment variables.

• Claude
  • macOS: reuses Claude Code OAuth from Keychain
  • other platforms: forwards ANTHROPIC_API_KEY when present
• Codex
  • forwards OPENAI_API_KEY when present
  • or runs codex login --with-api-key inside the sandbox when an API key is present
  • without API credentials, the default app-server transport reuses the active Codex login through short-lived auth-only stdio sessions; codex exec retains its cached ~/.codex/auth.json behavior
  • keyless Codex evals automatically use that managed login for plain and tool-using judges; an explicit evaluate(..., { llm }) still wins
• Cursor
  • forwards CURSOR_API_KEY when set
  • macOS: reuses cursor-agent login OAuth tokens from the login Keychain
  • surfaces a clear error when neither is available (run cursor-agent login or set CURSOR_API_KEY)
• OpenCode
  • defaults to anthropic/claude-sonnet-5 with an explicit or host ANTHROPIC_API_KEY and optional HTTPS /v1 proxy
  • supports model: 'openai/gpt-5.4' with an explicit or host OPENAI_API_KEY
  • when no OpenAI key or proxy is set, stages the validated OpenAI access record with a disabled refresh sentinel; it never copies the host refresh token or writes credentials back

If you set ANTHROPIC_BASE_URL, OPENAI_BASE_URL, or CURSOR_API_BASE_URL, set the matching API key too.

An experimental ChatGPT judge that reuses Codex-managed authentication is also available. It requires no API key, auth-file option, or second login and is selected automatically for keyless Codex evals.

Do not add copyFromHome: ['.codex'] just to reuse Codex authentication. Keyless app-server runs stage no credential file; the legacy exec transport stages only ~/.codex/auth.json. A deliberate full .codex copy remains supported, but it also imports host config, MCP definitions, plugins, caches, sessions, and other potentially large or sensitive state; Pathgrade prints a warning when requested.

Override credentials per test with env:

const agent = await createAgent({
    agent: 'claude',
    env: {
        ANTHROPIC_API_KEY: process.env.MY_ANTHROPIC_KEY!,
    },
});

Transport (Codex only)

Codex supports two transports and Pathgrade defaults to app-server:

• app-server (default) — uses codex app-server and keeps native thread state. Required for AskUserReaction handshakes (request_user_input reaches the model). Without API credentials it acquires and refreshes access sessions through the active Codex login.
• exec — uses codex exec and re-injects the transcript every turn. Kept for stateless CI matrices that don't need the handshake.

Precedence: createAgent({ transport }) > PATHGRADE_CODEX_TRANSPORT env > default (app-server). An invalid env value throws at createAgent time.

const agent = await createAgent({
    agent: 'codex',
    transport: 'exec', // opt out of app-server
});

If transport: 'exec' is resolved and any AskUserReaction is present in ConverseOptions.reactions, the conversation fails fast before turn 1 — the handshake cannot fire under exec. Set allowUnreachableReactions: true on runConversation to silence the guard.

Migrating from exec to app-server:

• Sign in with codex login for subscription-backed app-server and judge calls, export OPENAI_API_KEY for API billing, or select transport: 'exec' for the old cached-auth behavior.
• The noninteractive-user-question runtime policy no longer attaches under app-server. Snapshots that captured model output influenced by that policy text may need re-recording.
• MAX_TURN_RETRIES does not apply under app-server — a crashed turn ends the conversation with completionReason: 'agent_crashed'.

Plugin Setup

Create a vitest.config.ts with the built-in Vitest adapter plugin:

import { defineConfig } from 'vitest/config';
import { pathgrade } from '@wix/pathgrade/adapters/vitest';

export default defineConfig({
    plugins: [pathgrade({ timeout: 120 })],
});

The plugin registers the setup hooks Pathgrade needs, wires in the reporter, and automatically cleans up agent workspaces after each test.

For Pathgrade CLI behavior, use pathgrade.config.ts:

export default {
    runner: {
        adapter: 'vitest', // default; use 'jest' for Jest projects
        args: [],
    },
    evals: {
        include: ['**/*.eval.ts'],
        exclude: ['**/fixtures/**'],
    },
    affected: {
        global: ['package.json', 'yarn.lock'],
    },
};

First Eval

Write an eval file such as hello.eval.ts:

import * as fs from 'fs';
import * as path from 'path';
import { describe, it, expect } from 'vitest';
import { createAgent, check, evaluate } from '@wix/pathgrade';

describe('hello world', () => {
    it('agent creates the requested file', async () => {
        const agent = await createAgent({
            agent: 'claude',
            workspace: path.join(__dirname, 'fixtures'),
        });

        await agent.prompt('Create a file called hello.txt with the text "Hello, world!"');

        const result = await evaluate(agent, [
            check('hello.txt exists', ({ workspace }) =>
                fs.existsSync(path.join(workspace, 'hello.txt'))),
        ]);

        expect(result.score).toBe(1);
    });
});

Run your evals:

npx pathgrade run

pathgrade run is the recommended wrapper: it loads .env, warns when no auth is configured, and adds Pathgrade-specific flags such as --changed, --diagnostics, and --verbose. Plain npx vitest run or direct Jest runs work too if you configure the Pathgrade adapter hooks yourself.

Core Concepts

• Agent: the coding agent under test, such as Claude, Codex, or Cursor
• Workspace: an isolated directory where the agent works, optionally seeded from fixtures
• Scorer: a function or judge that evaluates output or behavior
• Evaluation: the aggregated result of one or more scorers, returned as a score from 0.0 to 1.0

For protocol-neutral multi-agent graphs, see Evaluating agent flows.

Scorers

Scorers evaluate the agent's output and behavior. evaluate() runs all scorers and computes a weighted average between 0.0 and 1.0.

Use check() for binary requirements, score() for partial credit, judge() for rubric-based evaluation, and toolUsage() when the workflow matters as much as the final output.

check() - Boolean gate

check('tests-pass', async ({ runCommand }) => {
    const { exitCode } = await runCommand('npm test');
    return exitCode === 0;
}, { revision: { contract: 1 } });

score() - Partial credit

score('coverage', async ({ runCommand }) => {
    const { stdout } = await runCommand('npx coverage-summary');
    return parseFloat(stdout) / 100;
}, { revision: { contract: 1 } });

judge() - Rubric evaluation

judge('workflow-quality', {
    rubric: `Did the agent read the file before editing? (0-0.5)
Was the fix minimal and correct? (0-0.5)`,
});

Function-bearing check() and score() scorers require explicit canonical-JSON revision data for authoritative baseline comparisons. A judge() with a function-valued input requires the same declaration. Imported scorer helpers belong in comparisonInputs; closed-over values belong in revision. Judge scorers also support:

• retry for transient judge failures
• includeToolEvents when the rubric should see the tool trace
• input for extra context such as generated file contents or command output
• tools to let the judge LLM read workspace artifacts itself via a bounded tool-use loop (readFile, listDir, grep, getToolEvents)

Example with artifact-backed input:

judge('output-quality', {
    rubric: 'Is the generated markdown correct and complete?',
    includeToolEvents: true,
    input: async ({ artifacts }) => ({
        'output.md': await artifacts.read('output.md'),
    }),
});

Example with tool-using judge (the judge reads the file itself — no pre-computed probe):

judge('spec-structure', {
    rubric: `Read artifacts/spec.md. Score 1.0 if it contains Intent Hierarchy,
             Functional Requirements, and API Surface sections. 0.33 per section.`,
    tools: ['readFile'],
});

Tool-using judges currently require the Anthropic HTTP provider (ANTHROPIC_API_KEY); other providers produce a clean provider_not_supported error. See the User Guide for the full tool list, failure codes, and the migration recipe from input-helper probes.

toolUsage() - Tool event matching

toolUsage('expected-workflow', [
    { action: 'read_file', min: 1, weight: 0.3 },
    { action: 'edit_file', min: 1, weight: 0.3 },
    { action: 'run_shell', commandContains: 'test', min: 1, weight: 0.4 },
]);

Conversations

Pathgrade supports claude, codex, cursor, and opencode. Set the backend per test via createAgent({ agent: 'claude' }), or omit agent and use PATHGRADE_AGENT as the process-wide fallback.

OpenCode trusted-internal backend

OpenCode v1 is only for fixtures, prompts, skills, and generated MCP mocks controlled by trusted Pathgrade maintainers. It is not a security sandbox: OpenCode and Bash children run with the host user's filesystem, process, and network authority and receive the provider credentials. Do not use it for external repositories, user-supplied prompts, or adversarial fixtures.

const agent = await createAgent({
    agent: 'opencode',
    opencodeExecutable: '/absolute/path/to/opencode-v1.18.14',
});

The backend supports anthropic/claude-sonnet-5 (default), anthropic/claude-sonnet-4-6, and openai/gpt-5.4, generated stdio mcpMock servers, and clean hosts without system-managed OpenCode configuration. TypeScript callers must select agent: 'opencode' explicitly when supplying its executable or configuration. Selecting GPT-5.4 opts into the ChatGPT Codex endpoint/account plane. A custom OPENAI_BASE_URL always requires a matching key and never receives the local OAuth bearer token. Explicit API keys use only an explicitly paired base URL; they never inherit an ambient host proxy.

Pathgrade's default persona, summarization, and plain-judge helpers follow the selected OpenCode provider. GPT-5.4 API-key runs use the OpenAI helper provider and never fall through to Claude CLI. Local OAuth authenticates only the isolated OpenCode process; OAuth-only runs must inject an LLM for helper calls or provide an explicit OPENAI_API_KEY. Tool-using judges still require the Anthropic HTTP provider. See typed harness configuration for production profiles and shared execution directories.

agent.prompt() - One shot

Send a single instruction and let the agent work to completion:

const agent = await createAgent({ agent: 'claude', workspace: 'fixtures' });
await agent.prompt('Create a file called hello.txt with the text "Hello, world!"');

startChat() - Imperative

Drive the conversation yourself:

const chat = await agent.startChat('Set up a new TypeScript project.');
await chat.reply('Use strict mode and add eslint.');
if (await chat.hasFile('tsconfig.json')) {
    await chat.reply('Now add a build script.');
}
chat.end();

runConversation() - Scripted or persona-driven

Drive the loop with reactions:

const result = await agent.runConversation({
    firstMessage: 'I want to create a new feature.',
    maxTurns: 12,
    until: async ({ hasFile }) => await hasFile('project-brief.md'),
    reactions: [
        { when: /goal/i, reply: 'Solve a user pain point' },
        { when: /audience/i, reply: 'Self-Creator' },
    ],
});

Or let a persona answer on the user's behalf:

const result = await agent.runConversation({
    firstMessage: 'I want to create a new feature.',
    maxTurns: 12,
    until: async ({ hasFile }) => await hasFile('project-brief.md'),
    persona: {
        description: 'A product manager who communicates concisely.',
        facts: ['The feature is for online stores'],
    },
});

runConversation() also supports stepScorers, so long conversations can be graded at intermediate milestones instead of only at the end.

Advanced SDK Features

Pathgrade exposes a few useful features that are easy to miss from the basic examples:

• createAgent({ skillDir, workspace }) stages a real skill and a fixture workspace into the sandbox, which is how Pathgrade's skill examples are evaluated.
• createAgent({ debug: true }) preserves the final workspace under the backward-compatible pathgrade-debug/<test-name>/ path; when you use runConversation(), it also writes run-snapshot.json.
• createAgent({ debug: { retainRuns: 5 } }) opts into managed run retention under pathgrade-debug/runs/<run-id>/<test-name>/. Use pathgrade clean --debug, --keep=N, and --dry-run to clean marked, inactive debug runs safely.
• evaluate.fromSnapshot(snapshotPath, scorers) re-runs grading against a saved snapshot without re-running the agent. For trusted live deterministic scoring, evaluate(agent, scorers, { deterministicToolEvidence: 'live' }) exposes exact transient MCP arguments only through ctx.toolEvents; agent.log stays sanitized, and redaction-blocked argument predicates fail closed.
• previewReactions(messages, reactions) lets you inspect which scripted reactions would fire offline.
• conversationWindow on agents and personas keeps long transcripts bounded with summarization instead of sending the full conversation every turn.
• copyIgnore and DEFAULT_COPY_IGNORE let you control what gets copied into the sandbox when seeding from large fixtures or skill directories.

See sdk-showcase for a single example suite that demonstrates these APIs together.

MCP Mock Servers

Simulate MCP tools when testing Claude, Codex app-server, or Cursor evals:

import { mockMcpServer } from '@wix/pathgrade/mcp-mock';

const mock = mockMcpServer({
    name: 'weather',
    tools: [{
        name: 'get_weather',
        description: 'Get weather for a city',
        annotations: { readOnlyHint: true },
        when: 'weather',
        response: { temp: 72, unit: 'F' },
    }],
});

const agent = await createAgent({ agent: 'claude', mcpMock: mock });

Read-only metadata is explicit: add annotations.readOnlyHint: true only when the operation being modeled is read-only. Unannotated and explicitly mutating mock tools retain the runtime's normal approval behavior.

To evaluate through the receipt-backed mock host with Claude or Codex app-server, pass an ordered mcpMockApprovalRules array. Use an empty array for read-only-only mocks. Mutation scenarios require explicit rules; the first matching rule wins:

const agent = await createAgent({
  agent: 'codex',
  transport: 'app-server',
  mcpMock: mock,
  mcpMockApprovalRules: [
    { serverName: 'sessions', toolName: 'remove_session',
      argumentsContaining: { mode: 'delete' }, decision: 'deny' },
    { serverName: 'sessions', toolName: 'remove_session',
      argumentsContaining: { mode: 'archive' }, decision: 'approve' },
  ],
});

Use getMcpApproval() for the simulated-user decision and wasMcpToolInvoked() for receipt-backed execution (true, false, or undefined). An mcp_tool_call event by itself proves only that an attempt was observed. Mock responses are fixed fixtures; this mode does not model stateful mutation or make later reads observe earlier writes.

Direct MCP scenario machine v2

ScenarioMachineV2 is the versioned stateful MCP contract. Its parent-owned host controls matching, transitions, cancellation, evidence, and replay. Cases are unordered, disjoint JSON constraints; callbacks and regexes are rejected. Concurrent calls involving a transition-capable tool poison the scenario. This safety behavior is a fixed V2 invariant and is not author-configurable.

import {
  compileScenario,
  getFinalScenarioState,
  getScenarioStateTimeline,
  wasScenarioCaseMatched,
  type ScenarioMachineV2,
} from '@wix/pathgrade';

declare const scenario: ScenarioMachineV2;
const compiled = compileScenario(scenario);
if (!compiled.ok) throw new Error(JSON.stringify(compiled.diagnostics));
declare const evidence: Parameters<typeof getFinalScenarioState>[0];
wasScenarioCaseMatched(evidence, { caseId: 'CreateDocument' });
getScenarioStateTimeline(evidence);
getFinalScenarioState(evidence);

Evidence helpers validate the hash chain and reject empty, corrupt, or cross-trial evidence. Provider lifecycle events are corroboration only.

Legacy migration window

Existing mcpMock descriptors remain supported with no scheduled removal. They retain ordered regexes, fallback behavior, and receipts. V2 migration is explicit; receipt settlement has no v2 authority or new features. Removing the legacy executor would require a new owner-approved ADR and a major release.

The public live author path requires an explicit agent. It supports Claude, Codex app-server, OpenCode ACP, and Cursor ACP when the checked-in exact runtime lock for the current provider, platform, architecture, and version is admitted. Codex exec is unsupported. OpenCode requires its existing absolute opencodeExecutable option and uses the default API-key model for Scenario V2; provider-specific scenario flags and profile controls are not public.

CLI

pathgrade run [--changed] [--since=<ref>] [--changed-files=<path>] [--adapter=<name|path>] [--attempts=N] [--diagnostics] [--verbose] [--quiet] [-- runner-args]
pathgrade init [--force]
pathgrade validate <file.eval.ts>
pathgrade validate --affected
pathgrade analyze [--skill=<name>] [--dir=<path>]
pathgrade affected [--since=<ref>] [--changed-files=<path>] [--explain] [--json]
pathgrade preview [browser] [--last=N] [--filter=text]
pathgrade preview-reactions --snapshot <run-snapshot.json> --reactions <file.ts>
pathgrade report [--results-path=<path>] [--no-comment] [--comment-id=<id>] [--details-url=<url>] [--expected-head-sha=<sha>] [--strict]

Useful details:

• pathgrade run --changed computes affected evals first, writes selection metadata to .pathgrade/selection.json, and only then launches the selected runner adapter. Vitest is the default adapter; Jest is selected with runner.adapter: 'jest' or --adapter=jest.
• pathgrade run --attempts=3 reruns every selected case in three sequential, fresh runner processes. The CLI value overrides attempts in config; arguments after -- still belong to the runner.
• pathgrade preview browser starts a local viewer on http://localhost:3847.
• pathgrade report posts or updates a PR comment in GitHub Actions; locally it prints the markdown report and then the numeric pass rate. Provider orchestrators can add a --details-url, suppress stale updates with --expected-head-sha, and opt into surfaced API failures with --strict.
• pathgrade validate --affected is a strict mode for CI: every discovered eval must either live under a SKILL.md anchor or export valid __pathgradeMeta. Affected selection always includes a discovered eval when its own file changes, including comment-only edits. You do not need to add the eval's path to deps or extraDeps: those lists control subject dependencies. Existing selection reasons keep precedence; an eval selected only by its own changed path has reason eval-changed in pathgrade affected --json and an explanation in --explain. Discovery exclusions still apply. This default can increase CI runs for eval-only edits. Declare methodology files separately from affected-selection dependencies with __pathgradeMeta.comparisonInputs, for example comparisonInputs: ['evals/fixtures/**', 'evals/scorers/**']. An empty array explicitly declares a self-contained eval. Pathgrade hashes the raw eval source plus each declared glob, matched repository-relative path, and raw file bytes; .git internals and .pathgrade outputs are excluded. Missing, invalid, unmatched, escaping, or unreadable inputs suppress numeric deltas. deps, extraDeps, inferred skill roots, and global triggers affect selection only and never definition identity. Run pathgrade --help for the full help text.

Configuration

// pathgrade.config.ts
export default {
    attempts: 1, // opt in to runner-owned repetitions
    runner: { adapter: 'vitest', args: [] },
    evals: {
        include: ['**/*.eval.ts'],   // default
        exclude: ['**/fixtures/**'], // replaces the default exclude list if set
    },
    affected: { global: ['package.json', 'yarn.lock'] },
    ci: { threshold: 0.8 },
};

Pathgrade reads pathgrade.config.* for CLI and affected-selection behavior. runner.adapter and --adapter=<name|path> select the runner; --adapter wins over config. attempts defaults to 1 and must be a positive integer. Built-in adapters vitest, jest, and node-test support repeated attempts; third-party invocation adapters must advertise supportsRepeatedAttempts: true and produce the normalized child snapshot contract.

New reports use schema version 3. They add a versioned task inventory with every selected eval file, collection completeness, stable task keys, and bounded non-scoring reasons. They also preserve every attempt with case_id, attempt_id, and attempt_index; include the display-safe runner_outcome; publish mean_reward; and publish finite-sample pass@k only for complete binary attempts. runner_status and threshold_status expose the two gates independently, and canonical status passes only when runner assertions pass and any configured threshold passes. Runner assertions, runner diagnostics, native runner references, and full flow traces remain transient or in trace artifacts rather than being copied into consolidated public reports. A partial reward or incomplete attempt makes pass@k explicitly unavailable. Report versions 1 and 2 remain readable, but they cannot be promoted as authoritative task-accounted baselines and must be rerun.

Third-party runner adapters are supported through @wix/pathgrade/adapter-kit. Adapter names resolve as follows:

• vitest, jest, node-test: built-in adapters
• demo: package @wix/pathgrade-adapter-demo, resolved from the project
• @scope/pathgrade-adapter-demo or another specifier containing /: package specifier, resolved from the project
• ./local-adapter.mjs or /abs/local-adapter.mjs: local adapter module

External adapter modules used by pathgrade run must export createPathgradeInvocationAdapter({ config }), returning a RunnerInvocationAdapter. Modules used by lower-level orchestration can also export createPathgradeAdapter(), returning a RunnerAdapter with discover, invoke, and collectNormalizedRunSnapshot.

Vitest runner behavior still belongs in vitest.config.ts:

import { pathgrade } from '@wix/pathgrade/adapters/vitest';

pathgrade({
    timeout: 300,                // seconds, default: 300
    reporter: 'cli',             // 'cli' | 'browser' | 'json'
    diagnostics: false,          // print full diagnostics for passing evals too
    verbose: false,              // stream live per-turn events to stderr while evals run
});

Jest runner behavior still belongs in jest.config.*. Pathgrade injects only the setup and reporter it needs when you use pathgrade run --adapter=jest; transforms, test environment, module resolution, and ESM/TypeScript support remain your Jest config's job.

For direct Jest runs, configure the same entry points explicitly:

// jest.config.mjs
export default {
    setupFilesAfterEnv: ['@wix/pathgrade/adapters/jest/setup'],
    reporters: ['default', '@wix/pathgrade/adapters/jest/reporter'],
};

Notes:

• Legacy @wix/pathgrade/plugin and @wix/pathgrade/plugin/vitest imports remain as compatibility fallbacks, but new config should use @wix/pathgrade/adapters/vitest and pathgrade.config.*.
• reporter: 'browser' writes results JSON and opens the viewer automatically after the run.
• SDK-only consumers can import @wix/pathgrade without installing Vitest or Jest. Adapter users need their selected runner available.

Environment Variables

| Variable | Purpose | |----------|---------| | ANTHROPIC_API_KEY | Claude auth; explicit or host API-key auth for OpenCode's default model | | OPENAI_API_KEY | Codex auth; optional explicit or host API-key auth for OpenCode GPT-5.4, and required with OPENAI_BASE_URL | | CURSOR_API_KEY | Cursor auth and the required key when using CURSOR_API_BASE_URL | | ANTHROPIC_BASE_URL | Custom Anthropic-compatible endpoint | | OPENAI_BASE_URL | Custom OpenAI-compatible endpoint | | CURSOR_API_BASE_URL | Custom Cursor-compatible endpoint | | PATHGRADE_AGENT | Fallback agent for all tests (claude, codex, cursor, or opencode). createAgent({ agent }) wins over this. | | PATHGRADE_CODEX_TRANSPORT | Fallback Codex transport (exec or app-server). createAgent({ transport }) wins over this. | | PATHGRADE_VERBOSE | 1 enables live per-turn streaming to stderr | | PATHGRADE_DIAGNOSTICS | 1 prints full diagnostics for passing evals too | | NO_COLOR | Disable ANSI colors |

Experimental node:test Adapter

node-test is a small proof adapter for validating the runner boundary without Vitest. It is intentionally narrow: eval files import the test wrapper from @wix/pathgrade/adapters/node-test, run through Node's built-in test runner, and still produce the normal .pathgrade/results.json and trace artifacts.

import { test } from '@wix/pathgrade/adapters/node-test';
import { createAgent, evaluate, check } from '@wix/pathgrade';

test('minimal node proof', async () => {
    const agent = await createAgent({ workspace: process.cwd() });
    await evaluate(agent, [check('passes', () => true)]);
});

Run it with:

pathgrade run --adapter=node-test

This proof validates Pathgrade's adapter, lifecycle, result capture, and reporting boundaries. It does not imply Mocha, Playwright, or runnerless CLI support.

pathgrade run loads .env from the working directory automatically.

CI / GitHub Actions

Run evals on every PR and post results as a PR comment:

jobs:
  eval:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # required for affected selection
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - name: Run affected evals
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: npx pathgrade run --changed
      - name: Post PR report
        if: always()
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: npx pathgrade report
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pathgrade-reports
          path: .pathgrade/

• fetch-depth: 0 is required for --changed; shallow clones break merge-base resolution.
• Evals under a SKILL.md are tracked automatically; use __pathgradeMeta for cross-skill or non-standard dependencies.
• Set ci: { threshold: 0.8 } in pathgrade.config.ts to fail the run when the mean test score drops below your threshold.

See the User Guide - CI Integration for the full reference.

Links

• User Guide - full API reference and usage patterns
• start-chat - multi-turn conversation
• sdk-showcase - advanced SDK features in one suite
• tool-judge-demo - judge({ tools }) reading workspace artifacts

Note on AI provider dependencies

Pathgrade is released under the MIT license. It can use third-party AI agent tools and SDKs, including @anthropic-ai/claude-agent-sdk, Codex CLI, and cursor-agent. Those tools, services, and authentication methods have separate provider terms; in particular, the Claude Agent SDK is governed by Anthropic's Commercial Terms unless Anthropic specifies another license for a component.

License

MIT