@unit-test-evaluator/cli

v0.0.5

Published

14 days ago

CLI tool for evaluating unit tests using LLM inference

0High
0Medium
0Low

rensilvaau

monk-eee

cli unit-test evaluation llm testing

Unit Test Evaluator CLI

A CLI tool that uses LLM inference and Stryker mutation testing to evaluate the quality of unit tests relative to their source files, based on customizable criteria. Both the LLM evaluation and the mutation run happen in a single command and emit a single JSON output.

Features

LLM-Powered Evaluation: Uses Azure OpenAI or GitHub Models to assess test quality against your criteria. Auto-detects github when GITHUB_TOKEN is set, otherwise uses azure.
Language-aware judging: The judge prompt is framed with the project's language and test framework, and code fences are tagged accordingly — so Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/etc.), and C# (.NET) tests are each evaluated against their own idioms rather than defaulting to JS/Jest. The runner is auto-detected from your mapping's source-file extensions when --test-runner is omitted (.py→pytest, .cs→dotnet, JS/TS→jest, refined to vitest/mocha/jasmine from package.json); the detected choice is always printed with instructions to override.
Integrated Stryker Mutation Testing: Pass --mutation and Stryker runs in the same job. Mutate patterns are auto-derived from the mapping; a sandbox-safe Stryker + jest + tsconfig bundle is generated under .stryker-tmp-config/ so the run never inherits a broken repo-root config.
Honest failure reporting: When Stryker fails, the underlying error is surfaced in metadata.mutationError and combinedOverallScore is omitted. No silent zeros, no stale-report leaks.
Variance reduction: --judge-runs N repeats each criterion N times and takes the median; --judge-panel a,b,c dispatches to a diverse-judge panel and takes the median across models. Default is single-shot for fast iteration; either flag enables the higher-confidence mode.
Deterministic advisor grounding (opt-in): --advisor-grounding shells out to a Python static-analysis advisor to compute a CodeProfile of facts (archetype, side-effect boundaries, cyclomatic complexity V(G), a basis-path test-count expectation) and injects them into the judge prompt as ground truth, so the model doesn't re-derive (and disagree about) those facts on every run. Off by default; when enabled it fails loudly if the advisor can't be reached — never scores without the ground truth.
Parallel Processing across criteria for each file.
0–5 Scoring per criterion with reasoning.
Customizable Criteria with structured description blocks.
Flexible File Mapping with cwd-independent path resolution (paths are resolved relative to the mapping file).
JSON, CSV, Excel output formats (auto-detected from file extension).
Test Coverage Analysis via --coverage-dir (folder-level Jest coverage).
Criteria, coverage, and mutation averages plus a weighted combined overall score when all three components are present.
Run telemetry captured into results.metadata.telemetry (host, user, git, CI, and package identity) so every persisted evaluation is traceable to the machine, repo, and commit that produced it. Collection is best-effort and never fails a run; embedded credentials are stripped from remote URLs and only a whitelist of CI variables is read (never the full environment).
Envelope output shape ({results, metadata}) that matches what the HTTP API in api/ returns — the CLI's on-disk JSON is byte-for-byte the same as the API response body.

Using the solution

1. CLI package

Use the root package when you want to run evaluations locally or from automation as a command-line tool.

Installation

The CLI is published to the public npm registry as @unit-test-evaluator/cli. Install it globally to expose the unit-test-evaluator command:

npm install -g @unit-test-evaluator/cli

The global install is self-contained: Stryker, jest, and ts-jest all ship with the CLI. The project you're evaluating does not need to install them.

For a portable install, npm pack produces a tarball you can drop anywhere: npm install -g ./unit-test-evaluator-cli-0.0.4.tgz.

Building from source

npm install
npm run build
npm install -g .          # exposes the `unit-test-evaluator` command globally

Publishing a new release? See DEVELOPERS.md.

Prerequisites & caveats

What the developer's machine needs:

Node.js ≥ 18. The CLI is a Node program — there's no static binary.
For provider auth setup: use az CLI for Azure token flows, and use the GitHub web UI to create a fine-grained PAT with the Models account permission (Read-only) for GitHub Models.
For .NET targets only: the dotnet SDK on PATH. Stryker.NET (dotnet-stryker) is lazy-installed via dotnet tool install on first --mutation run — that's the idiomatic .NET pattern. An npm package can't bundle .NET tooling.
For --advisor-grounding only: Python 3 with the advisor package importable (default invocation python -m advisor.cli). Override with --advisor-command. Not needed unless you enable grounding.

What the project being evaluated needs:

For JS/TS targets: nothing CLI-specific. The CLI ships its own jest + ts-jest + Stryker and generates a self-contained .stryker-tmp-config/ bundle per run. Your tests still have to be jest-runnable in principle (test syntax, imports) — the CLI provides the runner, it cannot make incompatible tests pass.
For .NET targets: a valid .csproj/.sln plus a runnable test project (xunit / nunit / mstest). Pass them via --package, --dotnet-project, --dotnet-test-project.
For coverage scoring: pre-generated coverage JSON in <coverage-dir>/coverage-final.json (Jest's --coverage default output). The CLI reads coverage, it does not run it.

Other things worth knowing up front:

First mutation run is slow. Stryker + jest startup is ~30s minimum even for one file. That's Stryker, not the CLI.
Mutation failures are surfaced, not hidden. When Stryker fails the underlying error is in results.metadata.mutationError and combinedOverallScore is omitted (not zeroed). No silent zeros, no phantom successes.
Coverage failures are also surfaced. Missing or unreadable coverage leaves affected entries with coverage: null rather than synthesizing fake numbers.
Harmless Stryker teardown warning. After Done in N seconds. you may see WARN ChildProcessTestRunnerProxy ... Injector is already disposed ... Stryker will ignore this error. This is an upstream race in @stryker-mutator/core's child-process disposal; the mutation run has already completed successfully and the report JSON is valid. We don't suppress it because filtering Stryker's stderr could hide real failures.

Authentication

Pick one provider. The CLI auto-detects github when GITHUB_TOKEN is set, otherwise azure. You can force a provider with --provider github|azure.

GitHub Models (local dev — getting a token)

The GitHub Models API requires the models:read permission. That permission only exists on fine-grained personal access tokens — it is not a classic PAT scope (you will not find models in the classic scope list), and gh auth refresh --scopes models fails with invalid_scope.

Create a fine-grained PAT at https://github.com/settings/personal-access-tokens/new:

Go to Settings → Developer settings → Personal access tokens → Fine-grained tokens.
Click Generate new token, give it a name and an expiry.
Under Repository access, select Public Repositories (read-only). A fine-grained PAT is only valid if it is scoped to some repository access; "Public Repositories" is enough even if you will use the token in a private repo — repository access is unrelated to Models and the actual grant comes from the account permission in the next step.
Under Permissions → Account permissions, find Models and set it to Read-only. (It is an account permission, not a repository permission — it will not appear in the Repository permissions list.)
Click Generate token, copy it, and export it:

# Bash / zsh
export GITHUB_TOKEN="<fine-grained PAT with Models: Read-only>"

# PowerShell
$env:GITHUB_TOKEN = "<fine-grained PAT with Models: Read-only>"

The CLI calls https://models.github.ai/inference/chat/completions. Default model: gpt-4o. See https://models.github.ai/catalog/models for other model ids.

Azure OpenAI

export AZURE_OPENAI_ENDPOINT="https://<resource>.openai.azure.com"
# Option A — Azure AD bearer token (preferred, no secrets):
export AZURE_OPENAI_ACCESS_TOKEN="$(az account get-access-token \
  --resource https://cognitiveservices.azure.com --query accessToken -o tsv)"
# Option B — API key:
export OPENAI_API_KEY="<your-key>"

--access-token takes precedence over --api-key.

Usage

Basic LLM-only evaluation (preferred: direct pairs)

The recommended way to specify source/test files is with repeatable --pair <source-file=test-file> flags — no separate mapping file to create or keep in sync:

unit-test-evaluator evaluate \
  --pair src/utils/stringUtils.ts=tests/utils/stringUtils.test.ts \
  --pair src/hooks/useAuth.ts=tests/hooks/useAuth.test.ts \
  --package package.json \
  --output output/evaluation-results.json

If you omit --criteria, the CLI uses the packaged evaluation-criteria.json. If you omit --model, it uses the configured default model: gpt-4o.

Recommended: stick with the defaults for comparable results. Both --criteria and --model are optional, and you can override them. However, the recommended approach is to run with the standard packaged criteria and the default model (gpt-4o). Scores are only directly comparable across runs, files, and projects when they are produced under the same criteria and the same judge model — changing either makes the resulting scores incomparable to runs that used the defaults. Override them only when you deliberately want an isolated, non-comparable experiment.

Alternative: a mapping file

For large or reusable file sets you can pass a mapping file instead of individual pairs:

unit-test-evaluator evaluate \
  --mapping examples/stringutils-only.json \
  --package package.json \
  --output output/evaluation-results.json

LLM + Stryker mutation in one job

unit-test-evaluator evaluate \
  --mapping examples/stringutils-only.json \
  --package package.json \
  --output output/evaluation-results.json \
  --mutation

Mutate patterns are derived from the mapping keys. The CLI writes a self-contained Stryker bundle to .stryker-tmp-config/ (jest config + tsconfig + stryker.conf.json) and runs npx stryker run <generated-config> --mutate <derived-patterns>. You can override either piece with --stryker-config <path> or --mutate <patterns...>.

With coverage

unit-test-evaluator evaluate \
  --mapping examples/stringutils-only.json \
  --package package.json \
  --coverage-dir . \
  --output output/results.csv \
  --format csv \
  --mutation

Custom scoring weights

unit-test-evaluator evaluate \
  --mapping examples/stringutils-only.json \
  --package package.json \
  --mutation \
  --llm-weight 0.3 \
  --mutation-weight 0.5 \
  --coverage-weight 0.2

Options

Required

Exactly one of:
- -m, --mapping <path>: JSON file mapping source files to test files
- --pair <source-file=test-file>: direct source/test pair input; repeat as needed

Project file (one required; auto-detected if omitted)

-p, --package <path>: package.json for JS/TS, .csproj or .sln for .NET

Output

-o, --output <path>: Output file path (default: output/evaluation-results.json)
-f, --format <format>: json (default), csv, or excel. Auto-detected from file extension.

LLM provider

--provider <name>: azure or github (auto-detects github when GITHUB_TOKEN is set)
--model <name>: Model / deployment name (default: gpt-4o)
--github-token <token>: GitHub PAT for GitHub Models (or set GITHUB_TOKEN)
-k, --api-key <key>: Azure OpenAI API key (or set OPENAI_API_KEY)
--access-token <token>: Azure AD bearer token (or set AZURE_OPENAI_ACCESS_TOKEN). Takes precedence over --api-key.
-e, --azure-endpoint <url>: Azure OpenAI endpoint (or set AZURE_OPENAI_ENDPOINT)
-t, --timeout <seconds>: Per-evaluation HTTP timeout (default: 120)
--temperature <number>: LLM temperature (default: 0)
--judge-runs <number>: Repeated evaluations per criterion. Default 1 (single-shot). Set to 3+ to take median across runs for variance reduction.
--judge-panel <models>: Comma-separated panel of model IDs (e.g. openai/gpt-4o,openai/gpt-4o-mini,meta/llama-3.3-70b-instruct). Overrides --model. Total calls per criterion = panel.length × judge-runs, median over all. Pre-baked panels live under llm.recommendedPanels in config/eval.config.json.

Deterministic grounding (advisor)

--advisor-grounding: Ground the judge with the deterministic advisor CodeProfile + expectations assessment. Off by default. When enabled, the advisor is invoked once per source file (shared across all criteria); its facts — archetype, side-effect boundaries, cyclomatic complexity V(G), and a basis-path expectation of at least V(G) test cases for independent paths — are injected into the judge prompt as ground truth. Requires the Python advisor to be importable; if it errors, the file fails loudly rather than scoring without ground truth (no silent fallback).
--advisor-command <cmd>: Command used to invoke the advisor CLI (default: python -m advisor.cli). The CLI runs <cmd> analyze --source <src> --test <test> and parses the emitted JSON.

Defaults for both flags come from the advisor block in config/eval.config.json (defaultEnabled, defaultCommand). Set advisor.defaultEnabled: true to turn grounding on for every run.

Coverage

--coverage-dir <path>: Directory to run test coverage for (enables coverage analysis)

Mutation testing

--mutation: Enable Stryker in the same job
--use-existing-mutation-report: Load an existing reports/mutation/mutation.json instead of re-running Stryker
--mutate <patterns...>: Override the auto-derived mutate globs
--stryker-config <path>: Use your own Stryker config instead of the generated sandbox bundle
--mutation-timeout <seconds>: Stryker per-run timeout (default: 600)
--max-concurrency <number>: Max concurrent Stryker test runners
--test-runner <runner>: jest (default), vitest, jasmine, mocha, or dotnet. pytest and unittest are also accepted — they set the language/framework context for the judge (see Language-aware judging) — but Stryker cannot mutate Python, so combining pytest/unittest with --mutation fails loudly rather than silently skipping mutation. When omitted, the runner is auto-detected from the mapping's source-file extensions (.py→pytest, .cs→dotnet, JS/TS→jest/vitest/mocha/jasmine); pass this flag to override. Python's unittest is never auto-selected — pass --test-runner unittest explicitly if you use the stdlib framework.
--dotnet-project <path>: Source .csproj for Stryker.NET --project
--dotnet-test-project <path>: Test .csproj for Stryker.NET --test-project

Scoring weights

--llm-weight <number> (default: 0.4)
--mutation-weight <number> (default: 0.4)
--coverage-weight <number> (default: 0.2)

Weights are normalized if they don't sum to 1.0. When --mutation is omitted the defaults flip to llm=0.6 / coverage=0.4.

Other commands

# Validate inputs without running an evaluation
unit-test-evaluator validate \
  --mapping examples/stringutils-only.json \
  --package package.json

# Scaffold a Stryker config (only useful if you want to hand-edit one
# instead of using the auto-generated sandbox bundle)
unit-test-evaluator init-stryker --test-runner jest --mutate "src/**/*.ts"

Input File Formats

1. File Mapping JSON

Maps source files to their corresponding test files:

{
  "src/utils/stringUtils.ts": "tests/utils/stringUtils.test.ts",
  "src/components/Button.tsx": "tests/components/Button.test.tsx",
  "src/hooks/useAuth.ts": "tests/hooks/useAuth.test.ts"
}

2. Package.json

Standard Node.js package.json file containing your project dependencies. The tool extracts all dependencies (dependencies, devDependencies, peerDependencies) for context during evaluation.

3. Evaluation Criteria JSON

Defines the criteria and scoring methodology with structured descriptions:

{
  "description": "Custom evaluation criteria for unit test quality",
  "criteria": [
    {
      "name": "Assertion Specificity & Strength",
      "analysis_target": "test file",
      "description": [
        {
          "name": "Precise Assertions",
          "details": "Tests use specific, targeted assertions rather than generic ones",
          "signal": "expect(result.status).toBe('success') vs expect(result).toBeTruthy()"
        },
        {
          "name": "Multiple Verification Points",
          "details": "Each test verifies multiple aspects of the expected behavior",
          "signal": "Checking both return value and side effects"
        }
      ],
      "weight": 1.0
    },
    {
      "name": "Test Structure & Independence",
      "analysis_target": "test file",
      "description": [
        {
          "name": "Isolated Tests",
          "details": "Each test can run independently without relying on other tests",
          "signal": "No shared state between test cases"
        }
      ],
      "weight": 1.0
    }
  ]
}

Output Formats

The tool supports multiple output formats with comprehensive data:

JSON Format

Every run produces a single envelope, identical in shape to what the HTTP API in api/ returns:

{
  "results": {
    "evaluations": [
      {
        "sourceFile": "examples/src/utils/stringUtils.ts",
        "testFile": "examples/tests/utils/stringUtils.test.ts",
        "scores": [
          {
            "criterion": "Assertion Specificity & Strength",
            "analysis_target": "test file",
            "score": 4,
            "reasoning": "Tests use specific assertions but could include more edge cases..."
          }
        ],
        "averageScore": 4.2,
        "coverage": {
          "lines":      { "total": 100, "covered": 85, "percentage": 85 },
          "functions":  { "total": 10,  "covered": 9,  "percentage": 90 },
          "branches":   { "total": 20,  "covered": 18, "percentage": 90 },
          "statements": { "total": 120, "covered": 100, "percentage": 83 }
        },
        "mutation": {
          "mutationScore": 72.55,
          "killed":   37,
          "survived": 10,
          "noCoverage": 4,
          "totalMutants": 51
        },
        "combinedScore": 75.1
      }
    ],
    "criteriaAverages":   { "Assertion Specificity & Strength": 3.8, "Test Structure & Independence": 4.1 },
    "coverageAverages":   { "lines": 83.5, "functions": 88.2, "branches": 85.7, "statements": 86.1 },
    "mutationAverages":   { "mutationScore": 72.55, "killed": 37, "survived": 10, "noCoverage": 4, "totalMutants": 51 },
    "overallAverage":     3.95,
    "combinedOverallScore": 75.1,
    "totalEvaluations":   1,
    "metadata": {
      "timestamp":       "2026-06-02T14:30:00.000Z",
      "model":           "gpt-4o",
      "criteria":        { "description": "...", "criteria": ["..."] },
      "mutationEnabled": true,
      "mutationError":   null,
      "scoringWeights":  { "llmScore": 0.4, "mutationScore": 0.4, "coverageScore": 0.2 },
      "temperature":     0,
      "judgeRuns":       1,
      "elapsedMs":       45123,
      "tokenUsage":      { "prompt": 12000, "completion": 800, "total": 12800 },
      "telemetry":       {
        "collectedAt": "2026-06-02T14:30:00.000Z",
        "host":    { "platform": "win32", "arch": "x64", "nodeVersion": "v20.11.0", "cliVersion": "0.0.4" },
        "user":    { "osUsername": "jdoe", "gitUserEmail": "[email protected]" },
        "git":     { "isRepo": true, "branch": "main", "commitShort": "a1b2c3d", "dirty": false, "remoteOrigin": "https://github.com/org/repo.git" },
        "ci":      { "provider": "github-actions", "runId": "12345", "repository": "org/repo" },
        "package": { "name": "my-project", "version": "1.4.0" }
      }
    }
  },
  "metadata": {
    "timestamp":  "2026-06-02T14:30:00.000Z",
    "pairCount":  1,
    "durationMs": 45123
  }
}

results.metadata.mutationError is null on success and carries the underlying Stryker error on failure.
combinedOverallScore is omitted (not zeroed) when any required component (LLM / mutation / coverage) is missing — per the no-fallback policy in agents.md.
results.metadata.telemetry records the host/user/git/CI/package context of the run. It is best-effort: any field that cannot be resolved is simply omitted, and the whole block is undefined only if collection is wholly impossible — it never fails an evaluation. Credentials embedded in git.remoteOrigin (https://user:token@…) are stripped, and CI capture reads only a fixed whitelist of provider variables, never the full environment.

CSV Format

The CSV output includes:

Individual file evaluations with scores, coverage, and mutation data
CRITERIA AVERAGES row showing average scores for each criterion
COVERAGE AVERAGES row showing project-wide coverage metrics
MUTATION AVERAGES row showing overall mutation testing metrics
COMBINED SCORE column showing weighted quality score
OVERALL AVERAGE row showing the overall evaluation score

Excel Format

Includes multiple sheets:

Test Evaluations: Main results with formatting
Summary: Statistical overview with performance rankings
Criteria Details: Detailed breakdown of each criterion

Scoring Scale

Each criterion is scored on a scale of 0-5:

5: Excellent - Exceeds expectations
4: Good - Meets expectations with minor improvements needed
3: Adequate - Meets basic expectations
2: Below Average - Some significant issues
1: Poor - Major issues that need addressing
0: Very Poor - Criterion not addressed or fundamentally flawed

Environment Variables

| Variable | Used by | |--------------------------------|---------------------------------------------------| | GITHUB_TOKEN | GitHub Models provider — fine-grained PAT with the Models account permission (Read-only). Scope repository access to Public Repositories to make the token valid. | | AZURE_OPENAI_ENDPOINT | Azure OpenAI endpoint URL | | AZURE_OPENAI_ACCESS_TOKEN | Azure AD bearer token (preferred over API key) | | OPENAI_API_KEY | Azure OpenAI API key (alternative to access token)|

Coverage Analysis

Pass --coverage-dir <path> to enable per-file coverage:

Automatic Detection: Tries npm run test:coverage first, then falls back to npx jest --coverage.
Folder-Level Execution: Runs coverage once for the directory (more efficient than per-file).
Per-File Data: Extracts individual file coverage from the project-wide report and matches it to mapping entries.
No mock data: If coverage analysis fails, the affected files have coverage: null and combinedScore is omitted for them. The error is surfaced — never silently substituted.

Coverage metrics include:

Lines Coverage: Percentage of executed lines
Functions Coverage: Percentage of called functions
Branches Coverage: Percentage of executed branches
Statements Coverage: Percentage of executed statements

Mutation Testing (Stryker Integration)

The CLI integrates with Stryker Mutator so the LLM evaluation and the mutation run happen in the same job, against the same mapping, producing a single output JSON.

What is mutation testing?

Mutation testing introduces small changes (mutants) to your source code and checks if your tests detect them. A high mutation score means tests are actually catching bugs.

Enabling mutation testing

unit-test-evaluator evaluate \
  --mutation \
  --mapping examples/stringutils-only.json \
  --criteria evaluation-criteria.json \
  --package package.json

That single command:

Derives --mutate glob patterns from the mapping keys (resolved relative to the mapping file).
Writes a self-contained Stryker bundle to .stryker-tmp-config/:
- stryker.conf.json (json + html + cleartext reporters, perTest coverage)
- jest.config.js (rootDir = your project, roots derived from the mapping's source + test dirs, ts-jest transform pointing at the bundle's own tsconfig)
- tsconfig.json (minimal, no rootDir constraint that could break in Stryker's sandbox copy)
Cleans stale reports/mutation*.json artifacts so a prior run's report can't silently masquerade as this run's result.
Runs npx stryker run .stryker-tmp-config/stryker.conf.json --mutate <patterns>.
Loads the fresh mutation report and merges per-file metrics into the eval output.

If Stryker fails, results.metadata.mutationError carries the underlying error and combinedOverallScore is omitted. No silent zeros.

Overriding the generated bundle

Provide your own config and the bundle generator is skipped:

unit-test-evaluator evaluate --mutation --stryker-config ./stryker.conf.json ...

Or override just the mutate patterns:

unit-test-evaluator evaluate --mutation --mutate "src/**/*.ts" "!src/**/*.d.ts" ...

Scaffold a hand-edited config

unit-test-evaluator init-stryker --test-runner jest --mutate "src/**/*.ts"

This writes stryker.conf.json you can customize. Pass it via --stryker-config to use it instead of the generated sandbox bundle.

Mutation metrics

Mutation Score: % of mutants killed by tests (higher is better)
Killed: Number of mutants detected
Survived: Number of mutants that escaped — the interesting ones
No Coverage: Mutants in code not exercised by any test
Timeout: Mutants that caused a test timeout
Total Mutants: Total mutations generated

Combined Scoring Model

When mutation testing is enabled, the tool calculates a combined score that weighs multiple quality dimensions:

Default Weights (with mutation)

| Dimension | Weight | Description | |-----------|--------|-------------| | LLM Score | 0.4 | AI-powered evaluation of test quality criteria | | Mutation Score | 0.4 | Objective measure from Stryker mutation testing | | Coverage Score | 0.2 | Code coverage percentage |

Weights without Mutation

| Dimension | Weight | Description | |-----------|--------|-------------| | LLM Score | 0.6 | AI-powered evaluation of test quality criteria | | Coverage Score | 0.4 | Code coverage percentage |

Custom Weights

You can customize the weights to match your quality priorities:

# Prioritize mutation testing
npx unit-test-evaluator evaluate \
  --mutation \
  --llm-weight 0.2 \
  --mutation-weight 0.6 \
  --coverage-weight 0.2 \
  ...

# Prioritize LLM evaluation
npx unit-test-evaluator evaluate \
  --mutation \
  --llm-weight 0.5 \
  --mutation-weight 0.3 \
  --coverage-weight 0.2 \
  ...

Combined Score Formula

Combined Score = (LLM_Score/5 × llm_weight) + 
                 (Mutation_Score/100 × mutation_weight) + 
                 ((Lines% + Branches%) / 200 × coverage_weight)

Final Score = Combined Score × 100 (scaled to 0-100)

Examples

See the examples/ directory:

examples/stringutils-only.json — minimal smoke mapping (1 source file, 1 test file)
examples/file-mapping.json — multi-file example mapping
examples/src/ and examples/tests/ — the source and test files those mappings reference

The canonical 7-criterion criteria set lives at evaluation-criteria.json in the repo root. It is the single source of truth used by the CLI, the API, and npm run example.

2. API

Use the API when you want to call the evaluator remotely over HTTP instead of installing or running the CLI locally.

The API is a thin HTTP wrapper around this CLI. It stages inline pairs to a temporary directory, invokes unit-test-evaluator evaluate, and returns the CLI's JSON envelope as the HTTP response body. Same evaluator, same Stryker integration, same scoring model.

Key differences from the CLI:

You send source/test content inline instead of passing local file paths.
The API runs the packaged evaluator inside the deployed container.
Runtime configuration comes from the API deployment environment.
REST API authentication is handled separately from MCP authentication.

Primary REST endpoints:

GET /health - health check and criteria load check.
GET /criteria - returns the configured evaluation criteria.
POST /evaluate - evaluates inline source/test pairs.
GET /docs - Swagger UI.
GET /openapi.json - OpenAPI document.

Minimal POST /evaluate shape:

{
  "pairs": [
    {
      "codeFileName": "src/utils/stringUtils.ts",
      "sourceCode": "export function trimValue(value: string) { return value.trim(); }",
      "testCode": "import { trimValue } from './stringUtils'; test('trims', () => expect(trimValue(' x ')).toBe('x'));"
    }
  ],
  "coverage": {
    "summary": {}
  },
  "options": {
    "mutation": false
  }
}

API deployment resources:

Service documentation: api/README.md
Deployment guide: api/deployment/README.md
Deployment workflow: .github/workflows/deploy-api.yml

3. MCP

Use MCP when you want GitHub Copilot Chat or another MCP client to call the evaluator as a tool.

The deployed MCP endpoint is:

https://unittest-eval-foundry.services.ai.azure.com/

Configure VS Code or another MCP client with the HTTP MCP server URL:

{
  "servers": {
    "unit-test-evaluator-api": {
      "type": "http",
      "url": "https://unittest-eval-foundry.services.ai.azure.com/mcp"
    }
  }
}

Key differences from the REST API:

The MCP surface exposes tools, not REST endpoints.
The MCP client should only need the server URL.
The deployed API handles GitHub OAuth for MCP.
GitHub OAuth client information and secrets are configured server-side in the API deployment environment.
MCP evaluate_tests uses the authenticated GitHub token for GitHub Models.

Available MCP tools:

api_health - confirms the MCP endpoint is reachable and can load evaluation criteria.
evaluate_tests - evaluates inline source/test pairs with the same evaluator capabilities exposed by the API.

Developing the solution

Local development

# Install dependencies
npm install

# Run in development mode
npm run dev -- evaluate --mapping examples/stringutils-only.json --criteria evaluation-criteria.json --package package.json

# Build for production
npm run build

# Type check
npm run type-check

Error handling

Error Handling

Validates all input files exist before starting.
Continues evaluation when individual files fail — the failing file is recorded with its error, the rest still run.
Mutation failures surface in results.metadata.mutationError; combinedOverallScore is omitted rather than zeroed.
Coverage failures leave the affected entries with coverage: null rather than synthesizing fake numbers.
API timeouts and rate-limit errors are reported, not swallowed.

Performance

Parallel Processing: Criteria for each file are evaluated concurrently.
Rate Limiting: Built-in concurrency caps to respect provider rate limits.
Timeout Protection: Configurable per-evaluation (--timeout) and per-Stryker-run (--mutation-timeout) timeouts.
Progress Tracking: Live progress indicators per file and per criterion.

Additional resources

Dev container

This repository includes a pre-configured dev container with all necessary dependencies:

Node.js & npm — for CLI development and testing
Python 3 & pip — for the Jupyter notebooks and Azure ML integration
Azure CLI — for Azure authentication and workspace management
Git and common CLI tools
Docker — for building and testing container images

Open the workspace in VS Code with the Dev Containers extension installed. Dependencies install from the public npm registry — no extra authentication required.

Unit Test MCP Server experimentation

This evaluator can be used with the Unit Test MCP Server from the UnitTestMCP repository.

Workflow:

Generate tests using the MCP server (via Copilot CLI or other clients).
Evaluate generated tests with this CLI.
Track quality metrics in Azure ML.
Iterate on test generation strategies.

Azure ML experimentation notebooks

Hypothesis Tests Skill

The repo ships a VS Code agent skill that turns a single evaluator finding into a controlled experiment. It picks one weak test-quality dimension (e.g. Assertion Specificity & Strength), runs N independent fix attempts — each isolated in its own git worktree — re-scores every attempt across all dimensions, and promotes the best one (or keeps the baseline when nothing beats it).

What it orchestrates:

| Tool | Role | |------|------| | Unit Test Evaluator MCP (hosted) / this CLI | Scores each source/test pair across the dimensions in evaluation-criteria.json. The hosted MCP server is the primary path; the CLI is the fallback and the only path with mutation scoring. | | Unit Test MCP (Jest / Pytest / .NET) | Generates and runs the actual test edits inside each experiment. |

Setup:

Add the hosted evaluator MCP server to .vscode/mcp.json (or your user MCP config) and reload VS Code so the tool is available:
```
{
  "servers": {
    "unit-test-evaluator-api": {
      "type": "http",
      "url": "https://unittest-eval-api.redglacier-30719b32.eastus.azurecontainerapps.io/mcp"
    }
  }
}
```
The hosted server runs the Azure provider server-side, so no GITHUB_TOKEN or Azure endpoint is needed for the MCP path. For the CLI fallback (or when you want mutation scoring), configure a provider as described in Authentication.
Enable the Unit Test MCP server in the same workspace so experiments can edit and run tests.
(Optional) Copy .github/skills/hypothesis-tests/config.example.json to config.json and edit the target pair, dimension, experiment count, and mutation gate. Defaults are sensible — dimension: "auto" picks the lowest-scoring criterion and experiments: 5.

Usage: in VS Code chat (agent mode), point the agent at a source/test pair and ask it to run a hypothesis test — for example:

Run a hypothesis test to improve src/utils/stringUtils.ts / tests/utils/stringUtils.test.ts.

The skill walks through 7 phases — baseline → hypothesis → spawn worktrees → run experiments → re-evaluate → compile a cross-dimension matrix → promote & clean up — and renders the results as score bars, a delta matrix, and a winner podium. See the skill definition for the full phase-by-phase contract.

Azure ML Experimentation Notebooks

The notebooks demonstrate end-to-end workflows for test generation experimentation and quality tracking:

notebooks/experiment.ipynb — main experimentation workflow
notebooks/meta-experiment.ipynb — meta-evaluation experiments
notebooks/jest-eval.ipynb — Jest/TS-specific evaluation runs
notebooks/dotnet-eval.ipynb — .NET / Stryker.NET evaluation runs
notebooks/api-test.ipynb — API smoke tests
notebooks/api-and-cli-no-mapping-test.ipynb — API + CLI inline-pairs (no mapping) tests

What they do:

Use GitHub Copilot CLI with MCP tools to generate unit tests
Evaluate test quality using this CLI
Log metrics, parameters, and artifacts to Azure Machine Learning via MLflow
Enable comparison of test generation strategies, models, and prompts

See the notebooks for setup instructions including Azure ML workspace configuration and authentication.

Contributing

See CONTRIBUTING.md for setup, repo conventions, branch/PR naming, and the pre-PR checklist. User-visible changes are tracked in CHANGELOG.md.

License

This project is licensed under the MIT License - see the LICENSE file for details.