ushman-corpus-runner

Build, run, and score synthetic fixture corpora that calibrate the ushman pipeline. This package is the runner; ushman-corpus is the read-only fixture dataset.

Runtime

• Bun is required at runtime. The package uses Bun.file, Bun.write, and Bun.YAML.
• The standalone CLI is useful for init, build, report, validate, pin, prune, and xv-check.
• measure is intentionally headless. It only works when a consumer injects a PipelineRunner.

Install

bun add ushman-corpus-runner

The npm bin entry (ushman-corpus) is a launcher inside this package; it is not a path to ushman-corpus the dataset. Fixture bytes live in the separate ushman-corpus git repo.

Corpus checkout

This package does not bundle fixtures. Resolve the dataset in one of these ways:

1. Published default — if you omit ushmanCorpus in the workspace package.json, commands fetch https://github.com/ragaeeb/ushman-corpus.git at branch v0.5.0 into ~/.cache/ushman/corpus/.
2. Explicit init — ushman-corpus init <workspace> --pin=v0.5.0 writes a pin and populates the cache (add --corpus=<path> for a local clone).
3. Monorepo dev — with ../ushman-corpus beside the workspace or this package, a sibling checkout is detected automatically.
4. Env override — USHMAN_CORPUS_DIR=/absolute/path/to/ushman-corpus for catalogue validation and default pin resolution.

Quick Start

ushman-corpus init ./calibration-workspace
ushman-corpus build ./calibration-workspace --fixtures=f001-synthetic

Local clone instead of the published git default:

ushman-corpus init <workspace> --corpus=/absolute/path/to/ushman-corpus --pin=local-2026-05-08
ushman-corpus build <workspace> --fixtures=f001-synthetic
ushman-corpus report <workspace> --format=md
ushman-corpus validate --catalogue=/absolute/path/to/ushman-corpus --retros-dir=/absolute/path/to/ushman/reviews/retros --format=md --strict
ushman-corpus xv-check <workspace>
ushman-corpus prune --keep-latest=2

CLI

ushman-corpus init <workspace> [--corpus=<path>] [--pin=<commit>] [--fetch=git|tarball] [--force]
ushman-corpus build <workspace> [--fixture=<id>] [--fixtures=<id,id>] [--tier=<T0,T1>] [--parallel=<n>]
ushman-corpus measure <workspace> [--fixture=<id>] [--fixtures=<id,id>] [--pipeline=<stage>] [--mode=jaccard|tree-edit] [--tier=<T0,T1>] [--parallel=<n>] [--no-stability-check] [--include-held-out|--xv-allow] [--xv-justification=<reason>]
ushman-corpus report <workspace> [--format=md|html|json] [--baseline=<path>] [--xv-allow] [--xv-justification=<reason>]
ushman-corpus validate [--catalogue=<path>] [--retros-dir=<path>] [--format=md|json|both] [--evidence-threshold=<n>] [--baseline=<path>] [--strict]
ushman-corpus pin <workspace> --commit=<sha-or-label> [--fetch=git|tarball] [--repo=<path>]
ushman-corpus xv-check <workspace>
ushman-corpus prune [--keep-latest=<n>] [--keep-tags=<tag,tag>]

--fixture and --fixtures are both accepted for compatibility with the extraction docs. --format is the canonical report flag; --out is still accepted as an alias.

build and measure both accept --parallel=<n>. Measurement work fans out across fixtures, but the returned and printed summaries stay in fixture selection order. --parallel must be an integer between 1 and the runtime cap derived from local CPU parallelism.

Catalogue Validation

• validate compares the published v4 catalogue artifacts against every retro in reviews/retros/*.md.
• --catalogue accepts a single artifact file, a family directory such as sdk-catalogue/, or the ushman-corpus repo root. The latest versioned artifact in each family is resolved automatically.
• When validating sdk-catalogue in isolation, the runner still resolves sibling runtime-fingerprint rules for runtime-gap suppression. If no sibling runtime catalogue can be found, validation fails fast instead of silently reporting inaccurate SDK gaps.
• --retros-dir should point at the retros/ directory itself. The validator reads *.md files and reports missing ## 3 / ## 5 sections as warnings.
• runtime-fingerprint rules can declare evidenceRetros per rule. The validator counts those citations, reports stale evidence, and treats runtime-family evidence gaps the same way it treats sdk-catalogue and vendor-globals.
• --format=both is the default. It prints a markdown report followed by a JSON payload. Use --format=json for CI or other machine readers.
• --evidence-threshold defaults to 2. Entries below that distinct-retro count are reported as low evidence, but the command still exits 0 as long as the inputs are readable and schema-valid.
• --strict returns a non-zero exit code whenever the report status is not pass.
• --baseline compares the current JSON report against a previous JSON report and returns non-zero if counts regress or matched coverage drops.
• Default path resolution is: explicit flag, then USHMAN_CORPUS_DIR / USHMAN_RETROS_DIR (or USHMAN_REPO_ROOT/reviews/retros for retros), then the local sibling checkout layout if it exists.

Held-Out Discipline

• Held-out fixtures are never scored implicitly.
• To score or report on held-out fixtures, pass both --xv-allow and --xv-justification=<reason>.
• Measurement runs persist that justification in measurements/<fixture>/xv-override.json and append an audit line to measurements/.xv-allow.log.
• Under parallel measurement runs, the shared .xv-allow.log is appended after worker completion and preserves fixture selection order for the recorded entries.
• xv-check validates registry/manifests and scans workspace sources, docs, prompts, and briefs for held-out references.

PipelineRunner

measure requires an injected PipelineRunner:

type PipelineRunner = {
  readonly run: (input: {
    fixtureRoot: string
    outputRoot: string
    timeoutMs?: number
  }) => Promise<{
    cleanedWorkspaceRoot: string
    elapsedMs: number
    perStageWallClockMs?: Partial<Record<PipelineStage, number>>
  }>
}

The runner must write stage outputs beneath:

<outputRoot>/stages/01-seed
<outputRoot>/stages/02-vendor-extract
<outputRoot>/stages/03-clean

Fixtures that declare vendorExpectations are also scored against:

<outputRoot>/stages/02-vendor-extract/vendor-boundaries.json

If that artifact is missing, malformed, or omits an expected vendor, the measurement is recorded as a scoring failure.

When measure --parallel is greater than 1, the injected runner must tolerate concurrent run() calls for different fixtures.

Execution Results

• buildFixture, runFixture, runCorpusMeasurement, and runCatalogueValidation all return discriminated unions with stable issues[].code values plus human-readable issues[].message text.
• runCorpusMeasurement returns measurement-run-aborted only for setup/preflight problems. A completed run returns measurement-run-complete and then reports per-fixture outcomes under outcomes.
• Score failures that still write measurements/<fixture>/calibration-result.json are represented only in failures, with the saved artifact exposed on persistedResult. They do not also appear in measurements.
• runFixture distinguishes setup problems (run-setup-failed) from pipeline execution crashes (run-pipeline-failed).
• runCorpusMeasurement(...).ok means "no fixture failures occurred." Skipped fixtures are tracked separately under skipped.

Primary issue-code families:

• Build: build-command-failed, build-io-error, build-lockfile-missing, build-stability-hash-mismatch
• Run: run-fixture-locked, run-setup-failed, run-pipeline-failed
• Measure: measure-held-out-requires-override, measure-pipeline-runner-missing, measure-setup-failed
• Score: score-unexpected-error, score-vendor-missing, score-vendor-artifact-invalid, score-vendor-import-plan-missing
• Validate: validate-input-invalid, validate-runtime-rules-missing, validate-baseline-regression, validate-strict-status, validate-report-build-failed

Public API

runCorpusMeasurement({
  corpusRoot,
  corpusCommit,
  stages,
  pipelineRunner,
  fixtureIds,
  noStabilityCheck,
  parallel,
  tiers,
  xvAllowJustification
}): Promise<RunCorpusMeasurementResult>

runCatalogueValidation({
  cataloguePath,
  retrosDir,
  baselineReport,
  evidenceThreshold,
  strict
}): Promise<CatalogueValidationResult>

scoreFixture(opts): Promise<ScoreFixtureResult>
loadCorpusManifest(corpusRoot): Promise<CorpusManifest>
loadFixtureManifest(fixtureDir): Promise<FixtureManifest>
listFixtures(corpusRoot, filters?): Promise<readonly FixtureManifest[]>
buildCalibrationAggregate(opts): CalibrationAggregate
renderCalibrationAggregate(aggregate, format): string
validateCatalogues(opts): Promise<CatalogueValidationReport>
findCatalogueValidationRegressions(current, baseline): CatalogueValidationRegression[]
renderCatalogueValidationReport(report, format): string
PIPELINE_STAGES
PIPELINE_STAGE_OUTPUT_DIRECTORIES

runCorpusMeasurement() defaults noStabilityCheck to false and validates parallel against the same runtime cap as the CLI.

Scorers

Identifier Match

• Input: reference source tree and recovered source tree, optionally with a precomputed file-structure mapping.
• Output: 0..1, plus per-file alignment counts and type-only exclusions.
• Known failure modes: parse failures degrade to empty token streams; anonymous default exports align approximately.

File Structure Match

• Input: reference source tree and recovered source tree.
• Output: 0..1, plus matched file mapping and unmapped file lists.
• Known failure modes: split/merged files are matched heuristically with a fixed 0.5 similarity threshold.

Semantic Distance

• Input: reference source tree and recovered source tree, optionally with a precomputed file-structure result.
• Output: 0..1 similarity score in jaccard or tree-edit mode.
• Known failure modes: syntax errors degrade to empty ASTs; tree-edit is an AST-sequence approximation rather than a full tree-edit algorithm.

Vendor Expectations

• Input: fixture-manifest vendorExpectations plus stages/02-vendor-extract/vendor-boundaries.json.
• Output: pass/fail validation metadata listing expected vendors, detected vendors, and any missing vendor boundaries.
• Known failure modes: missing or malformed vendor-boundaries.json fails every declared expectation; the check is skipped when stage stage-02-vendor-extract is not part of the run.

Aggregate Report

• Input: CalibrationResult fixtures plus an optional baseline aggregate.
• Output: per-tier percentiles, pass counts, regressions, improvements, and any vendor-expectation failures captured in the fixture results.
• Known failure modes: passed uses a report-only threshold of 0.8 across all three latest-stage scores.

Where This Fits

| | | |---|---| | Sister to | ushman-corpus | | Headless by design | The orchestrator owns the actual capture session and implements PipelineRunner | | Scope | Fixture build orchestration, pipeline measurement, scoring, and aggregate reporting |