@axiastudio/aioc-regression-cli

Experimental CLI for running AIOC regression suites from filesystem artifacts.

The CLI is intentionally thin: it loads baseline RunRecord files, builds a candidate harness from a descriptor, delegates executable behavior to an application adapter, then writes candidate records, comparisons, judge results, and a CI-friendly summary.

Usage

aioc-regression run \
  --records ./baseline/runrecords \
  --baseline ./baseline/harness.yaml \
  --candidate ./candidate/harness.yaml \
  --expectation ./expectation.yaml \
  --adapter ./adapter.mjs \
  --out ./out

Defaults:

• --mode live
• --input-mode recorded
• --fail-on fail

Use --dry-run to validate artifacts and build the candidate harness without running replay.

The --expectation file must contain a RunRegressionExpectation directly.

Adapter

The adapter must be a .js or .mjs ESM module with a default object export. TypeScript adapters are intentionally not loaded by v1.

import { allow, setupOpenAI, tool } from "@axiastudio/aioc";

export default {
  setup() {
    setupOpenAI();
  },

  createTools() {
    return {
      "example://tool/get_age_range": tool({
        name: "get_age_range",
        description: "Return the learner age range.",
        execute: async () => ({ ageRange: "8-10" }),
      }),
    };
  },

  createRunOptions() {
    return {
      policies: {
        toolPolicy: () => allow("allow_example_tool"),
      },
    };
  },
};

Optional adapter hooks:

• createJudge(context) returns a RunJudge.
• createComparisonOptions(context) returns compareRunRecords(...) options.
• onMissingToolCall(input) handles strict or hybrid replay misses.

Output

out/
  summary.json
  candidates/
    case-name.runrecord.json
  comparisons/
    case-name.comparison.json
  judges/
    case-name.judge.json
  results/
    case-name.result.json

summary.json is the compact artifact intended for CI. Full records and comparisons remain available for inspection and offline analysis.