AriaFlow Eval

@ariaflowagents/eval provides deterministic replay and assertions for AriaFlow transcript events.

This package is intentionally separate from @ariaflowagents/core to avoid runtime bloat.

What It Solves

• Validate event contracts without depending on exact LLM wording.
• Replay stored .jsonl transcripts in CI.
• Catch regressions in tool-call integrity and flow behavior.

Install

bun add @ariaflowagents/eval

Example

import { TranscriptReplay } from '@ariaflowagents/eval';

const replay = await TranscriptReplay.fromFile('./transcripts/run.jsonl');

replay
  .expectEventOrder(['input', 'tool-call', 'tool-result', 'done'])
  .expectToolCalled('start_order')
  .expectNoToolMismatches()
  .expectNoErrors()
  .expectDone();

Replay Tests from Stored Transcripts

Yes, this is the intended home for replay tests.

Typical workflow:

1. Run production-like examples and store transcript files.
2. Commit selected golden transcripts.
3. In CI, load those files with TranscriptReplay.
4. Assert structural contracts (event order, tool integrity, flow end behavior).

This makes tests stable even when model wording changes.

Golden Fixtures

Golden fixtures are committed in fixtures/golden/*.jsonl, with expectations in fixtures/golden.manifest.json.

Run the suite:

bun run --filter '@ariaflowagents/eval' test:golden

Workspace shortcut from repo root:

bun run test:golden