@fabric-harness/evals

Evaluation framework for Fabric Harness agents.

Define test cases, run them against your agents, and track quality over time.

Install

npm install @fabric-harness/evals @fabric-harness/sdk

Quick start

import { runEvalSuite, exactMatchScorer, llmAsJudgeScorer, jsonShapeMatchScorer } from '@fabric-harness/evals';

const result = await runEvalSuite({
  name: 'triage-quality',
  cases: [
    { id: 'sev-high', input: { issue: 'data loss in prod' }, expected: { severity: 'high' } },
    { id: 'sev-low',  input: { issue: 'typo in docs' },      expected: { severity: 'low' } },
  ],
  runner: async ({ case: c }) => triageAgent(c.input),
  scorers: [
    jsonShapeMatchScorer({ severity: 'string', summary: 'string' }),
    llmAsJudgeScorer({
      judge: anthropicProvider,
      model: 'anthropic/claude-haiku-4-5',
      rubric: 'Severity matches the actual urgency of the issue.',
    }),
  ],
  passThreshold: 0.8,
});

Scorers

• exactMatchScorer() — strict JSON-equality against case.expected.
• containsTextScorer(text?) — substring check on stringified output.
• regexMatchScorer(pattern, options?) — regex match against the stringified output.
• jsonShapeMatchScorer({ field: 'string' | 'number' | 'boolean' | 'array' | 'object' | nested }) — schema-shape check with detailed mismatch reasons.
• llmAsJudgeScorer({ judge, model, rubric, passThreshold? }) — LLM-as-judge with structural JSON parsing and graceful fallback on malformed responses.

Documentation

• Evals reference

License

Apache-2.0