@reaatech/agent-eval-harness-types

Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.

Canonical TypeScript domain types, Zod schemas, and interfaces for the agent-eval-harness ecosystem. This package is the foundational dependency of every other package in the monorepo.

Installation

npm install @reaatech/agent-eval-harness-types
# or
pnpm add @reaatech/agent-eval-harness-types

Feature Overview

• 19 domain type interfaces — Turn, Trajectory, EvalResult, JudgeScore, CostBreakdown, LatencyBudget, GoldenTrajectory, RegressionGate, and more
• 20 Zod schemas — runtime validation for every domain type with full type inference via z.infer
• Zero runtime dependencies beyond zod
• Dual ESM/CJS output — works with import and require
• Golden trajectory markers — golden, expected, and quality_notes fields on every turn
• CI gate types — threshold, baseline-comparison, and distribution gates with regression tracking
• Suite runner types — configuration, run status, comparison, and metric regression interfaces

Quick Start

import { TurnSchema, type Trajectory, type EvalResult } from '@reaatech/agent-eval-harness-types';

const turn = TurnSchema.parse({
  turn_id: 1,
  role: 'user',
  content: 'Hello',
  timestamp: '2026-04-15T00:00:00Z',
});

const trajectory: Trajectory = { turns: [turn], metadata: { total_turns: 1 } };

API Reference

Core Types

| Name | Type | Description | |------|------|-------------| | Turn | interface | Single turn in a trajectory with role, content, timestamp, and optional tool calls, latency, and cost | | ToolCall | interface | Tool invocation with name, arguments, and optional result | | CostData | interface | Token usage and cost for a single turn | | Trajectory | interface | Complete agent execution with turns array and optional metadata | | EvalResult | interface | Evaluation result with overall score, per-metric scores, and issues | | EvalIssue | interface | Issue found during evaluation with type, severity, and description |

Judge Types

| Name | Type | Description | |------|------|-------------| | JudgeScore | interface | LLM judge scoring result with score, explanation, confidence, and calibration status |

Cost Types

| Name | Type | Description | |------|------|-------------| | CostBreakdown | interface | Full cost breakdown for a trajectory with LLM, tool, and per-turn costs | | TurnCost | interface | Cost breakdown for a single turn with token counts |

Latency Types

| Name | Type | Description | |------|------|-------------| | LatencyBudget | interface | Latency SLA budget with P50, P90, P99 thresholds and component breakdowns | | LatencyResult | interface | Latency measurement result with percentiles, violations, and SLA status | | LatencyViolation | interface | SLA violation record with turn ID, actual vs threshold values |

Golden Types

| Name | Type | Description | |------|------|-------------| | GoldenTrajectory | interface | Golden reference trajectory with versioning and quality markers |

Gate Types

| Name | Type | Description | |------|------|-------------| | RegressionGate | interface | Gate definition with threshold, baseline-comparison, or distribution types | | GateResult | interface | Single gate evaluation result with pass/fail and actual vs expected values |

Suite Types

| Name | Type | Description | |------|------|-------------| | EvalSuiteConfig | interface | Suite configuration with metrics, judge model, budgets, gates, and parallelism | | EvalRunStatus | interface | Suite run progress with status, completion counts, and timing | | RunComparison | interface | Comparison of two evaluation runs with metric diffs and significance testing | | MetricRegression | interface | Single regression with baseline and candidate values and change percentage |

Schemas

| Name | Type | Description | |------|------|-------------| | ToolCallSchema | ZodObject | Validates tool invocation structure | | CostDataSchema | ZodObject | Validates token counts and cost data | | TurnSchema | ZodObject | Validates turn structure with optional tool calls, latency, and golden markers | | TrajectoryMetadataSchema | ZodObject | Validates trajectory metadata | | TrajectorySchema | ZodObject | Validates complete trajectory (minimum one turn, optional metadata) | | EvalIssueSchema | ZodObject | Validates evaluation issue records | | EvalResultSchema | ZodObject | Validates evaluation results with metrics and issues | | JudgeScoreSchema | ZodObject | Validates judge scoring output | | CostBreakdownSchema | ZodObject | Validates cost breakdowns with per-turn cost arrays | | LatencyBudgetSchema | ZodObject | Validates latency budget configuration | | LatencyViolationSchema | ZodObject | Validates latency SLA violations | | LatencyResultSchema | ZodObject | Validates latency measurement results | | QualityMarkersSchema | ZodObject | Validates golden trajectory quality markers | | GoldenTrajectorySchema | ZodObject | Validates golden trajectories with nested trajectory and quality markers | | RegressionGateSchema | ZodObject | Validates regression gate definitions | | GateResultSchema | ZodObject | Validates gate evaluation results | | EvalSuiteConfigSchema | ZodObject | Validates suite configuration with nested latency budget and gates | | EvalRunStatusSchema | ZodObject | Validates suite run status | | MetricRegressionSchema | ZodObject | Validates metric regression records | | RunComparisonSchema | ZodObject | Validates run comparison results with statistical significance arrays |

Related Packages

| Package | Description | |---------|-------------| | @reaatech/agent-eval-harness-types | Shared domain types and Zod schemas | | @reaatech/agent-eval-harness-trajectory | Trajectory loading, evaluation, and golden comparison | | @reaatech/agent-eval-harness-tool-use | Tool-use validation and schema compliance | | @reaatech/agent-eval-harness-cost | Cost tracking, budgets, and reporting | | @reaatech/agent-eval-harness-latency | Latency monitoring, SLA enforcement, and optimization | | @reaatech/agent-eval-harness-judge | LLM-as-judge with calibration and consensus | | @reaatech/agent-eval-harness-golden | Golden trajectory management and curation | | @reaatech/agent-eval-harness-suite | Suite runner, results aggregation, and comparison | | @reaatech/agent-eval-harness-gate | CI regression gates with JUnit and GitHub output | | @reaatech/agent-eval-harness-mcp-server | MCP server with three-layer tool architecture | | @reaatech/agent-eval-harness-cli | Command-line interface | | @reaatech/agent-eval-harness-observability | OTel tracing, metrics, structured logging, and dashboards |

License

MIT