@trace2test/core

The core package provides the canonical trace model, deterministic JSONL normalizers, versioned YAML test specs, the offline assertion engine and replay runner, and terminal, JSON, and JUnit XML reporters. All exported canonical values are validated by traceSchema.

OpenAI Agents SDK JSONL

normalizeOpenAIAgentsTrace accepts the Agents SDK exporter envelope: an object: "trace" record followed by its object: "trace.span" records. Supported span_data.type values are agent, generation, function, guardrail, handoff, and custom. New SDK span types are retained as canonical custom spans so an import does not silently lose their payload.

Generic JSONL

normalizeGenericJsonlTrace accepts one span per non-empty line. Every line has these fields:

| Field | Required | Description | | --- | --- | --- | | trace_id | yes | Stable run identifier | | workflow_name | yes | Agent workflow name; must be consistent within a trace | | span_id | yes | Stable span/event identifier | | type | yes | agent, generation, tool/function, guardrail, handoff, or custom | | group_id | no | Conversation/group identifier; must be consistent within a trace | | trace_metadata | no | Trace-level JSON object, read from the first record | | parent_id | no | Parent span identifier | | started_at, ended_at | no | ISO 8601 timestamps | | metadata | no | Span-level JSON object | | error | no | Object with message and optional data |

Type-specific fields use the same names as the fixtures in src/__fixtures__: agent records use name, tools, handoffs, and output_type; generation records use input, output, model, model_config, and usage; tool records use name, input, output, and optional mcp_data; guardrails require name and triggered; handoffs use from_agent and to_agent; custom records require name and may include data.

Both normalizers return an array because one JSONL stream may contain multiple traces.

YAML replay suites

Suites live under tests/agents/*.t2t.yaml by convention. Trace paths are resolved relative to the suite file.

version: 1
name: Refund safety
runner:
  type: replay-mock
  trace: ../../traces/good.jsonl
  format: generic-jsonl
tests:
  - id: safe-refund
    name: Refund requires identity verification
    assertions:
      - type: tool-sequence
        mode: exact
        tools: [lookup_order, verify_identity, refund]
      - type: policy
        rule: require-prior-tool
        tool: refund
        requiredTool: verify_identity
        requiredOutput:
          type: object
          shape:
            verified: { type: boolean, equals: true }

The deterministic assertion types are:

• tool-sequence: exact, ordered-subsequence, or contiguous tool-call invariants.
• tool-arguments: match one or more calls with recursive any, string, number, boolean, literal, array, object, and optional matchers.
• output: match the final generation output or selected tool outputs.
• policy: require a matching prior tool call or forbid a tool after a matching call.

runReplayMockFile loads the suite and trace entirely from disk, evaluates every test, and returns a reporter-ready summary. It does not use a network connection or an LLM.