Arena

Framework-agnostic multi-agent scenario simulation and evaluation.

Arena is a measurement layer for multi-agent systems. Define a scenario suite once, run it under any orchestration strategy, and get comparable quality / safety / autonomy / cost / latency metrics — with statistical significance tests and CI regression gates. It is deliberately framework-agnostic: no LangGraph, no Vercel AI SDK, no SaaS platform required. Bring your own agents through a thin adapter, or use the built-in scripted adapters for fully reproducible runs.

Same suite, many orchestration strategies, comparable numbers.

Why Arena?

The existing JS/TS evaluation tools all lock you into one framework or platform:

• @langwatch/scenario — bound to LangWatch SaaS and the LangGraph ecosystem.
• reddial — tied to the Vercel AI SDK.
• @perceo/perceo — coupled to Temporal + Supabase infrastructure.
• @agentesting/agentest — built around a specific agent runtime.

None of them are a framework-agnostic measurement layer. None of them let you run the same scenario suite under two orchestration strategies and compare the results head-to-head. And none of them measure autonomy — the human-in-the-loop ratio that matters for agent systems that collaborate with people.

Arena fills that gap:

• Framework-agnostic — AgentAdapter is the only contract between your agents and the engine.
• Deterministic — seeded PRNG and a message-bus scheduler mean the same input always produces the same event stream.
• Comparable — run one suite under strategy A and strategy B; get a side-by-side table with paired t-test and Wilcoxon signed-rank significance.
• Autonomy-aware — measures the human intervention rate and autonomous decision share out of the box.
• CI-ready — RegressionGate thresholds with JSON report and non-zero exit codes, plus a ready-made GitHub Action.

Features

| Area | What you get | | --- | --- | | Scenario | Roles (agents + human roles), task goal, environment state, initial messages, max turns, timeout, per-scenario seed. | | ScenarioSuite | Write suites in YAML, JSON or TypeScript. Seeded for reproducibility. | | SimulationRunner | Message-bus-driven deterministic scheduler: turn-based execution, full event log, timeout and deadlock detection, scripted customer turns. | | AgentAdapter | Framework-agnostic contract. Adapters for OpenAI Agents SDK, LangGraph.js, Vercel AI SDK, plus createScriptedAdapter / createFunctionAdapter / createHumanAdapter. | | MetricsEngine | Five dimensions: quality, safety, autonomy, cost, latency. Autonomy = autonomous decisions / (autonomous + human interventions). | | ComparisonReport | Same suite under two strategies, side-by-side table with Δ, Δ%, p-values (paired t-test + Wilcoxon) and significance verdicts. | | RegressionGate | Threshold assertions over metrics, JSON-serializable results, non-zero exit code for CI. Bundled GitHub Action. | | CLI | arena run, arena compare, arena report (bin name arena). | | Packaging | ESM + CJS, full type declarations, zero runtime dependencies (optional adapter deps are peer + optional). |

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        Scenario Suite (YAML / TS)                   │
│              scenarios: roles, goal, env, messages, turns           │
└────────────────────────────────┬────────────────────────────────────┘
                                 │ loads
┌────────────────────────────────▼────────────────────────────────────┐
│                        SimulationRunner                             │
│   message bus ──► turn scheduler ──► timeouts ──► deadlock detect   │
│   emits SimulationEvent[] ──► ScenarioRunResult[]                   │
└────────────────────────────────┬────────────────────────────────────┘
        ┌────────────────────────┴────────────────────────┐
        ▼                                                  ▼
┌───────────────┐                                  ┌───────────────┐
│ Strategy A    │                                  │ Strategy B    │
│ AgentAdapter  │                                  │ AgentAdapter  │
│ (scripted /   │                                  │ (scripted /   │
│  OpenAI SDK / │                                  │  OpenAI SDK / │
│  LangGraph /  │                                  │  LangGraph /  │
│  AI SDK /     │                                  │  AI SDK /     │
│  custom)      │                                  │  custom)      │
└───────────────┘                                  └───────────────┘
        └────────────────────────┬────────────────────────┘
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        MetricsEngine                                 │
│   quality │ safety │ autonomy │ cost │ latency                       │
│   (rubric / PII / human-ratio / token price / latency quantiles)     │
└────────────────────────────────┬────────────────────────────────────┘
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│   ComparisonReport      RegressionGate          CLI + GitHub Action │
│   Δ, Δ%, p-values       min/max thresholds      arena run/compare/   │
│   t-test + Wilcoxon     exit 0/1 for CI         report               │
└─────────────────────────────────────────────────────────────────────┘

Installation

npm install @noahisarider/arena

The core runtime has zero dependencies. Optional adapters pull their framework lazily — install them only if you use them:

npm install agents              # OpenAI Agents SDK adapter
npm install ai @ai-sdk/openai   # Vercel AI SDK adapter
npm install @langchain/langgraph

Quick start

1. Write a scenario suite

suite.yaml:

id: customer-service
seed: 7
scenarios:
  - id: billing-refund
    goal: Resolve the duplicate charge and issue a refund.
    maxTurns: 8
    timeoutMs: 30000
    roles:
      - id: tier1
        name: Tier 1 Support
        systemPrompt: First line of support for billing inquiries.
      - id: manager
        name: Manager
        isHuman: true
        systemPrompt: Authorized to approve refunds.
    initialMessages:
      - id: m1
        role: user
        from: customer
        to: tier1
        content: "I was charged twice for my subscription."

2. Define a strategy

strategy-autonomous.ts:

import { createScriptedAdapter } from '@noahisarider/arena';

export const strategy = {
  name: 'autonomous',
  adapters: {
    tier1: createScriptedAdapter('tier1', [
      { match: /billing|charge|double/i, reply: 'Refund issued.', done: true },
    ]),
    manager: createScriptedAdapter('manager', [
      { match: /.*/, reply: 'Standing by.' },
    ]),
  },
  runner: { turnOrder: 'conversational' },
  metrics: { pricing: { inputPerMTok: 0.5, outputPerMTok: 1.5 } },
};

3. Run it

arena run suite.yaml --strategy strategy-autonomous.ts

Scenario       | Status    | Quality | Safety | Autonomy | Interv. | Cost($)   | Latency(ms)
---------------|-----------|---------|--------|----------|---------|-----------|------------
billing-refund | completed |   1.000 |  1.000 |    1.000 |   0.000 |   0.00000 |            0

Suite "customer-service" (seed 7) — 1 scenario
  quality   mean 1.000 (p95 1.000)
  safety    mean 1.000 (p95 1.000)
  autonomy  mean 1.000 (intervention rate 0.000)
  cost      mean $0.00000 total (p95 $0.00000)
  latency   mean 0ms per run (p95 0ms)

4. Compare two strategies

arena compare suite.yaml \
  --a strategy-human-first.ts \
  --b strategy-autonomous.ts --md

| Metric | human-first | autonomous | Δ | Δ % | p (t) | p (w) | Better | | --- | --- | --- | --- | --- | --- | --- | --- | | Quality | 1.0000 | 1.0000 | 0.0000 | +0.0% | 1.0000 | 1.0000 | TIE | | Safety | 1.0000 | 1.0000 | 0.0000 | +0.0% | 1.0000 | 1.0000 | TIE | | Autonomy | 0.8333 | 1.0000 | +0.1667 | +20.0% | 0.4226 | 1.0000 | TIE | | Cost | 0.0000 | 0.0000 | 0.0000 | +0.0% | 1.0000 | 1.0000 | TIE | | Latency | 0.3333 | 0.3333 | 0.0000 | +0.0% | 1.0000 | 1.0000 | TIE |

5. Enforce it in CI

gate.yaml:

thresholds:
  aggregates.quality.mean:
    min: 0.8
  aggregates.autonomy.mean:
    min: 0.4

arena run suite.yaml --strategy strategy.ts --gate gate.yaml && echo "gate passed"

Failing gates exit with code 1. See the bundled GitHub Action below for a turnkey CI step.

CLI

arena run <suite> [options]                    Run a suite with one strategy.
arena compare <suite> --a <a.ts> --b <b.ts>    Run one suite under two strategies.
arena report <results.json> [options]          Render a saved results file.

| Option | Meaning | | --- | --- | | --strategy <file> | Strategy config (TS/JS/YAML/JSON) with adapters. | | --a <file> / --b <file> | Strategy configs A and B for compare. | | --out <file> | Write full report data as JSON. | | --json | Print report data as JSON to stdout. | | --md | Print a Markdown table (compare, report). | | --gate <file> | Regression gate thresholds (YAML/JSON/TS). Exit 1 on failure. | | --seed <n> | Override the simulation seed. | | --alpha <n> | Significance level for compare (default 0.05). | | --metric <k,...> | Limit comparison to quality,safety,autonomy,cost,latency. |

Scenario DSL

Suite

| Field | Type | Description | | --- | --- | --- | | id | string | Required. Unique suite id. | | name / description | string | Optional metadata. | | seed | number | Default seed for scenarios without their own. | | scenarios | Scenario[] | Required, non-empty. |

Scenario

| Field | Type | Description | | --- | --- | --- | | id | string | Required. Unique within the suite. | | goal | string | Required. What the agents should accomplish. | | roles | Role[] | Required, non-empty. | | environment | object | Initial shared state, merged into every adapter's view. | | initialMessages | Message[] | Messages present at simulation start. | | maxTurns | number | Hard cap on turns before the run stops. | | timeoutMs | number | Wall-clock timeout in ms. | | seed | number | Per-scenario seed (falls back to suite seed). | | metadata | object | Free-form. Used by default scorers (see below). |

Role

| Field | Type | Description | | --- | --- | --- | | id | string | Required. Role/agent id. | | name / description | string | Optional. | | systemPrompt | string | Optional system instructions. | | isHuman | boolean | When true, the role is played by a human and drives the autonomy metric. | | capabilities | string[] | Optional, for rubric scorers and adapters. |

Message

| Field | Type | Description | | --- | --- | --- | | id | string | Stable id. | | role | 'system' \| 'user' \| 'assistant' \| 'tool' \| 'human' | Message role. | | from | AgentId \| 'system' \| 'customer' | Sender. | | to | AgentId \| 'customer' \| 'human' \| null | Recipient; null broadcasts. | | content | string | Message body. | | metadata | object | Optional payload. |

Scenario metadata hooks

The default scorers read two metadata keys:

• qualityChecks — rubric items evaluated against all message text:

  metadata:
    qualityChecks:
      - label: issued-refund
        match: "refund"        # substring, RegExp, or function
        weight: 2

• blockedPhrases — safety violations (alongside default PII patterns for email, phone, SSN, credit card):

  metadata:
    blockedPhrases:
      - "share your password"
      - "credit card number"

• customerScript — deterministic scripted customer turns injected by turn:

  metadata:
    customerScript:
      - atTurn: 2
        content: "I still get 'invalid credentials'."
        to: tier2

Turn scheduling

• turnOrder: 'conversational' (default) — follow output.to handoffs, fall back to round-robin.
• turnOrder: 'round-robin' — cycle through roles in declaration order.

Handoff to 'human' requires a role with isHuman: true; otherwise the run ends as a deadlock.

Metrics

Autonomy

Defined as the share of decisions made autonomously:

autonomy = autonomous decisions / (autonomous decisions + human interventions)
interventionRate = human interventions / (autonomous decisions + human interventions)

A decision is any assistant message; an intervention is any human message. A strategy that escalates everything to a human manager scores low on autonomy; a strategy that resolves everything itself scores 1.0.

Other dimensions

| Dimension | Meaning | | --- | --- | | quality | Weighted rubric score. Defaults to completion + agent-reply + no-error checks when no qualityChecks metadata is present. | | safety | Fraction of agent/human messages that avoid PII and blockedPhrases. | | cost | USD from token usage, either adapter-reported or computed via metrics.pricing. | | latency | Total run time plus per-turn mean and p50/p90/p95/p99 from agent-turn-end events. |

Significance tests

ComparisonReport runs a paired t-test and a Wilcoxon signed-rank test between the two strategies' per-scenario metric vectors, and marks a metric as A, B or TIE at the configured alpha.

Programmatic API

import {
  SimulationRunner,
  MetricsEngine,
  RegressionGate,
  createScriptedAdapter,
} from '@noahisarider/arena';

const runner = new SimulationRunner({ turnOrder: 'conversational' });
const run = await runner.runSuite(suite, adapters, { seed: 7 });

const engine = new MetricsEngine({ pricing: { inputPerMTok: 0.5, outputPerMTok: 1.5 } });
const metrics = engine.metricsForSuite(run, suite);

const gate = RegressionGate.fromYaml(`
thresholds:
  aggregates.quality.mean: { min: 0.8 }
`);
const result = gate.evaluate(metrics);
console.log(result.passed); // boolean

Adapters

import { createOpenAIAgentsAdapter } from '@noahisarider/arena/adapters';
import { createLangGraphAdapter } from '@noahisarider/arena/adapters';
import { createAiSdkAdapter } from '@noahisarider/arena/adapters';

• createOpenAIAgentsAdapter({ role, agent }) — wraps an OpenAI Agents SDK Agent (lazy agents import).
• createLangGraphAdapter({ role, graph | invoke, formatResult? }) — wraps a compiled LangGraph graph or an invoke function.
• createAiSdkAdapter({ role, model, system?, tools? }) — wraps a Vercel AI SDK language model (lazy ai import).
• createScriptedAdapter(role, rules, opts) — deterministic rule-driven adapter for reproducible runs.
• createFunctionAdapter(role, fn) — wrap any function as an adapter.
• createHumanAdapter(role, reply?) — a human role that emits human-intervention events.
• defineAdapter(adapter) — identity helper with full type checking for hand-written adapters.

GitHub Action

The repo ships a ready-made composite action for CI gating.

steps:
  - uses: noahisarider/arena/action@v1
    with:
      suite: demo/customer-service/suite.yaml
      strategy: demo/customer-service/strategies/human-first.ts
      gate: demo/customer-service/gate.yaml

For a two-strategy comparison:

steps:
  - uses: noahisarider/arena/action@v1
    with:
      suite: demo/customer-service/suite.yaml
      a: demo/customer-service/strategies/human-first.ts
      b: demo/customer-service/strategies/autonomous.ts
      gate: demo/customer-service/gate.yaml

Inputs: suite (required), strategy (for run), a + b (for compare), gate (required), version (npm package version, default latest).

Comparison with existing tools

| | @noahisarider/arena | @langwatch/scenario | reddial | @perceo/perceo | @agentesting/agentest | | --- | --- | --- | --- | --- | --- | | Framework-agnostic | Yes — AgentAdapter | No — LangGraph ecosystem | No — Vercel AI SDK | No — Temporal + Supabase | No — specific agent runtime | | No SaaS/platform dependency | Yes | No — LangWatch SaaS | Yes | No — infra required | Yes | | Same suite across strategies | Yes — arena compare | Partial | No | No | No | | Autonomy metric | Yes — intervention rate / decision share | No | No | No | No | | Deterministic & seeded | Yes — seeded PRNG + message bus | Partial | Partial | Partial | Partial | | Regression gate + CI | Yes — RegressionGate + GitHub Action | Partial | No | No | No | | ESM + CJS + types | Yes | — | — | — | — | | Zero runtime deps | Yes | No | No | No | No |

Demo

A complete, runnable example lives in demo/customer-service: a customer service center with 3 roles (Tier 1, Tier 2, human Manager) and an A/B comparison between a human-first escalation strategy and a fully autonomous strategy.

arena run demo/customer-service/suite.yaml \
  --strategy demo/customer-service/strategies/human-first.ts

arena compare demo/customer-service/suite.yaml \
  --a demo/customer-service/strategies/human-first.ts \
  --b demo/customer-service/strategies/autonomous.ts --md

Development

npm install
npm run check     # lint + typecheck + test + build
npm run coverage  # test with coverage thresholds (90%+)
npm run attw      # verify ESM/CJS/type exports

CI runs lint, typecheck, coverage, build and export verification on every push/PR. Publishing is automatic when a v* tag is pushed (with npm provenance).

License

MIT