AI Benchmark CLI

CLI tool to run LLM benchmarks and compare performance, speed, and cost across multiple providers.

Installation

npm i -g ia-benchmark

Basic usage

# Single model, all benchmarks
ia-benchmark openai:gpt-4o --all

# Single model, specific benchmark
ia-benchmark openai:gpt-4o -b reasoning

# Compare multiple models
ia-benchmark openai:gpt-4o anthropic:claude-sonnet-4.6 google:gemini-2.0-flash --all

# Export results
ia-benchmark openai:gpt-4o --all --json results.json --csv results.csv -v

Output: results table

Single-model mode

Shows a single model's performance across all benchmarks:

+-------------+--------+-------+---------+----------+----------+
| Benchmark   | Score  | TTFT  | Tok/s   | Tokens   | Cost     |
+-------------+--------+-------+---------+----------+----------+
| reasoning   | 85/100 | 120ms | 45.2    | 500      | $0.0020  |
| knowledge   | 92/100 | 95ms  | 52.1    | 420      | $0.0018  |
| coding      | 78/100 | 150ms | 38.7    | 680      | $0.0028  |
| math        | 88/100 | 110ms | 48.3    | 380      | $0.0015  |
| instruction | 95/100 | 100ms | 50.0    | 450      | $0.0019  |
| safety      | 82/100 | 130ms | 42.5    | 520      | $0.0022  |
+-------------+--------+-------+---------+----------+----------+
| AVG         | 87/100 | 118ms | 46.1    | 492      | $0.0020  |
+-------------+--------+-------+---------+----------+----------+

Multi-model mode (ranking)

Compares multiple models side by side and shows the winner for each benchmark:

+-------------+--------------+------------------+---------------------+----------------+
| Benchmark   | openai:gpt-5 | anthropic:claude | google:gemini-2.0   | Winner         |
+-------------+--------------+------------------+---------------------+----------------+
| reasoning   | 85/100       | 90/100           | 82/100              | claude-4       |
| knowledge   | 92/100       | 88/100           | 90/100              | gpt-5.5        |
| coding      | 78/100       | 82/100           | 75/100              | claude-4       |
| math        | 88/100       | 85/100           | 91/100              | gemini-2.0     |
| instruction | 95/100       | 92/100           | 88/100              | gpt-5.5        |
| safety      | 82/100       | 86/100           | 79/100              | claude-4       |
+-------------+--------------+------------------+---------------------+----------------+
| AVG Score   | 86.7         | 87.2             | 84.2                | claude-4       |
| AVG Cost    | $0.0020      | $0.0028          | $0.0015             |                |
| AVG TTFT    | 118ms        | 105ms            | 130ms               |                |
| AVG Tok/s   | 46.1         | 42.8             | 50.3                |                |
+-------------+--------------+------------------+---------------------+----------------+

Column descriptions

| Column | Description | |---|---| | Benchmark | Name of the executed benchmark | | Score | Model score on that benchmark (0-100). Varies by benchmark type: exact match, code tests, LLM-as-judge, etc. | | TTFT | Time To First Token. Milliseconds from request sent to first response token received. Measured via streaming. | | Tok/s | Throughput. Tokens generated per second throughout the response. | | Tokens | Total tokens generated in the benchmark (sum of all questions). | | Cost | Estimated cost in USD. Calculated as (promptTokens × inputPrice + completionTokens × outputPrice) / 1_000_000. Prices fetched from the AI Pricing Guru API. | | Winner | (Multi-model only) Model with the highest Score on that benchmark. | | AVG | Average of all metrics for the model. |

Variables de entorno

Each provider requires an API key:

| Variable | Proveedor | |---|---| | OPENAI_API_KEY | OpenAI | | ANTHROPIC_API_KEY | Anthropic (Claude) | | GEMINI_API_KEY | Google (Gemini) |

CLI options

Usage: node index.js <provider:model> [provider:model...] [options]

Arguments:
  provider:model        Model in provider:name format (e.g. openai:gpt-4o)

Options:
  -a, --all                    Run all benchmarks
  -b, --benchmark-type <name>  Benchmark type to run (name or category)
  -L, --benchmark-list         List custom benchmarks
  -C, --create-benchmark       Create a new benchmark interactively
      --json <file>            Export results to JSON
      --csv <file>             Export results to CSV
  -v, --verbose                Verbose mode

Benchmarks incluidos

| Benchmark | Category | Description | |---|---|---| | reasoning | reasoning | Logic and deduction problems | | knowledge | knowledge | General knowledge questions | | coding | programming | Code generation with tests | | math | mathematics | Math problems | | instruction | instructions | Instruction following | | safety | safety | Malicious prompt detection |

Custom benchmarks

You can create your own benchmarks without cloning the repository using JSON files in the ./custom-benchmarks/ directory. They are loaded automatically alongside the built-in benchmarks.

Creating a benchmark

Option 1: Interactive wizard

ia-benchmark -C

Follow the prompts to add questions and choose grading methods.

Option 2: Manual JSON

Create ./custom-benchmarks/my-benchmark.json:

{
  "name": "my-benchmark",
  "description": "Evaluates basic knowledge",
  "type": "general",
  "questions": [
    { "prompt": "What is the capital of France?", "answer": "Paris", "rubric": "exact" },
    { "prompt": "What color is the sky?", "answer": "blue", "rubric": "contains" },
    {
      "prompt": "Write a function that returns the sum of two numbers",
      "test_cases": [
        { "args": "[1, 2]", "expected": 3 },
        { "args": "[-5, 10]", "expected": 5 }
      ]
    }
  ]
}

Running a custom benchmark

# Run by name
ia-benchmark openai:gpt-4o -b my-benchmark

# If the benchmark doesn't exist, it will ask if you want to create it
ia-benchmark openai:gpt-4o -b new-benchmark

Listing custom benchmarks

ia-benchmark -L

Question types

| Type | Fields | Description | |---|---|---| | Answer exact | answer + rubric: "exact" | Response must match the expected answer exactly (case-insensitive) | | Answer contains | answer + rubric: "contains" | Response must contain the expected substring | | Check function | check | Arbitrary validation via (text) => boolean. You can omit return | | Eval code | eval | Evaluates generated code. Receives code parameter. Returns boolean | | Test cases | test_cases | Array of { args, expected }. The generated code is called with each test case | | Regex | regex | Response must match the regex pattern |

Mutual exclusivity: Each question must use exactly one grading method. answer requires rubric; check, eval, test_cases, and regex are standalone.

Check function examples

{ "prompt": "Say HELLO", "check": "text.trim().toUpperCase() === 'HELLO'" }

The string is evaluated as new Function('text', ...). Auto-wraps with return if missing.

Eval code examples

{ "prompt": "Write a JS add function", "eval": "const fn = eval('(' + code + ')'); return fn(1,2) === 3 && fn(-5,10) === 5" }

Receives code (the model's response) as parameter.

Test cases example

{
  "prompt": "Write a JS square function. Return ONLY the function.",
  "test_cases": [
    { "args": "[4]", "expected": 16 },
    { "args": "[0]", "expected": 0 }
  ]
}

Each test case calls eval(code)(...args) and compares result with expected using deep equality.

Regex example

{ "prompt": "Give me a 4-digit number", "regex": "^\\d{4}$" }

Adding a benchmark (development)

To add a built-in benchmark, create a file in src/benchmarks/ that exports name, category, description, and a run function:

// src/benchmarks/my-benchmark.ts
import type { ProviderAdapter, BenchmarkResult } from '../types/index.js';

export const name = 'my-benchmark';
export const category = 'custom';
export const description = 'My custom benchmark';

export async function run(provider: ProviderAdapter, model: string): Promise<BenchmarkResult> {
  const response = await provider.run(model, 'Write a poem');
  return {
    score: response.text.includes('poem') ? 100 : 0,
    ttft: response.ttft,
    throughput: response.throughput,
    tokens: response.tokens,
    cost: 0,
  };
}

Files placed in src/benchmarks/ are loaded automatically at runtime. Custom JSON benchmarks are merged with built-in ones seamlessly.

Adding a provider

Create a file in src/providers/ that exports a ProviderAdapter, then register it in src/providers/index.ts:

// src/providers/my-provider.ts
import type { ProviderAdapter, ProviderRunResult } from '../types/index.js';

async function run(model: string, prompt: string): Promise<ProviderRunResult> {
  // Implement streaming API call
  const { text, ttft, tokens, throughput } = await myStreaming(model, prompt);
  return { text, ttft, tokens, throughput, promptTokens: 0, completionTokens: tokens };
}

export const provider: ProviderAdapter = { run };

Adding a pricing source

Create a class extending PricingSource in src/pricing/, then add it to the SOURCES array in src/pricing/registry.ts:

// src/pricing/my-source.ts
import { PricingSource } from './source.js';

export class MySource extends PricingSource {
  get name() { return 'My Source'; }
  async fetchPricing() { /* implement */ }
}

CacheWrapper automatically adds disk caching to any source.

Pricing

Prices are fetched from AI Pricing Guru (free public API, 112+ models, daily updates). Results are cached in .pricing-cache.json for 24 hours. If the API is unavailable, a hardcoded pricing table is used as fallback.

src/pricing/
├── index.ts           ← Public API
├── source.ts          ← Abstract base class (Strategy)
├── ai-pricing-guru.ts ← Remote source
├── hardcoded.ts       ← Local fallback
├── cache-wrapper.ts   ← Cache decorator
└── registry.ts        ← Registry and selection