Reprobench

Reprobench is a reproducible benchmark runner powered by Nix flakes.

It helps you run, compare, guard, and publish benchmark results from pinned development environments.

Why Reprobench?

Benchmark results are only meaningful when run in a consistent environment. Reprobench ties together:

• Pinned environments via Nix flakes — no more "it was faster on my machine"
• Structured results — benchmarks are stored as JSON for easy diffing and archiving
• Comparison — compare current results against a baseline with percentage deltas
• Markdown reports — generate tables ready to paste into GitHub READMEs or PRs
• CI guards — detect performance or size regressions automatically
• Extensible — designed to support multiple runtimes, languages, and environments

Installation

pnpm add -D reprobench

Or install globally:

npm install -g reprobench

Quick Start

pnpm add -D reprobench
pnpm reprobench init
pnpm reprobench run
pnpm reprobench compare
pnpm reprobench report

Commands

reprobench init

Initialize a reprobench.config.json in the current directory. If flake.nix is found, defaults to manager: "nix".

reprobench init
reprobench init --force            # overwrite existing config
reprobench init --manager nix      # force Nix manager
reprobench init --manager local    # force local manager

Also creates the bench/results/ directory.

reprobench run

Run all benchmark tasks defined in the config.

reprobench run

Each task's stdout is saved as JSON (or raw text) to the configured output path.

reprobench compare [baseline] [current]

Compare two benchmark result files.

reprobench compare          # uses config paths
reprobench compare bench/results/baseline.json bench/results/latest.json
reprobench compare --json   # JSON output

Example output:

size
✓ single-small: 166 -> 139 bytes (-16.27%)

speed
✓ encode batch: 7700 -> 16000 ops/s (+107.79%)

reprobench report

Generate a Markdown report from benchmark results.

reprobench report
reprobench report --input bench/results/latest.json --output bench/results/report.md

Example output:

## Benchmark Results

| group | benchmark    |  value | unit  |
| ----- | ------------ | -----: | ----- |
| size  | single-small |    139 | bytes |
| speed | encode batch | 16,000 | ops/s |

reprobench guard

Check guard conditions. Exits with code 1 if any guard fails.

reprobench guard

Example output:

✓ single-small: 139 bytes <= 200 bytes
✗ encode batch: 12000 ops/s < 14000 ops/s

reprobench doctor

Check the project configuration and environment.

reprobench doctor

Verifies config existence, schema validity, file paths, package.json scripts, and more.

Configuration

Nix-powered benchmark execution

Set environment.manager to "nix" to run benchmark tasks inside the Nix dev shell. This ensures the benchmark runs in the exact pinned environment defined by your flake.nix.

{
  "environment": {
    "manager": "nix",
    "flake": ".",
    "shell": "default"
  },
  "tasks": {
    "bench": {
      "command": "pnpm bench",
      "output": "bench/results/latest.json"
    }
  }
}

reprobench run will execute:

nix develop .#default --command bash -lc "pnpm bench"

Full configuration example

{
  "$schema": "https://reprobench.dev/schema.json",
  "project": "my-library",
  "environment": {
    "manager": "nix",
    "flake": ".",
    "shell": "default"
  },
  "tasks": {
    "bench": {
      "command": "pnpm bench",
      "output": "bench/results/latest.json"
    }
  },
  "compare": {
    "baseline": "bench/results/baseline.json",
    "current": "bench/results/latest.json"
  },
  "report": {
    "input": "bench/results/latest.json",
    "output": "bench/results/report.md"
  },
  "guards": {
    "benchmarks": {
      "encode batch": {
        "min": 14000,
        "unit": "ops/s"
      },
      "single-small": {
        "max": 200,
        "unit": "bytes"
      }
    }
  }
}

Benchmark Result Format

Benchmark tasks should output JSON to stdout in this format:

{
  "project": "my-library",
  "timestamp": "2024-01-01T00:00:00Z",
  "benchmarks": [
    {
      "name": "encode single-small",
      "group": "speed",
      "value": 16038,
      "unit": "ops/s"
    },
    {
      "name": "batch-homogeneous-256",
      "group": "size",
      "value": 5316,
      "unit": "bytes"
    }
  ]
}

Supported units and directions

| Unit | Direction | | ------- | ---------------- | | bytes | lower-is-better | | ms | lower-is-better | | s | lower-is-better | | ops/s | higher-is-better | | count | higher-is-better | | (other) | higher-is-better |

You can override the direction per entry using the direction field:

{
  "name": "my-bench",
  "value": 42,
  "unit": "custom",
  "direction": "lower-is-better"
}

CI Usage

Check CI

name: Check

on:
  pull_request:
  push:
    branches:
      - main

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: DeterminateSystems/nix-installer-action@main
      - run: nix develop --command pnpm install --frozen-lockfile
      - run: nix develop --command pnpm check

Benchmark CI

name: Benchmark

on:
  pull_request:
  push:
    branches:
      - main

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: DeterminateSystems/nix-installer-action@main
      - run: nix develop --command pnpm install --frozen-lockfile
      - run: nix develop --command pnpm reprobench run
      - run: nix develop --command pnpm reprobench guard

If environment.manager is set to "nix", Reprobench itself will enter the configured Nix dev shell before running each benchmark task — no need to wrap the run step manually.

Nix Usage

This project provides a Nix flake for reproducible development:

nix develop
pnpm install
pnpm check

The dev shell includes Node.js 22, pnpm, and git.

Run reprobench directly via Nix

You can run reprobench without installing it via npm:

nix run github:minagishl/reprobench -- --help
nix run github:minagishl/reprobench -- init
nix run github:minagishl/reprobench -- compare baseline.json latest.json

Formatter / Linter

This project uses Oxc tools for formatting and linting.

• Formatter: Oxfmt
• Linter: Oxlint

pnpm fmt
pnpm fmt:check
pnpm lint
pnpm lint:fix

No Biome, ESLint, or Prettier configuration files are present in this project.

Example Output

Compare

size
✓ single-small: 166 -> 139 bytes (-16.27%)

speed
✓ encode batch: 7700 -> 16000 ops/s (+107.79%)

Report (Markdown)

## Benchmark Results

| group | benchmark    |  value | unit  |
| ----- | ------------ | -----: | ----- |
| size  | single-small |    139 | bytes |
| speed | encode batch | 16,000 | ops/s |

Compare JSON

{
  "comparisons": [
    {
      "name": "single-small",
      "group": "size",
      "unit": "bytes",
      "baseline": 166,
      "current": 139,
      "delta": -27,
      "deltaPercent": -16.27,
      "direction": "lower-is-better",
      "improved": true
    }
  ]
}

License

This project is licensed under the MIT License - see the LICENSE file for details.