perfshield

perfshield is a build-system-agnostic benchmarking tool for comparing two versions of a JavaScript library across JS runtimes. It borrows statistical ideas from Tachometer (confidence intervals, adaptive sampling) and focuses on runtime JS engines (Node/V8 today).

What it does

• Builds your benchmark bundle using your own build command.
• Saves a baseline bundle (prepare).
• Builds the current bundle and compares it to the baseline (compare).
• Reports results in console and/or JSON.
• Exits with code 1 when a regression is detected (both relative and absolute CIs exclude 0 in the slower direction).

Requirements

• Node.js (used for the CLI and the Node engine).
• A build command that emits a single ESM benchmark bundle.

Install

pnpm add -D perfshield

Quick start

1. Create a benchmark module that exports benchmarks:

export const benchmarks = [
  {
    name: "sum",
    iterations: 1,
    async fn() {
      let total = 0;
      for (let i = 0; i < 10_000; i += 1) {
        total += i;
      }
      return total;
    },
  },
];

2. Create a perfshield.config.json:

{
  "artifactsDir": ".perfshield",
  "build": {
    "command": "pnpm run build:bench",
    "output": "dist/bench.js"
  },
  "engines": [
    {
      "name": "node",
      "command": "node"
    }
  ],
  "sampling": {
    "minSamples": 30,
    "timeoutMs": 10000,
    "conditions": [0]
  },
  "report": {
    "formats": ["console", "json"]
  }
}

3. Run the workflow:

perfshield prepare --config perfshield.config.json
perfshield compare --config perfshield.config.json

4. (Optional) Calibrate sampling defaults based on the prepared baseline:

perfshield calibrate --config perfshield.config.json

This prints a JSON snippet with recommended sampling values you can paste into your config.

Benchmark bundle contract

The build output must be a single ESM file that exports:

export const benchmarks = [
  {
    name: "example",
    fn: async () => {},
    setup: async () => {},
    teardown: async () => {},
    unit: "ms",
    iterations: 1,
    metadata: { tag: "optional" },
  },
];

Rules:

• benchmarks is required.
• fn is required.
• setup/teardown run once per sample (per benchmark, per version).
• If setup throws, the sample fails and the run is aborted.
• iterations repeats fn inside a single sample to reduce noise.

Artifacts

• prepare stores a baseline bundle in artifactsDir (default .perfshield).
• compare uses the stored baseline and the newly built bundle.

Reports

Supported formats:

• console: human‑readable summary.
• json: machine‑readable report.

If any benchmark shows a regression (both relative and absolute CIs exclude 0 in the slower direction), the process exits with code 1.

Examples

See examples/simple for a minimal setup you can run locally.