@composy/benchmark

v0.0.2

Published

8 days ago

Benchmark framework for Node.js projects with TypeScript API, CLI, reports, history storage, and CI regression analysis.

0High
0Medium
0Low

swimly

benchmark ldesign performance tinybench typescript

`@composy/benchmark`

面向 Node.js 与 TypeScript 项目的基准测试工具包，提供 tinybench 封装、CLI、报告导出、历史记录与阈值校验。 @composy/benchmark is a benchmark toolkit for Node.js and TypeScript projects, with a tinybench-based API, CLI, report export, history storage, and threshold checks.

功能概览 / Features

统一的 TypeScript API 与 CLI：既能在代码中直接运行 benchmark，也能通过 ldbench 扫描 *.bench.ts 文件。Unified TypeScript API and CLI: run benchmarks directly in code or scan *.bench.ts files with ldbench.
支持 console、json、markdown、csv 四种报告格式。Supports console, json, markdown, and csv reports.
支持历史记录保存、报告对比与阈值失败中断。Supports history persistence, report comparison, and threshold-based failure checks.
支持 run --json、doctor --json、history --json、compare --json、clean --json 与 config-validate --json，更适合 CI 和脚本消费。Supports run --json, doctor --json, history --json, compare --json, clean --json, and config-validate --json for CI and scripts.
提供报告摘要与结果索引 API，方便上层 UI、自动化和自定义分析复用。Provides report summary and result index APIs for UI, automation, and custom analysis.
内置本地 Dashboard：首轮诊断、历史浏览、经校验的 JSON 导入导出、报告健康度、环境感知对比、多指标趋势，以及从浏览器运行已选配置。Includes a local dashboard for first-run diagnostics, validated JSON import/export, report health, environment-aware comparisons, multi-metric trends, and configured runs from the browser.
当前 CLI 已接通 pattern、ignore、outputDir、historyDir、storage、reporters、defaults、thresholds、env、parallel、plugins、git、ci。The current CLI actively consumes these settings, with storage.path taking precedence over the legacy historyDir fallback. Git metadata controls and CI regression reporting are applied when their respective settings are enabled.
locale configures the shared translator and is respected by benchmark modules and plugins calling t(). CLI text, the embedded Dashboard page, console/Markdown reports, CI summaries, and annotations consistently follow the selected language; JSON and CSV schemas remain language-neutral.

审计与改进 / Audit And Improvements

本轮整理重点修复了以下问题。This maintenance pass focused on the following issues.

旧版 README、benchmark.schema.json、示例配置与当前实现脱节，已按真实配置结构重写。The old README, schema, and sample configs were outdated and are now aligned with the real implementation.
config.parallel 与 config.env 之前只有类型定义，CLI 运行时没有实际消费，现在已接入 run 命令。 config.parallel and config.env previously existed only in types; they are now consumed by the run command.
多个运行时模块存在历史遗留乱码注释、全局 process 引用和 ESLint 规则问题，已统一整理。Several runtime modules had legacy mojibake comments, global process usage, and ESLint issues; those have been cleaned up.
补齐了关键模块的中文注释，统一了更清晰的类型与错误处理写法。Key modules now have clearer Chinese comments and more explicit typing and error handling.
新增只读 doctor 诊断和机器可读 JSON 输出，便于本地排障与 CI 集成。A read-only doctor command and machine-readable JSON output improve local troubleshooting and CI integration.
CLI 命令注册改为 registry 结构，并导出 createBenchmarkCli() 供宿主注入自定义命令。CLI command registration now uses a registry and exposes createBenchmarkCli() for host-defined commands.

安装 / Installation

pnpm add -D @composy/benchmark

要求 / Requirements:

Node.js >= 18
推荐使用 pnpm pnpm is recommended.

CLI 快速开始 / CLI Quick Start

初始化工作区配置与示例 benchmark。Initialize workspace config and a sample benchmark.

pnpm exec ldbench init

运行当前工作区的 benchmark 文件。Run benchmark files in the current workspace.

pnpm exec ldbench run
pnpm exec ldbench run --json

诊断当前工作区配置、benchmark 文件发现和报告目录。Diagnose workspace config, benchmark discovery, and report directories.

pnpm exec ldbench doctor
pnpm exec ldbench doctor --json
pnpm exec ldbench config-validate --json
pnpm exec ldbench config-validate --config .ldesign/benchmark.config.ts --json

输出文件报告。Write file-based reports.

pnpm exec ldbench run --report json --out .ldesign/benchmark/reports
pnpm exec ldbench run --report markdown --out .ldesign/benchmark/reports
pnpm exec ldbench run --report csv --out .ldesign/benchmark/reports

写入历史记录并查看历史。Persist and inspect history.

pnpm exec ldbench run --history
pnpm exec ldbench history --limit 5
pnpm exec ldbench history --limit 5 --json
pnpm exec ldbench history --tag release --limit 5

启动可视化工作台。默认只绑定本机地址，页面可以诊断首轮配置、查看或导入历史、导出 JSON、检查报告质量和运行环境、选择两份报告对比、跟踪同一任务的多指标趋势，并运行当前配置。Start the visual dashboard. It binds to loopback by default and can diagnose the workspace, inspect or import history, export JSON, inspect report health and runtime compatibility, compare two reports, track multi-metric task trends, and run the current configuration.

pnpm exec ldbench dashboard
pnpm exec ldbench dashboard --config .ldesign/benchmark.config.ts --port 4178
pnpm exec ldbench dashboard --read-only

dashboard 会在终端输出访问地址。浏览器不能传入命令、配置路径或 benchmark 路径；导入 JSON 会经过报告校验，单次最多 5 MiB。--read-only 会禁用运行和导入，适合只分享报告；只有显式传入 --host 0.0.0.0 时才会监听非本机网络。 The command prints the URL in the terminal. The browser cannot supply a command, config path, or benchmark path; imported JSON is validated and limited to 5 MiB. --read-only disables browser-triggered runs and imports; it listens beyond the local machine only when --host 0.0.0.0 is explicitly supplied.

对比两份 JSON 报告。Compare two JSON reports.

pnpm exec ldbench compare .ldesign/benchmark/history/baseline.json .ldesign/benchmark/history/current.json
pnpm exec ldbench compare baseline.json current.json --threshold 3 --json

按阈值文件校验。Validate results against a threshold file.

pnpm exec ldbench run --threshold ./thresholds.json
pnpm exec ldbench run --threshold ./thresholds.json --json

run --json 会输出 { ok, command, report, historyId, threshold, plugins, health }。health 会列出失败/超时、RME和尾延迟信号；任一任务失败或超时、或阈值未通过时，ok 为 false 且进程以非零状态退出。配置加载、文件扫描或运行错误会输出 { ok: false, command: 'run', error }。 run --json emits { ok, command, report, historyId, threshold, plugins, health }. health includes failed/timeout, RME, and tail-latency signals; a failed or timed-out task, or a threshold failure, makes ok false and the process exits non-zero. Config loading, discovery, and runtime errors emit { ok: false, command: 'run', error }.

--out 指向具体文件（例如 report.json）时只能同时使用一种文件报告格式；需要导出 JSON、Markdown 和 CSV 时，请将 --out 指向目录。 When --out points to a file (for example report.json), use exactly one file report format. Point it at a directory when exporting JSON, Markdown, and CSV together.

按 suite 或任务标签预览清理范围。Preview cleanup for selected suites or task tags.

pnpm exec ldbench clean --suite api --tag release --dry-run

--suite 和 --tag 均支持逗号分隔的多个值；指定两个筛选项时，报告需要同时命中 suite 和 tag。Markdown/CSV 报告会根据同名 JSON 报告的元数据判断是否命中，缺少同名 JSON 时会保守保留。Both options accept comma-separated values. When both are present, a report must match both its suite and tag filters. Markdown/CSV reports use the metadata in their sibling JSON report and are kept when no sibling JSON exists.

监听文件变化并自动重跑。Watch files and rerun automatically.

pnpm exec ldbench run --watch

Benchmark 文件约定 / Benchmark Module Contract

CLI 默认会加载 *.bench.ts 或 *.bench.js 模块，并调用它们的默认导出函数。The CLI loads *.bench.ts and *.bench.js modules and invokes their default export.

import type { BenchmarkConfig, BenchmarkRunner } from '@composy/benchmark'
import { createBenchmark } from '@composy/benchmark'

export default async function registerBenchmarks(
  runner: BenchmarkRunner,
  config: BenchmarkConfig
): Promise<void> {
  const suite = createBenchmark('array-ops', config.defaults)

  suite.add('push', () => {
    const values: number[] = []
    for (let index = 0; index < 1_000; index += 1) values.push(index)
  })

  suite.add('concat', () => {
    const values = Array.from({ length: 1_000 }, (_, index) => index)
    void values.concat(1_001)
  })

  runner.addSuite('array-ops', suite)
}

约定 / Contract:

默认导出函数签名为 (runner, config) => void | Promise<void>。The default export signature is (runner, config) => void | Promise<void>.
通过 runner.addSuite(name, suite) 注册一个 benchmark 套件。Register a suite through runner.addSuite(name, suite).
config.defaults 会作为套件级默认参数传入 createBenchmark。 config.defaults can be forwarded into createBenchmark as suite-level defaults.

配置示例 / Configuration Example

推荐在 .ldesign/benchmark.config.ts 中使用 TypeScript 配置。Using a TypeScript config in .ldesign/benchmark.config.ts is the recommended setup.

import type { BenchmarkConfig } from '@composy/benchmark'

const config = {
  pattern: ['src/**/*.bench.ts', '.ldesign/benchmarks/**/*.bench.ts'],
  ignore: ['**/node_modules/**', '**/dist/**', '**/.vitest-build/**'],
  outputDir: '.ldesign/benchmark/reports',
  storage: {
    type: 'json',
    path: '.ldesign/benchmark/history',
    retention: {
      maxCount: 100,
      maxAge: 30,
      suites: [],
      tags: [],
      branches: [],
      names: [],
    },
  },
  reporters: ['console', 'json', 'markdown'],
  env: {
    NODE_ENV: 'benchmark',
  },
  thresholds: {
    'array-ops::push': {
      minOpsPerSecond: 200_000,
      maxRme: 5,
    },
    'array-ops::concat': {
      maxAvgTime: 0.05,
      maxP95: 0.08,
    },
  },
  defaults: {
    time: 1_000,
    iterations: 10,
    warmup: 5,
    collectMemory: false,
    retainSamples: true,
    concurrency: 'task',
    timeout: 30_000,
    retries: 1,
  },
  parallel: {
    enabled: true,
    maxWorkers: 4,
    isolate: true,
  },
  git: {
    enabled: true,
    trackCommit: true,
    trackBranch: true,
  },
  ci: {
    enabled: true,
    provider: 'github',
    failOnRegression: true,
    regressionThreshold: 5,
    annotations: true,
  },
  plugins: ['statistics', 'trend-analysis'],
} satisfies BenchmarkConfig

export default config

defaults.timeout 为每次 benchmark 任务调用的超时上限；超时会生成 timeout 状态并保留错误信息。defaults.retries 表示单次调用失败后允许的额外尝试次数，重试耗尽后才将任务标记为失败。两项配置也可以直接传给 createBenchmark，非法的非正超时或负重试次数会在创建 benchmark 时立即拒绝。

defaults.timeout limits each benchmark task invocation; a timeout is reported with status: 'timeout'. defaults.retries is the number of extra attempts after a failed invocation. The same options can be passed to createBenchmark, and invalid non-positive timeouts or negative retry counts are rejected before the benchmark starts.

配置字段速查 / Config Field Summary

| 字段 / Field | 类型 / Type | 默认值 / Default | 当前状态 / Status | 说明 / Notes | | ------------ | --------------------------------------------------------------- | ----------------------------------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------- | | pattern | string \| string[] | ['**/*.bench.{js,ts}'] | 已接线 / Active | benchmark 文件 glob。 | | ignore | string[] | ['**/node_modules/**', '**/dist/**', '**/build/**'] | 已接线 / Active | 搜索 benchmark 文件时忽略的路径。 | | outputDir | string | .ldesign/benchmark/reports | 已接线 / Active | 文件报告输出目录。 | | historyDir | string | .ldesign/benchmark/history | 兼容回退 / Fallback | 未设置 storage.path 时作为 --history 的保存目录。 | | reporters | ('console' \| 'json' \| 'markdown' \| 'csv')[] | ['console'] | 已接线 / Active | run 命令默认报告格式。 | | defaults | Partial<BenchmarkOptions> | { time: 1000, iterations: 10, warmup: 5 } | 已接线 / Active | 套件默认运行参数。 | | thresholds | Record<string, BenchmarkThreshold> | undefined | 已接线 / Active | 支持 Suite::Task 与 Task 两种 key。 | | env | Record<string, string> | undefined | 已接线 / Active | run 命令执行期间注入环境变量。 | | parallel | { enabled?: boolean; maxWorkers?: number; isolate?: boolean } | undefined | 已接线 / Active | enabled/maxWorkers 控制并发；isolate: true 让 CLI 按 benchmark 文件启动独立 Node.js 进程。 | | ci | object | undefined | 已接线 / Active | 启用后生成 CI 摘要、GitHub 注释；优先使用 --compare，否则自动选择最近的兼容历史报告。 | | git | object | undefined | 已接线 / Active | 控制报告中的 Git commit/branch 元数据采集。 | | plugins | string[] | undefined | 已接线 / Active | run 命令支持受控的内置插件名称。 | | storage | { type?: 'json'; path?: string; retention?: object } | undefined | 已接线 / Active | --history 使用 JSON 存储；path 覆盖 historyDir，可按天数/数量和 suite/tag、Git 分支或报告名称清理旧记录。 | | locale | object | undefined | 已接线 / Active | 配置共享 translator；CLI、内嵌 Dashboard、控制台/Markdown 报告和 CI 摘要/annotations 支持中英文。 |

阈值规则 / Threshold Rules

thresholds 的 key 支持两种写法。Threshold keys support two styles.

Suite::Task：只匹配指定套件下的单个任务。 Suite::Task: match one task under one suite.
Task：匹配所有同名任务。 Task: match every task with the same name.

支持的阈值字段 / Supported threshold fields:

maxAvgTime
minOpsPerSecond
maxRme
maxP95
maxP99
maxMemoryDelta

运行前会校验配置与外部阈值文件：数值必须有限且处于有效范围，parallel.maxWorkers 必须是正整数，环境变量值必须是字符串。 Invalid configuration and external threshold files are rejected before a run: numeric values must be finite and in range, parallel.maxWorkers must be a positive integer, and environment values must be strings.

CLI 数值参数也会严格校验：history --limit 必须是正整数，compare --threshold 必须是大于等于 0的有限数字；非法输入会直接失败，不会静默套用默认值. CLI numeric options are strict as well: history --limit must be a positive integer and compare --threshold must be a finite number at least 0; invalid input fails instead of silently falling back.

支持 --json 的命令在参数、配置或文件读取失败时，会输出稳定错误包络 { ok: false, command, error } 并返回非零状态，不会在 stdout 混入 Node 堆栈。Commands supporting --json emit a stable { ok: false, command, error } envelope and a non-zero exit status for parameter, configuration, or file-read failures, without mixing a Node stack trace into stdout.

启用 ci 后，run 会优先使用 --compare 指定的 baseline；未指定时，从 history 自动选择早于当前、环境可比较且（双方都有记录时）同 Git 分支的最近报告。run --json 会在 ci.baseline 中标出 explicit 或 history 来源。 When CI is enabled, explicit --compare input takes precedence; otherwise the latest earlier history report with a comparable environment and matching recorded branch is selected.

文件级进程隔离 / File-Level Process Isolation

const config = {
  parallel: {
    enabled: true,
    maxWorkers: 4,
    isolate: true,
  },
}
void config

ldbench run 会在独立 Node.js 进程中重新加载每个 benchmark 文件，再按文件发现顺序合并报告。这适用于需要隔离全局状态或高内存任务；不会改变直接使用 ParallelBenchmarkRunner API 时的进程模型。 ldbench run reloads each benchmark file in an independent Node.js process and merges reports in discovery order. The direct ParallelBenchmarkRunner API keeps its existing process model.

内置插件 / Built-In Plugins

plugins 当前只加载受控的内置插件名称，不会按配置任意动态导入外部模块。未知插件会进入 run --json 的 plugins.warnings。 plugins currently loads controlled built-in plugin names only. It does not dynamically import arbitrary external modules from config. Unknown names are reported through plugins.warnings in run --json.

支持的插件 / Supported plugins:

statistics
trend-analysis
memory-analysis
regression-detector

编程 API / Programmatic API

如果你不想使用 CLI，也可以在代码里直接创建和运行 benchmark。If you do not want to use the CLI, you can run benchmarks programmatically.

import { createBenchmark, createRunner } from '@composy/benchmark'

const stringSuite = createBenchmark('string-ops', {
  time: 500,
  iterations: 20,
  warmup: 3,
})

stringSuite.add('startsWith', () => {
  void 'ldesign-benchmark'.startsWith('ldesign')
})

stringSuite.add('regex', () => {
  void 'ldesign-benchmark'.startsWith('ldesign')
})

const runner = createRunner({
  printResults: false,
  parallel: true,
  maxConcurrency: 2,
})

runner.addSuite('string-ops', stringSuite)

const report = await runner.runAll()
console.log(report.suites.map(suite => suite.name).join(', '))

生成摘要，避免每个上层工具重复遍历报告。Create a summary without duplicating report traversal in every higher-level tool.

import { summarizeBenchmarkReport } from '@composy/benchmark'

const summary = summarizeBenchmarkReport(report)
console.log(summary.totalTasks, summary.fastestTask?.key)

扩展 CLI 时可以复用内置命令并注入自定义命令。When extending the CLI, reuse built-in commands and inject custom commands.

import { createBenchmarkCli } from '@composy/benchmark'

const cli = createBenchmarkCli({
  commandRegistrars: [
    app =>
      app.command('custom').action(() => {
        console.log('custom command')
      }),
  ],
})

cli.parse()

公开的主要 API / Major exported APIs:

createBenchmark(name, options)
bench(name, tasks, options)
createRunner(options)
checkThresholds(report, thresholds)
summarizeBenchmarkReport(report) / createReportResultIndex(report)
analyzeBenchmarkReportHealth(report) / compareBenchmarkEnvironments(baseline, current)
diagnoseBenchmarkWorkspace(options)
startBenchmarkDashboard(options)
createBenchmarkCli(options)
consoleReporter / jsonReporter / markdownReporter / csvReporter
loadConfig / loadWorkspaceConfig / validateConfig

报告、历史与对比 / Reports, History, And Compare

console：直接打印到控制台。Prints the results directly to the terminal.
json：最适合做历史记录与自动化比对。Best for history storage and automated comparisons.
markdown：适合直接贴进 PR、Issue 或 Wiki。Good for PRs, issues, and Wiki pages.
csv：适合导入表格软件或数据分析工具。Useful for spreadsheets and downstream analysis tools.

开发与打包 / Development And Packaging

本包开发时建议使用以下命令。Use the following commands during package maintenance.

pnpm run type-check
pnpm run lint:check
pnpm run test:run
pnpm run build

说明 / Notes:

lint 使用 @antfu/eslint-config。 lint uses @antfu/eslint-config.
build 通过 @composy/pack（pack build）驱动，基于 .ldesign/pack.config.ts 薄配置产出 ESM、CJS 与 .d.ts。 build is driven by @composy/pack (pack build) with the thin .ldesign/pack.config.ts, producing ESM, CJS, and .d.ts.
npm 发布内容还包含 README.md 与 MIT LICENSE；可通过 npm pack --dry-run --ignore-scripts 预览最终文件清单。
type-check 与 lint 使用本地包装脚本，从工作区共享依赖中解析 typescript、eslint 与 prettier。 type-check and lint resolve shared workspace tooling through local wrapper scripts.

目录结构 / Package Layout

构建与质量工具的权威配置集中在 .ldesign/：pack.config.ts 供 @composy/pack 使用， eslint.config.js 与 tsconfig.json 分别供 ESLint 和 TypeScript 使用。根目录同名文件是供 CLI 自动发现的薄转发入口。 The canonical build and quality configuration is collected in .ldesign/: pack.config.ts is used by @composy/pack, while eslint.config.js and tsconfig.json configure ESLint and TypeScript. The root files with matching names are thin CLI discovery shims.

src/
  analysis/      报告对比与回归分析 / comparison and regression analysis
  benchmark/     benchmark 核心实现与类型 / benchmark core and types
  cli/           命令行入口与子命令 / CLI entry and commands
  config/        配置加载与校验 / config loading and validation
  errors/        统一错误类型 / error hierarchy
  integrations/  Git、环境、CI 集成 / Git, environment, and CI integrations
  plugins/       内置插件与通知插件 / built-in and notification plugins
  reporting/     console/json/markdown/csv 报告器 / report generators
  runtime/       通用运行时工具 / runtime utilities
  storage/       历史记录存储 / history storage

更多示例请查看以下文件。For more examples, inspect the following files.

benchmark.config.example.json
benchmark.config.example.yaml
benchmark.schema.json
src/cli/commands/initCommand.ts

通过 `@composy/cli` 统一接入

接入类型：bin
统一命令：ldesign benchmark
命令别名：bench、bm
包内原生 bin：ldbench

当前包通过独立 bin 接入，统一命令会转发到包自身 CLI。

pnpm add -D @composy/cli
ldesign benchmark --help
ldesign tools run benchmark --help