@composy/benchmark

v0.0.1

Published

5 days ago

Benchmark framework for Node.js projects with TypeScript API, CLI, reports, history storage, and CI regression analysis.

Downloads

0High
0Medium
0Low

swimly

benchmark ldesign performance tinybench typescript

`@composy/benchmark`

面向 Node.js 与 TypeScript 项目的基准测试工具包，提供 tinybench 封装、CLI、报告导出、历史记录与阈值校验。 @composy/benchmark is a benchmark toolkit for Node.js and TypeScript projects, with a tinybench-based API, CLI, report export, history storage, and threshold checks.

功能概览 / Features

统一的 TypeScript API 与 CLI：既能在代码中直接运行 benchmark，也能通过 ldbench 扫描 *.bench.ts 文件。Unified TypeScript API and CLI: run benchmarks directly in code or scan *.bench.ts files with ldbench.
支持 console、json、markdown、csv 四种报告格式。Supports console, json, markdown, and csv reports.
支持历史记录保存、报告对比与阈值失败中断。Supports history persistence, report comparison, and threshold-based failure checks.
支持 run --json、doctor --json、history --json 与 compare --json，更适合 CI 和脚本消费。Supports run --json, doctor --json, history --json, and compare --json for CI and scripts.
提供报告摘要与结果索引 API，方便上层 UI、自动化和自定义分析复用。Provides report summary and result index APIs for UI, automation, and custom analysis.
内置本地 Dashboard：首轮诊断、历史浏览、经校验的 JSON 导入导出、报告健康度、环境感知对比、多指标趋势，以及从浏览器运行已选配置。Includes a local dashboard for first-run diagnostics, validated JSON import/export, report health, environment-aware comparisons, multi-metric trends, and configured runs from the browser.
当前 CLI 已接通 pattern、ignore、outputDir、historyDir、reporters、defaults、thresholds、env、parallel、plugins。The current CLI actively consumes pattern, ignore, outputDir, historyDir, reporters, defaults, thresholds, env, parallel, and plugins.
ci、git、storage、locale 等配置字段已保留在类型层，便于后续扩展，但目前 CLI 尚未完全接线。Fields such as ci, git, storage, and locale remain in the type surface for forward compatibility, but are not fully wired into the CLI yet.

审计与改进 / Audit And Improvements

本轮整理重点修复了以下问题。This maintenance pass focused on the following issues.

旧版 README、benchmark.schema.json、示例配置与当前实现脱节，已按真实配置结构重写。The old README, schema, and sample configs were outdated and are now aligned with the real implementation.
config.parallel 与 config.env 之前只有类型定义，CLI 运行时没有实际消费，现在已接入 run 命令。 config.parallel and config.env previously existed only in types; they are now consumed by the run command.
多个运行时模块存在历史遗留乱码注释、全局 process 引用和 ESLint 规则问题，已统一整理。Several runtime modules had legacy mojibake comments, global process usage, and ESLint issues; those have been cleaned up.
补齐了关键模块的中文注释，统一了更清晰的类型与错误处理写法。Key modules now have clearer Chinese comments and more explicit typing and error handling.
新增只读 doctor 诊断和机器可读 JSON 输出，便于本地排障与 CI 集成。A read-only doctor command and machine-readable JSON output improve local troubleshooting and CI integration.
CLI 命令注册改为 registry 结构，并导出 createBenchmarkCli() 供宿主注入自定义命令。CLI command registration now uses a registry and exposes createBenchmarkCli() for host-defined commands.

安装 / Installation

pnpm add -D @composy/benchmark

要求 / Requirements:

Node.js >= 18
推荐使用 pnpm pnpm is recommended.

CLI 快速开始 / CLI Quick Start

初始化工作区配置与示例 benchmark。Initialize workspace config and a sample benchmark.

pnpm exec ldbench init

运行当前工作区的 benchmark 文件。Run benchmark files in the current workspace.

pnpm exec ldbench run
pnpm exec ldbench run --json

诊断当前工作区配置、benchmark 文件发现和报告目录。Diagnose workspace config, benchmark discovery, and report directories.

pnpm exec ldbench doctor
pnpm exec ldbench doctor --json

输出文件报告。Write file-based reports.

pnpm exec ldbench run --report json --out .ldesign/benchmark/reports
pnpm exec ldbench run --report markdown --out .ldesign/benchmark/reports
pnpm exec ldbench run --report csv --out .ldesign/benchmark/reports

写入历史记录并查看历史。Persist and inspect history.

pnpm exec ldbench run --history
pnpm exec ldbench history --limit 5
pnpm exec ldbench history --limit 5 --json

启动可视化工作台。默认只绑定本机地址，页面可以诊断首轮配置、查看或导入历史、导出 JSON、检查报告质量和运行环境、选择两份报告对比、跟踪同一任务的多指标趋势，并运行当前配置。Start the visual dashboard. It binds to loopback by default and can diagnose the workspace, inspect or import history, export JSON, inspect report health and runtime compatibility, compare two reports, track multi-metric task trends, and run the current configuration.

pnpm exec ldbench dashboard
pnpm exec ldbench dashboard --config .ldesign/benchmark.config.ts --port 4178
pnpm exec ldbench dashboard --read-only

dashboard 会在终端输出访问地址。浏览器不能传入命令、配置路径或 benchmark 路径；导入 JSON 会经过报告校验，单次最多 5 MiB。--read-only 会禁用运行和导入，适合只分享报告；只有显式传入 --host 0.0.0.0 时才会监听非本机网络。 The command prints the URL in the terminal. The browser cannot supply a command, config path, or benchmark path; imported JSON is validated and limited to 5 MiB. --read-only disables browser-triggered runs and imports; it listens beyond the local machine only when --host 0.0.0.0 is explicitly supplied.

对比两份 JSON 报告。Compare two JSON reports.

pnpm exec ldbench compare .ldesign/benchmark/history/baseline.json .ldesign/benchmark/history/current.json
pnpm exec ldbench compare baseline.json current.json --threshold 3 --json

按阈值文件校验。Validate results against a threshold file.

pnpm exec ldbench run --threshold ./thresholds.json
pnpm exec ldbench run --threshold ./thresholds.json --json

run --json 会输出 { ok, command, report, historyId, threshold }；配置加载、文件扫描或运行错误会输出 { ok: false, command: 'run', error } 并保持非零退出码。 run --json emits { ok, command, report, historyId, threshold }; config loading, file discovery, or runtime errors emit { ok: false, command: 'run', error } and keep the non-zero exit code.

监听文件变化并自动重跑。Watch files and rerun automatically.

pnpm exec ldbench run --watch

Benchmark 文件约定 / Benchmark Module Contract

CLI 默认会加载 *.bench.ts 或 *.bench.js 模块，并调用它们的默认导出函数。The CLI loads *.bench.ts and *.bench.js modules and invokes their default export.

import type { BenchmarkConfig, BenchmarkRunner } from '@composy/benchmark'
import { createBenchmark } from '@composy/benchmark'

export default async function registerBenchmarks(
  runner: BenchmarkRunner,
  config: BenchmarkConfig
): Promise<void> {
  const suite = createBenchmark('array-ops', config.defaults)

  suite.add('push', () => {
    const values: number[] = []
    for (let index = 0; index < 1_000; index += 1) values.push(index)
  })

  suite.add('concat', () => {
    const values = Array.from({ length: 1_000 }, (_, index) => index)
    void values.concat(1_001)
  })

  runner.addSuite('array-ops', suite)
}

约定 / Contract:

默认导出函数签名为 (runner, config) => void | Promise<void>。The default export signature is (runner, config) => void | Promise<void>.
通过 runner.addSuite(name, suite) 注册一个 benchmark 套件。Register a suite through runner.addSuite(name, suite).
config.defaults 会作为套件级默认参数传入 createBenchmark。 config.defaults can be forwarded into createBenchmark as suite-level defaults.

配置示例 / Configuration Example

推荐在 .ldesign/benchmark.config.ts 中使用 TypeScript 配置。Using a TypeScript config in .ldesign/benchmark.config.ts is the recommended setup.

import type { BenchmarkConfig } from '@composy/benchmark'

const config = {
  pattern: ['src/**/*.bench.ts', '.ldesign/benchmarks/**/*.bench.ts'],
  ignore: ['**/node_modules/**', '**/dist/**', '**/.vitest-build/**'],
  outputDir: '.ldesign/benchmark/reports',
  historyDir: '.ldesign/benchmark/history',
  reporters: ['console', 'json', 'markdown'],
  env: {
    NODE_ENV: 'benchmark',
  },
  thresholds: {
    'array-ops::push': {
      minOpsPerSecond: 200_000,
      maxRme: 5,
    },
    'array-ops::concat': {
      maxAvgTime: 0.05,
      maxP95: 0.08,
    },
  },
  defaults: {
    time: 1_000,
    iterations: 10,
    warmup: 5,
    collectMemory: false,
    retainSamples: true,
    concurrency: 'task',
    timeout: 30_000,
    retries: 1,
  },
  parallel: {
    enabled: true,
    maxWorkers: 4,
  },
  plugins: ['statistics', 'trend-analysis'],
} satisfies BenchmarkConfig

export default config

配置字段速查 / Config Field Summary

| 字段 / Field | 类型 / Type | 默认值 / Default | 当前状态 / Status | 说明 / Notes | | ------------ | --------------------------------------------------------------- | ----------------------------------------------------- | --------------------------- | ---------------------------------------------------------- | | pattern | string \| string[] | ['**/*.bench.{js,ts}'] | 已接线 / Active | benchmark 文件 glob。 | | ignore | string[] | ['**/node_modules/**', '**/dist/**', '**/build/**'] | 已接线 / Active | 搜索 benchmark 文件时忽略的路径。 | | outputDir | string | .ldesign/benchmark/reports | 已接线 / Active | 文件报告输出目录。 | | historyDir | string | .ldesign/benchmark/history | 已接线 / Active | --history 产物的保存目录。 | | reporters | ('console' \| 'json' \| 'markdown' \| 'csv')[] | ['console'] | 已接线 / Active | run 命令默认报告格式。 | | defaults | Partial<BenchmarkOptions> | { time: 1000, iterations: 10, warmup: 5 } | 已接线 / Active | 套件默认运行参数。 | | thresholds | Record<string, BenchmarkThreshold> | undefined | 已接线 / Active | 支持 Suite::Task 与 Task 两种 key。 | | env | Record<string, string> | undefined | 已接线 / Active | run 命令执行期间注入环境变量。 | | parallel | { enabled?: boolean; maxWorkers?: number; isolate?: boolean } | undefined | 部分接线 / Partially active | 当前 CLI 使用 enabled 与 maxWorkers 控制 runner 并发。 | | ci | object | undefined | 预留 / Reserved | 类型保留，便于后续集成 CI 报告流程。 | | git | object | undefined | 预留 / Reserved | 目前报告环境信息仍由运行时自动采集。 | | plugins | string[] | undefined | 部分接线 / Partially active | run 命令支持内置插件名称。 | | storage | object | undefined | 预留 / Reserved | 当前历史记录主要由 historyDir 驱动。 | | locale | object | undefined | 预留 / Reserved | 国际化能力已存在，CLI 尚未从配置自动切换语言。 |

阈值规则 / Threshold Rules

thresholds 的 key 支持两种写法。Threshold keys support two styles.

Suite::Task：只匹配指定套件下的单个任务。 Suite::Task: match one task under one suite.
Task：匹配所有同名任务。 Task: match every task with the same name.

支持的阈值字段 / Supported threshold fields:

maxAvgTime
minOpsPerSecond
maxRme
maxP95
maxP99
maxMemoryDelta

内置插件 / Built-In Plugins

plugins 当前只加载受控的内置插件名称，不会按配置任意动态导入外部模块。未知插件会进入 run --json 的 plugins.warnings。 plugins currently loads controlled built-in plugin names only. It does not dynamically import arbitrary external modules from config. Unknown names are reported through plugins.warnings in run --json.

支持的插件 / Supported plugins:

statistics
trend-analysis
memory-analysis
regression-detector

编程 API / Programmatic API

如果你不想使用 CLI，也可以在代码里直接创建和运行 benchmark。If you do not want to use the CLI, you can run benchmarks programmatically.

import { createBenchmark, createRunner } from '@composy/benchmark'

const stringSuite = createBenchmark('string-ops', {
  time: 500,
  iterations: 20,
  warmup: 3,
})

stringSuite.add('startsWith', () => {
  void 'ldesign-benchmark'.startsWith('ldesign')
})

stringSuite.add('regex', () => {
  void 'ldesign-benchmark'.startsWith('ldesign')
})

const runner = createRunner({
  printResults: false,
  parallel: true,
  maxConcurrency: 2,
})

runner.addSuite('string-ops', stringSuite)

const report = await runner.runAll()
console.log(report.suites.map(suite => suite.name).join(', '))

生成摘要，避免每个上层工具重复遍历报告。Create a summary without duplicating report traversal in every higher-level tool.

import { summarizeBenchmarkReport } from '@composy/benchmark'

const summary = summarizeBenchmarkReport(report)
console.log(summary.totalTasks, summary.fastestTask?.key)

扩展 CLI 时可以复用内置命令并注入自定义命令。When extending the CLI, reuse built-in commands and inject custom commands.

import { createBenchmarkCli } from '@composy/benchmark'

const cli = createBenchmarkCli({
  commandRegistrars: [
    app =>
      app.command('custom').action(() => {
        console.log('custom command')
      }),
  ],
})

cli.parse()

公开的主要 API / Major exported APIs:

createBenchmark(name, options)
bench(name, tasks, options)
createRunner(options)
checkThresholds(report, thresholds)
summarizeBenchmarkReport(report) / createReportResultIndex(report)
analyzeBenchmarkReportHealth(report) / compareBenchmarkEnvironments(baseline, current)
diagnoseBenchmarkWorkspace(options)
startBenchmarkDashboard(options)
createBenchmarkCli(options)
consoleReporter / jsonReporter / markdownReporter / csvReporter
loadConfig / loadWorkspaceConfig / validateConfig

报告、历史与对比 / Reports, History, And Compare

console：直接打印到控制台。Prints the results directly to the terminal.
json：最适合做历史记录与自动化比对。Best for history storage and automated comparisons.
markdown：适合直接贴进 PR、Issue 或 Wiki。Good for PRs, issues, and Wiki pages.
csv：适合导入表格软件或数据分析工具。Useful for spreadsheets and downstream analysis tools.

开发与打包 / Development And Packaging

本包开发时建议使用以下命令。Use the following commands during package maintenance.

pnpm run type-check
pnpm run lint:check
pnpm run test:run
pnpm run build

说明 / Notes:

lint 使用 @antfu/eslint-config。 lint uses @antfu/eslint-config.
build 通过 scripts/runPack.mjs 调用工作区内的 tools/tsup-config，产出 ESM、CJS 与 .d.ts。 build delegates to tools/tsup-config through scripts/runPack.mjs, producing ESM, CJS, and .d.ts.
type-check 与 lint 使用本地包装脚本，从工作区共享依赖中解析 typescript、eslint 与 prettier。 type-check and lint resolve shared workspace tooling through local wrapper scripts.

目录结构 / Package Layout

src/
  analysis/      报告对比与回归分析 / comparison and regression analysis
  benchmark/     benchmark 核心实现与类型 / benchmark core and types
  cli/           命令行入口与子命令 / CLI entry and commands
  config/        配置加载与校验 / config loading and validation
  errors/        统一错误类型 / error hierarchy
  integrations/  Git、环境、CI 集成 / Git, environment, and CI integrations
  plugins/       内置插件与通知插件 / built-in and notification plugins
  reporting/     console/json/markdown/csv 报告器 / report generators
  runtime/       通用运行时工具 / runtime utilities
  storage/       历史记录存储 / history storage

更多示例请查看以下文件。For more examples, inspect the following files.

benchmark.config.example.json
benchmark.config.example.yaml
benchmark.schema.json
src/cli/commands/initCommand.ts

通过 `@composy/cli` 统一接入

接入类型：bin 统一命令：ldesign benchmark 命令别名：bench、bm 包内原生 bin：ldbench 当前包通过独立 bin 接入，统一命令会转发到包自身 CLI。

pnpm add -D @composy/cli
ldesign benchmark --help
ldesign tools run benchmark --help