pi-perf

Pi-native performance benchmarking toolkit for pi.dev extensions — profiling, benchmarking, and CI reporting.

Installation

pi install npm:pi-perf

What It Does

pi-perf provides deterministic performance measurement tools for pi.dev extensions. No LLM calls — all measurements are pure timing, memory, and CPU instrumentation.

Three tools:

• perf_profile — Run a tool N times, collect timing/memory/CPU stats, detect outliers via IQR
• perf_bench — Run a benchmark suite (JSON-defined test cases), compare against baselines for regression detection
• perf_report — Generate CI-friendly reports (JSON/Markdown/HTML) with configurable pass/fail thresholds

Tools

perf_profile

Profile the execution of a pi.dev extension tool.

Parameters:

• toolName (string, required) — Name of the tool to profile
• args (object, optional) — Arguments to pass to the tool
• iterations (number, optional, default: 10) — Number of measured iterations (1-1000)
• warmup (boolean, optional, default: true) — Run 3 warmup iterations before measuring

Example:

Use the perf_profile tool with toolName="my_tool", iterations=20, warmup=true

Returns: Duration stats (avg, min, max, P50/P95/P99), memory delta, CPU usage, outlier list.

perf_bench

Run a benchmark suite against pi.dev extensions.

Parameters:

• suitePath (string, optional) — Path to a benchmark suite JSON file (default: discovers in ./perf-suites/)
• compare (string, optional) — Path to a baseline result JSON for comparison
• output (string, optional, default: "markdown") — Output format: json, markdown, or table

Example:

Use the perf_bench tool with suitePath="./perf-suites/ci.json", compare="./baselines/v1.json"

Suite JSON format:

{
  "name": "my-suite",
  "description": "Regression tests",
  "cases": [
    { "name": "fast-tool", "toolName": "tool_a", "iterations": 10 },
    { "name": "slow-tool", "toolName": "tool_b", "iterations": 5, "warmup": false }
  ]
}

Comparison thresholds:

• Regressions: >15% slower → FAIL
• Warnings: >5% slower → WARN
• Improvements: >10% faster → noted

perf_report

Generate a performance report with CI-friendly output.

Parameters:

• format (string, required) — Output format: json, markdown, or html
• outputPath (string, optional) — Write report to file (default: return inline)
• thresholds (object, optional) — Pass/fail thresholds as metric:value pairs

Available threshold metrics:

• avg_duration_ms — Average execution time
• p95_duration_ms — 95th percentile latency
• p99_duration_ms — 99th percentile latency
• memory_delta_mb — Memory usage change
• outliers_count — Number of statistical outliers

Example:

Use the perf_report tool with format="markdown", thresholds={"avg_duration_ms": 50, "p95_duration_ms": 100}

CI Integration

Use with pi-ci to add performance gates to your CI pipeline:

- name: Performance check
  run: |
    pi --provider xiaomi --model mimo-v2.5-pro --api-key "$API_KEY" \
      -e . -p "Run perf_bench with suitePath='./perf-suites/ci.json' then perf_report with format='json' and thresholds={\"avg_duration_ms\":50}" \
      --no-session

Resources

• npm
• GitHub
• pi.dev

License

MIT