@cliwatch/cli

The CLIWatch command-line tool — manage benchmarks, view results, and get AI-agent-friendly documentation for setting up CLI tests.

Install

npm install -g @cliwatch/cli

Quick start

# Interactive setup — creates cli-bench.yaml + optional GitHub Actions workflow
cliwatch init

# Non-interactive (for AI agents)
cliwatch init --cli myapp --ci

# Validate your config
cliwatch validate

# View recent bench runs
cliwatch runs

# View pass rate trends
cliwatch trend --cli myapp

Commands

| Command | Description | |---------|-------------| | cliwatch init | Scaffold a new cli-bench.yaml (interactive) | | cliwatch validate | Validate your cli-bench.yaml config | | cliwatch runs | List recent bench runs | | cliwatch runs <id> | Show details of a specific run | | cliwatch trend --cli <name> | Show pass rate trend for a CLI | | cliwatch skills | List documentation topics | | cliwatch skills <topic> | Show a specific topic (e.g. setup-tests, assertions) |

Skills (AI-agent documentation)

cliwatch skills outputs structured documentation designed for AI coding assistants. Topics:

• setup-tests — how to write effective CLI benchmark tests
• assertions — all 10 assertion types with examples
• yaml-reference — complete cli-bench.yaml schema
• ci-integration — GitHub Actions setup
• providers — supported LLM providers via Vercel AI Gateway
• thresholds — pass rate threshold configuration
• help-modes — injected vs discoverable vs none
• troubleshooting — common issues and fixes

# AI agents: read the setup guide
cliwatch skills setup-tests

# JSON output for programmatic use
cliwatch skills --json

Environment variables

| Variable | Description | |----------|-------------| | CLIWATCH_API_KEY | API key from app.cliwatch.com |

Changelog

0.2.0

• Interactive cliwatch init with CLI auto-detection, GitHub Actions setup, --cli flag for agents
• system_prompt field in cli-bench.yaml for custom LLM context
• AI_GATEWAY_API_KEY replaces per-provider keys (Vercel AI Gateway)
• cliwatch validate shows context-aware messages

0.1.0

Initial release.

See CHANGELOG.md for full history.

Related

• @cliwatch/cli-bench — the benchmark runner
• docs.cliwatch.com — full documentation
• app.cliwatch.com — dashboard

License

MIT