@edspencer/cli-conformance
v0.2.1
Published
Black-box conformance checks for the CLI contract that Ed's local tools share: --version, --help, exit codes, service status, busy and serve.
Maintainers
Readme
cli-contract
One CLI shape for Ed's local tools: Paddock, Managers, Hallway, Local Review, BragDoc and Localidraw. Each runs on a laptop, some in a home lab too, and each should feel the same around running it: the same commands, flags, exit codes and output for serving, installing as a service, logs, checks and updating.
paddock service install # run it in the background, at login
paddock service status # is it running, which version, is it busy?
paddock update # check it's safe, stop, install the new version, start, roll back if it fails
paddock busy # 0 idle, 75 busy, 3 not running: for scripts, Ansible and Watchtower
paddock doctor # what's wrong, and how to fix it- SPEC.md is the contract.
- ROADMAP.md is where each tool stands, and the order they get there in.
Each tool copies the reference code (reference/) into its own repo rather than depending on a
package. Localidraw allows no runtime dependencies, and a fix to six products' lifecycle code is six small PRs either
way. schemas/ has the JSON schemas for the --json output the spec requires.
The conformance runner
@edspencer/cli-conformance checks a tool from the
outside. It packs the tool (npm pack, so prepack builds it), installs the tarball into a temporary prefix, and
runs the installed bin with a throwaway HOME, NO_COLOR, and no TTY on stdin. It's a devDependency:
npm install --save-dev @edspencer/cli-conformance
npx cli-conformance # in the tool's checkout; exit 0 passed, 1 failed, 2 a usage or config error
npx cli-conformance --json # the report as one JSON document on stdout; progress on stderr
npx cli-conformance --only version,busy
npx cli-conformance --package localidraw@latest --config cli-contract.json # a published version insteadIn CI, after the build: - run: npx cli-conformance.
Each tool keeps a cli-contract.json at its root, saying which checks apply and why any don't. Every check must be
listed, so a new check in a newer runner fails until the tool says whether it applies. The update checks are the
exception: they're opt-in (below).
{
"$schema": "./node_modules/@edspencer/cli-conformance/schemas/cli-contract.schema.json",
"name": "Local Review",
"serve": { "args": ["serve", "--port", "{port}", "--data-dir", "{home}/data"] },
"env": { "SOME_TOKEN": null },
"checks": {
"version": true,
"help": true,
"unknown-flag": true,
"service-status": true,
"service-status-json": true,
"busy": { "skip": "no busy command yet (wave D)" },
"busy-json": { "skip": "no busy command yet (wave D)" },
"serve-banner": true,
"serve-health": true
}
}| Check | Passes when |
|---|---|
| version | --version and -v print the version from package.json and a newline, nothing else, and exit 0 |
| help | --help and -h print something on stdout and exit 0 |
| unknown-flag | --no-such-flag-cli-conformance exits 2; stderr's first line is <bin>: …, and the usage follows it |
| service-status | service status exits 4 with nothing installed |
| service-status-json | service status --json exits 4 and prints one document matching schemas/service-status.schema.json, with installed and running false |
| busy | busy exits 3 with nothing running |
| busy-json | busy --json exits 3 and prints one document matching schemas/busy.schema.json, with running false |
| serve-banner | serve's first stdout line is <name> <version>: <url> |
| serve-health | GET <url>/api/health answers 200 with { "ok": true, "version": "<version>" } |
The update checks (opt-in)
A tool with update lists these too. Unlisted, they're neither run nor reported. They need a network to the real
registry only for the tool's dependencies.
The runner makes two more builds from the installed copy of the tool (its files, with every package.json whose
version is the tool's bumped): B, a minor higher, and C, a patch above B, whose serve exits at once. A local
registry serves all three, and sends every other package to the real one. npm's prefix, cache and registry are the
runner's own, and the runner checks that npm root -g is inside its temporary folder before anything installs.
| Check | Passes when |
|---|---|
| update-up-to-date | with latest at this version, update --yes exits 0 and installs nothing |
| update-check-json | with latest at B, update --check --json exits 0 with one document matching schemas/update-check.schema.json (B available, npm-global, canUpdate), and installs nothing |
| update-no-tty | update without --yes exits 2, with <bin>: … and then the usage on stderr |
| update-busy | with a stand-in instance whose /api/busy says busy, found through update.runtimeFile, update --yes exits 75 and installs nothing |
| update-npx | run from a copy under …/_npx/<hash>/node_modules, update --yes exits 5 |
| update-checkout | run from a copy outside any node_modules, update --yes exits 5 |
| update-not-writable | with the global folder read-only, update --yes exits 5 and prints a command with sudo (skipped as root, and on Windows) |
| update | update --yes exits 0, and the installed bin then says B |
| update-to-lower | at B with latest at A, update --yes stays at B; update --yes --to A goes to A |
| update-service | with the service installed (update.serviceInstall) at A, update --yes exits 0, B answers /api/health on its port, and service status --json agrees |
| update-rollback | the same, to C: it exits 6, and A is installed and answering again |
update-service and update-rollback install the tool's real service, under the real home, so they run only where
CI_REAL_SERVICE=1 and CI are set: a CI runner's real-service job. Elsewhere they're skipped, saying why.
"update": {
"runtimeFile": "{home}/.local-review/runtime.json",
"serviceInstall": ["service", "install", "--port", "{port}"],
"timeoutMs": 300000
}update.runtimeFileis where the tool looks for itsruntime.json(SPEC §4), forupdate-busy.update.serviceInstallinstalls the service on a free{port}(this is the default). The runner then askshttp://127.0.0.1:{port}/api/health, and runsservice uninstallafterwards whatever happened.update.timeoutMsbounds eachupdate(default 300 000).
Settings
serve.argsstart the tool in the foreground (default["serve", "--port", "{port}"]).{port}is a free port, also given asPORT;{home}is the throwaway home and{tmp}a scratch folder (the commands' working folder). The runner stops it afterwards with SIGTERM (a tree kill on Windows), then SIGKILL after 10 seconds.envadds variables to every command, or removes them withnull. A tool that would spend money or touch real data when it starts (Paddock's drive mode, an API key) sets that here.timeoutMs(default 15 000) bounds each command;serve.timeoutMs(default 30 000) bounds the wait forserve's first line and for/api/health. A command that's still running then is killed and fails its check.- The home is throwaway, but the machine isn't.
XDG_RUNTIME_DIRandDBUS_SESSION_BUS_ADDRESSare removed, sosystemctl --usercan't reach the real user's services, butlaunchctlon macOS still sees the real login session. A tool should decide "installed" from the unit file in its home, as the spec'sstatusdoes. CI runners have nothing installed either way.
examples/localidraw.cli-contract.json is Localidraw 0.4's settings: four checks pass, and five wait for the commands it doesn't have yet.
MIT.
