@speqkit/plugin-cli

The terminal surface: run, report, validate, list, capabilities.

# speq.yaml
plugins:
  - cli

speq run [--env ci] [--test <file|glob>]... [--suite <dir>]... [--tags a,b] [--name a,b] [--advisory a,b] [--reporter a,b] [--workers N] [--shard i/n] [--watch] [--json] [--color|--no-color]
speq report [--run <id>] [--list] [--reporter a,b]
speq validate [--json]
speq list [--shard i/n] [--json]
speq capabilities [--json]
speq schema [--out FILE]

A flag is written --test x or --test=x; --test and --suite may be written more than once, and a --test with a * in it is a pattern over the files under its leading directory — suites/menu/*.yaml, or **/smoke-*.yaml under suites/. A flag a command does not take is refused, with the nearest one it does: a run that did something other than what it was asked, with nothing saying so, is the same fault as --workers 8 quietly running one. Output is coloured when stdout is a terminal and NO_COLOR is unset; --color and --no-color override that, FORCE_COLOR too.

Four flags choose the tests

--test takes a file, --suite a directory, --tags a label. --name takes the test's own name, which is the only one of the four that addresses a single test:

speq run --name 'menu.create[jpy]'

Reading a report and wanting to re-run exactly that row is the commonest thing anybody does, and until this flag it meant running the file and watching the other nine. All four apply to run, validate and list alike, and the files and directories are joined: --suite suites/orders --test suites/health.yaml is both, with --tags and --name applied to the lot.

--advisory chooses what the exit code answers for

--tags narrows what runs. This narrows what the verdict covers, and runs everything either way:

speq run --env ci --advisory menu,admin,devops

Every selected test still runs, still prints and still counts in the totals. A test carrying one of these tags does not set the exit code; every other test does. The run then says which rule produced the code it exited with:

7 passed - 1 failed in 2.1s
2 of 8 test(s) advisory (--advisory menu,admin,devops); exit 0: 1 of them red, and nothing blocking is

The case it exists for is a gate on a pull request. A project that tags its suites by zone wants red in the zone this branch touched to block the merge and red anywhere else to be loud without blocking — because on the first wave of coverage a flake next door otherwise stops delivery, the working rule becomes "run it again", and a fortnight later there is no gate at all. The cost is named rather than hidden: somebody else's regression can reach main, and a run after deployment is what catches it.

Only the advisory half is named, and a test carrying none of those tags blocks. That asymmetry is deliberate: a zone somebody forgot to add to the list must not quietly become a zone nobody checks.

A test is advisory when everything it answers for is advisory. One tag out of two is not enough. A suite tagged [backend, menu] — publishing is the backend's, "the guest can see it" is the menu app's — answers for the work in hand on a branch that touched either, so it blocks on both. The lenient reading costs more than it looks: on a branch touching only the menu, that suite is the one covering the menu app, and sparing it for its backend tag makes the whole run advisory and the job green whatever happens.

The flag takes tags, not zones, and it cannot tell one kind of tag from another. A suite tagged [backend, readonly], where the second says it creates nothing and can therefore be run against production, is advisory only when readonly is named too. So compute the list rather than typing it — every tag carried by the tests that answer for no zone I touched — which is what a CI job with a file-to-zone map does anyway:

speq run --env ci --advisory "$(node scripts/gate.mjs advisory-tags)"

Advisory red is never silent. It is in the console — the test's header says advisory before it runs, not after it goes red — in the totals, in the JUnit file, and in --json under advisory. What it does not do is decide.

What this replaces is two runs, one with || true after it. That pays two run ids, two JUnit files, the list of zones written down twice — the second copy is the one nobody updates when a zone is added — and a || true that swallows the binary being missing exactly as readily as a red test.

It is a plugin, and that is the point

Remove it and the framework still runs — from a VS Code extension, from a TUI, from someone's own harness. The kernel keeps only the commands that must work before plugins are loaded (install, add, link, doctor); everything a person thinks of as using the framework lives here and could be replaced wholesale.

It publishes the cli service, which is how any other plugin contributes a command without depending on a terminal existing:

import type { CommandHost } from '@speqkit/plugin-api'

ctx.inject(['cli'], (services) => {
  const cli = services.cli as CommandHost
  cli.register('db:seed', { summary: 'load fixtures', run: async () => 0 })
})

inject and not a plain lookup: in VS Code there is no cli service, the callback never fires, and the plugin stays perfectly usable.

The console output is an ordinary reporter

It used to subscribe to the event bus directly, which meant the default path went around defineReporter and left the mechanism untested by anything a user actually runs. It is now registered as the reporter named console and is the default value of --reporter, so the most common command in the framework exercises the extension point every time.

--reporter console,junit drives both. An unknown name fails before the first test rather than after the last one.

The console reporter holds a test's output until the test is over and then prints the block whole, because with --workers above one two tests are open at once and printing each event as it arrives puts one test's steps under another test's header.

A step printed with a suite beside it — . tenant (http) suites/menu — belongs to that suite rather than to a test: it is the suite's own setup or cleanup, and there is no test header above it to inherit.

--workers N runs N suites at once

One by default, and one is not a placeholder. Every other runner defaults to the CPU count because its bottleneck is the local processor; speq's is somebody else's service, and --workers 8 is eight times the load on the system under test. Step timeouts start firing where they did not fire in sequence, which means the number could change a verdict — so nothing guesses it, and there is no auto.

Concurrency is between suites and nowhere else. A test runs whole, interleaved with nothing; the tests inside one suite stay sequential, so a suite's hooks and its suite-scoped resources behave exactly as they did. A suite that fails frees its slot rather than stopping the run.

--workers 0, --workers auto and --workers 2.5 are refused before anything is discovered, rather than quietly falling back to one — a twenty-minute suite that pretends to obey is worse than one that says no.

--shard i/n gives this machine its slice

--workers is one machine doing more at once. --shard is n machines each doing part of it — an independent run apiece, with its own events.jsonl and its own JUnit XML, merged by CI the way it merges everything else. They compose: speq run --shard 2/4 --workers 3 is the second quarter, three suites at a time.

It is not a fifth selection flag. The other four say which tests you care about; this one says you care about all of them and there are n machines. So it applies to what discovery returned, and it applies last — sharding a selection is a sensible thing to want, selecting out of a shard is not:

speq list --test suites/orders/matrix.yaml --shard 1/2   # half of that file

The slice is by test, not by file, and the slices are contiguous. Slicing by file would keep a file whole, but a table of a thousand cases is one test in one file — so the case shards exist for would be the one case they could not split. What that costs is a file on a boundary, whose suite-scoped resources are then set up in both shards; a contiguous cut pays it for at most n-1 files in the whole run, where i % n would pay it for every file with more than one test in it. And the cost is already there one level up: a shard is a separate process, so a directory suite's setup already runs once per shard however the slice is cut. One sentence covers both — a shard is an independent run, and every suite that has work in it opens in it.

speq list --shard i/n takes the flag too, because the property worth checking — n shards between them run each test exactly once — is checkable without running anything.

--shard 2, --shard 0/4, --shard 5/4 and --shard a/b are refused before discovery. A machine that quietly ran the whole suite after being asked for a quarter of it is four times the work with nothing saying so.

One thing to know when the shards are on one machine: reports/<runId>/ is per run and never collides, but junit.xml is a stable path on purpose, so four shards in one working directory overwrite one file. Give each a different junit.output, or do what CI does and put them on four machines.

--json answers a program

run, validate and list each take --json and write one document to stdout instead of prose to the terminal.

speq validate --json

{
  "checked": 12,
  "diagnostics": [
    {
      "file": "suites/orders/create.yaml",
      "path": "steps[0].type",
      "code": "unknown-step-type",
      "message": "unknown step type 'htpp'",
      "hint": " — did you mean 'http'?"
    }
  ]
}

code is the field to switch on. The message is a sentence written for a person and may be reworded in any release; the code is a slug and may not. It is what lets a caller tell a step type that does not exist from one whose input is malformed without matching substrings of coloured stderr — which is the difference between a generated suite that can be repaired and one that cannot. The kernel's codes are bare words (unknown-step-type, unknown-assertion, missing-field, unknown-field, duplicate-test-name, duplicate-step-id, test-has-no-steps, pending-needs-reason, case-has-no-id, …); anything a plugin's own validate found is prefixed with that plugin's short name — http/unknown-topic — so the two sets can never collide.

run --json prints one document when the run is over: the counts, and per test its identity and its failures, present and empty on a green one. What compared badly is in there, expected and actual included; everything else — every step, every artifact — is in the report already written under runDir. It also prints a document when it refuses to start, with "status": "invalid" and the diagnostics, or "status": "no-tests".

Two rules worth knowing. The document goes to stdout even when the news is bad — stderr keeps what went wrong with the command, such as a malformed --shard, which is a bug in the caller rather than a result to read. And --json replaces the default reporter, not a chosen one: --json --reporter junit still writes the XML, because a document on stdout and a file on disk answer different callers. Exit codes do not move: 0 passed, 1 something failed, 2 nothing ran.

speq capabilities is the grammar itself

speq capabilities          # for you
speq capabilities --json   # for a program

Every step type, assertion, value provider, reporter and loader the loaded plugins define, with the InputSchema each declared. In the terminal a star marks a field the schema requires; the rest of the schema is in --json.

The schemas have been in the registry since the plugin that owns them registered, and could not be reached from outside the process. So an editor offering completion, a palette in a panel and a system prompt describing speq to a model each carried a copy of the vocabulary — one that goes stale the moment somebody installs a plugin, and goes stale silently, because a suite written against the wrong vocabulary looks exactly like a suite with a typo in it. Asking the session instead means the answer is true for this project.

speq run --watch

Runs the selection, then runs it again on every change under the project, until Ctrl-C. The loop everybody runs by hand between two windows — save, switch, press up, press enter — and the pause in the middle is where the thought goes.

It watches the project and not what the run writes: reports and artifacts land under the same root, and a watcher that does not exclude them re-runs on its own output forever. Filesystem events are coalesced, so one editor save is one run and a git checkout of two hundred files is also one. A change that arrives mid-run is remembered and taken up when that run finishes, because a run against a real system is not something to have two of.

--watch and --json exclude each other: a stream of runs is not a document.

speq schema is the same grammar, for an editor

speq schema                        # to stdout
speq schema --out schema.json      # to a file, with the modeline to paste

A JSON Schema of this project's test files: the ten spine fields, and a branch per step type and assertion carrying that plugin's own InputSchema. Point yaml-language-server at it and a misspelled bodyRaw: is underlined as you type, with completion on every word the loaded plugins define.

# yaml-language-server: $schema=../schema.json

It is generated rather than shipped, because a step type is a word a plugin registers at load time: no file in a package could list the ones this project has. Which is also its honest limit — it is a snapshot, and it goes stale when the plugins change, so it is a command somebody re-runs and not something speq init writes once and leaves quietly wrong. And it is not the check: references, forward declarations, duplicate ids and each plugin's own validate are not expressible in JSON Schema. This is completion and a red squiggle; the answer is still speq validate.

A typed field accepts a whole ${…} beside its own type, because expected: "${want}" is a string in the file and an integer by the time the assertion sees it — an editor that underlined it would be underlining the correct way to write it.

speq plugins is the other half of the same question and stays what it is: who is loaded, grouped by owner. This one is grouped by kind and carries the schemas — what may be written, rather than who brought it.

It does not import the kernel

Every command here drives the running session through ctx.host — discover, validate, run, runs, replay. This package's manifest names only @speqkit/plugin-api, as a peer.

That is not tidiness. Importing speqkit put the kernel in this plugin's published dependencies, so speq install materialised a second copy into the store and pinned it in speq.lock; and it made this file call bootstrap() inside a process that had already booted one, loading every plugin twice into two registries that could not see each other. This plugin is what a third-party plugin gets copied from, so whatever it does, the ecosystem does.

--env and --speq-root

Both are parsed by the kernel before dispatch, not here. By the time a command runs, the session already has the right root and the right environment layered on top.

MIT.