@seldonqa/cli
v0.15.0
Published
Seldon QA - generate a QA test plan from any codebase and run AI agents against your live app, from your terminal or CI.
Maintainers
Readme
Seldon QA
Write your QA test plan as code, then run AI agents that use your product like a real user — from your terminal or CI.
seldon keeps a baseline of user-facing test cases in a committed seldon.yml. Point it at a live URL and autonomous agents exercise each case — click, type, navigate, verify — then hand back a clean, actionable report for you or your coding agent to act on.
npm install -g @seldonqa/cliWhy
AI is shipping more code than any team can test by hand. Seldon closes the gap: describe the journeys that matter once, in plain English, and agents run them against every deploy — telling you exactly what broke, why, and what to do about it.
Get started
npm install -g @seldonqa/cli
seldon login seldon_your_api_key # or: seldon config set-key <key>From inside a project (one baseline per project):
seldon init # scaffold a template seldon.yml
seldon cases add --name "Login works" \
--step "go to /login" --step "sign in" --criteria "lands on the dashboard"
seldon validate # schema, plus your plan's case ceilings, before you commit to a run
seldon run --url https://staging.example.com # prints the run id and returns
seldon logs --follow # watch the agents test, live
seldon report --md # clean, actionable resultsHighlights
- Test plan as code. Your cases live in a committed
seldon.yml— reviewed in PRs, versioned with your app.seldon initscaffolds it;seldon cases addappends more. - Agents that behave like users.
seldon rundrives your live app and judges each case PASS / FAIL / BLOCKED / TIMED_OUT. - Reports built for action. Every failure carries an
expected/observedpair and a single concrete recommendation —--mdfor pull requests,--jsonfor coding agents, colorized plain text for the terminal. - Watch it think.
seldon logs --followstreams the agent's reasoning, framed by test case and step. - Start and walk away.
seldon runsubmits and returns the run id. Check in withseldon report, watch withseldon logs -f, stop it withseldon cancel. Add--waitto stay for the report. - Never loses a run. The id is printed the moment it exists, and if the submit itself times out the CLI goes and finds the run rather than leaving one burning with nobody holding its id.
- CI-native gate.
seldon run --fail-on failexits non-zero so a red QA run can block a merge (and implies--wait).
Configuration — seldon.yml
Your baseline is a single committed file at the project root — the plain-English cases themselves, not id overrides:
name: web-app # optional lineage name; defaults to the git remote owner/repo
labels: [nightly, team-web] # optional run tags for profiling; merged with `--label`
run:
url: https://staging.example.com # default target; override per run with --url
username: [email protected] # optional login the agent signs in with
password: hunter2 # optional; prefer a managed secret in prod
testCases:
- name: Login works
id: login-works # optional stable id; defaults to a slug of the name
url: /login # optional per-case path (relative) or absolute URL override
steps:
- go to /login
- enter valid credentials and submit
criteria: the user is signed in and lands on the dashboardReading a report
The report lists every case worst-first, then expands each failure:
Seldon QA — Checkout & account flows · 2/6 passed (33%) · ❌2 · 🚫1 · ⌛1
FAIL BLOCKER TC-201 Complete payment
BLOCKED TC-301 View order history
PASS TC-101 Add item to cart
TC-201 Complete payment
acceptance: A valid card places the order and shows a confirmation.
expected: Order is placed and a confirmation screen appears.
observed: The Pay button spins indefinitely; no order is created.
recommendation: Wire the Pay button's submit handler to /orders and surface a failure toast.
repro: Add an item, open /checkout, enter a valid card, click Pay- Verdicts:
PASS·FAIL(a real defect, with aseverity) ·BLOCKED(couldn't be evaluated — e.g. a missing precondition) ·TIMED_OUT. - recommendation appears on
FAIL/BLOCKEDcases: the single strongest next step to get the case passing. --fullexpands every failing case (the terminal view caps at 5 by default);--jsonemits the same data as a stable object.
In CI
seldon run --url "$PREVIEW_URL" --fail-on fail--fail-on fail returns a non-zero exit code when any case fails; --fail-on any also gates on blocked or timed-out cases. --fail-on implies --wait, because there is no verdict to gate on until the run ends — a gate that returned immediately would always pass. With --json the report is stable, machine-readable output you can post to a pull request or hand a coding agent for an automated fix loop.
Detaching, checking in, and stopping
seldon run starts the run and returns. A run takes minutes, and a terminal held open for it is one
timeout away from losing track of it entirely:
Run started — exp_01HX…
progress seldon report --id exp_01HX…
live seldon logs -f --id exp_01HX…
stop seldon cancel --id exp_01HX…The id is written to .seldon/last-run, so the same three commands work with no --id at all.
--wait keeps the command attached and prints the report at the end; --fail-on and a local --url
turn that on for you, because a gate needs a result to gate on and a tunnel dies with the process
holding it.
If the submit itself times out, the CLI does not report that as a failed run — it isn't one. Aborting the request does not cancel the server, so it looks for the run the submit may have started and names it:
The server didn't answer in time, so this command never learned the run's id.
That does NOT mean nothing started — looking for it now ...
Found it — exp_01HX… (Running). It is running 42 case(s).It claims a run only when exactly one recent run matches this project; with two it tells you to look
at seldon runs --running rather than pointing seldon cancel at a coin flip.
Plan limits
Your workspace has two ceilings, both counted in test cases: how many one run may dispatch, and how
many you may have in flight at once. They follow your commercial tier (trial is capped; usage and
enterprise are not) and live in the backend, so they can be raised for your account without a CLI
release.
seldon validate checks a file against the ones that actually apply to you, before you commit to a
run:
$ seldon validate
Not runnable — 250 test cases.
case_limit_exceeded
This run would dispatch 250 test cases, but the trial plan allows 20 per run. Mark the cases you
want with 'only: true' (or 'skip: true' on the rest) to bring it to 20 or fewer, then run again.
plan 'trial' · 20 case(s) per run · 20 in flight · 0 running nowseldon run enforces the same ceilings and refuses in the same words — before anything is written,
so an oversized suite costs nothing. A concurrency refusal clears on its own: wait for the current
run, or clear the decks with seldon cancel --all (every in-flight run your key can see; one that
can't be stopped doesn't hold up the rest). --local skips the server check (schema only, no API
key needed) and --json prints the whole verdict for scripts.
One answer, whichever surface you ask
A broken seldon.yml is caught in one of two places: by the CLI's own parser before anything is
sent, or by the server after it is. You should not be able to tell which. seldon validate,
seldon run, the MCP validate_seldon_yaml and run_seldon tools, and the GitHub App's PR check
all report the same code, point at the same field, and give the same advice:
$ seldon validate --local
seldon.yml isn't valid at testCases[0].criteria: must be a non-empty string. Every test case needs
a non-empty name, at least one step, and a single criteria; only/skip must be literal true or false.// MCP validate_seldon_yaml, same file
{ "ok": false, "code": "invalid_field", "field": "testCases[0].criteria", "error": "…the same sentence…" }Only the parser's own detail differs between local and server (must be a non-empty string versus
Field required), because the two genuinely detect it differently. Everything a tool branches on,
and every word of advice, is identical — and pinned by a conformance fixture both repos hold
byte-identically, so it cannot drift.
Four codes are server-only, because the local parser sees the file and nothing else: your plan's
ceilings (case_limit_exceeded, concurrency_limit_exceeded), the resolved target
(no_target_url), and credentials (invalid_credentials — they are vaulted server-side and never
part of the plan the CLI parses).
Every refusal carries a stable code alongside its sentence. The full catalogue — what each code means
and what fixes it — is in docs/seldon-validation-errors.md.
Local targets — automatic tunnels
Point --url at a local address — localhost, *.localhost, any 127.x.x.x, 0.0.0.0, ::1, or a bare :PORT — and Seldon exposes it to the QA agent through a temporary, authenticated Cloudflare tunnel for the duration of the run. No deploy, no manual tunnelling:
seldon run --url http://localhost:3000First the CLI checks that something is actually listening there. If nothing answers, it stops before doing anything else: Nothing is answering at http://localhost:3000. Start your app (or fix the port in the URL), then run this again. Then you'll see a one-line warning that the local address will be exposed and a [Y/n] prompt (defaults to yes). The public hostname is protected by Cloudflare Access, so only this run's agent can reach it, and the tunnel is torn down automatically when the run finishes or you press Ctrl-C. In CI or any non-interactive shell the prompt is skipped automatically; pass --auto-approve to skip it explicitly:
seldon run --url localhost:3000 --auto-approve # e.g. in a CI jobRemote --urls are used as-is (no tunnel).
Ctrl-C during a tunneled run deletes the tunnel straight away and exits. The run itself keeps going on the server but can no longer reach your app, so the CLI tells you to cancel it: Detached — the local tunnel was closed, so run <id> can no longer reach your app. Cancel it with: seldon cancel --id <id>. Pressed before the run has started, it says Stopped before the run started — nothing was left running on the server. If the CLI is killed outright (kill -9, or the terminal is closed), it can't clean up: the tunnel stays open until the server reclaims it, and cloudflared may keep running (pkill -f 'cloudflared tunnel run' stops it).
A local URL always waits. The tunnel only stays open while seldon run is running, so returning early would close it under a run that had just started. run therefore stays attached for a local target even without --wait, and says so (waiting because the target is local — its tunnel is only open while this command runs). Point --url at a public URL if you want to detach.
Tunnels are short-lived. Each tunnel has a hard maximum lifetime (30 minutes by default) and is reclaimed by the server once it's exceeded, regardless of the run's state — so the run must finish within that window. When a tunnel comes up the CLI prints the exact cap for your account (… active for up to 30 min; the run must finish within that window). Runs that legitimately need longer should target a deployed/public --url instead of a local one.
If your account has too many tunnels open at once you'll get Tunnel limit reached — wait a minute and re-run; if the same local target is already being provisioned you'll get a short retry hint. Neither starts a run — just re-run once things settle. If the retry hint doesn't go away, an earlier tunnel for that address couldn't be fully cleaned up and is still holding it; it's cleared by the server's scheduled tunnel cleanup. Meanwhile the same app is reachable under another address, e.g. 127.0.0.1 instead of localhost.
One local target, one run at a time. Runs against the same local URL in the same workspace share a single tunnel. That includes two terminals, a terminal and a CI job, or two teammates. Whichever run finishes first closes the tunnel, and the other run loses access to your app. Until this is fixed, run them one after another.
Usability studies
Alongside QA, a project can keep usability studies as code: personas from your workspace attempt a goal on your app, and you get back a summary of where they struggled. Each study is one committed <name>.usability.yml file at the project root, next to seldon.yml. Every key is required except version and credentials:
version: 1 # optional
experiment:
title: Trial signup — first-time visitors
goal: Sign up for a free trial and invite one teammate to your workspace.
hypothesis: The pricing step is where hesitation will show up.
type: Onboarding # Onboarding | Task Completion | Churn | Conversion |
# Navigation Clarity | Feature Adoption | Trust Signals
target:
device_type: Website # Website only for now
frame_type: Desktop # Desktop | Mobile
url: https://staging.example.com/signup
# Optional login (for apps that gate behind sign-in). Committed like a .env value
# (fine for staging/dev; use a managed secret for production). The agents sign in before testing.
credentials:
username: [email protected]
password: hunter2
# To force a logged-out run, use credentials: none.
personas: # at least one persona name from your workspace
- Budget-conscious founder
- Skeptical IT adminseldon init --type usability trial-signup # writes ./trial-signup.usability.yml
seldon init --type usability trial-signup --dir studies # …or into another directory (created if missing)
seldon validate --file trial-signup.usability.yml
seldon run --file trial-signup.usability.yml # waits, then prints a short report (--md / --full / --json)
seldon report # re-show the last run's report
seldon personas # the persona names your workspace has (--limit / --offset to page)Which kind of file? A file named
*.usability.ymlis always a usability study, so a study with a YAML error or a misspelled key gets usability errors rather than QA ones. Otherwise the CLI decides from the content: a top-levelexperiment:block is a usability study,testCases:is a QA suite. Pass--type usability(or--type qa) torun/validateto make that explicit; a contradicting--typeis an error.Checked like the server checks it. Unknown keys are an error at every level, listing the allowed ones, so a typo such as
credential:can't be dropped silently.versionmust be1.target.url, and--urlonseldon run, must be a full http(s) URL such ashttps://staging.example.comorhttp://localhost:3000(notlocalhost:3000).Names, not ids. Personas and the experiment type are written by name; the server resolves them when the run starts. A typo fails the run before anything is created.
Local URLs (
localhost,127.0.0.1, …), whether intarget.urlor--url, are exposed through the same temporary Cloudflare tunnel as a QA run, after the same[Y/n]prompt (skipped with--auto-approve, in CI, or when not interactive). The study's starting page keeps its path:http://localhost:3000/signupruns ashttps://<tunnel-host>/signup. The same tunnel lifetime applies (30 minutes by default, so the study must finish within it, andseldon runwaits exactly as long as its tunnel lives), and--no-waitisn't allowed. The tunnel closes as soon as all persona runs have finished (All runs finished — closed the local tunnel.), before the report is written.Apps that start a service worker before rendering may load blank through the tunnel. A dev setup like Mock Service Worker under
vite devwaits for its worker before rendering, and on backends without the agent's worker fix (blok-maind7c82e59d) the worker's own requests are refused by Cloudflare Access, so the page stays blank for the personas even though it works in your own browser. Until your backend has that fix, run studies against a production build (vite build && vite preview, ornext build && next start), or turn the service worker off for the study. Withvite preview, put your API proxy and allowed hosts underpreview(preview.proxy,preview.allowedHosts), notserver. To check a tunnel yourself,scripts/tunnel-access-check.shin this repo opens one, prints its Cloudflare Access headers so you can browse it, and runs a few curl checks.Not yet supported for usability runs:
--fail-on(there's no pass/fail gate),--label/--name, and App and StaticAsset studies (device_typemust beWebsite).When a study can't start (for example a persona name that doesn't exist in your workspace), the CLI lists each name that didn't resolve, or each invalid field, on its own line, and nothing is created.
Credentials work like QA's
run.username/run.password: put ausernameandpassword(always together) undertarget.credentials, and the personas sign in before testing. Any other key is an error. Leavecredentialsout and the server uses a login saved in the app whose site matches the target URL, if there is one; otherwise the personas stay logged out. Usecredentials: noneto make sure they stay logged out even when a saved login matches. A localhost target is reached through the tunnel's address, which no saved login matches, so put the login in the file.The report prints only when the run left one, in
seldon runand inseldon report:- With a report: the CLI prints it. If the run didn't end as
Completed, the report is labelled partial and the exit code is 1. - Without a report: the CLI prints why and the app link, doesn't fetch the report, and exits 1.
- No report covers a run whose persona runs all failed, a stalled run, a launch whose setup failed (safe to re-run), and a run whose dispatch was lost. The last two are caught about 30 seconds after launch.
- With a report: the CLI prints it. If the run didn't end as
The report is short by default. It shows the study's question, the one-line answer and the top three findings (severity, title, personas and the recommended fix), then a link to the full report in the app:
Seldon — Can new visitors sign up and invite a teammate? Mostly — the invite step confused two of three personas. Top findings (3 of 5): HIGH Invite button has no label — Budget-conscious founder, Skeptical IT admin Fix: Label the button … … +2 more findings Full report: https://<base URL>/experiments/<run id>The link is the server's own link to the run in the app:
seldon rungets it when it launches the study, andseldon reportgets it from the experiment (or, on a backend that doesn't send it yet, from.seldon/last-run-app-url, saved when this project launched that run). Only when neither is available is it built from the CLI's base URL (--base-url, elseSELDON_BASE_URL, else the saved config), which opens the app only when the app and the API share a host.--mdprints the Markdown summary instead (capped, for pull requests and agents),--fullprints it with every finding and observation, and--jsonprints the summary object.seldon reporton a run that's still going says there's no report yet.- With
--json, nothing is printed to stdout when there's no report, so check the exit code.
seldon logs --followstreams a study's agent thinking and stops when the run ends.
Commands
| Command | What it does |
| --- | --- |
| seldon init | Scaffold a template seldon.yml |
| seldon init --type usability [name] [--dir <dir>] | Scaffold a <name>.usability.yml study |
| seldon cases add | Append a test case (--name --step … --criteria) |
| seldon cases | List the cases in the current seldon.yml |
| seldon validate [--file <file>] [--type qa\|usability] | Check seldon.yml — schema locally, then plan ceilings and login server-side (--local for schema only, --json for the full verdict) |
| seldon run --file <name>.usability.yml | Run a usability study and print a short report with a link to the app |
| seldon run [--url <url>] | Ingest seldon.yml and start it, printing the run id and returning; --wait to stay for the report, --fail-on to gate (implies --wait), --label <tag> to profile. A local --url is auto-exposed via a Cloudflare tunnel (--auto-approve to skip the prompt) and always waits |
| seldon report [--md \| --json] [--full] | Show the report for the last run (or --id <run>); for a usability run, a short report with a link to the app (--md / --full for the Markdown summary) |
| seldon logs --follow | Stream the agent's live reasoning |
| seldon runs | List QA runs with color-coded status |
| seldon cancel [--id <run>] · --all | Stop this project's last run, a specific one, or every in-flight run your key can see |
| seldon list | List baselines your key can see |
| seldon personas [--limit <n>] [--offset <n>] | List the personas your key can see (name and description, 10 at a time), for a study's personas: list |
| seldon links | List the links your key can see, with a masked preview of each stored login |
| seldon links create --name <n> --url <u> | Register a link, optionally with a login (--username, and --password-stdin rather than --password) |
| seldon links update <id> · --clear-credentials | Rotate or remove the login on a link, or rename/repoint it |
| seldon links delete <id> | Remove a link and the credential stored on it |
| seldon config · login | Manage credentials and settings |
| seldon api <operation> | Call any Blok API operation directly |
Run seldon --help or seldon <command> --help for the full reference.
Links: the logins your runs sign in with
A link pairs a site with the login a QA run should use. When a seldon.yml declares no login of its
own, a run matches its target URL against this registry — which is how a credential reaches a run
without being committed to a repo.
printf '%s' "$STAGING_PASSWORD" | seldon links create \
--name Staging --url https://staging.example --username [email protected] --password-stdin
seldon linksPrefer --password-stdin. --password <value> works, but it lands in shell history and is
visible in the process table for the life of the command.
An empty password is refused rather than obeyed. Server-side an empty password clears the stored
one, and the way to write that by accident is --password "$STAGING_PASSWORD" in CI with the
variable unset — which would wipe a working credential and send the next run out signed out. Use
--clear-credentials when you mean it.
A stored password is never readable again. What comes back is a masked preview, enough to tell one credential from another:
ID NAME URL CREDENTIALS EXPERIMENTS
dom-1 Staging (default) https://staging.example.com qa-a******** / ********a27b 12The halves are masked from opposite ends: the username shows its first characters (up to 4, never all of it — a 4-character username shows 3, a 1-character one shows none) and the password shows its last (nothing below 8 characters, then up to 4 and never more than a quarter). The mask is a fixed 8 characters either way, so neither preview discloses a length.
Every preview is rendered by the server and printed verbatim; the CLI masks nothing itself, so there
is exactly one rule and it cannot drift between surfaces. A link registered before previews existed
prints set (no preview) — its login works, there is just none on file until the password is
re-saved.
Tagging runs for profiling
Attach freeform labels to a run — persistent ones in seldon.yml (labels: [...]) and per-invocation ones on the command line — to slice usage later:
seldon run --label nightly --label team-webLabels are merged (file + flags), de-duplicated, and shown in seldon runs alongside each run.
Staying up to date
seldon --version prints the installed version. When a newer release is available, the CLI shows a one-line upgrade nudge with the command that matches how you installed it — npm install -g @seldonqa/cli or brew upgrade seldon. The check is cached and never blocks a command; silence it with NO_UPDATE_NOTIFIER=1 (it's already quiet in CI and when output is piped).
Coding agents (MCP)
Seldon is also available as an MCP server, so coding agents can author baselines, run them, and read reports directly — the same report you see, rendered as Markdown — enabling a closed test-and-fix loop. See @blok/mcp.
Requirements
- Node.js 20 or newer
- A Seldon API key
Run seldon --help to get started.
