npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2026 – Pkg Stats / Ryan Hefner

@seldonqa/cli

v0.15.0

Published

Seldon QA - generate a QA test plan from any codebase and run AI agents against your live app, from your terminal or CI.

Readme

Seldon QA

Write your QA test plan as code, then run AI agents that use your product like a real user — from your terminal or CI.

seldon keeps a baseline of user-facing test cases in a committed seldon.yml. Point it at a live URL and autonomous agents exercise each case — click, type, navigate, verify — then hand back a clean, actionable report for you or your coding agent to act on.

npm install -g @seldonqa/cli

Why

AI is shipping more code than any team can test by hand. Seldon closes the gap: describe the journeys that matter once, in plain English, and agents run them against every deploy — telling you exactly what broke, why, and what to do about it.

Get started

npm install -g @seldonqa/cli
seldon login seldon_your_api_key       # or: seldon config set-key <key>

From inside a project (one baseline per project):

seldon init                            # scaffold a template seldon.yml
seldon cases add --name "Login works" \
  --step "go to /login" --step "sign in" --criteria "lands on the dashboard"
seldon validate                        # schema, plus your plan's case ceilings, before you commit to a run
seldon run --url https://staging.example.com   # prints the run id and returns
seldon logs --follow                   # watch the agents test, live
seldon report --md                     # clean, actionable results

Highlights

  • Test plan as code. Your cases live in a committed seldon.yml — reviewed in PRs, versioned with your app. seldon init scaffolds it; seldon cases add appends more.
  • Agents that behave like users. seldon run drives your live app and judges each case PASS / FAIL / BLOCKED / TIMED_OUT.
  • Reports built for action. Every failure carries an expected / observed pair and a single concrete recommendation — --md for pull requests, --json for coding agents, colorized plain text for the terminal.
  • Watch it think. seldon logs --follow streams the agent's reasoning, framed by test case and step.
  • Start and walk away. seldon run submits and returns the run id. Check in with seldon report, watch with seldon logs -f, stop it with seldon cancel. Add --wait to stay for the report.
  • Never loses a run. The id is printed the moment it exists, and if the submit itself times out the CLI goes and finds the run rather than leaving one burning with nobody holding its id.
  • CI-native gate. seldon run --fail-on fail exits non-zero so a red QA run can block a merge (and implies --wait).

Configuration — seldon.yml

Your baseline is a single committed file at the project root — the plain-English cases themselves, not id overrides:

name: web-app                          # optional lineage name; defaults to the git remote owner/repo
labels: [nightly, team-web]            # optional run tags for profiling; merged with `--label`
run:
  url: https://staging.example.com     # default target; override per run with --url
  username: [email protected]             # optional login the agent signs in with
  password: hunter2                    # optional; prefer a managed secret in prod
testCases:
  - name: Login works
    id: login-works                    # optional stable id; defaults to a slug of the name
    url: /login                        # optional per-case path (relative) or absolute URL override
    steps:
      - go to /login
      - enter valid credentials and submit
    criteria: the user is signed in and lands on the dashboard

Reading a report

The report lists every case worst-first, then expands each failure:

Seldon QA — Checkout & account flows · 2/6 passed (33%) · ❌2 · 🚫1 · ⌛1

FAIL      BLOCKER  TC-201  Complete payment
BLOCKED            TC-301  View order history
PASS               TC-101  Add item to cart

TC-201  Complete payment
  acceptance:     A valid card places the order and shows a confirmation.
  expected:       Order is placed and a confirmation screen appears.
  observed:       The Pay button spins indefinitely; no order is created.
  recommendation: Wire the Pay button's submit handler to /orders and surface a failure toast.
  repro:          Add an item, open /checkout, enter a valid card, click Pay
  • Verdicts: PASS · FAIL (a real defect, with a severity) · BLOCKED (couldn't be evaluated — e.g. a missing precondition) · TIMED_OUT.
  • recommendation appears on FAIL / BLOCKED cases: the single strongest next step to get the case passing.
  • --full expands every failing case (the terminal view caps at 5 by default); --json emits the same data as a stable object.

In CI

seldon run --url "$PREVIEW_URL" --fail-on fail

--fail-on fail returns a non-zero exit code when any case fails; --fail-on any also gates on blocked or timed-out cases. --fail-on implies --wait, because there is no verdict to gate on until the run ends — a gate that returned immediately would always pass. With --json the report is stable, machine-readable output you can post to a pull request or hand a coding agent for an automated fix loop.

Detaching, checking in, and stopping

seldon run starts the run and returns. A run takes minutes, and a terminal held open for it is one timeout away from losing track of it entirely:

Run started — exp_01HX…
  progress seldon report --id exp_01HX…
  live     seldon logs -f --id exp_01HX…
  stop     seldon cancel --id exp_01HX…

The id is written to .seldon/last-run, so the same three commands work with no --id at all. --wait keeps the command attached and prints the report at the end; --fail-on and a local --url turn that on for you, because a gate needs a result to gate on and a tunnel dies with the process holding it.

If the submit itself times out, the CLI does not report that as a failed run — it isn't one. Aborting the request does not cancel the server, so it looks for the run the submit may have started and names it:

The server didn't answer in time, so this command never learned the run's id.
That does NOT mean nothing started — looking for it now ...
Found it — exp_01HX… (Running). It is running 42 case(s).

It claims a run only when exactly one recent run matches this project; with two it tells you to look at seldon runs --running rather than pointing seldon cancel at a coin flip.

Plan limits

Your workspace has two ceilings, both counted in test cases: how many one run may dispatch, and how many you may have in flight at once. They follow your commercial tier (trial is capped; usage and enterprise are not) and live in the backend, so they can be raised for your account without a CLI release.

seldon validate checks a file against the ones that actually apply to you, before you commit to a run:

$ seldon validate
Not runnable — 250 test cases.

  case_limit_exceeded
  This run would dispatch 250 test cases, but the trial plan allows 20 per run. Mark the cases you
  want with 'only: true' (or 'skip: true' on the rest) to bring it to 20 or fewer, then run again.

plan 'trial' · 20 case(s) per run · 20 in flight · 0 running now

seldon run enforces the same ceilings and refuses in the same words — before anything is written, so an oversized suite costs nothing. A concurrency refusal clears on its own: wait for the current run, or clear the decks with seldon cancel --all (every in-flight run your key can see; one that can't be stopped doesn't hold up the rest). --local skips the server check (schema only, no API key needed) and --json prints the whole verdict for scripts.

One answer, whichever surface you ask

A broken seldon.yml is caught in one of two places: by the CLI's own parser before anything is sent, or by the server after it is. You should not be able to tell which. seldon validate, seldon run, the MCP validate_seldon_yaml and run_seldon tools, and the GitHub App's PR check all report the same code, point at the same field, and give the same advice:

$ seldon validate --local
seldon.yml isn't valid at testCases[0].criteria: must be a non-empty string. Every test case needs
a non-empty name, at least one step, and a single criteria; only/skip must be literal true or false.
// MCP validate_seldon_yaml, same file
{ "ok": false, "code": "invalid_field", "field": "testCases[0].criteria", "error": "…the same sentence…" }

Only the parser's own detail differs between local and server (must be a non-empty string versus Field required), because the two genuinely detect it differently. Everything a tool branches on, and every word of advice, is identical — and pinned by a conformance fixture both repos hold byte-identically, so it cannot drift.

Four codes are server-only, because the local parser sees the file and nothing else: your plan's ceilings (case_limit_exceeded, concurrency_limit_exceeded), the resolved target (no_target_url), and credentials (invalid_credentials — they are vaulted server-side and never part of the plan the CLI parses).

Every refusal carries a stable code alongside its sentence. The full catalogue — what each code means and what fixes it — is in docs/seldon-validation-errors.md.

Local targets — automatic tunnels

Point --url at a local address — localhost, *.localhost, any 127.x.x.x, 0.0.0.0, ::1, or a bare :PORT — and Seldon exposes it to the QA agent through a temporary, authenticated Cloudflare tunnel for the duration of the run. No deploy, no manual tunnelling:

seldon run --url http://localhost:3000

First the CLI checks that something is actually listening there. If nothing answers, it stops before doing anything else: Nothing is answering at http://localhost:3000. Start your app (or fix the port in the URL), then run this again. Then you'll see a one-line warning that the local address will be exposed and a [Y/n] prompt (defaults to yes). The public hostname is protected by Cloudflare Access, so only this run's agent can reach it, and the tunnel is torn down automatically when the run finishes or you press Ctrl-C. In CI or any non-interactive shell the prompt is skipped automatically; pass --auto-approve to skip it explicitly:

seldon run --url localhost:3000 --auto-approve   # e.g. in a CI job

Remote --urls are used as-is (no tunnel).

Ctrl-C during a tunneled run deletes the tunnel straight away and exits. The run itself keeps going on the server but can no longer reach your app, so the CLI tells you to cancel it: Detached — the local tunnel was closed, so run <id> can no longer reach your app. Cancel it with: seldon cancel --id <id>. Pressed before the run has started, it says Stopped before the run started — nothing was left running on the server. If the CLI is killed outright (kill -9, or the terminal is closed), it can't clean up: the tunnel stays open until the server reclaims it, and cloudflared may keep running (pkill -f 'cloudflared tunnel run' stops it).

A local URL always waits. The tunnel only stays open while seldon run is running, so returning early would close it under a run that had just started. run therefore stays attached for a local target even without --wait, and says so (waiting because the target is local — its tunnel is only open while this command runs). Point --url at a public URL if you want to detach.

Tunnels are short-lived. Each tunnel has a hard maximum lifetime (30 minutes by default) and is reclaimed by the server once it's exceeded, regardless of the run's state — so the run must finish within that window. When a tunnel comes up the CLI prints the exact cap for your account (… active for up to 30 min; the run must finish within that window). Runs that legitimately need longer should target a deployed/public --url instead of a local one.

If your account has too many tunnels open at once you'll get Tunnel limit reached — wait a minute and re-run; if the same local target is already being provisioned you'll get a short retry hint. Neither starts a run — just re-run once things settle. If the retry hint doesn't go away, an earlier tunnel for that address couldn't be fully cleaned up and is still holding it; it's cleared by the server's scheduled tunnel cleanup. Meanwhile the same app is reachable under another address, e.g. 127.0.0.1 instead of localhost.

One local target, one run at a time. Runs against the same local URL in the same workspace share a single tunnel. That includes two terminals, a terminal and a CI job, or two teammates. Whichever run finishes first closes the tunnel, and the other run loses access to your app. Until this is fixed, run them one after another.

Usability studies

Alongside QA, a project can keep usability studies as code: personas from your workspace attempt a goal on your app, and you get back a summary of where they struggled. Each study is one committed <name>.usability.yml file at the project root, next to seldon.yml. Every key is required except version and credentials:

version: 1               # optional
experiment:
  title: Trial signup — first-time visitors
  goal: Sign up for a free trial and invite one teammate to your workspace.
  hypothesis: The pricing step is where hesitation will show up.
  type: Onboarding        # Onboarding | Task Completion | Churn | Conversion |
                          #           Navigation Clarity | Feature Adoption | Trust Signals
target:
  device_type: Website    # Website only for now
  frame_type: Desktop     # Desktop | Mobile
  url: https://staging.example.com/signup
  # Optional login (for apps that gate behind sign-in). Committed like a .env value
  # (fine for staging/dev; use a managed secret for production). The agents sign in before testing.
  credentials:
    username: [email protected]
    password: hunter2
  # To force a logged-out run, use credentials: none.
personas:                 # at least one persona name from your workspace
  - Budget-conscious founder
  - Skeptical IT admin
seldon init --type usability trial-signup             # writes ./trial-signup.usability.yml
seldon init --type usability trial-signup --dir studies   # …or into another directory (created if missing)
seldon validate --file trial-signup.usability.yml
seldon run --file trial-signup.usability.yml          # waits, then prints a short report (--md / --full / --json)
seldon report                                         # re-show the last run's report
seldon personas                                       # the persona names your workspace has (--limit / --offset to page)
  • Which kind of file? A file named *.usability.yml is always a usability study, so a study with a YAML error or a misspelled key gets usability errors rather than QA ones. Otherwise the CLI decides from the content: a top-level experiment: block is a usability study, testCases: is a QA suite. Pass --type usability (or --type qa) to run / validate to make that explicit; a contradicting --type is an error.

  • Checked like the server checks it. Unknown keys are an error at every level, listing the allowed ones, so a typo such as credential: can't be dropped silently. version must be 1. target.url, and --url on seldon run, must be a full http(s) URL such as https://staging.example.com or http://localhost:3000 (not localhost:3000).

  • Names, not ids. Personas and the experiment type are written by name; the server resolves them when the run starts. A typo fails the run before anything is created.

  • Local URLs (localhost, 127.0.0.1, …), whether in target.url or --url, are exposed through the same temporary Cloudflare tunnel as a QA run, after the same [Y/n] prompt (skipped with --auto-approve, in CI, or when not interactive). The study's starting page keeps its path: http://localhost:3000/signup runs as https://<tunnel-host>/signup. The same tunnel lifetime applies (30 minutes by default, so the study must finish within it, and seldon run waits exactly as long as its tunnel lives), and --no-wait isn't allowed. The tunnel closes as soon as all persona runs have finished (All runs finished — closed the local tunnel.), before the report is written.

  • Apps that start a service worker before rendering may load blank through the tunnel. A dev setup like Mock Service Worker under vite dev waits for its worker before rendering, and on backends without the agent's worker fix (blok-main d7c82e59d) the worker's own requests are refused by Cloudflare Access, so the page stays blank for the personas even though it works in your own browser. Until your backend has that fix, run studies against a production build (vite build && vite preview, or next build && next start), or turn the service worker off for the study. With vite preview, put your API proxy and allowed hosts under preview (preview.proxy, preview.allowedHosts), not server. To check a tunnel yourself, scripts/tunnel-access-check.sh in this repo opens one, prints its Cloudflare Access headers so you can browse it, and runs a few curl checks.

  • Not yet supported for usability runs: --fail-on (there's no pass/fail gate), --label / --name, and App and StaticAsset studies (device_type must be Website).

  • When a study can't start (for example a persona name that doesn't exist in your workspace), the CLI lists each name that didn't resolve, or each invalid field, on its own line, and nothing is created.

  • Credentials work like QA's run.username / run.password: put a username and password (always together) under target.credentials, and the personas sign in before testing. Any other key is an error. Leave credentials out and the server uses a login saved in the app whose site matches the target URL, if there is one; otherwise the personas stay logged out. Use credentials: none to make sure they stay logged out even when a saved login matches. A localhost target is reached through the tunnel's address, which no saved login matches, so put the login in the file.

  • The report prints only when the run left one, in seldon run and in seldon report:

    • With a report: the CLI prints it. If the run didn't end as Completed, the report is labelled partial and the exit code is 1.
    • Without a report: the CLI prints why and the app link, doesn't fetch the report, and exits 1.
    • No report covers a run whose persona runs all failed, a stalled run, a launch whose setup failed (safe to re-run), and a run whose dispatch was lost. The last two are caught about 30 seconds after launch.
  • The report is short by default. It shows the study's question, the one-line answer and the top three findings (severity, title, personas and the recommended fix), then a link to the full report in the app:

    Seldon — Can new visitors sign up and invite a teammate?
    
    Mostly — the invite step confused two of three personas.
    
    Top findings (3 of 5):
      HIGH    Invite button has no label — Budget-conscious founder, Skeptical IT admin
              Fix: Label the button
      …
      … +2 more findings
    
    Full report: https://<base URL>/experiments/<run id>

    The link is the server's own link to the run in the app: seldon run gets it when it launches the study, and seldon report gets it from the experiment (or, on a backend that doesn't send it yet, from .seldon/last-run-app-url, saved when this project launched that run). Only when neither is available is it built from the CLI's base URL (--base-url, else SELDON_BASE_URL, else the saved config), which opens the app only when the app and the API share a host. --md prints the Markdown summary instead (capped, for pull requests and agents), --full prints it with every finding and observation, and --json prints the summary object.

    • seldon report on a run that's still going says there's no report yet.
    • With --json, nothing is printed to stdout when there's no report, so check the exit code.
  • seldon logs --follow streams a study's agent thinking and stops when the run ends.

Commands

| Command | What it does | | --- | --- | | seldon init | Scaffold a template seldon.yml | | seldon init --type usability [name] [--dir <dir>] | Scaffold a <name>.usability.yml study | | seldon cases add | Append a test case (--name --step … --criteria) | | seldon cases | List the cases in the current seldon.yml | | seldon validate [--file <file>] [--type qa\|usability] | Check seldon.yml — schema locally, then plan ceilings and login server-side (--local for schema only, --json for the full verdict) | | seldon run --file <name>.usability.yml | Run a usability study and print a short report with a link to the app | | seldon run [--url <url>] | Ingest seldon.yml and start it, printing the run id and returning; --wait to stay for the report, --fail-on to gate (implies --wait), --label <tag> to profile. A local --url is auto-exposed via a Cloudflare tunnel (--auto-approve to skip the prompt) and always waits | | seldon report [--md \| --json] [--full] | Show the report for the last run (or --id <run>); for a usability run, a short report with a link to the app (--md / --full for the Markdown summary) | | seldon logs --follow | Stream the agent's live reasoning | | seldon runs | List QA runs with color-coded status | | seldon cancel [--id <run>] · --all | Stop this project's last run, a specific one, or every in-flight run your key can see | | seldon list | List baselines your key can see | | seldon personas [--limit <n>] [--offset <n>] | List the personas your key can see (name and description, 10 at a time), for a study's personas: list | | seldon links | List the links your key can see, with a masked preview of each stored login | | seldon links create --name <n> --url <u> | Register a link, optionally with a login (--username, and --password-stdin rather than --password) | | seldon links update <id> · --clear-credentials | Rotate or remove the login on a link, or rename/repoint it | | seldon links delete <id> | Remove a link and the credential stored on it | | seldon config · login | Manage credentials and settings | | seldon api <operation> | Call any Blok API operation directly |

Run seldon --help or seldon <command> --help for the full reference.

Links: the logins your runs sign in with

A link pairs a site with the login a QA run should use. When a seldon.yml declares no login of its own, a run matches its target URL against this registry — which is how a credential reaches a run without being committed to a repo.

printf '%s' "$STAGING_PASSWORD" | seldon links create \
  --name Staging --url https://staging.example --username [email protected] --password-stdin
seldon links

Prefer --password-stdin. --password <value> works, but it lands in shell history and is visible in the process table for the life of the command.

An empty password is refused rather than obeyed. Server-side an empty password clears the stored one, and the way to write that by accident is --password "$STAGING_PASSWORD" in CI with the variable unset — which would wipe a working credential and send the next run out signed out. Use --clear-credentials when you mean it.

A stored password is never readable again. What comes back is a masked preview, enough to tell one credential from another:

ID      NAME               URL                           CREDENTIALS                  EXPERIMENTS
dom-1   Staging (default)  https://staging.example.com   qa-a********  /  ********a27b          12

The halves are masked from opposite ends: the username shows its first characters (up to 4, never all of it — a 4-character username shows 3, a 1-character one shows none) and the password shows its last (nothing below 8 characters, then up to 4 and never more than a quarter). The mask is a fixed 8 characters either way, so neither preview discloses a length.

Every preview is rendered by the server and printed verbatim; the CLI masks nothing itself, so there is exactly one rule and it cannot drift between surfaces. A link registered before previews existed prints set (no preview) — its login works, there is just none on file until the password is re-saved.

Tagging runs for profiling

Attach freeform labels to a run — persistent ones in seldon.yml (labels: [...]) and per-invocation ones on the command line — to slice usage later:

seldon run --label nightly --label team-web

Labels are merged (file + flags), de-duplicated, and shown in seldon runs alongside each run.

Staying up to date

seldon --version prints the installed version. When a newer release is available, the CLI shows a one-line upgrade nudge with the command that matches how you installed it — npm install -g @seldonqa/cli or brew upgrade seldon. The check is cached and never blocks a command; silence it with NO_UPDATE_NOTIFIER=1 (it's already quiet in CI and when output is piped).

Coding agents (MCP)

Seldon is also available as an MCP server, so coding agents can author baselines, run them, and read reports directly — the same report you see, rendered as Markdown — enabling a closed test-and-fix loop. See @blok/mcp.

Requirements

  • Node.js 20 or newer
  • A Seldon API key

Run seldon --help to get started.