@haystackeditor/cli

haystack verify runs your app with and without your change, explores both side by side, and reports where the change shows up and what it broke.

Quick Start

npm install -g @haystackeditor/cli
haystack login   # GitHub sign-in: open the URL it prints and enter the code
haystack init    # shows what it will change, then starts onboarding your app
haystack verify  # after a change: crawl the app with and without it

There is no configuration file: Haystack works out how to run your app from what the repository already keeps working (lockfiles, Dockerfiles, compose files, package scripts). Or have your coding agent do the setup — paste this into Claude Code, Codex or Cursor:

Set up Haystack for me: fetch https://haystack.sh/agents.md and follow it

Requires Node.js 22.18 or newer.

After a command finishes in an interactive terminal, the CLI checks npm at most once a day and prints one line on stderr when a newer version exists. It never runs with --json, when stdout is not a TTY, or when CI is set. Set HAYSTACK_NO_UPDATE_CHECK=1 to turn it off.

The CLI sends limited operational events to PostHog by default so Haystack can measure setup reliability. Events contain a pseudonymous install identifier (a random id generated once and stored in ~/.haystack/telemetry-id; it is not derived from your username or hostname), CLI/version/platform fields, command-step and error categories, and numeric outcomes. They exclude source code, repository names, local paths, usernames, hostnames, and raw error text. IP enrichment is disabled. Set DO_NOT_TRACK=1 or HAYSTACK_TELEMETRY_DISABLED=1 to turn this telemetry off.

For coding agents

Most users of this CLI are coding agents. What to know:

• After a change, two steps inside the checkout (nothing needs to be committed or pushed):
  1. haystack verify is the handshake: it shows what the change touches (first its flows, the behaviors it changes), what production and real users do there, and Haystack's ideas of what could break, then asks you to steer (exit 4). haystack verify flow <n> drills into one flow: its code, the pages it shows on with their real use, and what its code saw in production.
  2. haystack verify --intent "<the task, in the user's words>" --idea "<as you would tell a tester>" explores the app with your steering: keep the ideas worth trying, add your own (repeat --idea); yours are explored first.
• Keep the app's seed in .haystack.json: "data": { "seed": "<one command>", "files": ["<every file it reads>"] }. Haystack runs it before each check, and the handshake shows what it made or its error.
• First time in a repository, haystack init --yes sets it up and starts onboarding the app. haystack verify onboarding shows where onboarding is. When onboarding asks a question, answer it with haystack verify answer <question-id> <choice>.
• --json means pure stdout. stdout carries exactly one JSON document and everything else goes to stderr. A failure still prints the error envelope (haystack schema error) and exits 1. haystack schema <name> prints any command's contract.
• When Haystack gets in your way, tell us in a sentence and keep going: haystack feedback "<what happened>". The team sees it at once.

haystack verify shows the product blast radius of your current change: where it shows up in the running app, and what broke. Inside a git checkout (nothing needs to be committed or pushed), run:

haystack verify
haystack verify --no-wait
haystack verify --json

It captures the checkout, committed and uncommitted changes together, exactly as the stop hook does: the repository from origin, the base at git merge-base HEAD origin/<default branch>, and the working tree as a deterministic snapshot commit, so the same tree always names the same capture. It then finds your crawl of that exact capture: the same code and the same change title (HEAD's subject, which tells the crawl's judge what the change is; committing or rewording keeps the code but changes the title). When there is none, or the one it finds was cancelled, stopped before finishing or is being cancelled, or the stop hook only built it, it asks the service for the crawl (the service starts it, runs the stopped one again once it has shut down, or turns the hook's prepared run into a crawl that starts from the app the hook already built), says so in one line, and follows that crawl; it never shows a crawl of another revision or another title.

A crawl builds the app with and without your change, starts from the changed code, reaches it in the running app, explores outward with both builds side by side, and double-checks and judges every difference. haystack verify waits for the crawl, printing each step as it happens and each bug the moment the crawl finds it (each read is held by the service until the crawl changes, so nothing waits on a polling interval). It finishes when the crawl answers, as soon as its time is up, without waiting for its machines to shut down; there is no time limit, and Ctrl-C stops waiting, never the crawl. It then prints:

• a headline: how many bugs it found, none, or why the crawl could not finish;
• Where your change shows up in the app: every changed spot as file:line, how far the crawl got with it (never reached, loaded but never ran, shown on screen, ran, ran when its control was pressed, a difference seen, a difference seen and confirmed, judged a bug), and the steps and page that reached it;
• What the crawl found: bugs first, each with a one-sentence reason and the steps to see it;
• What the crawl never ran: changed files no path ran, and how many of the functions your change affects ran.

--no-wait prints the crawl's current state and returns. --account <login> picks a saved account, and --repo owner/repo names the repository when origin does not. --json prints the same report as one document: the headline, the findings (bugs first, each with its steps), where the change showed up, what became of each idea, and what never ran (haystack schema verify). --raw prints the crawl's whole record as the service returns it, for scripts; it is large (haystack schema verify-raw). Exit codes: 0 when the crawl answered or finished (the bugs it found are in the output) or is still running under --no-wait; 2 when it ended without finishing (stopped early, or cancelled) or the machines it used could not be proven shut down; 1 when the command failed or no crawl could be started; 3 when onboarding is blocked; 4 for the handshake (run it again with --intent); 5 when the app copy is broken and the change could not be tested.

The stop hook. haystack init installs a Claude Code Stop hook that runs haystack verify precompute --hook whenever an agent turn stops. It captures the checkout within five seconds and hands the request to a detached sender. The service then builds the app with and without the change, works out what the change touches (the brief the handshake shows) and sets up starting points next to it, and stops there: it never crawls. haystack verify then crawls from that prepared app instead of building it again. No run cancels another: each runs to its end unless it is cancelled. haystack verify precompute (without --hook) submits the same capture in the foreground and prints one line for the run it started.

Retained fleet cases remain available through their exact case IDs:

haystack verify cases <run-id> --repo owner/repo
haystack verify explore <run-id> <case-id> --repo owner/repo

CLI Commands

haystack init

Sets the repository up for haystack verify and starts onboarding the app. It shows each change as a diff and makes it only with --yes (or a yes at the prompt): Claude Code's Stop hook in the per-developer .claude/settings.local.json (kept out of commits), and a short note in AGENTS.md telling coding agents to run haystack verify after a change. Running it again changes only what is missing and shows where onboarding is.

haystack init              # Preview, then confirm
haystack init --yes --json # Apply without asking; one JSON document (haystack schema init)
haystack verify onboarding # Where onboarding is; --wait follows it

When initial setup has run successfully, the coding agent reviews the evidence from haystack verify onboarding against the repository. Recorded commands are evidence, not instructions to execute. Check the build, services, sign-in and workflow, then accept the exact review with haystack verify onboarding --accept-review <review-id>. To correct setup, write complete JSON notes with run, signIn and workflow (preserving the notes that still apply), then run haystack verify onboarding --revise-review <review-id> --notes <file> --wait. Haystack runs the corrected setup and returns fresh evidence to review. The coding agent handles setup details; the person supplies product choices or access. Setup becomes Ready after acceptance. Existing setups remain usable.

Exit codes: 0 set up, 1 failed, 2 changes shown but not made, 3 onboarding blocked, 4 not logged in, 5 the Haystack GitHub App is not installed, 6 set up but onboarding stopped before finishing (run it again).

haystack login

Authenticate with GitHub (required for setup and secrets):

haystack login

This uses GitHub's device flow - you'll get a code to enter at github.com/login/device. A coding agent runs haystack login --no-wait --json to get the URL and code without waiting, shows them to you, then runs haystack login to finish.

# Log out (removes stored credentials)
haystack logout

haystack feedback

Tell the Haystack team something went wrong, confused you, or is missing. It lands with the team at once, with the CLI version, your platform and the checkout's repository.

haystack feedback "verify said the app never started, but it was up on :3000"
haystack feedback --run cv_<id> "the bug it reported is the intended change"
echo "longer report" | haystack feedback -

haystack db profile

Describe your production Postgres database so Haystack can build a full-size stand-in for testing, without any personal data leaving your environment. You run the profiler yourself, inside your own network, against your own database; it writes a JSON file you read before you send it.

# The connection string is read from the variable you name, never from a flag.
export DATABASE_URL='postgres://[email protected]:5432/app'
haystack db profile --url-env DATABASE_URL --owner-table public.users --out profile.json

# Read exactly what would be sent, in plain English.
haystack db profile show profile.json

# Send it for one repository.
haystack db profile upload profile.json --repo acme/app

Which database it is used for. Uploaded without --dependency, the profile belongs to the repository, and Haystack builds the stand-in from it for the repository's one Postgres database, whatever id onboarding gave it. When the repository has several (or none), haystack verify hosted start says the stand-in was not used, and why, naming the ids; upload again with --dependency <id> to pick one. A profile uploaded with --dependency <id> is used for that database; when both apply, the newer upload is used. haystack verify hosted start prints one line saying which profile the stand-in was built from, or why it was not used, for every repository that has uploaded a profile.

Connection strings. Any postgres:// or postgresql:// URL that node-postgres accepts works, including a Unix socket (postgresql://profiler_readonly@/app?host=/var/run/postgresql).

What it does to your database. It only reads. Every query is a single SELECT in its own READ ONLY transaction (the session default is read only too, and the transaction's read-only state is checked before each query), with a statement timeout and a 5-second lock timeout so it never queues behind a migration. Each transaction is rolled back. All of them read one REPEATABLE READ snapshot, so every number describes the same moment even while your application writes; one extra connection holds that snapshot open for the run (on a primary, rows deleted during the run are cleaned up by vacuum only after it ends). A role with SELECT on the application tables is enough, and is what we recommend; tables with row-level security active for that role, or that the role cannot read, stop the run with their names. It works on Postgres 12 and later (tested on 16).

Run it on a read replica. It works on a streaming-replication hot standby (tested on Postgres 16): nothing it runs writes, takes a write lock or creates a temporary table, and each transaction names its isolation level (REPEATABLE READ), so a replica whose default is serializable still answers. A replica does not count dead rows, so a profile taken there marks each table's dead rows as unavailable and says why; everything else is the same as on the primary. A replica may cancel a long query when replaying the primary's changes cannot wait for it (canceling statement due to conflict with recovery), and the run's snapshot is held for its whole length; the run then stops and names the fix: turn on hot_standby_feedback on the replica, or raise its max_standby_streaming_delay above --statement-timeout.

What leaves your database. The database computes every count and pattern; no row ever reaches the profiler. A column's actual values are written only when all three hold:

1. the column is a category: at most --category-max-distinct distinct values, counted exactly over the whole table even when the table is otherwise sampled (never estimated, so a column with many rare values cannot pass);
2. the column is not personal. Personal means names, emails, phones, addresses, free text (notes, comments, descriptions, messages), secrets, tokens and passwords, IP and network addresses, dates of birth and government IDs. It is decided from the column's name, from its type (inet, cidr, macaddr), and from value shapes the database measures (the share of values shaped like an email, phone number, IP address, secret, SSN, card number or "First Last" name, and the number of words per value). When in doubt a column is personal, and the profile records why;
3. each value is shared by at least --category-min-owners owners. Owners are the distinct rows of --owner-table (your users or organizations) reached by following foreign keys from the column's table, and only owners that exist there count (a key left dangling under a NOT VALID foreign key owns nothing). A table with no foreign-key path to the owner table keeps no values, shapes, JSON keys, percentiles or shares of true. Without --owner-table, each row counts as an owner and the profile says so; a value one heavy user repeats in thousands of rows then passes, so name an owner table whenever you have one.

Values your schema declares (enum labels, CHECK (col IN (...)) lists) need no data and are kept with their share of rows. Everything else leaves only as numbers: row counts (and whether each was counted exactly or is the planner's estimate), how each table was read (in full, or the sample percentage and seed), table/index/TOAST sizes, dead-row share (or why it is unavailable), null share, distinct count, average width, the share of true in each boolean column, and the 5th/25th/50th/75th/95th percentiles of lengths and numbers (never the minimum or maximum; percentiles and the share of true are withheld for a column with fewer than --category-min-owners owners, and number percentiles are withheld for personal columns, because a percentile of birth dates is a birth date). Character shapes (Aa Aa, +9 (9) 9-9) and JSON key paths leave only when shared by at least --category-min-owners owners, and a JSON key path never leaves when any key on it looks like an email, phone number, IP address, SSN, card number or secret (15551234567.verified stays home). Odd rows (wrong type, oversize, bad encoding, rare nulls, rare shapes, missing JSON keys) are counted and described without their values, even for a single row. Index definitions are sent with every literal replaced by ?, since a partial index can name a value (WHERE email <> '[email protected]' is sent as WHERE (email <> ?::text)); the column names, operators and types stay. Settings are an allowlist of planner and storage settings; connection, authentication, SSL and file settings are never read.

show prints every string the file carries. upload refuses a file with any field show does not print, a file whose percentiles include anything but p5 to p95, and a file with a literal left in an index definition. Haystack's server checks the file again and also refuses one made with a looser rule than at most 50 distinct values and at least 50 owners, a kept value its row share shows fewer than --category-min-owners rows hold, and any kept value shaped like an email, phone number, IP address, SSN, card number, token or password hash, whatever its column's type, or that does not fit its column's declared type (an integer column keeps only integers, a boolean column true or false, a date or time column dates or times; a column of any other type than text, numbers, booleans, dates and times keeps no data values, and enum labels leave as schema values). The profiler never writes such a value itself: however many owners share it (a date like 2031-07-19 has a phone number's shape, and so does any number of seven or more digits), it is withheld like a value too few owners share. An accepted file is stored once, under its SHA-256, in Haystack's results storage.

Knobs and what turning them does:

| Flag | Default | Turning it up | Turning it down | |------|---------|---------------|-----------------| | --owner-table <schema.table> | none (rows are owners) | n/a | n/a; naming one makes the owner threshold count people or organizations instead of rows, which is stricter and what we recommend. | | --schema <name> (repeatable) | every non-system schema | n/a | Profiles fewer schemas; foreign-key paths to the owner table only run through profiled tables. | | --category-max-distinct <n> | 50 (1 to 1000) | More columns count as categories and can keep values (each still needs enough owners); upload accepts at most 50. | Fewer columns keep values; more leave only as numbers and shapes. | | --category-min-owners <n> | 50 (5 to 1,000,000) | Stricter: fewer values, shapes, JSON keys and percentiles pass. | More rare values pass; each one is shared by fewer people. Not below 5; upload requires at least 50. | | --scan-max-mb <mb> | 256 | Bigger tables are read in full: exact counts and more rare-but-common-enough values, at the cost of more reading on your database. | Tables above this are read through TABLESAMPLE SYSTEM of about this size (same pages for every query). Owner counts in a sample are lower than the truth, so fewer values pass (never more). Row counts of sampled tables come from the planner's statistics, so a sampled table must have been analyzed; the profile records the sample percentage and seed and marks the row count as an estimate. Distinct counts of a sampled table are estimated from the sample (a single-column primary or unique key counts one per row, and a validated single-column foreign key counts the parent rows it references), except a column that could be a category: its distinct values are counted exactly, which reads that table in full once for all such columns (the run says which tables). | | --concurrency <n> | 2 (1 to 8) | Faster, more simultaneous load on your database. | Gentler; 1 runs one query at a time. Either way one more connection holds the run's snapshot. | | --statement-timeout <seconds> | 120 | Allows longer queries on big tables. | Caps each query sooner; a query that hits it stops the run and names the table. |

Every failure is one line naming what failed. Nothing is written unless the whole profile succeeds. On a primary, dead-row shares come from pg_stat_user_tables; a table with rows but no counts there (statistics reset) stops the run and asks for ANALYZE on that table.

How It Works

1. haystack init starts onboarding: Haystack reads the repository, builds the app, starts it with its databases and services, prepares its data and test accounts, and proves one workflow works. Your coding agent reviews the setup evidence and accepts it or supplies corrections for Haystack to run again. It needs the Haystack GitHub App on the repository's owner.
2. After a change, haystack verify builds the app at the base and with your change on Haystack's isolated machines (no internet access), explores both side by side starting from the changed code, and judges every difference.
3. It prints where your change shows up, the bugs it found with the steps to see each, and the changed code it never ran. See https://haystack.sh/docs.

License

MIT