testforge

AI-powered QA automation. Point it at a project and it scans the code, drafts a journey doc for each feature, and generates seeded Jest and Playwright tests that match the project's own structure.

Works in any project, with any AI CLI (defaults to Claude).

Install

npm install -D @activate-core/test-forge
```

testforge calls a local AI CLI to do its work. By default it uses
`claude -p`, so make sure the Claude CLI is installed and logged in.
To use a different CLI, see [Configuration](#).

## Quick start

From your project root:

```bash
testforge setup      # one-time: inspect project, map it, draft journey docs
testforge generate   # generate tests + seed files from the docs

After setup, review the drafted docs in docs/ and fix anything the AI got wrong — they are the source of truth for the tests. Then run generate.

When your code changes later:

testforge update     # re-check and refresh only what changed

Working on one journey

update is the whole-project command: if the code has moved it re-maps, re-drafts every doc and regenerates. Most of the time you want one journey, and both steps take --journey (or --area):

# the spec for one journey changed — re-draft just that doc, then its tests
testforge discover --journey payment
testforge generate --journey payment

# adding a journey that the reference now describes
testforge discover --journey refund-flow
testforge generate --journey refund-flow

# everything in one area
testforge generate --area ops

A feature that arrives on its own document

A ticket turns up with a PDF, or a spec for one feature. Point discover at it directly — you do not name the journey, the AI reads the document and works out which journey it describes:

testforge discover --reference ./docs/tickets/zh-4521.pdf

A reference only writes the journeys it actually covers. The ticket's PDF has never heard of your other journeys, so their docs are left exactly as they are, and the project's own source of truth stays where it was. TestForge then remembers that this one journey came from that document, and refreshes it from there on every later update — not from the project spec, which still does not describe it.

When a reference really is the whole product's new specification, say so:

testforge discover --reference ./docs/prd-v2.md --replace

A journey is its doc: generate reads docs/ and treats every markdown file there as one journey. So a doc you write by hand — for a feature the reference does not cover yet, or one taken from a ticket — works exactly like a drafted one. Write the doc, then generate --journey <name>.

If you edited a doc yourself, skip discover entirely and go straight to generate --journey <name>.

Regeneration replaces: when a journey is regenerated, test files for checks that no longer exist are deleted — but only files TestForge wrote. Your own tests are never touched.

Commands

| Command | What it does | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | testforge setup | One-time setup: runs init, map, and discover in order, with progress. | | testforge generate | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated. | | testforge update | Refreshes the map, docs and tests when the source code or the reference has changed. Skips everything if neither has (no AI calls). Broad by design — for one journey, see below. |

Advanced (run a single step)

| Command | What it does | | -------------------- | ----------------------------------------------------------- | | testforge init | Inspect the project and write testforge.config.json only. | | testforge map | Rebuild the project map (testforge.md) only. | | testforge discover | Redraft the per-journey docs in docs/ only. | | testforge env | Rebuild .env.testing.example from the docs only. | | testforge run | Run the generated tests (--journey / --area / --env-file). | | testforge report | Publish results to the dashboard — --results after a run, --inventory to register every generated check. |

Flags worth knowing

| Flag | On | What it does | | -------------------------- | ---------------------- | ---------------------------------------------------------------------------- | | --journey <name> | discover, generate, run | Work on one journey instead of all of them. | | --area <name> | discover, generate, run | Work on one area (api, web, ops…). | | --force | generate | Regenerate even when the doc has not changed. | | --refresh-docs | update | Re-draft the docs even when nothing detectably changed — see below. | | --reference <file\|url> | setup | Base the docs on a spec instead of the code map. | | --source map | setup | Base the docs on testforge.md, no prompt. |

When the reference cannot be checked

update compares two things: a hash of your source code, and a hash of the reference the docs came from. A reference it can read (a file, or a plain URL) is compared directly, so a spec that moved while the code stood still is detected — that case used to pass silently as "everything is up to date".

Some references cannot be read at all. A Notion or Confluence link answers a plain fetch with the application, not the document: no page content, and a build stamp that changes on every vendor deploy. Hashing that would be worse than not checking — it would report "unchanged" while the spec moved, and "changed" every time the vendor shipped. TestForge refuses it and tells you what it could not check:

No source changes detected.

  Note: the reference could not be checked — app.notion.com serves an app
  shell, not the document — its content needs the API.
  If the specification has changed, run `testforge update --refresh-docs`.

How it works

setup prepares the project in three stages:

1. init — detects the layout (monorepo or single app, where tests live, JS or TS) and writes testforge.config.json.
2. map — writes testforge.md: a short list of journeys and a summary of the data layer (database, models, key fields). The AI reads this instead of re-scanning all the code each time.
3. discover — writes one rich doc per journey into docs/ — fields, actions, expected results, edge cases, security notes, and the data a test needs.

generate turns each doc into a test. A journey with Type: api becomes a Jest test (HTTP + database checks); Type: web becomes a Playwright test. Other areas use the framework set in config. It also generates a seed file that provisions the test data each test needs.

generate hands each file it writes to the project's own ESLint with --fix before moving on. Formatting differences are corrected in place — CI runs lint before it runs a test, and a file that is correct but indented differently still fails the build. Anything the linter cannot fix (an unused declaration, say) is reported as a warning against that journey, because a declaration nothing checks is worth a person's attention. If the project has no ESLint, the step is skipped.

update hashes your source code AND the reference the docs came from, and compares both to the last run. Neither changed means no work and no AI calls; either one changed triggers a refresh, and it says which. A reference it cannot read is reported as exactly that, never as "up to date".

Source of truth

The journey docs in docs/ are the source of truth for the tests — generate reads the docs and nothing else. So where the docs come from decides everything.

By default the docs are derived from your code (the AI reads the project, writes testforge.md, then drafts the docs from it). That is fast, but it inherits whatever the code does: if a flow is buggy or a scenario is mishandled, the AI faithfully documents the wrong behavior, and the tests then lock that wrong behavior in.

So when the docs are drafted — during testforge setup (or testforge discover) — testforge asks what they should be based on:

What should the journey docs be based on?
  [1] testforge.md (what the AI derived from your code) (default)
  [2] An external reference (a file/PDF in scope, a website, or a Notion link)

Choose [2] to give testforge an authoritative reference of the expected behavior — a spec or requirements doc in the repo, a PDF, a website, or a Notion page. testforge then:

1. reads the reference and compares it against testforge.md,
2. writes a full comparison to testforge.diff.md and prints a short summary of the differences,
3. asks you to confirm, and on confirmation drafts the docs from the reference — expected behavior comes from the reference, while routes, fields, models, and env var names come from the code/map,
4. generate then turns those docs into tests as usual.

The reference becomes the source of truth — expected behavior from a human or client, not just what the code happens to do. The choice is remembered in test-artifacts/source.json, so a later testforge update refreshes the docs from the same source without asking again. Your review of testforge.diff.md stays as an audit of what differed and why.

Non-interactive use (CI, scripts):

testforge setup --source map              # base the docs on testforge.md, no prompt
testforge setup --reference ./spec.md     # base the docs on a reference, no prompts
testforge setup --reference https://…     # a URL or Notion link works too

The same flags work on testforge discover. With no flags and no terminal attached, testforge does not block — it falls back to testforge.md and prints a note.

Running the generated tests

testforge generates the tests and seed files — you run them with your own test runner, the way your project already does. testforge does not run tests for you; that keeps it out of your test setup and CI.

A typical run:

1. Provide test credentials. The tests and seeds read the database URL and any secrets from environment variables — never hardcoded.

   testforge setup writes a .env.testing.example for you. It lists every env var the journeys need, grouped by area (api / web / ops), and annotates each one with the journeys that use it — so you know exactly which credentials a feature needs, and can skip the ones for features you don't run. For example:

   # ========================================
   # API
   # ========================================
   DB_URL=              # used by: (all)
   TOKEN_SECRET=        # used by: auth, user
   STRIPE_SECRET_KEY=   # used by: payments, subscription

   # ========================================
   # WEB
   # ========================================
   StripePublicKey=     # used by: payments

Copy it to .env.testing, fill in real values, and keep that file out of version control (add it to .gitignore). Re-running setup or update refreshes the example as your code changes.

Note: some credentials aren't static — e.g. a user access/refresh token from a login flow. Those belong in your own test setup (a global setup or seed step), not in the env file.

Keep this file out of version control (add it to .gitignore). The exact variable names come from your own code — the journey docs and the project map list the env var names each journey needs.

2. Run the seeds to put known test data in the database. Run the seed files in seedOutput (see config) with your runtime, e.g.:

   npx tsx path/to/seed/<journey>.seed.ts   # or: node <journey>.seed.js

3. Run the tests with the project's own command, e.g.:

   # api (Jest)
   NODE_ENV=testing npx jest

   # web (Playwright)
   npx playwright test

The test runner produces its own pass/fail report.

Configuration

testforge setup writes testforge.config.json for you. You can edit it:

{
  "aiCommand": "claude -p",
  "testOutput": {
    "api": "packages/api/test",
    "web": "packages/webapp/tests",
    "ops": "packages/ops/tests"
  },
  "framework": {
    "api": "jest",
    "web": "playwright",
    "ops": "playwright"
  },
  "seedOutput": "packages/api/test/helpers/seed",
  "testExtension": ".test.ts",
  "externalModules": ["shipengine", "@customerio/cdp-analytics"]
}

• aiCommand — the CLI used to call the AI. Any tool that reads a prompt from stdin and prints the reply works, so testforge is not tied to one provider. testforge appends the right flags per phase: the setup phases (init/map/discover) run with edit permission because the model writes files, while the generation phases (tests/seeds) run with no tools (--allowedTools "" for Claude) so the model just prints code instead of trying to create files — which would otherwise leak "the write wasn't approved…" prose into the output.
• testOutput — where tests go, per area. Areas are not fixed — use whatever your project has (api, web, ops, lambda, ...).
• framework — which test framework each area uses.
• seedOutput — where seed files go.
• testExtension — .test.ts or .test.js.
• externalModules — the third-party network services this project talks to, named as a test would import them. A generated backend test that imports one without mocking it is rejected before it is written: left alone it would reach the live service, so it breaks offline and can fire real charges, messages or shipments. testforge already knows the common SDKs (payment, SMS, email, cloud, identity) and this list is added to them, never instead of them — so name only what is particular to your product. Every domain has something: a shipping carrier, a CMS, a ticketing or streaming backend, an ad network.

Notes

• Generated tests and docs are a strong draft, not final. Review them — the AI leaves TODO comments where it could not be certain.
• Secrets are never written to docs, configs, or tests. Connections use environment variable names only; real values come from your environment at run time.

License

MIT