testforge

AI-powered QA automation. Point it at a project and it scans the code, drafts a journey doc for each feature, and generates seeded Jest and Playwright tests that match the project's own structure.

Works in any project, with any AI CLI (defaults to Claude).

Install

npm install -D @activate-core/test-forge
```

testforge calls a local AI CLI to do its work. By default it uses
`claude -p`, so make sure the Claude CLI is installed and logged in.
To use a different CLI, see [Configuration](#).

## Quick start

From your project root:

```bash
testforge setup      # one-time: inspect project, map it, draft journey docs
testforge generate   # generate tests + seed files from the docs

After setup, review the drafted docs in docs/ and fix anything the AI got wrong — they are the source of truth for the tests. Then run generate.

When your code changes later:

testforge update     # re-check and refresh only what changed

Commands

| Command | What it does | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | testforge setup | One-time setup: runs init, map, and discover in order, with progress. | | testforge generate | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated. | | testforge update | Detects source-code changes and refreshes the map, docs, and tests when needed. Skips everything if nothing changed (no AI calls). |

Advanced (run a single step)

| Command | What it does | | -------------------- | ----------------------------------------------------------- | | testforge init | Inspect the project and write testforge.config.json only. | | testforge map | Rebuild the project map (testforge.md) only. | | testforge discover | Redraft the per-journey docs in docs/ only. | | testforge env | Rebuild .env.testing.example from the docs only. |

How it works

setup prepares the project in three stages:

1. init — detects the layout (monorepo or single app, where tests live, JS or TS) and writes testforge.config.json.
2. map — writes testforge.md: a short list of journeys and a summary of the data layer (database, models, key fields). The AI reads this instead of re-scanning all the code each time.
3. discover — writes one rich doc per journey into docs/ — fields, actions, expected results, edge cases, security notes, and the data a test needs.

generate turns each doc into a test. A journey with Type: api becomes a Jest test (HTTP + database checks); Type: web becomes a Playwright test. Other areas use the framework set in config. It also generates a seed file that provisions the test data each test needs.

update hashes your source code and compares it to the last run. No change means no work and no AI calls; a change triggers a refresh.

Source of truth

The journey docs in docs/ are the source of truth for the tests — generate reads the docs and nothing else. So where the docs come from decides everything.

By default the docs are derived from your code (the AI reads the project, writes testforge.md, then drafts the docs from it). That is fast, but it inherits whatever the code does: if a flow is buggy or a scenario is mishandled, the AI faithfully documents the wrong behavior, and the tests then lock that wrong behavior in.

So when the docs are drafted — during testforge setup (or testforge discover) — testforge asks what they should be based on:

What should the journey docs be based on?
  [1] testforge.md (what the AI derived from your code) (default)
  [2] An external reference (a file/PDF in scope, a website, or a Notion link)

Choose [2] to give testforge an authoritative reference of the expected behavior — a spec or requirements doc in the repo, a PDF, a website, or a Notion page. testforge then:

1. reads the reference and compares it against testforge.md,
2. writes a full comparison to testforge.diff.md and prints a short summary of the differences,
3. asks you to confirm, and on confirmation drafts the docs from the reference — expected behavior comes from the reference, while routes, fields, models, and env var names come from the code/map,
4. generate then turns those docs into tests as usual.

The reference becomes the source of truth — expected behavior from a human or client, not just what the code happens to do. The choice is remembered in .testforge/source.json, so a later testforge update refreshes the docs from the same source without asking again. Your review of testforge.diff.md stays as an audit of what differed and why.

Non-interactive use (CI, scripts):

testforge setup --source map              # base the docs on testforge.md, no prompt
testforge setup --reference ./spec.md     # base the docs on a reference, no prompts
testforge setup --reference https://…     # a URL or Notion link works too

The same flags work on testforge discover. With no flags and no terminal attached, testforge does not block — it falls back to testforge.md and prints a note.

Running the generated tests

testforge generates the tests and seed files — you run them with your own test runner, the way your project already does. testforge does not run tests for you; that keeps it out of your test setup and CI.

A typical run:

1. Provide test credentials. The tests and seeds read the database URL and any secrets from environment variables — never hardcoded.

   testforge setup writes a .env.testing.example for you. It lists every env var the journeys need, grouped by area (api / web / ops), and annotates each one with the journeys that use it — so you know exactly which credentials a feature needs, and can skip the ones for features you don't run. For example:

   # ========================================
   # API
   # ========================================
   DB_URL=              # used by: (all)
   TOKEN_SECRET=        # used by: auth, user
   STRIPE_SECRET_KEY=   # used by: payments, subscription

   # ========================================
   # WEB
   # ========================================
   StripePublicKey=     # used by: payments

Copy it to .env.testing, fill in real values, and keep that file out of version control (add it to .gitignore). Re-running setup or update refreshes the example as your code changes.

Note: some credentials aren't static — e.g. a user access/refresh token from a login flow. Those belong in your own test setup (a global setup or seed step), not in the env file.

Keep this file out of version control (add it to .gitignore). The exact variable names come from your own code — the journey docs and the project map list the env var names each journey needs.

2. Run the seeds to put known test data in the database. Run the seed files in seedOutput (see config) with your runtime, e.g.:

   npx tsx path/to/seed/<journey>.seed.ts   # or: node <journey>.seed.js

3. Run the tests with the project's own command, e.g.:

   # api (Jest)
   NODE_ENV=testing npx jest

   # web (Playwright)
   npx playwright test

The test runner produces its own pass/fail report.

Configuration

testforge setup writes testforge.config.json for you. You can edit it:

{
  "aiCommand": "claude -p",
  "testOutput": {
    "api": "packages/api/test",
    "web": "packages/webapp/tests",
    "ops": "packages/ops/tests"
  },
  "framework": {
    "api": "jest",
    "web": "playwright",
    "ops": "playwright"
  },
  "seedOutput": "packages/api/test/helpers/seed",
  "testExtension": ".test.ts"
}

• aiCommand — the CLI used to call the AI. Any tool that reads a prompt from stdin and prints the reply works, so testforge is not tied to one provider. testforge appends the right flags per phase: the setup phases (init/map/discover) run with edit permission because the model writes files, while the generation phases (tests/seeds) run with no tools (--allowedTools "" for Claude) so the model just prints code instead of trying to create files — which would otherwise leak "the write wasn't approved…" prose into the output.
• testOutput — where tests go, per area. Areas are not fixed — use whatever your project has (api, web, ops, lambda, ...).
• framework — which test framework each area uses.
• seedOutput — where seed files go.
• testExtension — .test.ts or .test.js.

Notes

• Generated tests and docs are a strong draft, not final. Review them — the AI leaves TODO comments where it could not be certain.
• Secrets are never written to docs, configs, or tests. Connections use environment variable names only; real values come from your environment at run time.

License

MIT