Premortem

A QA auditor and test-scaffolding engine for AI-generated codebases.

Point it at a repo. It finds what the model didn't test, ranks it by what actually breaks, and writes you the tests.

Quick start · Why · How it works · Rules · Architecture · Contributing · Limitations

  PRE MORTEM  vibe-shop  express

   F   0/100   |   5 files  •  333 lines  •  116 findings  •  0.10s

  This needs a hardening pass before it goes anywhere near real users.
  19 critical issues should be fixed today. Security carries the most risk (F).

  BREAKDOWN ─────────────────────────────────────────────────────────────
  Security         F    0  ░░░░░░░░░░░░░░░░░░  38 findings   worst: critical
  Reliability      F    0  ░░░░░░░░░░░░░░░░░░  32 findings   worst: high
  Maintainability  F    2  ░░░░░░░░░░░░░░░░░░  16 findings   worst: high
  Performance      F   23  ████░░░░░░░░░░░░░░  11 findings   worst: medium
  Testability      F   33  ██████░░░░░░░░░░░░   5 findings   worst: high
  API Contracts    F   37  ███████░░░░░░░░░░░   7 findings   worst: medium
  Accessibility    D   60  ███████████░░░░░░░   5 findings   worst: medium

  FIX THESE FIRST ───────────────────────────────────────────────────────

  1. ● CRITICAL  DELETE /api/users/:id changes state but no authorisation
                 check was found in the handler.
     server.js:84:1  sec/missing-auth-check  risk 124
     → Put an auth middleware in front of the route and an ownership check
       inside it — authentication answers "who are you", authorisation
       answers "may you touch this row".
     ✓ test: Call the endpoint with no token, an expired token, and a valid
       token belonging to a different user. All three must fail with 401/403
       before the happy-path test matters.

The problem

AI writes code that runs. It does not write code that survives. And the failure modes are boringly predictable, which means they are mechanically detectable:

| What the model reliably does | What it reliably skips | | --- | --- | | Produces a working happy path | Every unhappy path | | Wires up an endpoint | Auth checks, input validation, status codes | | Writes a React component | Loading state, error state, a11y, stable keys | | Hardcodes a value to make it run | Moving it to config before the commit | | Says "you can add tests later" | Tests |

Premortem is not a general-purpose linter. Every rule answers exactly one question:

What would a QA engineer catch here that the model didn't?

That framing is why each finding ships with a test hint — the test that would have caught it — and why the tool can then go and write that test for you.

Quick start

npm install -g premortem-audit

cd any-repo-you-like
premortem scan .                              # audit the whole repo
premortem gen .                               # scaffold tests for the riskiest paths
premortem probe . -u http://localhost:3000    # prove the findings against a running app

Or without installing anything:

npx premortem-audit scan .

The package is premortem-audit; the command is premortem (with pm as a short alias). npm rejects unscoped names that are punctuation-variants of an existing package, and an unrelated pre-mortem already holds that shape.

No config needed. No API key needed. Zero runtime dependencies — the install is one package, not four hundred. It walks the repo the way git sees it, works out what framework you're on, and reports.

Try it on the deliberately-broken demo app that ships with the source:

git clone https://github.com/itsmepriyu/premortem && cd premortem
npm install && npm run build
npm run demo

What it does

1. Audits the whole codebase

premortem scan . walks every file the project would ship — honouring .gitignore at every level — and runs 81 rules across seven categories: security, reliability, testability, API contracts, accessibility, maintainability and performance.

Findings are not a flat list of warnings. Each one is scored:

risk = severity × confidence × blast-radius

Confidence is a first-class, visible field. Static analysis without full type resolution is inherently probabilistic; pretending otherwise is how tools lose trust. A tentative finding is shown and labelled as tentative.

Blast radius comes from where the code lives: path keywords (auth, payment, admin), whether the file registers routes or is middleware, whether it's a script or a demo (penalised) — and fan-in from the import graph, because a module twenty files depend on carries more of the app's weight than a leaf.

The result is a 0–100 score normalised per 1,000 lines, so a 40k-line app isn't automatically graded worse than a 400-line one. Three hard ceilings override the arithmetic: any critical caps you at 55, any high at 84, and no tests at all at 79.

2. Maps your HTTP surface

It extracts every endpoint from Express, Fastify, Koa, Hono, both Next.js routers, Flask and FastAPI — then reads each handler for the four signals that matter:

  Method  Path                   Auth    Validation  Errors  Status
  GET     /api/products          ✗       ✗           ✗       ✗
  POST    /api/login             ✗       ✗           ✗       ✗
  DELETE  /api/users/:id         ✗       ✗           ✗       ✗

A red mark isn't proof of a bug. It means the check wasn't visible in the handler — which is exactly the list of things worth testing.

3. Ranks your test gaps

Coverage percentage tells you how much code ran, not how much behaviour is protected. What you actually want is an ordered list: of everything untested, which hurts most if it breaks?

  TEST GAPS (ranked by what breaks worst)
   80  src/auth.js          auth    15 open findings
   78  server.js            api     serves 12 endpoints, 75 open findings
   40  src/ProductList.jsx  ui      20 open findings

4. Writes the tests

premortem gen . turns the audit into work you can actually do. It doesn't write tests for "the code" — it writes tests for the specific doubts the scan produced. An endpoint with no auth check gets an unauthenticated-request test; a handler that reads three body fields gets three missing-field tests.

Every generated case is traceable:

test.describe('DELETE /api/users/:id', () => {
  // Declared at server.js:84 (express)

  // WHY: No authorisation check was visible in the handler; this proves whether one exists.
  // FROM: sec/missing-auth-check
  test('rejects an unauthenticated request', async ({ request }) => {
    const res = await request.delete(BASE_URL + '/api/users/' + id, {});
    // No Authorization header is sent. A state-changing endpoint must refuse it.
    expect([401, 403], 'unauthenticated request').toContain(res.status());
  });

Output is deterministic — same input, same output, no API key required, and it cannot hallucinate an endpoint that doesn't exist.

Optional: --ai enrichment with Gemini

--ai additionally sends each scaffold and the source it tests to Gemini, which fills in TODOs, corrects expected status codes from the real contract, and removes cases the source proves are already handled:

premortem gen . --ai

- expect(res.status(), 'happy path').toBeLessThan(400);   // TODO: replace
+ // NOTE: server.js:31 returns HTTP 200 with an array of products.
+ expect(res.status()).toBe(200);
+ expect(Array.isArray(await res.json())).toBe(true);

Credentials go in a .env file at the project root (see .env.example):

GEMINI_API_KEY=AQ.Ab...      # Google Cloud key -> Vertex AI
GCP_PROJECT_ID=project-51... # only needed for Google Cloud keys

An AI Studio key (AIza...) works too — the client detects which surface the key belongs to and probes Vertex express mode, Vertex regional and the Gemini Developer API in turn, then remembers whichever answered. The key is sent in the x-goog-api-key header rather than a ?key= query parameter, so it stays out of proxy logs and shell history.

Default model is gemini-3.7-flash; override with --model or GEMINI_MODEL.

The model is asked to improve a scaffold, never to invent tests from nothing, and is explicitly instructed never to weaken an assertion to make a test pass. Every enriched file is stamped AI-DRAFTED, HUMAN-REVIEW REQUIRED, and every changed expectation carries a // NOTE: explaining why. Enrichment failing never fails the command — you keep the deterministic scaffold.

5. Proves it against a running app

The scan infers. The probe proves.

premortem probe . --base-url http://localhost:3000

It fires real negative and boundary requests — missing credentials, forged tokens, malformed JSON, type confusion, hostile strings, oversized payloads — and reports what actually came back, plus a security-header audit.

Read-only by default. No POST/PUT/PATCH/DELETE is ever sent without --allow-mutations, because this points at a live server that may not be yours to break.

How it works

                    ┌──────────────────────────────────────────┐
   repo root ─────► │  walker      .gitignore-aware discovery   │
                    │  profiler    framework/deps/tests/CI      │
                    └───────────────────┬──────────────────────┘
                    ┌───────────────────▼──────────────────────┐
                    │  lexer       JS/TS/JSX + HTML + Python    │
                    │  ↳ mask      blank strings/comments/JSX   │
                    │  structure   fns, imports, try/catch, JSX │
                    │  routes      Express/Next/Fastify/Flask   │
                    └───────────────────┬──────────────────────┘
                    ┌───────────────────▼──────────────────────┐
                    │  engine      pooled, cached, suppressible │
                    │  ↳ 81 rules across 7 categories           │
                    └───────────────────┬──────────────────────┘
                    ┌───────────────────▼──────────────────────┐
                    │  score       risk model → 0-100 + grade   │
                    │  baseline    new-vs-existing debt         │
                    │  testmap     coverage gap analysis        │
                    └───────┬───────────────────────┬──────────┘
              ┌─────────────▼──────────┐  ┌─────────▼──────────────┐
              │ reporters              │  │ generators             │
              │ console/md/html/       │  │ Playwright + API specs  │
              │ json/sarif/junit/csv   │  │ (+ optional AI enrich)  │
              └────────────────────────┘  └────────────────────────┘

The one interesting engineering decision

There is no AST parser. The analyser is built on a hand-written masking lexer, for three reasons that all matter in this specific problem domain:

1. Vibe-coded repos frequently don't parse. Half-finished JSX, a literal // ...rest of the code, mismatched braces. A parser throws and you get nothing. A lexer degrades gracefully and still reports the other 400 lines. There's a test for exactly this.
2. Zero runtime dependencies. A QA tool you aim at untrusted code should not drag 300 transitive packages into the room with it. npm i -g premortem installs one thing.
3. Every rule is a pattern rule — and patterns need exactly one thing raw grep can't give them: did this match land in real code, or inside a string, comment or JSX text run?

So the lexer emits span records, and buildMask() produces a parallel string with every literal blanked to spaces of identical length — byte offsets stay aligned with the original file:

// source
const q = `SELECT * FROM users WHERE id = ${userId}`;  // TODO: eval this

// source.code (rules scan this)
const q = `                                  ${userId}`;

That one trick kills the entire class of false positives that makes grep-based scanners unusable. It handles nested template literals, regex-vs-division disambiguation, JSX text containing apostrophes (<p>don't panic</p> breaks naive scanners), expression containers, and attribute strings. source.codeStr is the same mask but keeps string contents, because route paths and SQL text live inside strings.

docs/ARCHITECTURE.md covers the module map, the risk model and the rest of the design in detail.

Commands

Every command takes an optional [path], defaulting to the current directory, and --help. premortem help <command> prints the same detail without running anything.

| Command | What it does | | --- | --- | | scan | Audit a codebase, write reports, exit with a CI-actionable code | | gen | Scaffold Playwright/API/unit tests for the riskiest paths | | probe | Negative-test a running application | | rules | Browse the catalogue, or explain one rule with before/after | | baseline | Record today's findings as accepted debt | | watch | Re-audit on every file change | | init | Write a starter config, CI workflow and git hook | | bench | Measure the rules against a labelled corpus |

Exit codes are the same everywhere: 0 clean · 1 findings at or above --fail-on · 2 usage error · 3 internal error. Nothing else is ever returned, so a CI step can branch on the code without parsing output.

premortem scan

Walks every file the project would ship, honouring .gitignore, runs the full rule set, and scores each finding by severity × confidence × blast radius.

premortem scan .
premortem scan . --format html,sarif --out .premortem
premortem scan . --fail-on high                          # the CI gate
premortem scan . --rule sec --all                        # security only, nothing elided
premortem scan . --baseline .premortem/baseline.json     # fail only on new findings

| Flag | Effect | | --- | --- | | -f, --format <list> | console, markdown, html, json, sarif, junit, csv, github | | -o, --out <dir> | Where written reports go (default .premortem/) | | --fail-on <severity> | Exit 1 at this severity or worse. never to always exit 0 | | --min-score <n> | Exit 1 if the 0-100 score falls below this | | --min-severity, --min-confidence | Hide findings below a threshold, rather than failing on them | | -b, --baseline <file> | Report only findings absent from the baseline | | -r, --rule <id>, --off <id> | Run only, or skip, these rule ids or prefixes | | --include, -x, --exclude <glob> | Narrow or widen the file set, gitignore syntax | | -a, --all | Print every finding instead of the top few per group | | -q, --quiet | The one-line summary only — useful in a pre-commit hook | | --no-cache | Ignore the incremental cache and rescan everything | | -j, --concurrency <n> | Files scanned in parallel | | --no-routes, --no-gaps | Drop the HTTP surface or test-gap sections from the report |

--min-severity and --fail-on are different questions: the first is what you want to look at, the second is what should break the build. Setting the first to hide a problem does not stop the second from catching it.

premortem gen

Turns the audit into work you can do. Every generated case traces back to the finding or route signal that motivated it and carries a // WHY: comment saying what it proves.

premortem gen .                          # api + e2e + unit
premortem gen . --kind api --dry-run     # see the plan, write nothing
premortem gen . --ai                     # enrich with Gemini

| Flag | Effect | | --- | --- | | -k, --kind <list> | api, e2e, unit (default: all three) | | -o, --out <dir> | Where the specs are written | | --max-files <n> | Cap on generated spec files | | --base-url <url> | Baked into the generated specs | | --config | Also emit a playwright.config.ts | | --dry-run | Print the plan and exit | | --force | Overwrite files generated by a previous run | | --ai | Send each scaffold plus its source to Gemini for enrichment | | --model <id>, --ai-max-files <n> | Which model, and how many files to spend on it |

The deterministic path needs no API key and no network. --ai is strictly additive: it rewrites assertions and adds cases, everything it touches is labelled AI-drafted, and if the call fails you still get the deterministic scaffold. See Installing it elsewhere for the GEMINI_API_KEY setup.

premortem probe

The scan infers; the probe proves. It maps your HTTP surface statically, then sends real requests — missing credentials, malformed bodies, type confusion, hostile strings — and reports what actually came back.

premortem probe . --base-url http://localhost:3000
premortem probe . -H "Authorization: Bearer $TOKEN"
premortem probe . --only /api --allow-mutations

| Flag | Effect | | --- | --- | | -u, --base-url <url> | Required. The running application | | -H, --header <h> | Extra header, repeatable | | --only <path>, --skip <path> | Restrict by path prefix | | --allow-mutations | Permit POST/PUT/PATCH/DELETE — this changes data | | --timeout <ms>, -j, --concurrency <n>, --max-requests <n> | Rate and blast-radius limits | | -a, --all | List passing probes as well as failures |

Read-only by default. Nothing that mutates state is sent unless you ask for it, because this points at a live server that may not be yours to change. Point it at a staging environment.

premortem rules

premortem rules                       # the whole catalogue
premortem rules sec/sql-injection     # one rule, with before/after code
premortem rules -C security -v        # one category, full why/fix/test text
premortem rules -f markdown           # what generates docs/RULES.md

| Flag | Effect | | --- | --- | | -f, --format | console or markdown | | -C, --category <name> | security, reliability, testability, contracts, accessibility, maintainability, performance | | -s, --severity <level> | Filter by severity | | -v, --verbose | Include the full explanation for every rule listed |

Every rule explains why it fires, how to fix it, and what test would have caught it. If a finding doesn't make sense, this is the command that answers it.

Note that a rule's id prefix is not always its category: a11y/img-alt is in accessibility, api/undocumented-surface is in contracts, and vibe/* is in maintainability. --rule matches the id prefix; --category matches the category.

premortem baseline

premortem baseline .                 # writes .premortem/baseline.json
premortem baseline . -o debt.json

Records today's findings as accepted debt. See Adopting on an existing codebase for why this matters more than it sounds.

premortem watch

premortem watch .
premortem watch . --fail-on high --debounce 500

Re-audits on every file change, using the incremental cache so only what you touched is rescanned. --debounce <ms> controls how long it waits for the file system to settle.

premortem init

premortem init .                  # premortem.config.json
premortem init . --ci --hook      # + GitHub Actions workflow + pre-commit hook

--force overwrites an existing config. The generated workflow uploads SARIF, so findings appear inline on the pull request diff.

premortem bench

Answers the question unit tests cannot: did this change make the output better or worse on real code? Mostly of interest if you are changing rules — see The benchmark for the full story.

premortem bench                                   # the bundled corpus
premortem bench -v                                # per-fixture detail
premortem bench --corpus ~/projects               # your own checkouts
premortem bench --update                          # accept the current output

| Flag | Effect | | --- | --- | | --corpus <dir> | Directory holding one subdirectory per project | | --snapshot <file> | Baseline to compare against | | -u, --update | Write the current output as the new snapshot | | --strict | Fail on any drift, not just regressions | | -v, --verbose | Show every fixture, its grade and its finding count |

Output formats

premortem scan . --format console,html,markdown,json,sarif,junit,csv

• console — ranked, grouped, colour-aware, degrades cleanly when piped
• html — a single self-contained interactive file. No network requests, no build step. Filterable, searchable, dark/light aware. Email it or attach it to a CI run.
• sarif — GitHub ingests this natively; findings appear inline on the PR diff
• junit — every CI product renders this, so the audit shows up next to your unit tests
• markdown — shaped for a PR comment, with everything below the fold in <details>

CI

- run: npx premortem scan . --format console,sarif --fail-on high
- uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: .premortem/premortem.sarif

Exit codes: 0 clean · 1 findings at/above --fail-on · 2 usage error · 3 internal error.

Adopting on an existing codebase

Nobody adopts an analyser that opens with 900 findings — the build goes red, someone adds --no-verify, and the tool is dead. So:

premortem baseline .                                   # accept today's debt
premortem scan . --baseline .premortem/baseline.json   # fail only on NEW findings

The existing debt still appears in the report. It's deferred, not forgiven. Fingerprints are content-based, so they survive the whole file shifting down by an import.

Silencing a rule

// premortem-disable-next-line sec/hardcoded-secret -- test fixture, not a real key
const key = 'sk_test_abc123';

The trailing -- reason is optional, but an unexplained suppression is its own kind of smell. Also supported: -line, -file, and rule prefixes (sec silences all of sec/*).

Configuration

premortem.config.json, or a "premortem" block in package.json:

{
  "failOn": "high",
  "minScore": 70,
  "exclude": ["**/*.stories.tsx"],
  "rules": {
    "vibe/leftover-debug": "low",
    "a11y/form-label": "off",
    "security": "critical"
  }
}

Keys accept a rule id, a prefix (sec/*), a category (security), or *. More specific wins. An explicit override always beats a rule's own per-finding severity.

Rules

81 rules across 7 categories. Full catalogue with before/after examples: docs/RULES.md, or premortem rules <id> for any single one.

Two are worth calling out as genuinely specific to this problem, with no equivalent in any standard linter:

vibe/ai-placeholder — catches // ... rest of the implementation here, your-api-key-here, // TODO: implement this, // In a real application you would…. These are the literal fingerprints of code pasted out of a chat window and never finished. Unlike a normal TODO they usually mean the code path doesn't work at all — it just compiles.

vibe/undeclared-dependency — the code imports a package that isn't in package.json. This is the most common reason generated code fails on a colleague's machine: it works locally because a transitive dependency happened to install it, and the build breaks the moment that changes. One-line check, no standard toolchain performs it.

Use as a library

import { audit } from 'premortem';

const report = await audit('./my-app', { minSeverity: 'high' });

console.log(report.scorecard.grade, report.scorecard.score);
for (const f of report.findings) {
  console.log(`${f.file}:${f.line}  ${f.severity}  ${f.message}`);
}

The whole engine is ordinary functions over plain data — lexer, rules, scoring, reporters and generators are all separately importable.

Honest limitations

Every static analyser has these. Most don't write them down.

• Heuristic, not sound. No type resolution, no cross-file dataflow. That's why confidence is a labelled, first-class field rather than something hidden behind a threshold.
• Taint tracking is intra-function only. Input laundered through a helper function is missed. sec/sql-injection catches the common shape, not every shape.
• The probe only tests what it can reach. It won't find authorisation bypasses behind a login wall unless you hand it a token with --header.
• Generated tests are drafts. They encode the scanner's assumption about intent, which may be wrong for your application. That's why the workflow is explicitly draft-then-review, why every case carries a // WHY: comment, and why nothing is auto-committed. A test asserting the wrong thing is worse than no test.
• Python support is structural only — routes, functions, secrets, debug statements. The deep rules are JS/TS-family.
• It does not auto-fix. Rewriting someone's code from a heuristic is how you break production. The credible product is evidence plus tests.

Installing it elsewhere

Three ways to get premortem onto a machine, in order of convenience:

npm install -g premortem-audit              # from the registry
npx premortem-audit scan .                  # no install at all
npm install -g github:itsmepriyu/premortem    # straight from a git checkout

Whichever route you take, the binary installed is premortem (with pm as a short alias).

The git route works because prepare builds on install, so a clone needs no extra steps.

To publish a new version yourself:

npm login
npm version patch          # or minor / major — updates package.json
npm publish                # prepublishOnly runs build + 219 tests + a self-audit first

prepublishOnly is a real gate: it refuses to publish if the build fails, if any test fails, or if Premortem finds a critical issue in its own source.

Contributing

Contributions are welcome, and the most valuable one is probably not what you'd guess:

False-positive reports. A rule that fires on correct code is worse than a missing rule, because it teaches people to ignore the whole tool. If Premortem flagged something that's actually fine, open an issue with the snippet — those are treated as bugs and turned into regression tests.

After that: new rules, route extraction for unsupported frameworks, and language support.

CONTRIBUTING.md has the bar a rule has to clear, how to write one, and which of the three scan surfaces to use (this trips everyone up at least once).

• Architecture — how it fits together and why
• Rule catalogue — all 81, generated from the source
• Security policy — including the design commitments this project intends to keep
• Changelog

Development

npm install
npm run build
npm test              # 259 tests
npm run selfscan      # the tool audits itself; CI fails on any critical
npm run demo          # scan the deliberately-broken example app
npm run docs:rules    # regenerate docs/RULES.md from the registry
npm run bench         # measure the rules against the labelled corpus
npm run check         # source hygiene + version consistency

docs/RULES.md is generated from the rule registry and checked in CI, so the documentation cannot drift from the behaviour.

The benchmark

A unit test tells you a rule does what its author meant. It cannot tell you whether the author was right, because the same person writes both. Every false positive fixed in this project was found by running the tool across real repositories and diffing the aggregate.

premortem bench makes that repeatable. bench/fixtures/ holds small projects, each with a .bench.json declaring what it is:

// bench/fixtures/clean-express/.bench.json
{ "kind": "clean", "note": "idiomatic Express API, parameterised queries, schema validation" }

// bench/fixtures/dirty-react/.bench.json
{ "kind": "dirty", "expect": ["sec/dangerous-html", "rel/missing-ui-states", "a11y/img-alt"] }

A clean fixture is correct code, so any finding on it is a false positive by definition — this is the half that matters, because a rule that misses a bug is a gap, but a rule that fires on working code is what gets the tool uninstalled. A dirty fixture declares the rules that must fire, so a narrowed regex cannot quietly kill detection.

The result is compared to bench/snapshot.json. New findings on clean code and lost detections on dirty code fail; anything else is reported as drift you are expected to explain in one sentence. Accept an intended change with npm run bench:update and commit the diff.

You can point it at your own code:

premortem bench --corpus ~/projects --snapshot ~/premortem-baseline.json

Unlabelled directories are recorded but not judged, which makes this a way to see what a rule change does to thousands of real files before it ships.

License

MIT