crashtest
v1.4.0
Published
In-process adversarial QA for agent-built app code - seeded redteam/fuzz/guard campaigns against real handlers, no sockets, CI-fails-on-regression ledger.
Maintainers
Readme
crashtest
We break it so they can't. In-process adversarial QA for agent-built software - no network, no bans, no invoice.
npmjs.com/package/crashtest/ MIT - the npm package is the version truth; this file never hand-copies a release number (crashtest#51)
Crashtest attacks your project's real handlers - the way a security engineer would, minus the live server. Every attack is an in-process Request through your own routing, against an in-memory backend, with op counters attached. It never opens a socket. That invariant is the whole safety story.
60-second usage
npx crashtest init # labs/crashledger.json + labs/hardening.md stub - commit them
# write crashtest.config.js: routes (your real handlers) + fake env (see DEPLOY.md)
npx crashtest run --module-plan # dry-run: which attacks WILL fire? which packs need arming?
npx crashtest run # verdicts print live; report + ledger update
# exit 0 unless a NEW EXPLOITED finding or a REFUSED->EXPLOITED regressionThe config behind all of that, in its own file (crashtest.config.js):
// crashtest.config.js - the shape
import { buildRoutes, ENV_SPEC } from './src/app.js';
export default {
routes: buildRoutes(), // (ctx) => Response handlers (Pages-Functions shape)
env: ENV_SPEC, // fake backend: D1 tables, KV, fake secrets
adapters: { /* hono: app, express: app, fastify: await app.ready() */ },
seed: 0xC0FFEE, // deterministic: same commit -> same findings
};Seed campaigns (built in)
| Library | Attacks shipped | |---|---| | auth | identity-spoof-headers (54-spoof matrix: denied->allowed flip = bypass) / secret-watermark (fake env secrets must never leak) / privilege-escalation (denied writes must not flip on field mutations) | | input | proto-pollution-body (Object.prototype canary via hostile JSON keys) / never-fail-mutations (seeded hostile bodies; floor = never 500, never an uncaught throw) / oversized-field (1 MiB bodies) | | guards | hygiene-no-store (surviving 2xx writes need explicit Cache-Control) / burst-wall (walls must carry Retry-After; any break is a finding) | | state | enumeration-oracle (foreign ids must deny - no cross-tenant reads) / id-collision-alias (second owner same key must conflict or fork - overwrites caught by DB snapshot diff) | | volume | credential-stuffing / storage-fill / cost-amplification - every attack reports backend ops/request (bound via config, default 12) | | taxonomy | abuse-not-5xx (abuse shapes must land as honest 4xx - never 5xx, never a throw) / errors-stay-impersonal (error bodies must not parrot attacker canaries or leak stack frames) |
Verdicts: REFUSED (defense held) / EXPLOITED (you got hit) - BOUNDED (accepted risk - requires owner + bound or it's a finding).
Reports: labs/hardening.md (human-first) + labs/crashledger.json (CI gate: fail only on a new EXPLOITED or a regression of something previously REFUSED - accepted risks never re-alarm you).
Framing the working shape
Bring your own scenarios (ESM modules in labs/scenarios/ or listed in scenarioDirs) - contract: { name, description, attacks: [{ name, run(harness, ctx) -> verdict }] }.
Framework dispatch targets work in-process - no server, no sockets, ever:
adapters: {
hono: honoApp, // app.request()
express: expressApp, // app.handle() (routes must end their response)
fastify: await fastifyApp.ready(), // app.inject() (routes exist post-ready)
// wordpress: { wpRoot }, // subprocess spike - PHP worker, NDJSON stdio, no sockets
}CI as a GitHub Action
- uses: dhaupin/[email protected] # composite: install + run + gate
with:
config: crashtest.config.js
run-tests: 'true' # optional: also run your own testsFails on a NEW EXPLOITED finding or a REFUSED->EXPLOITED regression - never on accepted risks. (A moving v0 major tag is planned; pin the version for now.)
Programmatic use (drive the library API)
A run root that already builds its own Harness can drive the campaign directly instead of shelling out to crashtest run. The CLI flags map onto opts:
import { createRequire } from 'node:module';
import { pathToFileURL } from 'node:url';
import path from 'node:path';
import { buildTargetHarness } from 'crashtest'; // package root -> src/cli.js
// Resolve the runner from the installed package dir (the exports map exposes
// ./package.json but not the internal subpaths, so locate the dir first).
const require = createRequire(import.meta.url);
const dir = path.dirname(require.resolve('crashtest/package.json'));
const { runCampaign } = await import(pathToFileURL(path.join(dir, 'src', 'campaigns', 'runner.js')).href);
const target = (await import('./crashtest.config.js')).default;
const run = await runCampaign(await buildTargetHarness(target), scenarios, {
timing: true, // == --per-row (per-row durationMs; off by default so ledgers stay byte-stable)
concurrency: 4, // == --concurrency (races rows that declare isolated: true)
shard: { index: 0, count: 4 }, // == --shard 0/4 (serial-phase partition; see below)
harnessFactory: (moduleName, attack) => buildTargetHarness(target), // fresh harness per racer
});harnessFactory(moduleName, attack) is called once per raced row and must return a Promise of an equivalently-seeded harness - build it from the same target config. shard: { index, count } partitions the serial rows (position % count === index); a row declaring fanout: true runs in every shard on its own slice, so merge fanout rows by worst verdict per position. The full contracts live in reference/scenario-contract.md.
This repo dogfoods itself
npx crashtest run attacks fixtures/demo/ (hardened + the deliberately-broken draft), gated in CI against the committed labs/crashledger.json. The committed labs/hardening.md is the live report; provenance on every npm release traces package -> commit (verify the badge on npm).
The dogfood goes further: the shipped site pack packs/public/crashtest-site/ attacks everything the repo actually ships to crashtest.creadev.org - the real /api/version Pages Function, the deployed _headers (CSP, HSTS, COOP), the prerendered dist HTML, sitemap, manifest and robots.txt. Its first run caught a real bug (hostile registry shapes echoed straight through as a badge 200); the validation floor now in web/functions/api/version.js is guarded forever by test/site-pack.test.js, which pins the two-way proof: a porous route EXPLOITED, the hardened one REFUSED. A pack runs from any run root - point packs: at a directory on disk.
Known-vulnerability awareness ships the same way, and just as offline: packs/public/cve-feed/ carries a curated KEV/NVD snapshot (27 entries, budget-capped) and sweeps the target for banner disclosure (Server, X-Powered-By, stack-tell bodies) and vulnerable-range matches against a hardcoded version line. Run time reads only the committed snapshot.json - no sockets, same commit -> same findings; freshness is a maintainer-side sync, not an attack-time fetch. Its shape, byte/row budget and token grammar (the ws/answer word-boundary case) are pinned in test/cve-feed.test.js.
Parser-shaped packs follow the same teeth-first pattern: packs/public/qs-shapes/ attacks the bracket-notation wire itself (the express extended-query / urlencoded parser, any qs-style expander): a prototype-key canary riding __proto__/constructor.prototype bracket keys, and a deep-nesting ladder that must stay under the fuzz floor (never 5xx, never an uncaught throw). It sweeps every declared route the way cve-feed does, narrowed with ctx.routes, and its two-way teeth (a vulnerable parser EXPLOITEDed, hardened and bracket-blind wires honestly REFUSED) are pinned in test/qs-shapes.test.js.
Two more framework wire packs ship the same way: packs/public/vite-dev-shapes/ attacks the vite DEV wire's own signature, the /@fs/ module route: a grammar differential (plain endpoint gated, bypass-grammar suffixes like ?raw?? / ?import&raw?? served) and a content-shape oracle for the .env deny-list family, and packs/public/next-middleware-shapes/ attacks the Next.js middleware subrequest header: a known-grammar bypass must not turn a gated route into a 2xx, and a canary value that is NOT a certified chain must never unlock the gate (value-blindness is its own finding). Both sweep declared routes with differential oracles and are pinned two-way in test/vite-shapes.test.js and test/next-mw-shapes.test.js.
Landscape
| Tool | Mode | Gap crashtest fills | |---|---|---| | ffuf / nuclei / ZAP | live network | needs a server + traffic; banned by hosts | | semgrep / CodeQL | static | never runs the code; behavioral holes pass review | | crashtest | behavioral, in-process, pre-deploy | - |
Principles
- Attack code, not people. No live-fire, no third-party targets, ever.
- In-process or nothing. If it needs a socket, it's out of scope.
- Deterministic. Same commit -> same findings (seeded randomness).
- Markdown is the interface. Reports a human reads, JSON a CI reads.
- Zero dependencies by default. The attacker ships light.
- Accepted risks must have owners and bounds.
Surface: packs, plugins and presets (the ecosystem thread)
Self-contained packs - a pack in packs/<visibility>/<name>/ is the one extension surface, and its pack.json can bundle its own imports ("plugins": [npm-or-relative-specifiers]), so it ships entry + imports to the adopting repo with zero config wiring. A pack doesn't need to live in the attacked repo - packs: accepts any directory on disk (another repo's checkout, a vendored npm package):
// crashtest.config.js - one contract, three ways in
export default {
routes: buildRoutes(),
env: ENV_SPEC,
packs: ['./packs', '/abs/path/to/pack-owning-repo/packs'], // bundles included
plugins: ['some-shape-pack-npm-name', './shims/wallet-profile.js'], // escape hatch: no pack
};Builder presets - crashtest preset lovable|bolt|replit [--root <dir>] prints a starter config from what detect() actually finds in an exported repo (Supabase Edge Function layout, TanStack Start markers, the .replit run command, package.json shape) - prints it to stdout (nothing is written), and names plainly what it could NOT resolve rather than guessing a handler from the builder's name.
Pack-shape spec (packstruct@1). This repo implements packstruct@1 - the /packs manifest/entry/directory contract, frozen at SPEC v1 and transcribed byte-exact at labs/packstruct-backfill/SPEC-v1.md (transcribed from the packstruct v1.1.0 release commit 5e473120ed73c5a38b88644cc101be492d5dc2a2; the SHA is the transcript's provenance, no longer the install shape). The CI lane runs packstruct lint packs/ from the npm package, installed as an EXACT devDependency ("packstruct": "1.1.0" in package.json, no float - same commit -> same findings applies to the linter too). The pin sits on the published packstruct v1.1.0 release (OIDC-attested; verify-spec confirms the shipped spec/SPEC.md is byte-equal to the transcript, so the provenance SHA names the same bytes); any SPEC drift re-runs the same review rule - re-transcribe + re-derive the moved specKeys in one commit.
Docs map
- Full docs -> docs.creadev.org/crashtest (quick start, playbooks, catalog, ledger format - the same content crosslinked from the CLI help)
- Site -> crashtest.creadev.org (sourced from this repo's
/web, prerendered by prestruct) - Dashboard -> crashtest.creadev.org/dashboard (render any ledger or run in-browser - same section 7 rule, zero backend)
- FAQ -> crashtest.creadev.org/faq (the short answers: safety, SAST/pentest contrast, exit contract, packs)
- Release posts -> docs/discussions (one announcing-vX.Y.Z per release, discussions-ready, same shape as prestruct's)
- The workshop -> Creadev.org (crashtest is made by Rook @ Creadev.org - the same home as the docs and site surfaces above)
- Agent/human contributor conventions ->
AGENTS.md - Publish internals (npm Trusted Publishing, staged route, token policy) ->
docs/publish.md - Contract + task log ->
labs/prd-crashtest.md+labs/TASKS.md
License: MIT.
