Saffron

Gherkin-native test runner with zero-token cached replay and runtime AI healing.

Free to use · npm: saffron-ai · site

Write tests as plain .feature files. On the first run, an AI agent (Claude Agent SDK + Playwright MCP) executes each scenario in a real browser and records what it did into a cache — a human-reviewable JSON action list committed to git. Every run after that replays the cache with plain Playwright: zero AI calls, zero tokens, ~100ms per scenario.

When the UI changes and a cached step fails at runtime, the agent takes over mid-execution with full scenario context, adapts, and finishes the run — then files a cache proposal you accept or reject, like a snapshot test update. Reports state exactly what was adapted and what it cost.

The rules that keep it honest

• Assertions are sacred. By default no Then is ever healed — the AI may help reach an assertion, never make it pass; enforced mechanically, not by prompt. Projects may opt into adaptable-mid for mid-scenario checkpoints, but the final assertion block of a scenario is strict under every policy, forever.
• Three result states. Green = cached pass. Yellow = passed with AI adaptation (pending your review). Red = failed.
• Diagnose before adapting. On a step failure the agent first decides: UI drift (heal) or application defect (fail with a diagnosis).
• Caches are git artifacts. Proposals show diffs — including a suggested .feature edit when the written steps no longer match reality — and nothing is committed without you.
• Verified proposals. Every recording/heal is proof-replayed zero-AI before it's filed (with one bounded refinement pass on failure), so proposals arrive stamped verified ✓ or honestly UNVERIFIED ✗.
• Honest cost reporting. Reports break out prompt-cache reads/writes — the real bill of agent sessions — next to the in+out token count, per scenario and in totals.
• Trend memory. Every run appends to .saffron/history.jsonl; reports show deltas vs the previous run and 20-run sparklines, and Saffron flags chronic scenarios (healing repeatedly — re-record instead of paying again) plus recurring failure themes.

Step reuse: new scenarios get cheaper as your suite grows

Like a maturing Cucumber suite, most of a new feature file is steps you already have. Saffron derives a step index from committed caches and, when recording a new scenario, replays every seeded island of known steps zero-AI — the agent records only the genuinely novel steps, wherever they sit. Exact step text is the identity (Cucumber-style); quoted values may differ ("admin" seeds from "bob"); data-table steps seed on matching keys. Measured on a live site: a 6-step scenario with one novel step recorded for $0.15 / 6 AI calls, vs $2.07 for a full unseeded recording of comparable length.

Step sets (shipped): .saffron files — a superset dialect of Gherkin — add the StepSet: keyword for named, reusable step sequences invoked with StepSet <name> inside any scenario. Sets expand at parse time, so their steps cache and seed like ordinary steps; editing a set makes every invoking scenario honestly stale (re-recorded mostly seeded), and heal edits route to the set definition — one fix, every caller follows. Sets are project-wide: keep application-wide flows in a sets-only library file (convention: features/shared.steps.saffron) and invoke them from any feature.

IDE integration & authoring (shipped): saffron steps lists the project vocabulary with recorded/divergent/unrecorded badges (--snippets for native VS Code completion), saffron mcp serves it to AI assistants, saffron lsp brings completion/navigation/diagnostics to JetBrains (incl. Community editions via LSP4IJ) and Neovim, saffron author drafts feature files from prose in your own vocabulary, and duplicate wordings that record identical actions get behavior-proven rename proposals. The Saffron VS Code extension lives in the sibling repo saffron-vscode.

Data tables and doc strings are first-class: 2-column key/value tables parameterize the recording (<table:username>), multi-row record tables parameterize per cell (<table:1:firstName>), and """ doc strings record as <docstring> — so editing values or content replays at zero tokens, while structural changes (keys, headers, row counts) honestly re-record. Unambiguous params are scenario-wide, so a later assertion on a note's text follows content edits too. Secrets stay out of everything: {env:VAR} resolves from the environment (or a git-ignored .env) at replay, recordings and reports are masked back to the token, and missing variables fail fast by name.

Install

npm install --save-dev saffron-ai
npx playwright install chromium
npx saffron run

CI tip: saffron run --strict treats yellow (passed-with-adaptation) as a failure until its cache proposal is reviewed — cached-green-only builds.

Quick start (this repo)

npm install
npm run build
npx playwright install chromium

# start the demo app
node examples/demo-app/server.mjs 4173 &

# replay the example suite from its committed caches — zero AI
node dist/cli/index.js -p examples run

# delete a cache and watch the agent re-record it (needs Claude Code auth
# or ANTHROPIC_API_KEY)
rm examples/.saffron/cache/login/successful-login.json
node dist/cli/index.js -p examples run
node dist/cli/index.js -p examples accept --all

CLI

| Command | What it does | |---|---| | saffron run [paths] [--headed] [--filter @tag] [--no-agent] [--strict] [--browser b] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless --no-agent). Replay cross-browser with --browser firefox\|webkit, parallelize with --workers N, heal on a cheaper model with --heal-model. Exit 1 on red (and on yellow with --strict). | | saffron accept [file \| --all] [--with-feature-edit] [--propagate] | Promote cache proposals to committed caches; --with-feature-edit also rewrites the adapted steps in the .feature file (and keeps the cache in sync); --propagate applies the heal's locator fixes to every other cache using the same locator — one heal repairs N scenarios before they ever fail. No args: list pending proposals. | | saffron reject [file \| --all] | Discard proposals; the agent will try again next run. | | saffron steps [search] [--json] [--snippets] | List/search the step vocabulary (files + caches) with recorded/divergent/unrecorded badges and usage; --snippets writes .vscode/saffron.code-snippets for native VS Code completion. | | saffron mcp | Serve the vocabulary to AI assistants over stdio MCP (search_steps, list_step_sets) — e.g. claude mcp add saffron -- npx saffron mcp. | | saffron author <prose-file> [-o out] | Draft a .saffron feature file from plain-paragraph requirements, reusing the project's step vocabulary (AI; reports how many lines are seedable). | | saffron lsp | Run the Saffron language server over stdio — JetBrains (LSP4IJ/Ultimate), Neovim, any LSP editor: badge completion, StepSet go-to-definition, hover, diagnostics. | | saffron report | Open the latest HTML report. |

Configuration lives in saffron.config.json at your project root:

{
  "baseURL": "http://localhost:4173",
  "features": "features",
  "actionTimeoutMs": 5000,
  "retries": 1,
  "model": "claude-sonnet-5",
  "maxTurns": 100,
  "storageState": ".auth/state.json",
  "strict": false,
  "assertionPolicy": "strict",
  "verifyProposals": true,
  "reuseSteps": true,
  "snapshotMode": "none",
  "browser": "chromium",
  "workers": 1,
  "healModel": "claude-haiku-4-5"
}

storageState (also --storage-state) points to a Playwright storage-state JSON (cookies + localStorage) applied to every scenario's browser context, so cached replays and the agent start authenticated. Generate one with npx playwright open --save-storage=.auth/state.json <url>.

How it works

The full execution-model diagram is on the site: https://saffron-ai.lovable.app/docs

• Caches store per-step action lists with semantic targets (role + accessible name, nameRegex for volatile labels, CSS fallbacks) and <param> placeholders, so one cache serves every Examples row of a Scenario Outline.
• Dynamic-content vocabulary for real-world apps: captureText stores displayed values (prices, counters) under a name; expectDiffers compares against captured values; expectMatches/expectAttribute assert patterns instead of literals; {date±N} templates keep date-picker locators valid across days; the replayer follows links that open new tabs; native dialogs, file uploads, drag-and-drop, and iframe-scoped elements replay on all three engines.
• The agent and the replayer share one browser over CDP, so healing continues from the exact page state where replay failed.
• Every recorded action carries a page fingerprint (hash of the normalized aria snapshot) for token-free drift detection.

Project layout

.saffron/
  cache/         committed replay caches (commit these)
  proposals/     pending AI-generated cache updates (review these)
  history.jsonl  one line per run — trends (commit recommended)
  reports/       latest.html / latest.json (gitignore these)
features/        your .feature files

Roadmap

See the roadmap on the site — the single source of truth for milestones, statuses, and decision gates. Headlines: action-vocabulary additions as real-world failures surface them, the Level-2 shared step library (gated on Level-1 divergence data), LLM-less MCP replay, multi-provider agents, parallel workers, public npm/GitHub release, and the long-term open-core SaaS layer. Everything shipped so far is in the roadmap's "Where we are" table.

Documentation

• Website — what Saffron is, the honesty rules, economics, quickstart
• Documentation — the cache lifecycle, step sets & the .saffron dialect, the step vocabulary, and the guarantees — with diagrams
• Issues & questions
• Docs site: npm run docs builds a self-contained page at

License

Free to use — including commercially — under the Saffron Free Use License (see LICENSE). The source opens up later; a CLA will accompany that step.