e2e-autopilot

Autonomous end-to-end QA for Claude Code — installed once as a plugin, runs on any project. It discovers a web app's screens, drives them in a real browser, finds end-user-facing defects, fixes the safe ones, opens a PR, runs a reviewer↔fix loop, and involves the human only to merge.

Status

Design SHIP-approved (4-round Codex review). Phase 8 complete — on top of the 1.0.0 public release it lands the three residual "Later" items as three independent, off-by-default tracks: an opt-in gated-fix tier (human-confirm a requirement / visual-regression before it is fixed, behind two human gates), a richer domain vocabulary (numeric, exact/regex, and relational rules), and a read-only cross-run trend report. Phase 7 (1.0.0, the first public release) added two additive report-only oracle tiers that ride inside verify (both off by default): domain oracles (catch requirement bugs from committed, human-authored rules) and visual-regression baselines (screenshot drift per screen × breakpoint), plus marketplace submission-readiness (versioning, CHANGELOG.md, docs/RELEASE.md). Phase 6 made it run unattended in the cloud (preview baseUrl, Routines triggers, Slack MCP notify, hooks/hooks.json + .mcp.json, single-run lock, worktree GC, re-auth, retries — docs/agents/cloud-triggers.md); Phase 5 packaged the plugin; Phase 4 delivered all five CLI modes (discover, verify, fix, review, notify), real and unit-tested. See docs/agents/run-pipeline.md for the full end-to-end run.

Install as a plugin

The plugin is a veneer; the engine is the e2e CLI, which /e2e-autopilot:setup installs into your project.

/plugin marketplace add impetus-network/e2e-autopilot
/plugin install e2e-autopilot@e2e-autopilot-marketplace

Then, in the project you want to test:

/e2e-autopilot:setup      # install the CLI + chromium, write e2e.config.ts, store creds
/e2e-autopilot:run        # discover -> verify -> fix -> review loop -> notify, stop at "PR ready to merge"

/setup writes e2e.config.ts + e2e.fixtures.ts into your project (never into this repo) and stores creds via the AuthAdapter (keychain + gitignored .env); the plugin ships no secret. Validate the packaging with claude plugin validate . --strict.

How it works

trigger (/e2e-autopilot:run or schedule)
  -> boot the app locally    (config.target.boot; auto-probe/spawn/teardown)
  -> discover screens        (e2e-qualifier + e2e CLI)
  -> verify (Playwright, console/network/a11y)
  -> defect? -> confidence + class gate
       auto-fixable -> e2e-mechanic fixes -> PR -> e2e-inspector review loop (cap 3)
       report-only  -> report to human
  -> Slack notify "PR ready to merge"
-> you review + merge

• Engine: agent-browser (explore) + Playwright (durable specs) + selector-only self-heal.
• Core: a deterministic Node e2e CLI owns config, fixtures, adapters, app boot (config.target.boot), Playwright, artifacts, secrets, confidence scoring. Agents own judgment.
• Portable: per project you run /e2e-autopilot:setup (writes e2e.config.ts + creds) then /e2e-autopilot:run.

CLI (foundation)

The deterministic core lives in src/ and ships as the e2e bin.

pnpm build
node bin/e2e.mjs            # runs discover + verify
node bin/e2e.mjs verify     # a single mode
node bin/e2e.mjs verify --base-url https://pr-123.preview.app   # cloud/preview target
node bin/e2e.mjs discover verify --lock                          # serialize runs per project
node bin/e2e.mjs notify --job <id> --pr-url <url>

Modules: contracts/schemas (Zod data contracts), config/{validate,load} (load a project's e2e.config), explore/* (ExplorationAdapter + agent-browser / Playwright-crawl), discover/* (multi-strategy screen map), verify/* (oracles + reproduce + reproSpec codegen + Playwright capture), notify/* (redact + scrubbed summary + Slack), jobs/writer (.e2e/jobs/<id> artifacts), triage/confidence (score + fix-gate), cli/run (mode router). All five handlers are real — see docs/agents/run-pipeline.md.

Run in the cloud

The same e2e.config.ts serves localhost and the cloud — the only run-time difference is the target URL and where secrets come from. Cloud is a target + a trigger + two plugin files, not a rewrite (docs/agents/cloud-triggers.md).

1. Onboard with target.mode: "preview" and a non-UI auth adapter (apiSession / testBypass — cloud is a fresh clone with no localhost access).
2. Store secrets in the Routine / Claude secret store (E2E_BASE_URL, E2E_SLACK_TOKEN, E2E_SLACK_TEAM_ID, E2E_ACCOUNT_*) — never in .env, never committed. The plugin's .mcp.json references ${E2E_SLACK_TOKEN}, never a literal.
3. Create a Routine (/schedule or claude.ai/code/routines) with prompt /e2e-autopilot:run --base-url ${E2E_BASE_URL} — 5-field cron (1-hour minimum) or a GitHub PR event. Shapes: routines/{nightly,on-pr}.example.json.
4. On fire: the base URL resolves + preflights → the usual discover→verify→fix→loop → the crew posts the scrubbed summary.md to Slack via MCP → the run stops at "PR ready to merge". No autonomous merge, ever.

Notify channels (pick one per run): a local webhook (E2E_SLACK_WEBHOOK_URL, the CLI posts) or a cloud Slack MCP connector (the crew posts the CLI-scrubbed summary). Only the scrubbed summary.* ever leaves the machine on either. An optional, opt-in inbound bridge (services/slack-inbound/) maps a Slack command → a run trigger (signature-verified, allowlisted; it never merges or posts results).

Local mode is unchanged: no flag/env ⇒ config.target.baseUrl + webhook notify.

Auth (AuthAdapter)

The sole session provider. createAuthProvider(config) exposes acquireRole(role), returning a per-role Playwright storageState path under auth.storageStateDir (gitignored). Adapters, selected by auth.adapter:

• apiSession (email/password, non-UI) — POSTs creds to auth.apiSession.loginUrl, captures Set-Cookie (and an optional JSON token into localStorage). The path CI/cloud use.
• testBypass — guarded test-only token seed (needs E2E_TEST_BYPASS=1).
• manualStorageState — loads a storageState a human saved once.
• uiOidc — stub until the browser ExplorationAdapter lands.

Credentials resolve from <envPrefix>_EMAIL / <envPrefix>_PASSWORD via env / .env (keychain optional). Secrets and tokens are never logged or committed.

Fix → review loop (Phase 3)

The deterministic loop machinery lives in the CLI; the agents supply judgment.

• e2e fix --job <id> — gate every bug, write dispatch.json (eligible vs report-only). Never edits code.
• e2e review --worktree <path> — run the mechanized hard-checks (product code changed, no weakened assertions, tests green) → review-checks.json. Takes --tests-green from the caller (the orchestrator runs the suite; the CLI does not); it records the verdict and exits 0 even on a failed check (read .passed), exiting non-zero only on an infra error such as a bad --base or a git failure.
• Worktree isolation (src/fix/worktree.ts), per-job budget (src/fix/budget.ts), and a capped state machine (src/loop/state.ts, cap 3) drive the mechanic↔inspector loop. See docs/agents/fix-review-loop.md.

Domain oracles (report-only, off by default)

Beyond the implicit oracles, catch requirement bugs — the app renders cleanly but does the wrong thing for your product. The qualifier reads CONTEXT.md / docs/ and distills each checkable requirement into a DomainRule in a closed vocabulary (present | absent | text-matches | count-equals | count-at-least | ordering | attribute-equals); a human reviews and commits e2e.oracles.ts. At run time the CLI only checks the constrained rule — no prose is interpreted. Enable with config.oracles; a violation becomes a reproduced, report-only requirement bug traceable to its rule id — never auto-fixed. Runbook: docs/agents/domain-oracles.md.

Visual regression (report-only, off by default)

Per screen × breakpoint, capture a screenshot baseline and compare against it with Playwright's bundled diff (zero new deps). Enable with config.visual.enabled; a drift over threshold becomes a report-only visual-regression bug with -expected / -actual / -diff artifacts. A first run creates the baseline (not a defect); e2e verify --update-baselines regenerates them. Baselines are committed PNGs — capture against synthetic fixtures, mask dynamic/PII regions, and keep them in the committed baselineDir (never under .e2e/ or e2e/.auth/). Approval is a human review + commit; never auto-fixed. Runbook: docs/agents/visual-regression.md.

Gated fixes — human-confirmed (Phase 8, off by default)

Opt a requirement / visual-regression class into config.fix.gated and it becomes fixable — behind two human gates. A confident gated finding lands in dispatch.pendingConfirmation; nothing is touched until a human runs e2e confirm --job <id> --bug <n> (or --accept-baseline for a visual drift, or --decline). The confirmation is bound to a content digest of that exact finding, so it can't be replayed or mis-targeted. A confirmed requirement is fixed by the mechanic against a codegen'd domain reproSpec; a confirmed visual-regression is resolved by accept-baseline (a baseline-only PR, never a CSS patch). Both run the standard mechanic↔inspector loop and still stop at the human-merge gate — there is no autonomous merge. Absent config.fix ⇒ both stay strictly report-only. Runbook: docs/agents/gated-fix.md.

Cross-run trends (Phase 8, read-only)

e2e report [--window <N>] [--out <dir>] aggregates the scrubbed run-summary history across .e2e/jobs/* into a drift/recurrence trend — each recurring finding classified transient | flaky | persistent, so you can see whether a screen is chronically regressing. It reads only summaries + non-PII manifest fields (never raw bug JSON), re-scrubs through redact.ts, and writes .e2e/trends/report.{json,md}; --out copies the Markdown to a committed dir. No new data leaves the machine.

Releasing

Version is kept in parity across .claude-plugin/plugin.json, .claude-plugin/marketplace.json, and package.json (asserted by the guard test). The full submission-readiness checklist — both validate gates, the npm + github publish path, the clean-clone install dry-run, and the human-run community-marketplace submission — lives in docs/RELEASE.md. Changes are tracked in CHANGELOG.md.

Layout

docs/plans/2026-07-04-e2e-autopilot-plan.md   # the plan (source of truth)
docs/ideas/e2e-qualifier-idea.md              # origin concept
CHANGELOG.md                                   # versioned changelog
docs/RELEASE.md                                # release + marketplace checklist
AGENTS.md                                      # agent/session orientation

Crew

• e2e-qualifier — QA engineer (discover + test + codegen)
• e2e-mechanic — fix engineer (patch + PR)
• e2e-inspector — review + gate the PR
• e2e-onboarder — first-run setup wizard

See the plan for each agent's role / skills / tech-stack.