test-ez

Failing tests, made EZ — a portable, model-agnostic MCP that runs your Playwright e2e and Vitest unit suites, triages why each one failed (backend/API break · frontend change · real bug · flaky · snapshot drift · test-authoring · environment), and — on your approval — opens grouped fix PRs on GitHub.

It's data + deterministic logic only: it runs tests, parses reports, recovers network/console from traces, correlates with your git diff, and rule-classifies each failure with evidence + a suggested fix. Your AI client (Claude Code, Cursor, Windsurf, …) drives the conversation and applies the code edits. Nothing is project-specific — drop it into any Playwright + GitHub repo.

Install

test-ez is an MCP server. You register it once per AI host (Codex, Claude Code, Cursor, Windsurf, …); project config is saved separately, later. Restart the host after registering so it loads the server.

Want your agent to do it? Skip to Let your agent install it — paste one block and it detects your host and registers everything.

Recommended — one command per host (from npm)

test-ez is published on npm as a public package, so it installs with npx — no clone, no GitHub auth, no .npmrc.

Codex

codex mcp add test-ez -- npx -y @buyhatke-dev/test-ez serve

macOS, if codex isn't on your PATH: replace codex with /Applications/Codex.app/Contents/Resources/codex.

Claude Code

claude mcp add test-ez --scope user -- npx -y @buyhatke-dev/test-ez serve

Cursor / Windsurf / Claude Desktop / any MCP-compatible host — add this to your MCP config (.mcp.json, .cursor/mcp.json, or the host's config file):

{
  "mcpServers": {
    "test-ez": {
      "command": "npx",
      "args": ["-y", "@buyhatke-dev/test-ez", "serve"]
    }
  }
}

TEST_EZ_PROJECT_ROOT is optional — set it (a "env" block in the JSON, or -e TEST_EZ_PROJECT_ROOT=/abs/path on the mcp add command) only if your host launches from a directory other than your project root; otherwise the server uses the launch cwd / git toplevel.

Let your agent install it

Paste this whole block into your AI agent. It detects the host, registers test-ez, and hands you a restart message instead of stalling:

You are setting up the test-ez MCP server for whichever AI host you are running in. Do these steps in order.

STEP 1 — Detect the host
Work out which AI host you are running in: Codex, Claude Code, Cursor, Windsurf, or another MCP-compatible host.

STEP 2 — Register the server
test-ez is a PUBLIC npm package, so it installs with npx and needs no GitHub auth. Register an MCP server:
  Name:    test-ez
  Command: npx
  Args:    -y @buyhatke-dev/test-ez serve

If you cannot register MCP servers yourself, give the user exactly ONE terminal command for their detected host:
  Codex:        codex mcp add test-ez -- npx -y @buyhatke-dev/test-ez serve
  Claude Code:  claude mcp add test-ez --scope user -- npx -y @buyhatke-dev/test-ez serve

STEP 3 — Verify
List the host's MCP servers and confirm test-ez appears.

STEP 4 — Hand off for restart (IMPORTANT)
Registering an MCP server does NOT load it into this running session, and restarting the host ENDS this conversation. So do not call any test-ez tools now. Stop here and show the user this exact message (fill in the host name):

  ────────────────────────────────────────
  ✅ test-ez MCP registered.

  👉 NEXT STEPS (in your project repo, after restart):
     1. Restart <this AI host> now so it loads test-ez.
        (The first start may take a few seconds while npx fetches it — that's normal.)
     2. Say:  "set up test-ez here"        → writes .test-ez/config.json (one-time)
     3. Say:  "run my e2e tests using test-ez"
  ────────────────────────────────────────

Install from source (pre-release / private GitHub repo)

To run an unreleased build straight from Buyhatke/test-ez, install from GitHub. The repo is private, so npx must reach it without an interactive prompt: authenticate to GitHub first (gh auth login, or an SSH key / credential helper with access to the repo). Use the HTTPS spec if you authenticate over HTTPS, or the SSH spec if you use an SSH key:

Codex

# HTTPS:
codex mcp add test-ez -- npx -y github:Buyhatke/test-ez serve
# SSH:
codex mcp add test-ez -- npx -y git+ssh://[email protected]/Buyhatke/test-ez.git serve

Claude Code

# HTTPS:
claude mcp add test-ez --scope user -- npx -y github:Buyhatke/test-ez serve
# SSH:
claude mcp add test-ez --scope user -- npx -y git+ssh://[email protected]/Buyhatke/test-ez.git serve

If the HTTPS form makes your host prompt for a GitHub username every session, you authenticate over SSH — switch to the SSH form, or route GitHub HTTPS through your SSH auth once: git config --global url."[email protected]:".insteadOf "https://github.com/".

Initialise your project (once)

Easiest: in your repo, just tell your AI client "set up test-ez here" — it runs the init tool and writes .test-ez/config.json (no extra npx fetch).

Prefer the CLI? From your project root:

npx -y @buyhatke-dev/test-ez init
# unreleased build from the private repo: npx -y github:Buyhatke/test-ez init

Either way it detects your environment (git / GitHub / gh / Playwright / Vitest), suggests sensible defaults, and writes .test-ez/config.json.

Requirements

• Node ≥ 18 and a project with Playwright (@playwright/test, browsers installed via npx playwright install) and/or Vitest for unit suites.
• For fix PRs: a GitHub repo + the gh CLI, authenticated (gh auth login). Without git/GitHub, triage still works — only PR creation is disabled.

Troubleshooting install

• ENOVERSIONS / "no matching versions" on install — your .npmrc has a min-release-age guard (a supply-chain check that blocks just-published versions), and test-ez was published recently.
  • Permanent, scoped fix: run test-ez exception (or ask your client "run test-ez exception"). It asks first, then adds a min-release-age-exclude for only test-ez — your guard still protects every other package, and future updates won't be age-blocked. test-ez doctor shows whether the guard is active and whether test-ez is excepted.
  • First install (before test-ez is fetchable): add NPM_CONFIG_MIN_RELEASE_AGE=0 to the server's env block (overrides it for test-ez only), or wait until the version is older than your threshold — then run test-ez exception so you never hit it again.
• npmjs.com page 403s for the package — the registry API still works; check it with npm view @buyhatke-dev/test-ez.
• Verify the install anytime: npx -y @buyhatke-dev/test-ez doctor (or tell your client "run test-ez doctor") — prints version, registry reachability, environment, config status, and any warnings.

Quick start

Once it's installed, just talk to your AI client in plain language:

"Run my e2e tests using test-ez." (or "run my unit tests using test-ez")

It will run_e2e → give you a short pass/fail summary → and (after you say go) triage each failure with real evidence, propose fixes, and open_fix_pr for the clear-cut ones. Backend / env / snapshot failures are flagged only, never auto-fixed.

Tools

| Tool | Purpose | |---|---| | init | Detect environment + config, or persist config (set). | | run_e2e | Run a suite or a subset (tests, grep, project, lastFailed, onlyChanged, repeatEach). Forces JSON report + retain-on-failure traces into .test-ez/ regardless of your config. background:true returns immediately (poll with run_status); every run streams to a tailable run.log. command = run an exact shell command then a controlled capture. | | run_status | Live status of the latest (or named) run — phase (pre-step/running/done/failed), elapsed, approximate test counts + current test (parsed from the live reporter), and a log tail. Self-heals a run whose process died without finalizing. On a timed-out/crashed run, surfaces the salvaged partial results. | | list_failures | Failures + stats + skipped, with stable testIds. | | get_failure_detail | Full error/stack/stdout for one test. | | get_network_log | Network from the trace; first-party API failures highlighted. | | get_console_log | Console + uncaught page (JS) errors from the trace. | | get_snapshot | "Look closer" at one failure: the screenshot at the failed step (an image to read/see the real screen — banners, wrong screen, offline state) + the data-testids present in the DOM then. Turns a guess into evidence. | | get_changed_files | git diff vs base, tagged app-source/test/config. | | triage | Gather evidence + signals per failure (error + call log, distilled first-party network, changed-file correlation, trace action log / typed values, cross-browser) plus a labeled heuristic hint. The hint is a prior, not a verdict — the calling agent asserts the issue type. JSON + markdown. | | open_fix_pr | One PR per cause off the base branch, labelled, context in the body. Pass the agent's asserted type as each group's category. dryRun to preview. |

Classification is the agent's job, not the server's. triage deliberately does not return a final verdict — heuristics get fooled by red herrings (a cross-origin asset block, an unrelated app edit). The MCP gathers accurate, deterministic evidence; the LLM reasons over it and asserts the type (backend · frontend · flow-change · test-authoring · flaky · real-bug · infra).

Workflow

🗺️ Full end-to-end map (tools, modules, the e2e↔unit fork): docs/WORKFLOW.md.

1. init (first time).
2. run_e2e — reliable JSON + trace capture is automatic.
3. triage — read the evidence, then assert the issue type yourself (the hint is just a prior).
4. Approve.
5. Apply fixes for the clear-cut, fixable cases; leave ambiguous ones queued.
6. run_e2e { onlyChanged: true } as a touched-area regression gate; repeatEach: 3 to confirm flaky-suspects (must pass 3/3).
7. open_fix_pr with triage.suggestedGroups. Backend / env / snapshot failures are flagged only, never auto-PR'd.

Config (.test-ez/config.json)

{
  "baseBranch": "main",
  "label": "e2e AI fix",
  "defaultSuite": "main",
  "suites": {
    // COMMAND mode (preferred when you have a real test script): run it as-is,
    // test-ez only appends capture flags. Preserves the script's scope/wiring
    // (one browser via randomBrowser, build/deploy, synpress, …) — no fan-out.
    "test:e2e": { "command": "npm run test:e2e" },
    // BUILD mode: test-ez constructs `npx playwright test …` itself.
    "main":  {},                                   // {} → auto-discover playwright.config.*, run all projects
    "e2e":   { "config": "playwright/config.cjs", "project": "chromium",
               "preRun": "npm run deploy:test", "env": { "FOO": "bar" } }
  },
  "repeatEach": 3,
  "runTimeoutMs": 1200000,
  "redact": false,                 // mask secrets in report/triage/console? off (your machine) — PR bodies redact regardless
  "worktree": {
    "enabled": true,               // run/fix/PR in an isolated worktree (default on; false to opt out)
    "path": null,                  // default: sibling "<repo>.test-ez-wt"
    "branch": "test-ez-worktree",
    "ref": "base",                 // what to mirror: "base" | "current" | <ref>
    "autoNpmCi": true,             // reinstall deps only when the lockfile changes
    "installCmd": null,            // override; else derived (npm ci / yarn / pnpm)
    "copyDirs": []                 // extra heavy gitignored dirs to carry in whole (adds to the built-in list; false for none)
  }
}

A suite is a structured run-spec. A suite with a command runs your project's own script verbatim and only appends capture flags (--reporter=json, --trace, --output) — so its scope and wiring (e.g. --project="$(randomBrowser.mjs)", build/deploy, synpress) are preserved and there's no project fan-out. Otherwise preRun runs before tests (e.g. a deploy) and aborts the run if it fails; test-ez builds the playwright test invocation from the suite + selection params.

Long suites — never a black box, never a total loss

• Watch it: every run streams to .test-ez/work/runs/<suite>/run.log (tail -f it). For long suites pass run_e2e { background: true } to return immediately, then poll run_status for phase + live counts + current test.
• Timeout ≠ wasted run: if a run is killed before Playwright writes its JSON report (timeout/crash), test-ez salvages what ran from the live log + the on-disk traces — producedReport:false but partial carries the pass/fail/flaky counts, the failing tests (matched to their traces), and a rerunSubset of file:lines to re-run just the failures for a clean report. Raise timeoutMs, or shard via grep/tests, for a complete run.

Worktree mode (parallel-safe)

Worktree mode is on by default (disable with worktree.enabled: false, or per run with run_e2e { worktree: false }). Runs, fixes, and PRs happen in a persistent sibling git worktree instead of your main checkout — so you can keep working while test-ez operates. Each run:

1. syncs the worktree to the target ref (fetch → reset --hard <ref> → clean, keeping node_modules),
2. carries gitignored env/secret files from your main checkout into the worktree — top-level .env* (minus committed .example/.sample templates) by default, plus anything you list in worktree.copyFiles (e.g. a nested .env or a secrets.json). A fresh git worktree checkout otherwise lacks these, breaking deploy scripts / wallet setup / anything that reads them,
3. carries heavy wallet/auth caches the fresh checkout can't cheaply rebuild (see below),
4. reinstalls deps only when the lockfile hash changed (so the common case starts instantly), and
5. runs the suite there; reports/traces still land under the main repo's .test-ez/work/.

Wallet / auth caches (worktree.copyDirs)

Step 2 deliberately skips heavy directories — node_modules, dist, caches, browser profiles — because they're rebuilt anyway. But some gitignored directories are not cheaply rebuildable: a Synpress .cache-synpress holds pre-onboarded MetaMask/Phantom browser profiles plus the downloaded extensions, and recreating it needs the seed phrase, a headful Chrome and a ~50MB download. Without it the run's own setup step tries to rebuild it inside the worktree on every run — slow at best, and when it fails the run dies before a single test executes, so an environment problem shows up as a failing test suite.

test-ez carries these whole. .cache-synpress is on the built-in list; add your own with worktree.copyDirs: ["my-tool/.cache-state", …], or set "copyDirs": false to carry none. Entries add to the built-in list rather than replacing it.

Playwright's playwright/.auth/*.json storage state needs no entry — it's small enough that step 2 carries it like any other gitignored file.

The copy is copy-on-write cloned where the filesystem supports it (APFS on macOS, btrfs/xfs on Linux), so a 300MB cache lands in well under a second and costs no extra disk — while the worktree still gets its own copy, so a run that rewrites the cache can't corrupt your main checkout's. It's refreshed every run, since your main checkout is the source of truth.

What the worktree mirrors is configurable via worktree.ref (or per run, run_e2e { ref }):

| ref | Mirrors | Use | |---|---|---| | "base" (default) | origin/<baseBranch> | "is the important branch healthy" (CI-mirror) | | "current" | your checked-out branch HEAD (local commit, incl. unpushed) | "fix the tests on my branch" | | any literal ref | that branch / tag / SHA | fix tests as they'll run on release-x, etc. |

The run result echoes the resolved ref + SHA + commit subject, so it's never ambiguous what was tested; an unknown ref is rejected rather than silently falling back. triage and open_fix_pr follow the worktree the last run used — the fix PR targets the branch that was mirrored (so a current run PRs into your branch, not master; override with open_fix_pr { baseBranch }). Your main tree is never stashed or branch-switched. A run-lock (.test-ez/work/run.lock) keeps two runs from trampling the shared worktree. Apply fixes in the worktree path that run_e2e reports.

Note: ref carries committed work only. To test live uncommitted edits, either commit first or run with worktree off (in-place sees everything).

Safety

• Secret redaction is off by default for owner-facing output (report / triage / console) — it's your machine and your secrets, and masking them just hampers debugging; flip redact: true to mask. PR bodies are redacted regardless (auth headers, cookies, tokens, JWTs, emails, env secrets), since they're published to GitHub. Screenshots can't be text-redacted, so the report flags when it contains real data.
• open_fix_pr refuses on detached HEAD / in-progress rebase-merge / missing gh auth, stashes and restores your working tree (protecting unrelated changes), and supports dryRun.
• Snapshot diffs are never auto-updated.

License

MIT