TestTrace

Your AI coding assistant guesses selectors and the test fails. TestTrace gives it the real ones from your code, verified against your running app — and scores which elements keep breaking.

Two features:

• Locator MCP — a local MCP server that gives Cursor, Claude Code, or any MCP-compatible assistant ground-truth locators for your app's elements, sourced from your own code and confirmed against your running app.
• Locator Risk Score — combines git churn on your test repo with CI failure history to flag which locators are most likely to break next, with evidence for every score.

Runs entirely locally: a CLI plus a local stdio MCP server. No account, no hosting, no domain. Nothing leaves your machine.

Status: Gates 1–3 pass automated validation (self-check parser, self-check risk, self-check cli) against test-app ground truth and calibration fixtures. 210 tests. Human QA review on a production repo is still recommended. See Limitations.

Install

npm install -g testtrace

Or run it without installing:

npx testtrace scan ./my-app

Playwright is a dependency (used for verify); its browser binaries install automatically the first time you run a command that needs them.

Feature 1 — Locator MCP

1. Inspect your app (optional, informational)

testtrace inspect ./my-app

Detects framework, router type, language, UI library, test framework, package manager, monorepo layout, and your existing test-attribute convention (data-testid, data-qa, etc.).

2. Scan

testtrace scan ./my-app

Walks your app's routes — Next.js App Router, Pages Router, React Router (both the createBrowserRouter([...]) config API and the <Routes><Route> JSX API, including nested routes, dynamic params, and catch-alls), Vue Router (createRouter({ routes: [...] }), including nested/dynamic routes and lazy-loaded () => import(...) components), or SvelteKit's file-based routing (src/routes/, +page.svelte/+layout.svelte, [param]/[[optional]]/[...rest] segments) — parses each page to an AST, and extracts every interactive element with ranked locator candidates. Also writes a repository manifest (repo-manifest.json: file inventory, entry points, local import graph) and a bounded component composition tree per page for React (JSX import following), Vue/Svelte (local .vue/.svelte import following), and Angular (selector-map matching), up to depth 8. Writes .testtrace/ in your current directory — an index.json manifest plus one JSON file per route. Nothing is sent anywhere.

React Router has no file-system routing convention, so route definitions are found by scanning the whole project for createBrowserRouter/createHashRouter/createMemoryRouter calls and <Routes> JSX trees, wherever they live. A route's component only resolves if it's a traceable local import or declared in the same file — never guessed, so a component from a UI library or a dynamic reference is honestly skipped rather than misattributed. Next.js detection is gated on an actual next dependency in package.json, not just on an app//pages/ folder existing — plenty of plain React Router projects use a pages/ folder purely for organization, and that's not Next.js. SvelteKit detection is gated the same way, on an actual @sveltejs/kit dependency.

Vue detection works the same way, framework-appropriate: components declared as .vue single-file components are told apart from native HTML elements using Vue's own compiler AST (no naming-convention guessing), and a component reference resolves through Vue's automatic attribute fallthrough — a data-testid (or other locator-relevant attribute) passed at the call site and not declared as a prop lands on the component's own root element, the same way it would at runtime.

Svelte works differently, and deliberately so — verified by compiling real Svelte 5 components rather than assumed: unlike Vue, Svelte does not forward extra attributes to any element automatically. Forwarding only happens where the component author explicitly spreads a rest-props object ({...$$restProps} in Svelte 4, or {...rest} from let { a, ...rest } = $props() in Svelte 5) onto some element in their own template — wherever that is, root or not. A component reference resolves by finding that spread in the AST and merging the call site's own attributes into it; if no such spread exists, the call site's attributes genuinely aren't reachable, and that's reported as a caveat rather than silently guessed.

3. Verify against your running app (optional but recommended)

testtrace verify --url http://localhost:3000 [--storage-state auth.json] [--evidence]

Opens a browser, visits each discovered route, and confirms each element actually exists in the live DOM — marking it RESOLVED, CONDITIONAL (in source but not observed — likely a feature flag or permission gate), or UNRESOLVED. --storage-state reuses whatever Playwright auth state your own test suite already produces, so this tool never needs to understand your login flow. --evidence captures a screenshot and DOM snapshot per route.

4. Point your AI assistant at the MCP server

Generate a starter MCP config (merges with any existing servers):

testtrace connect --app ./my-app          # writes .cursor/mcp.json
testtrace connect --app ./my-app --claude # writes .mcp.json

Or add manually to your MCP client's config (e.g. Claude Code's .mcp.json, Cursor's MCP settings):

{
  "mcpServers": {
    "testtrace": {
      "command": "npx",
      "args": ["testtrace-mcp", "/absolute/path/to/my-app"]
    }
  }
}

The server exposes six tools:

| Tool | Purpose | |---|---| | get_page_map() | Every route discovered, with resolution status | | list_elements(page) | Every element on one route, with status and confidence | | get_page_composition(page) | Bounded component tree for one route (React/Vue/Svelte/Angular) | | get_locator(page, element_description) | Ranked candidates, confidence, evidence, and ready-to-use Playwright/Selenium/Cypress syntax | | get_risk_report({ top?, route? }) | Ranked risky elements with evidence (requires audit and/or import first) | | report_result(locator, outcome, error?) | Best-effort — lets the agent report back whether a locator actually worked |

Ask your assistant to write a test for a page. It calls get_page_map and get_locator instead of guessing.

Feature 2 — Locator Risk Score

1. Analyze git churn on your test repo

testtrace audit ./my-tests [--top 10] [--exclude-bare-tags]

Reads git log over your test files and scores each locator by how often it's changed, over what time span, by how many authors. No CI integration needed — this alone produces a ranked list within minutes of connecting.

2. Import CI test results

testtrace import ./artifacts/junit.xml
# or, for Playwright specifically:
testtrace import ./artifacts/playwright-report.json

Parses JUnit XML (the format every major CI system and test framework already emits) and extracts the locator from each failure message. Run this after every CI run — results accumulate.

JUnit XML undercounts real instability wherever retries are configured — confirmed empirically, not assumed, across all three frameworks. A flaky-on-purpose test (fails once, passes on retry) was run for real through Playwright's own reporter, Cypress's mocha-junit-reporter, and pytest's built-in --junitxml (driving Selenium), each with retries enabled. All three wrote the retried-then-passed test as a completely clean pass — failures="0", no <failure> element, no indication a retry ever happened. This isn't a bug in any one tool; JUnit XML's schema has no concept of "attempt" at all, so nothing that emits it can represent a recovered failure. Since retries exist specifically to absorb exactly this kind of flakiness, a locator that's genuinely intermittent can score as if it never failed once.

For Playwright, use --reporter=json instead — it preserves every attempt, including ones a test recovered from, and testtrace import reads it directly, counting each retried-but-recovered test execution as one instability event (not one per retry attempt — three retries in one CI run is one flake, not three).

Selenium and Cypress ingestion is JUnit-only, and investigation found no zero-effort fix for either — confirmed by testing, not assumed. For Cypress: neither mocha-junit-reporter nor mochawesome (a richer JSON reporter) records a retried-then-passed test as anything but a single clean pass, and even Cypress's own Module API — which has a field (test.attempts) specifically documented for this — only ever returned one attempt in testing, despite the retry demonstrably happening (timing matched the retry delay exactly). For Selenium via pytest + pytest-rerunfailures: pytest-json-report's output has the same blind spot, and --rerun-show-tracebacks (a flag that does print the failed attempt's traceback) only affects console text output — the JSON artifact itself is unchanged. Getting equivalent visibility for either would mean the customer writing custom test-repo instrumentation (a pytest hook, a Cypress plugin), not picking a different built-in reporter flag — out of proportion for what this tool promises (zero code changes in the customer's CI). Left as a known, permanent limitation for these two frameworks rather than forced.

3. Get the combined risk report

testtrace report [--top 10] [--json] [--exclude-bare-tags]

Validate Gate 1 parser accuracy (static ground-truth match + optional runtime verify against test-app):

testtrace self-check parser [--json] [--no-verify]

Validate Gate 2 calibration (automated QA ranking check against test-app + synthetic evidence):

testtrace self-check risk [--json]

Validate Gate 3 CLI usefulness (automated rubric — actionable locators, transparency flags, evidence sections, JSON mode, clear errors):

testtrace self-check cli [--json]

Or run the full local pipeline in one command (scan → optional verify → optional audit → optional import → report):

testtrace run \
  --app ./my-app \
  --verify-url http://localhost:3000 \
  --storage-state auth.json \
  --tests ./e2e \
  --junit ./artifacts/junit.xml \
  --top 10

Joins your scanned registry against both churn and failure history, and prints a ranked list like:

Locator Risk: 47/100          Confidence: HIGH

  "Apply promo code" button — /checkout
  Source: app/checkout/page.tsx:21

  Evidence
    • locator changed in 2 commit(s), 2 author(s), last changed 2026-08-11 (churn score 2.6)
    • 3 locator-related failure(s) in the last 30 days, last failure 2026-08-11

  Cause
    Located by accessible role/label (good practice) — but the element's accessible name may be changing, and there's no dedicated automation attribute yet.

  Fix
    Add data-qa="checkout-apply-promo-code" as a supplementary stable anchor        [1-line PR]

Also available as an MCP tool, get_risk_report({ top?, route? }), so your assistant can check an element's risk before it relies on that locator.

How confidence and risk scores are computed

Both scores are documented, evidence-backed approximations of formulas described only in prose in the original design spec — not literal reproductions. Every score is always returned with the evidence behind it; neither tool reports a number without justification. See src/confidence.ts and src/risk-score.ts for the exact formulas and the worked examples they're checked against.

Limitations

• Static analysis sees JSX/templates, not the rendered DOM — deep composition through component libraries may be unresolvable without verify.
• Duplicate accessible names on the same route (e.g. repeated "View" links in a table) are flagged AMBIGUOUS at scan time and refused as a single top locator by MCP — scope by row or add a test attribute.
• Conditional rendering and feature flags mean an element may not exist at runtime even if it's in source.
• Risk score constants are calibrated against the spec's worked example and an automated Gate 2 fixture (fixtures/gate2/) — not yet validated by a live QA team on a production repo.
• JUnit failure-message extraction covers Playwright, Selenium (JS/Java/Python/C#-shaped messages), and Cypress — an unrecognized format is left honestly unextracted, never guessed.
• JUnit XML ingestion undercounts instability wherever CI retries are configured — see the retry note under Feature 2, step 2. Playwright JSON ingestion closes this gap for Playwright; Selenium and Cypress are still JUnit-only and still have it.
• Angular is supported for route discovery and template analysis (standalone components with provideRouter / RouterModule). Vue 3 and Svelte 5 (SvelteKit) are also supported.
• React Router's file-based routing convention (Remix / React Router v7's newer file-route API) is different from the config-object and JSX APIs supported here, and isn't covered.
• All four frameworks trace component composition arbitrarily deep (bounded, cycle-safe, depth 8) to find interactive elements nested through wrapper components (list → item → button, container/presentational splits) — a component reference is explored regardless of whether it carries role/a click handler itself. Attribute/prop merging from the call site is still only ever attempted at the first hop, not carried through multiple levels: a deep-found element is reported using its own in-file candidates, flagged with a note that the call site's own attributes (e.g. a data-testid) weren't traced that far. Vue's merge is additionally root-only (its own inheritAttrs behavior); Svelte's is exact (only elements that actually spread {...rest}/{...$$restProps}); Angular has no merge step at all (the host tag is always a real DOM node). A component that renders more than one interactive element is no longer discarded as "ambiguous" — every one is now reported independently, and the existing duplicate-locator (AMBIGUOUS) detection catches any that would collide.
• Non-relative imports resolve through tsconfig.json's compilerOptions.paths (@/*, src/*, etc.), including one level of extends. Aliases defined only in a bundler config (vite.config.ts, webpack.config.js) with no tsconfig mirror are not read — those configs are executable code, not statically parsed.

Development

npm run build          # compile TypeScript
npm test                # run the test suite (includes Gate 1–3 validation)
npm run typecheck:tests # typecheck test files
testtrace self-check parser  # Gate 1 parser + verify check
testtrace self-check risk    # Gate 2 calibration check
testtrace self-check cli     # Gate 3 CLI usefulness check

License

ISC