AISentinelQA — MCP Server

A Model Context Protocol (stdio) server that lets IDE AI agents (Cursor / Windsurf / Claude Desktop, or any MCP-capable client) drive AISentinelQA directly from the editor.

This closes the #1 TestSprite gap: TestSprite is invoked from the IDE, but our platform's real execution engine, onboarding/understanding, traceability, ALM and SSO live behind a web app. This server exposes them as first-class MCP tools the agent can call — and, crucially, it acts as a specific user via a personal API token, so the same RBAC and team scoping apply.

Why this is the credibility play

Two kinds of tools, both real:

1. Local execution — run_ui_test does real headless-Chromium execution. It does not re-implement a browser harness; it imports and calls the platform's own runPlaywrightTest from ../test-runner-service/playwright-runner.ts, the exact code path the production runner uses.
2. Platform API (per-user token) — whoami, onboard_repo, get_coverage, run_suite, and analyze_failure call the running app's HTTP API authenticated with the user's ait_ token. The agent in Cursor operates the platform exactly as that user would.

Tools

| Tool | What it does | Kind | |------|--------------|------| | run_ui_test | Runs a UI test in real headless Chromium against a URL, walking each step and verifying expectations. | Local, real execution (imports the platform runner). | | generate_playwright_spec | Deterministically generates a runnable @playwright/test spec string with a // @tc:<id> marker + report annotation for traceability. | Local, generated, not executed. No LLM. | | list_capabilities | Reports platform capabilities, each marked verified vs available. | Local, honest report. | | whoami | Confirms the API token works and reports the authenticated user (id, name, email, role). | Platform API (per-user token). | | onboard_repo | Points the platform at a GitHub repo (read-only) and returns what it understands: stack/frameworks/routes/endpoints, a coverage estimate, and a gap-driven backlog of proposed tests. | Platform API (per-user token). | | get_coverage | Returns the requirement↔test↔result coverage matrix + summary (optionally scoped to a project). | Platform API (per-user token). | | run_suite | Triggers a real run of a saved test suite and returns the run record (status + pass/fail/skip counts). | Platform API (per-user token). | | analyze_failure | AI root-cause for a failed execution case: summary, likely cause, suggested fix, developer prompt. | Platform API (per-user token). |

Install from npm (zero clone — recommended)

Published as aisentinelqa-mcp. Anyone can run the server with no clone, no paths — just like TestSprite's published package. Add this to ~/.cursor/mcp.json (or Windsurf / Claude Desktop) and replace the token with your own ait_…:

{
  "mcpServers": {
    "aisentinelqa": {
      "command": "npx",
      "args": ["--yes", "aisentinelqa-mcp@latest"],
      "env": {
        "AISENTINELQA_API_URL": "https://your-instance.example.com",
        "AISENTINELQA_API_TOKEN": "ait_your_own_token"
      }
    }
  }
}

For run_ui_test (real Chromium), install the browser once: npx playwright install chromium.

Publishing updates (maintainer): bump version in package.json, then npm publish (the prepublishOnly script rebuilds dist/bin.js). Publishing needs 2FA — either an OTP (--otp=123456) or a Granular token with Read+Write to All packages and Bypass 2FA enabled.

Local-dev alternative (run from source, no npm): command: "npx", args: ["--yes", "tsx", "<ABSOLUTE path>/mini-services/mcp-server/index.ts"].

Setup (per-user token + config)

1. Create a personal API token

In the running app: Settings → API Tokens → New token. Copy the ait_… value — it is shown once. (Only its hash is stored; revoke any time.)

2. Point the MCP server at your instance

Two env vars enable the platform-API tools:

| Env var | Meaning | Default | |---------|---------|---------| | AISENTINELQA_API_URL | Base URL of your running app | http://localhost:3000 | | AISENTINELQA_API_TOKEN | Your ait_ token | (required for API tools) |

If AISENTINELQA_API_TOKEN is unset, the local tools (run_ui_test, generate_playwright_spec, list_capabilities) still work; the API tools return a clear "token not set" message instead of failing silently.

3. Add it to your IDE

See mcp.json for a ready-to-paste config. Same { "mcpServers": { … } } shape works for Cursor, Windsurf and Claude Desktop.

{
  "mcpServers": {
    "aisentinelqa": {
      "command": "npx",
      // Use an ABSOLUTE path to index.ts — Cursor does not reliably honor `cwd`,
      // so a relative "index.ts" + "cwd" fails to launch. `--yes` stops npx from
      // hanging on a tsx install prompt.
      "args": [
        "--yes",
        "tsx",
        "C:/ABSOLUTE/PATH/TO/AISentinelQA/mini-services/mcp-server/index.ts"
      ],
      "env": {
        "AISENTINELQA_API_URL": "http://localhost:3000",
        "AISENTINELQA_API_TOKEN": "ait_replace_with_your_token"
      }
    }
  }
}

Why an absolute path and not cwd? Cursor does not reliably honor the cwd field (especially in a window with no folder open), so args: ["tsx", "index.ts"] + cwd fails with Cannot find module …index.ts → connected=false. Passing the absolute path to index.ts is cwd-independent and works everywhere. tsx still resolves the file's imports + node_modules from the file's own directory.

Why npx tsx and not bun? run_ui_test launches Playwright Chromium, which hangs on Windows under bun's process pipe. Running the server under Node/tsx avoids that. (Pure-API setups can use either.)

| Client | Config file | |--------|-------------| | Cursor | ~/.cursor/mcp.json (global) or <project>/.cursor/mcp.json | | Windsurf | ~/.codeium/windsurf/mcp_config.json | | Claude Desktop | claude_desktop_config.json |

Walkthrough: onboard a project from Cursor

Open any project in Cursor and ask the agent things like:

1. "Use the aisentinelqa MCP — who am I?" → calls whoami, proving the token works and showing which user you're acting as.
2. "Onboard the GitHub repo acme/storefront and tell me what's tested and what's missing." → onboard_repo returns the detected stack, a coverage estimate, and a prioritized backlog of proposed test cases.
3. "Show me the coverage matrix." → get_coverage.
4. "Run the smoke suite <suiteId>." → run_suite triggers a real run.
5. "Generate a Playwright spec for the login flow, then run it for real against http://localhost:3000/login." → generate_playwright_spec + run_ui_test.
6. "Analyze the failing case in execution <id>." → analyze_failure.

This is the company-onboarding loop: connect a repo → understand it → see coverage gaps → generate and really run tests → trace results back.

Install & run locally

# from this directory (mini-services/mcp-server)
bun install                 # installs the SDK + zod (deps only; we run via tsx)
npx tsx index.ts            # starts the stdio server ("ready on stdio" to STDERR)

Real execution prerequisite: Chromium must be installed for the Playwright runner. If missing, run_ui_test returns an honest "browser not installed" result instead of crashing: npx playwright install chromium.

stdout is the MCP transport. This server never writes to stdout; all logs go to stderr.

Verify it works (live, over stdio)

# local tools only:
npx tsx verify-mcp.ts

# full end-to-end incl. live token auth (app must be running):
AISENTINELQA_API_URL=http://localhost:3000 \
AISENTINELQA_API_TOKEN=ait_xxx \
npx tsx verify-mcp.ts

verify-mcp.ts talks to the server exactly like a real IDE agent would:

1. spawns index.ts and connects over stdio;
2. lists tools — asserts all 8 are present;
3. calls list_capabilities;
4. calls generate_playwright_spec (asserts the // @tc: marker);
5. calls run_ui_test against VERIFY_URL (default https://example.com) → real headless Chromium;
6. if AISENTINELQA_API_TOKEN is set, calls whoami → live per-user token auth against the running app.

Exit code is 0 only when tools list, the spec generates, run_ui_test actually executed in a real browser, and (when a token is set) whoami returned the user.

Env overrides: VERIFY_URL, VERIFY_RUN_TIMEOUT_MS (default 240000).

Tests

bun test tests/

Unit tests cover the deterministic, browser-free helpers: spec generation (marker, annotation, locators, escaping, determinism) and the capability report. URL-building for the API client is covered by tests/mcp-client.test.ts at the repo root.

Honesty statement

• run_ui_test is real browser execution via the platform's own runner — not a mock. A failed step is reported as a real failure.
• The platform-API tools (whoami, onboard_repo, get_coverage, run_suite, analyze_failure) make real authenticated HTTP calls to the running app and surface the genuine response; each result is labeled with the HTTP status.
• generate_playwright_spec generates code; it does not run it and uses no LLM. Output is labeled "generated, not executed".
• list_capabilities distinguishes verified (proven here) from available (implemented in the repo but not exercised by this process).