QA Workbench

An MCP server for Bluecopa's QA team. Use it from Claude Code, opencode, or omp to turn Jira tickets into executable BEATS scenarios, run them, and report on the results, all without leaving the chat.

Status: v0.1 is usable locally and was trialled end to end against a live BEATS on 2026-10-15 (not yet published to npm). New here? Start with the hands-on guide. See the design spec and implementation plan.

Intent

The problem

Turning a Jira story into test coverage today means:

• reading the ticket and working out what to test,
• writing scenarios by hand,
• discovering at run time that a step doesn't exist in BEATS,
• triggering runs in the BEATS UI,
• piecing together results across Jira, BEATS, and chat.

Every hop between tools costs time, and nothing records which scenarios cover which ticket.

What the workbench is for

A QA engineer works in one place, their AI coding assistant, and the workbench gives that assistant safe, grounded access to BEATS:

• Author test cases. Draft Gherkin scenarios using only steps BEATS can actually execute, validated before they are saved.
• Generate test cases from Jira. Go from a ticket's acceptance criteria to validated DRAFT scenarios linked to that ticket.
• Run and watch. Trigger BEATS test plans or ad-hoc runs and wait for the results.
• Report. Get run summaries, regressions between runs, and Jira-to-scenario-to-result coverage.

Who it's for

• QA engineers: daily authoring and triage.
• Leads: coverage and regression reports.
• Later, automated agents (see Vision).

Success looks like

• Installing it in Claude Code, opencode, or omp takes one copy-paste config block.
• On real tickets, jira-to-scenarios produces scenarios that pass BEATS validation without hand edits and land in BEATS as DRAFT, tagged jira:<KEY>.
• A QA engineer can trigger a run, wait for it, and get a readable report from inside the chat.

Design

Principles

1. BEATS is the source of truth. Scenarios, runs, and evidence live in BEATS. The workbench stores nothing of its own.
2. The workbench is deterministic. It never calls an LLM. Tools do things (validate, create, run, report), and prompts carry the know-how. The thinking is done by the client's model, the one the engineer is already talking to.
3. Grounded generation. Scenarios may use only steps from the BEATS step catalog, and every scenario is validated against BEATS before it is saved. Behaviour the catalog can't express is reported as "needs new step handler", never approximated.
4. Humans gate what goes live. The workbench creates scenarios only as DRAFT. A person promotes them in the BEATS UI.
5. Safe to retry. Creation is all-or-nothing per batch, and re-running on the same ticket skips scenarios that already exist.
6. Traceable. Every generated scenario is tagged jira:<KEY> and generated-by:qa-workbench, so Jira → scenario → run can always be traced.
7. Client-agnostic. Everything is plain MCP (tools, prompts, resources), so it works the same in Claude Code, opencode, and omp.

How it fits together

 ┌──────────────── QA engineer's client (Claude Code / opencode / omp) ────────────────┐
 │                                                                                     │
 │   LLM  ── reads prompts & resources, decides what to do, asks the user to confirm   │
 │    │                                                                                │
 │    ├──► Atlassian MCP ───────────────► Jira (tickets, acceptance criteria)          │
 │    │                                                                                │
 │    └──► QA Workbench MCP (stdio, local)                                             │
 │            ├─ tools      catalog · validate · create DRAFT · run · wait · report    │
 │            ├─ prompts    jira-to-scenarios · review-scenarios · triage-run          │
 │            └─ resources  step catalog · Gherkin style guide                         │
 └────────────────────┬────────────────────────────────────────────────────────────────┘
                      │ HTTPS + bearer token
                      ▼
               BEATS server ──► agents ──► product under test
        (catalog, scenarios, test plans, runs, events)

The workbench has no Jira code. The client loads the Atlassian MCP alongside it, and the prompts tell the model to use both.

Code structure

src/core/   plain TypeScript library: BEATS client, scenario / run / report logic (no MCP imports)
src/mcp/    thin MCP adapter: tool, prompt, and resource registrations over core/

Keeping core/ free of MCP is deliberate: a future CLI, webhook handler, or factory dispatcher can reuse the same logic unchanged.

What it exposes

| Kind | Name | Purpose | |---|---|---| | Prompt | jira-to-scenarios | Ticket → acceptance criteria → catalog-only scenarios → validate (up to 3 fix rounds) → confirm → create DRAFT | | Prompt | review-scenarios | Find coverage gaps, duplicates, weak assertions | | Prompt | triage-run | Group a run's failures by likely cause; draft bug reports | | Tools | beats_list_templates, beats_get_step_catalog | What BEATS can execute | | Tools | beats_list_scenarios, beats_get_scenario, beats_validate_scenarios, beats_create_scenarios | Scenario authoring | | Tools | beats_list_environments, beats_list_test_plans, beats_run_test_plan, beats_quick_run, beats_get_run, beats_wait_for_run | Running | | Tools | report_run, report_run_diff, report_jira_coverage | Reporting | | Resources | beats://catalog/{templateId}, qa://guides/gherkin-style | Context for the model |

A typical session

You: run jira-to-scenarios for PRD-4599

Assistant: reads PRD-4599 through the Atlassian MCP, loads the step catalog, checks for existing jira:PRD-4599 scenarios, drafts three scenarios, validates them (fixing one step that wasn't in the catalog), and shows you an acceptance-criteria → scenario table, plus one criterion that "needs new step handler".

You: looks good, create them and run them on local

Assistant: creates three DRAFT scenarios, quick-runs jira:PRD-4599 on the chosen environment, waits, and returns the run report, with any failing step and its error (for example expected status 200 but was 403).

Deliberately out of scope (for now)

The workbench does not:

• edit or delete scenarios,
• promote DRAFT scenarios to active,
• create or modify test plans,
• write to Jira.

These stay with people, the BEATS UI, or the Atlassian MCP.

Vision

The workbench is step one. Each later step reuses the same core, prompts, and safety rules.

| Stage | What | Status | |---|---|---| | A · Core + BEATS connector | Local stdio MCP server, typed BEATS client, catalog/scenario/run tools | Built (v0.1) | | B · Test-case authoring | Jira → validated DRAFT scenarios; review workflow | Built (v0.1) | | C · Reporting | Run reports, run diffs, Jira coverage | Built (v0.1) | | E · AI evals | Evals for the models we use and for AI features in the product. Likely the one place with narrow in-workbench LLM calls (LLM-as-judge), possibly a Python sidecar | Needs its own design | | F · Factory mode | Unattended operation: event-driven (a Jira transition or merged PR triggers generate → validate → run → report back), batch (cover a whole epic, sprint, or release), and an agent fleet (a queue of QA tasks worked by parallel headless agents, using the same MCP server and prompts) | Needs its own design | | D · External TCMS | Connector to a separate test-case management system | Parked: BEATS is the TCMS |

Why factory mode needs no rewrite. In workbench mode the model in your chat drives the prompts. In factory mode a dispatcher starts a headless agent (claude -p, opencode run, or an Agent SDK worker) with the same MCP servers and the same prompt, for example jira-to-scenarios PRD-4599. The QA logic doesn't change; only who presses the button does.

What factory mode will still need:

• headless Jira authentication (the Atlassian MCP uses OAuth today),
• a service credential for BEATS (people sign in through the browser and their sessions renew for up to 30 days, which suits interactive use, not unattended workers),
• concurrency limits,
• a per-ticket log of what was generated,
• a hosted HTTP transport.

Known risks

• Periodic sign-in. Each person uses their own BEATS session (no shared secret). It renews automatically, but by design it needs a fresh browser sign-in (one click on Allow) every 30 days.
• Atlassian MCP availability. Jira workflows need the Atlassian MCP configured and authorized in each user's client.
• Catalog coverage. Generated scenarios can only be as good as the step catalog. "Needs new step handler" output should feed AgentTemplate work.

Getting started

Follow the hands-on guide. In short: build the workbench, run node dist/mcp/main.js login and click Allow in your browser, then add the server to your AI client. No BEATS checkout, Docker or copied tokens are needed.

Configure

| Variable | Config file key | Default | | |---|---|---|---| | BEATS_TOKEN | beatsToken | — | Set for you by qa-workbench login (your own BEATS session; renews automatically for up to 30 days). Setting BEATS_TOKEN in the environment overrides the saved session, so leave it unset. | | BEATS_URL | beatsUrl | https://beats-server-production.up.railway.app | Shared BEATS API | | BEATS_UI_URL | beatsUiUrl | https://beats.bluecopa.com | For links in reports | | BEATS_DEFAULT_TEMPLATE | defaultTemplate | — | Step catalog to use when a request doesn't name one | | BEATS_DEFAULT_PLATFORM | defaultPlatform | — | Platform new scenarios belong to (required to create scenarios unless named per request) |

The config file is ~/.config/qa-workbench/config.json (override the path with QA_WORKBENCH_CONFIG). Environment variables take precedence over it.

Add to your client

Until the package is published, build it locally (pnpm install && pnpm build) and sign in once with node dist/mcp/main.js login. The client then only needs the command:

Claude Code

claude mcp add qa-workbench -- node /ABSOLUTE/PATH/qa-workbench/dist/mcp/main.js

opencode (opencode.json)

{
  "mcp": {
    "qa-workbench": { "type": "local", "command": ["node", "/ABSOLUTE/PATH/qa-workbench/dist/mcp/main.js"] }
  }
}

omp: add a stdio MCP server with command node and args ["/ABSOLUTE/PATH/qa-workbench/dist/mcp/main.js"].

Also configure the Atlassian MCP in the same client for the Jira workflows.

Develop

pnpm install
pnpm test           # unit + MCP tests, plus a stdio smoke test of the built server (no BEATS needed)
pnpm test:contract  # against a live BEATS (skipped if unreachable or unauthorized); see docs/guides/local-sandbox.md
pnpm build

Documents

• Hands-on guide: access you need, setup, and the five workflows with real outputs
• Trial transcripts: every call and output from the 2026-10-15 live trial
• Local sandbox and demo agent: for maintainers who need a disposable BEATS
• Design spec: decisions, MCP surface, verified BEATS API facts
• Implementation plan: 13 test-first tasks
• BEATS: ~/repos/beats (API, authoring)