@msalaam/xray-qe-toolkit

QE toolkit for Xray Cloud — manage test definitions, sync to Jira/Xray, import Playwright results.

Table of Contents

• Overview
• How It Works
• Prerequisites
• Installation
• Quick Start
• CLI Commands
• Configuration
• tests.json Reference
• Test Sets, Plans & Executions
• Playwright Integration
• xray-mapping.json
• CI/CD Integration
• Programmatic API
• Troubleshooting

Overview

@msalaam/xray-qe-toolkit (CLI: xqt) manages the boundary between your local test definitions (tests.json) and Xray Cloud. It does not generate Playwright tests — those are created by a separate Copilot skill after Xray issue keys are known.

What it does:

• tests.json — single source of truth for test metadata (synced to Xray)
• Idempotent push — creates new tests and Test Sets, updates existing ones, never duplicates
• Test Set management — groups tests by feature, persistent across sprints
• Test Plan management — create per-sprint plans, link Test Sets
• Test Executions — auto-created per CI run, tagged by environment, linked to plan
• Playwright result import — JSON reporter output → Xray with step-level results
• Xray folder sync — organise the Test Repository from folder fields in tests.json
• Schema validation — validate tests.json as a CI gate before pushing
• Visual editor — browser-based editor for reviewing/editing tests.json

For the full spec-driven QE process — how openapi.yaml + business-rules.yaml become tests.json, how Playwright tests are created, and how sprints are managed — see SPEC-DRIVEN-APPROACH.md (scaffolded into every project by xqt init).

How It Works

openapi.yaml + business-rules.yaml
      ↓ (AI agent generates tests.json)
tests.json — structured test definitions
      ↓
xqt validate   ← schema check
      ↓
xqt push       ← creates/updates Tests + Test Sets in Xray
      ↓
xray-mapping.json       ← auto-populated with Jira issue keys
      ↓
Playwright API tests    ← created by Copilot (reads tests.json + xray-mapping.json)
      ↓
Sprint starts  ← xqt plan → link Test Sets to Test Plan
      ↓
CI runs tests  ← xqt import → Test Execution in Xray

Key design decisions:

• Test Sets are persistent — tests live in Test Sets grouped by feature
• Test Plans are per-sprint — create one per sprint, link relevant Test Sets
• Test Executions are ephemeral — every CI run creates a fresh execution
• Environments are labels — IOP-DEV, IOP-QA, IOP-PROD tag each execution
• tests.json is the source of truth — never edit test metadata directly in JIRA
• xray-mapping.json is auto-generated — maps local testId → JIRA key; never edit manually
• Playwright tests come after push — created only once xray-mapping.json has real Jira keys

Prerequisites

• Node.js >= 18.0.0
• Xray Cloud API credentials (Client ID + Client Secret) — from Apps → Xray → API Keys
• JIRA Cloud API Token — from https://id.atlassian.com/manage-profile/security/api-tokens

Installation

# As a dev dependency (recommended)
npm install --save-dev @msalaam/xray-qe-toolkit

# Globally
npm install -g @msalaam/xray-qe-toolkit

Quick Start

# 1. Scaffold a new QA project
npx xqt init

# 2. Fill in credentials
cp .env.example .env
# Edit .env with XRAY_ID, XRAY_SECRET, JIRA_* values

# 3. Place your API spec and business rules in resources/
#    resources/api-specs/openapi.yaml
#    resources/business-rules.yaml

# 4. Generate tests.json from openapi.yaml + business-rules.yaml
#    (done by AI agent / Copilot skill — reads both files, produces tests.json)

# 5. Validate and push to Xray
npx xqt validate
npx xqt push
#    → Tests + Test Sets created in Jira
#    → xray-mapping.json populated with issue keys

# 6. Create Playwright tests (done by Copilot skill — reads tests.json + xray-mapping.json)

# 7. When entering a sprint, create a Test Plan and link Test Sets
npx xqt plan --summary "Sprint 12 — My Service Regression"

# 8. Run tests and import results
npx playwright test
npx xqt import --file test-results/results.json --env IOP-QA

CLI Commands

All commands accept --verbose for debug output and --env <path> for a custom .env file path.

xqt init

Scaffold a QA project in the current directory. Idempotent — never overwrites existing files.

Creates: resources/ (API spec + business rules), tests/ (models/services/resources/specs), playwright.config.ts, tests.json, xray-mapping.json, .xrayrc, .env.example, XQT-GUIDE.md (xqt command reference), SPEC-DRIVEN-APPROACH.md (QE process guide).

npx xqt init

xqt edit

Launch a browser-based visual editor for tests.json. Use this to review, add, or update test definitions.

npx xqt edit
npx xqt edit --port 3000

| Flag | Description | Default | |---|---|---| | --port <n> | Port for local editor server | Random |

xqt plan

Create a new Xray Test Plan in JIRA. Automatically saves the key to .xrayrc and tests.json. create-plan is accepted as a legacy alias.

npx xqt plan --summary "My Service v2 Regression"
npx xqt plan --summary "Sprint 12 Smoke Tests" --version "2024.12"

| Flag | Description | |---|---| | --summary <text> | Test Plan title (required) | | --version <ver> | Fix version to associate | | --label <labels> | Comma-separated JIRA labels |

xqt push

Create or update tests in Xray Cloud, then sync Test Plan membership and the Xray folder structure. Optionally create a Test Execution linked to the plan in a single command. push-tests is accepted as a legacy alias.

npx xqt push
npx xqt push --plan APIEE-1234
npx xqt push --bulk                                             # force bulk import regardless of count
npx xqt push --plan APIEE-1234 --create-execution --execution-env IOP-QA

| Flag | Description | |---|---| | --plan <key> | Test Plan key (overrides .xrayrc testPlanKey) | | --bulk | Force bulk Xray REST import regardless of new-test count (useful when count is below bulkImportThreshold) | | --create-execution | Create a Test Execution linked to the Test Plan after pushing | | --execution-env <label> | Environment label for the created execution (e.g. IOP-QA) | | --execution-summary <text> | Custom summary for the created execution |

Behaviour:

• Tests in xray-mapping.json → updated in Xray
• Tests not in mapping → created in Xray, mapping entry added
• Tests with "skip": true → excluded entirely
• Test Sets auto-created from the testSet field and tests added to them (see below)
• Xray folder structure synced from folder fields
• With --create-execution, a new Test Execution is created and linked to the Test Plan

xqt pull

Pull test definitions from Xray Cloud and merge them into tests.json. Useful for onboarding to an existing Xray project. pull-tests is accepted as a legacy alias.

npx xqt pull --plan APIEE-1234
npx xqt pull --project APIEE --limit 200

| Flag | Description | |---|---| | --plan <key> | Fetch tests from a specific Test Plan | | --project <key> | Fetch all tests in a JIRA project | | --limit <n> | Max tests to fetch (default: 100) |

xqt exec

Pre-create a Test Execution with a specific set of tests before running them. Use --quiet to capture the key in CI for use with import --exec. create-execution is accepted as a legacy alias.

# Standard output
npx xqt exec --env IOP-QA

# CI mode — prints ONLY the key
EXEC_KEY=$(npx xqt exec --env IOP-QA --quiet)

# Specific test selection
npx xqt exec --env IOP-QA --plan APIEE-1234 --tests TC-001,TC-002

| Flag | Description | |---|---| | --env <label> | Environment label (e.g. IOP-QA) | | --plan <key> | Test Plan to link to (overrides .xrayrc) | | --tests <ids> | Comma-separated testIds or JIRA keys (default: all mapped) | | --summary <text> | Custom execution title | | --fix-version <ver> | JIRA Fix Version to stamp on the execution (e.g. 2.5.0) | | --quiet | Print only the execution key |

xqt import creates an execution automatically — this command is only needed when pre-selecting a specific subset of tests.

xqt import

Import test execution results into Xray Cloud. import-results is accepted as a legacy alias.

• Without --exec — creates a new Test Execution linked to the Test Plan
• With --exec — imports results INTO a pre-created execution

Supported: Playwright JSON (.json) and JUnit XML (.xml).

# Auto-create execution (standard CI)
npx xqt import --file test-results/results.json --env IOP-QA

# Import into a pre-created execution
npx xqt import --file test-results/results.json --exec APIEE-9876

# JUnit XML
npx xqt import --file test-results/results.xml --env IOP-PROD

# Full options — with Fix Version and git SHA
npx xqt import \
  --file test-results/results.json \
  --env IOP-QA \
  --plan APIEE-1234 \
  --fix-version "2.5.0" \
  --revision "a1b2c3d4"

| Flag | Description | |---|---| | --file <path> | Path to results file (required) | | --env <label> | Environment label (default: defaultEnvironment from .xrayrc) | | --plan <key> | Test Plan key (overrides .xrayrc; used when no --exec) | | --exec <key> | Import INTO an existing execution (from xqt exec) | | --fix-version <ver> | JIRA Fix Version name to stamp on the execution (e.g. 2.5.0) | | --version <ver> | Xray version label (free-form, separate from Fix Version) | | --revision <rev> | Build number or git SHA | | --summary <text> | Custom execution summary |

xqt sync

Sync the Xray Test Repository folder structure from folder fields in tests.json without touching test content. sync-folders is accepted as a legacy alias.

npx xqt sync

xqt status

Show local and remote project status: tests.json counts, mapping coverage, Test Plan info.

npx xqt status
npx xqt status --plan APIEE-1234

xqt validate

Validate tests.json against its schema. Exits with code 1 on errors — use as a CI gate before push-tests.

npx xqt validate
npx xqt validate --file path/to/tests.json

| Flag | Description | |---|---| | --file <path> | Override path to tests.json |

xqt pipeline

Generate an Azure Pipelines YAML template for the current project. gen-pipeline is accepted as a legacy alias.

npx xqt pipeline
npx xqt pipeline --output .azure/ci.yml

Configuration

Environment Variables (.env)

Copy .env.example to .env and fill in credentials. Never commit .env.

# ── Xray Cloud ─────────────────────────────────────────────────
XRAY_ID=your_xray_cloud_client_id
XRAY_SECRET=your_xray_cloud_client_secret

# ── JIRA ───────────────────────────────────────────────────────
JIRA_PROJECT_KEY=APIEE
JIRA_URL=https://your-org.atlassian.net
JIRA_API_TOKEN=your_jira_api_token
[email protected]

# ── Optional ───────────────────────────────────────────────────
# XRAY_REGION=us                    # "us" (default) or "eu"
# XQT_TEST_PLAN_KEY=APIEE-1234      # default Test Plan
# XQT_DEFAULT_ENV=IOP-DEV           # default environment label

| Variable | Required | Description | |---|---|---| | XRAY_ID | Yes | Xray Cloud Client ID | | XRAY_SECRET | Yes | Xray Cloud Client Secret | | JIRA_PROJECT_KEY | Yes | JIRA project key | | JIRA_URL | Yes | JIRA instance URL | | JIRA_API_TOKEN | Yes | JIRA API token | | JIRA_EMAIL | Yes | JIRA user email | | XRAY_REGION | No | us (default) or eu | | XQT_TEST_PLAN_KEY | No | Default Test Plan key | | XQT_DEFAULT_ENV | No | Default environment label |

Project Config (.xrayrc)

.xrayrc (JSON) at the project root. Created by xqt init, updated by xqt create-plan.

{
  "testsPath":            "tests.json",
  "mappingPath":          "xray-mapping.json",
  "testPlanKey":          "APIEE-1234",
  "defaultEnvironment":   "IOP-DEV",
  "environments":         ["IOP-DEV", "IOP-QA", "IOP-PROD"],
  "folderRoot":           "/MyService",
  "xrayRegion":           "us",
  "bulkImportThreshold":  50,
  "statusMapping": {
    "interrupted": "ABORTED",
    "skipped":     "TODO"
  }
}

| Field | Description | |---|---| | testPlanKey | Default Test Plan key for push and import | | defaultEnvironment | Fallback environment when --env is not provided | | environments | Allowed environment labels | | folderRoot | Base folder path in Xray Test Repository | | xrayRegion | us (default), eu, or au | | bulkImportThreshold | Min new-test count to use the Xray bulk REST API (default: 50) — faster for large suites | | statusMapping | Override default Playwright → Xray status mapping (see below) |

Playwright status mapping

By default the converter maps:

| Playwright status | Xray status | |---|---| | passed | PASSED | | failed | FAILED | | timedOut | FAILED | | interrupted | ABORTED | | skipped | TODO |

Override any value in .xrayrc:

{
  "statusMapping": {
    "skipped": "TODO",
    "interrupted": "FAILED"
  }
}

tests.json Reference

tests.json is the canonical list of test cases. It is generated from openapi.yaml + business-rules.yaml by an AI agent, then consumed by xqt push-tests to sync with Xray Cloud.

{
  "testPlan": {
    "key": "APIEE-1234",
    "summary": "Sprint 12 — My Service API Regression"
  },
  "tests": [
    {
      "test_id": "TC-MYSVC-BR-001",
      "type": "api",
      "skip": false,
      "tags": ["smoke", "regression"],
      "folder": "/MyService/HealthCheck/Validation",
      "testSet": "Health Check",
      "preconditions": ["APIEE-50"],
      "requirementKeys": [],
      "xray": {
        "summary": "Service returns 200 OK when healthy",
        "description": "Given Service is running, when GET /health is called, then Response is 200",
        "testType": "Generic",
        "definition": "Automated Playwright API test: GET /health — Service returns 200 OK when healthy",
        "priority": "High",
        "labels": ["smoke", "regression"]
      }
    }
  ]
}

Field Reference

| Field | Type | Required | Description | |---|---|---|---| | test_id | string | Yes | Stable local ID — maps to the Xray key in xray-mapping.json | | type | string | — | api / ui / e2e (informational) | | skip | boolean | — | true to exclude from push-tests | | tags | string[] | — | QE tags for categorisation and filtering. Allowed: regression, smoke, edge, critical, integration, e2e, security, performance, contract, functional, negative, positive, boundary, acceptance, sanity, data-driven, exploratory, accessibility | | folder | string | — | Xray repository folder path — must start with / | | testSet | string or string[] | — | Test Set name(s) in Jira — a single string or an array to assign the test to multiple Test Sets | | preconditions | string[] | — | JIRA keys of Xray Precondition issues to link to this test (e.g. ["APIEE-50"]) | | requirementKeys | string[] | — | JIRA keys this test covers (creates traceability links) | | xray.summary | string | Yes | Test case title (JIRA issue summary) | | xray.description | string | — | Detailed description | | xray.testType | string | — | Generic (default for automated) / Manual / Cucumber | | xray.definition | string | — | Generic test definition field in Xray | | xray.priority | string | — | Highest / High / Medium / Low / Lowest | | xray.labels | string[] | — | JIRA labels on the test issue |

Test Sets, Plans & Executions

Test Sets (persistent groupings)

Tests live in Test Sets in Jira. Each Test Set groups related tests by feature or area — they persist across every sprint.

Set the testSet field on each test in tests.json — use a string or an array of strings to assign the test to multiple sets:

{ "test_id": "TC-001", "testSet": "Client Lookup", ... }
{ "test_id": "TC-002", "testSet": "Client Lookup", ... }
{ "test_id": "TC-003", "testSet": ["Health Check", "Smoke"], ... }

When you run xqt push:

1. Tests are created/updated in Xray
2. For each unique testSet value, the Test Set JIRA issue is created automatically if it doesn't exist yet (safe to run on a fresh clone — existing sets are found by name, not duplicated)
3. Tests are added to their Test Set (idempotent — Xray ignores tests already in the set)
4. Test Set → JIRA key mappings (e.g. "Client Lookup" → APIEE-99) are stored in xray-mapping.json under _testSets

// xray-mapping.json (auto-managed)
{
  "_testSets": {
    "Client Lookup": { "key": "APIEE-99", "id": "123456" },
    "Health Check":  { "key": "APIEE-100", "id": "123457" }
  },
  "TC-001": { "key": "APIEE-200", "id": "234001" },
  ...
}

Test Sets are not deleted when tests are removed from tests.json. They persist in Jira as a record of all tests that have ever existed for that feature.

Test Plans (per-sprint)

A Test Plan scopes testing work for a specific sprint or release.

• Create with xqt plan --summary "Sprint 12 — My Service"
• Link relevant Test Sets to the Test Plan in Jira (manual step in Jira UI)
• Key stored in .xrayrc (testPlanKey)
• xqt import links executions to the active plan

xqt push performs a bi-directional sync with the Test Plan: new tests are added and tests removed from tests.json are removed from the plan. Use --plan <key> to override the key.

Test Executions (ephemeral per run)

A Test Execution represents a single CI run.

• Auto-created by xqt import — one new execution per run
• Pre-created by xqt exec for controlled test selection
• Tagged with environment labels (IOP-DEV, IOP-QA, IOP-PROD)
• Linked to the Test Plan automatically

Summary

| Concept | Lifecycle | Purpose | |---------|-----------|---------| | Test Set | Permanent | Groups tests by feature — where tests live | | Test Plan | Per-sprint | Scopes testing for a sprint — links Test Sets | | Test Execution | Per-CI-run | Records pass/fail for one run |

Three execution modes

Mode A — Auto (standard CI):

npx xqt import --file results.json --env IOP-QA

Mode B — Pre-create (controlled test selection):

EXEC_KEY=$(npx xqt exec --env IOP-QA --plan APIEE-1234 --quiet)
npx playwright test
npx xqt import --file results.json --exec $EXEC_KEY

Mode C — Push & execute (single command):

npx xqt push --plan APIEE-1234 --create-execution --execution-env IOP-QA
npx playwright test
npx xqt import --file results.json --env IOP-QA

Playwright Integration

Annotate tests with Xray keys

import { test, expect } from '@playwright/test';

test('GET /users returns 200', async ({ request }) => {
  test.info().annotations.push({ type: 'xray', description: 'APIEE-1234' });

  const response = await request.get('/users');
  expect(response.status()).toBe(200);
});

Or include the key in the test title:

test('[APIEE-1234] GET /users returns 200', async ({ request }) => { ... });

playwright.config.ts reporters

reporter: [
  ['json',  { outputFile: 'test-results/results.json' }],  // ← xqt import-results
  ['junit', { outputFile: 'test-results/results.xml'  }],
  ['html',  { open: 'never' }],
],

Test steps → Xray step results

Steps created with test.step() are mapped to Xray step results automatically. Screenshots captured on a failing step are attached directly to that step's evidence in Xray — giving per-step failure screenshots in the Xray UI. Traces and other attachments are attached at the test level.

test('Create and retrieve user', async ({ request }) => {
  test.info().annotations.push({ type: 'xray', description: 'APIEE-2001' });

  await test.step('POST /users — expect 201', async () => {
    const r = await request.post('/users', { data: { name: 'Alice' } });
    expect(r.status()).toBe(201);
  });

  await test.step('GET /users/:id — expect 200', async () => {
    const r = await request.get('/users/1');
    expect(r.status()).toBe(200);
  });
});

Folder Structure in Xray

Tests are organised in the Xray Test Repository using the folder field in tests.json:

{
  "test_id": "TC-MYSVC-BR-001",
  "folder": "/MyService/HealthCheck/Validation"
}

• Paths must start with /
• folderRoot in .xrayrc sets the base path
• Folders are created automatically by xqt push and xqt sync

CI/CD Integration

Azure DevOps pipeline

Generate a template: npx xqt pipeline

- script: npx xqt validate
  displayName: Validate tests.json

- script: npx playwright test
  displayName: Run tests
  continueOnError: true

- script: |
    npx xqt import \
      --file test-results/results.json \
      --env $(XQT_ENV) \
      --fix-version $(BUILD_BUILDNUMBER) \
      --revision $(Build.SourceVersion)
  displayName: Import results to Xray
  env:
    XRAY_ID:          $(XRAY_ID)
    XRAY_SECRET:      $(XRAY_SECRET)
    JIRA_PROJECT_KEY: $(JIRA_PROJECT_KEY)
    JIRA_URL:         $(JIRA_URL)
    JIRA_API_TOKEN:   $(JIRA_API_TOKEN)
    JIRA_EMAIL:       $(JIRA_EMAIL)

Required pipeline variables

| Variable | Description | |---|---| | XRAY_ID | Xray Cloud Client ID | | XRAY_SECRET | Xray Cloud Client Secret | | JIRA_PROJECT_KEY | JIRA project key | | JIRA_URL | JIRA instance URL | | JIRA_API_TOKEN | JIRA API token | | JIRA_EMAIL | JIRA user email | | XQT_ENV | Environment label (IOP-DEV / IOP-QA / IOP-PROD) | | XQT_BULK_THRESHOLD | Override bulk-import threshold (default: 50) |

Pre-create execution in pipeline (Mode B)

- script: |
    EXEC_KEY=$(npx xqt exec --env $(XQT_ENV) --quiet)
    echo "##vso[task.setvariable variable=EXEC_KEY]$EXEC_KEY"
  displayName: Pre-create Test Execution

- script: npx playwright test
  continueOnError: true

- script: |
    npx xqt import --file test-results/results.json --exec $(EXEC_KEY)
  displayName: Import results into pre-created execution

Programmatic API

Every function is exported for custom scripts:

import {
  loadConfig,
  authenticate,
  createTestExecution,
  importResultsMultipart,
  getTestPlan,
  syncTestPlan,
} from '@msalaam/xray-qe-toolkit';

const cfg = loadConfig();
const token = await authenticate(cfg);

const { key } = await createTestExecution(cfg, token, {
  summary: 'My custom execution',
  environments: ['IOP-QA'],
  testPlanKey: 'APIEE-1234',
});

Key exports: authenticate, createIssue, updateIssue, getIssue, getTest, getTests, getTestPlan, createTestPlan, addTestsToTestPlan, removeTestsFromTestPlan, addTestsToTestExecution, createTestExecution, importResultsMultipart, importTestsBulk, waitForImportJob, searchIssues, getTestSets, addPreconditionsToTest, syncTestPlan, syncFolders, buildAndPush, loadMapping, saveMapping, loadConfig, validateConfig.

xray-mapping.json

Auto-managed by xqt push. Maps local test_id → JIRA issue key and numeric Xray ID.

{
  "TC-MYSVC-BR-001": {
    "key": "APIEE-1234",
    "id":  "8765432"
  },
  "TC-MYSVC-BR-002": {
    "key": "APIEE-1235",
    "id":  "8765433"
  }
}

• Commit this file — the whole team must share the same mapping
• Used by Copilot skill to generate Playwright tests with real Jira keys
• Do not edit manually
• Re-push to regenerate a missing entry

Troubleshooting

Missing credentials

Error: Missing required config: xrayId, xraySecret

Ensure .env exists and is populated.

Test not found in mapping after push

npx xqt status
npx xqt push --verbose

Import results — 0 tests matched Ensure Playwright tests have xray annotations or the key in the title:

test.info().annotations.push({ type: 'xray', description: 'APIEE-1234' });

Validate fails in CI Fix schema errors before pushing. Common causes: missing testId, invalid priority, malformed folder path (must start with /).

xqt push — Test Repository folder errors

GraphQL errors: User doesn't have permissions to view/edit test repository for project

The JIRA user in JIRA_EMAIL needs Test Repository write access in Xray. Fix in Jira:

Project Settings → Xray → Test Repository → add your role or user to the allowed list. Folder sync errors are non-fatal — tests are still created/updated correctly.

Test Set creation returns HTTP 400 Ensure the Xray Test Set issue type exists in your JIRA project and that the user has permission to create it.

Project Settings → Issue Types — "Test Set" must be present.

updateTestType "not found" on new tests Xray Cloud is eventually consistent — a newly created issue may not be immediately visible to the GraphQL API. xqt push automatically retries with backoff. If failures persist, rerun xqt push (existing tests update, not duplicate).

All previous multi-word commands (push-tests, pull-tests, create-plan, create-execution, import-results, sync-folders, gen-pipeline, mcp-server) remain valid as aliases.

EU region Set XRAY_REGION=eu in .env or "xrayRegion": "eu" in .xrayrc.

"disallowed to impersonate" error JIRA_EMAIL must match the email of the user who created the Xray API Key.

License

MIT — see LICENSE