testcafe-reporter-xray-cloud

A TestCafe reporter plugin that automatically reports test execution results to Xray Cloud (Jira Xray Test Management). This reporter supports data-driven testing with iterations, multiple browsers, screenshots, videos, and defect tracking.

Features

• Automatic test execution reporting to Xray Cloud
• Data-driven testing support with test iterations and parameters
• Multi-browser execution reporting
• Screenshot and video attachments as evidence
• Defect linking - automatically associates tests with linked defects
• Test steps tracking with detailed results and automatic Actual Result population
• Partial iteration updates - supports running subset of iterations
• Comprehensive error reporting with formatted stack traces in step-level Actual Results

Installation

npm install testcafe-reporter-xray-cloud

Prerequisites

1. Xray Cloud account with API access
2. API credentials (Client ID and Client Secret) from Xray Cloud
   • Go to Xray Cloud > API Keys to generate credentials
3. Test Execution Key in Jira Xray where results will be reported

Configuration

Required Environment Variables

XRAY_CLIENT_ID=<your-client-id>
XRAY_CLIENT_SECRET=<your-client-secret>
XRAY_EXECUTION_KEY=<execution-key>

Or via Command Line Arguments

testcafe chrome tests/ \
  -r xray-cloud \
  --xray-client-id="<your-client-id>" \
  --xray-client-secret="<your-client-secret>" \
  --xray-execution-key="<execution-key>"

Optional Configuration

Test Plan Key

Link results to a specific test plan:

--xray-plan-key="<test-plan-key>"
# or
export XRAY_TEST_PLAN_KEY=<test-plan-key>

Version Number

Add version metadata to test execution:

--jiraVersionNumber="1.2.3"
# or
export JIRA_VERSION_NUMBER=1.2.3

Report Folder

Customize the folder where JSON reports are saved:

--xray-json-reports-folder="./custom-reports"
# Default: ./artifacts/reports/xray-json-reports

Usage

Basic Usage

testcafe chrome tests/ -r xray-cloud

With Multiple Browsers

testcafe chrome,firefox,edge tests/ -r xray-cloud

Test Naming Convention

Tests must include the Jira test key in the name to be reported to Xray:

test('NJ-12345 Login functionality should work correctly', async t => {
  // Test implementation
});

Data-Driven Tests with Iterations

For tests with multiple iterations (data-driven testing), use the following naming pattern:

test('NJ-12345 Iteration: 1 |{"browser":"Chrome","os":"Windows"}| Login test', async t => {
  // Test implementation
});

test('NJ-12345 Iteration: 2 |{"browser":"Firefox","os":"Mac"}| Login test', async t => {
  // Test implementation
});

Iteration naming format:

NJ-XXXXX Iteration: N |{parameters}| Test description

Where:

• N is the iteration number (1-indexed)
• {parameters} is a JSON object with iteration parameters
• Parameters will be sent to Xray as iteration metadata

Test Steps

The reporter automatically tracks test steps and their status:

test('NJ-12345 Multi-step test', async t => {
  await t.ctx.step('Login to application', async () => {
    // Step implementation
  });

  await t.ctx.step('Navigate to dashboard', async () => {
    // Step implementation
  });

  await t.ctx.step('Verify user data', async () => {
    // Step implementation
  });
});

Step Results with Errors

When a test step fails, the reporter automatically adds the error details to the Actual Result field of the failed step in Xray Cloud:

For tests with iterations:

• Error is added to the failed step within the specific iteration that failed
• If no step is explicitly marked as failed, error is added to the last step of the failed iteration

For tests without iterations:

• Error is added to the first step with FAILED status
• If no step is explicitly marked as failed, error is added to the last step

The error message includes:

• Full stack trace
• File path and line number
• Formatted code snippet showing where the failure occurred

This makes it easy to identify exactly which step failed and why, directly in Xray Cloud's UI.

Evidence Attachments

Screenshots and videos captured during test execution are automatically attached as evidence:

test('NJ-12345 Test with screenshots', async t => {
  await t.takeScreenshot('screenshot-1.png');
  // Screenshot will be attached to the test report
});

TestCafe's automatic video recording is also supported and will be attached to failed tests.

Advanced Features

Partial Iteration Runs

The reporter intelligently handles partial iteration runs. If you have a test with 5 iterations in Xray but only run 3:

• The reporter uses GraphQL API to update only the executed iterations
• Non-executed iterations retain their previous status
• Prevents overwriting all iterations with incomplete data

Defect Linking

Tests linked to defects in Xray will automatically have those defects included in the report:

1. Tests are linked to defects in Xray Cloud
2. Reporter fetches existing defect links
3. New results maintain defect associations
4. Defects are resolved from numeric IDs to Jira keys

Attached Tests

Support for reporting "attached tests" - additional test validations that are part of a main test:

// The reporter can track and report secondary test validations
// within a main test execution

Output

JSON Reports

The reporter generates JSON files in the specified folder (default: ./artifacts/reports/xray-json-reports):

Chrome_143_0_0_0_Sequoia_15-2025-12-10T02-41-09-155Z.json

Console Output

During execution, the reporter displays:

• Test execution progress
• Pass/fail status
• Error details with formatted stack traces
• Summary statistics

Final Summary

StartTime : 2025-01-26T09:00:00.000Z
EndTime   : 2025-01-26T09:15:30.000Z
Duration  : 15 minutes 30 seconds
UserAgents: Chrome 143.0.0.0 / Sequoia 15
TestCount : 25
Passed    : 23
Failed    : 2
Skipped   : 0
Warnings  : 0
XrayExeKey: NJ-12345

API Integration

The reporter integrates with multiple Xray Cloud APIs:

REST API v2

• Authentication
• Test execution import
• Dataset fetching

GraphQL API

• Test structure retrieval
• Defect mapping
• Iteration status updates
• Test run ID resolution

Error Handling

The reporter provides comprehensive error handling:

• Authentication errors: Clear messages about missing credentials
• Network errors: Detailed error logging for API failures
• Dataset mismatches: Validation of local vs. cloud dataset differences
• Iteration conflicts: Intelligent handling of partial iteration runs

Automatic Actual Result Population

When a test fails, error details are automatically added to the Actual Result field in Xray Cloud:

Example Actual Result in Xray:
┌─────────────────────────────────────────────────┐
│ Step: Verify location was changed               │
│ Status: FAILED                                  │
│                                                 │
│ Actual Result:                                  │
│ {code:title=Detailed Results|...}              │
│ AssertionError: expected false to be truthy    │
│                                                 │
│ Browser: Chrome 143.0.0.0 / Sequoia 15        │
│                                                 │
│ at DeviceSettingsPage.verifyPolicyNameExist    │
│ (/path/to/page.ts:509:56)                      │
│ {code}                                         │
└─────────────────────────────────────────────────┘

This makes debugging failed tests much easier as all error information is visible directly in each step's result.

Troubleshooting

Tests not appearing in Xray

• Verify test names include valid Jira test keys (e.g., NJ-12345)
• Check that XRAY_EXECUTION_KEY points to a valid execution
• Ensure API credentials have correct permissions

Dataset mismatch errors

Datasets for test NJ-12345 have different number of iterations
local: (3), cloud: (5), test will not be reported to Xray

Solution: Ensure local test data matches the dataset defined in Xray Cloud

Authentication failures

There was an issue requesting an XRay authentication token.
Please check if you have configuration for XRAY_CLIENT_ID and XRAY_CLIENT_SECRET.

Solution: Verify credentials are correctly set in environment variables or command line arguments

Development

Build

npm run build

Format

npm run format

Lint

npm run lint

Version History

Current version: 0.0.27

Recent enhancements:

• Automatic Actual Result population: Error details are now automatically added to the "Actual Result" field of failed steps in Xray Cloud
• Fixed test duplication issue: Tests with multiple iterations no longer appear duplicated in execution reports
• Improved iteration tracking: Enhanced status updates and tracking for data-driven tests
• Enhanced partial iteration runs: Better handling when running subset of iterations

License

MIT

Author

Beatriz Zamorano ([email protected])

Contributing

Issues and pull requests are welcome. Please ensure all tests pass and code is properly formatted before submitting.

Support

For issues or questions:

1. Check the troubleshooting section above
2. Review Xray Cloud documentation: https://docs.getxray.app/
3. Open an issue in the repository