Playwright TestRail Reporter with Multi-Project Support

A Playwright reporter that integrates with TestRail via API and supports multiple projects and test suites.

This reporter automatically creates test runs and updates test results in TestRail based on your Playwright test execution.

Key Features

• 🔄 Multi-project and multi-suite support
• 📦 Support for both projects with test suites and projects with a single repository structure
• 🏷️ Test case mapping via tags
• 🪜 Tagged test steps support
• 📊 Automatic test run creation and updating
• 🔍 Comprehensive error reporting
• 📝 Automatic run closing (optional)
• 🖼️ Attachment support (optional)

Setup

Installation

npm install --save-dev playwright-reporter-testrail

Playwright Setup

1. Get your TestRail credentials:

   • Domain (e.g., https://yourdomain.testrail.io)
   • Email
   • Password or API Key

2. Configure the reporter in your playwright.config.ts:

import { PlaywrightTestConfig } from '@playwright/test';

const config: PlaywrightTestConfig = {
  reporter: [
    ['playwright-reporter-testrail', {
      domain: 'https://yourdomain.testrail.io',
      username: 'your-email',
      password: 'your-password',
      includeAllCases: false,
      includeAttachments: false,
      closeRuns: false,
      createEmptyRuns: false,
      apiChunkSize: 10,
      runNameTemplate: 'Playwright Run #{date}'
    }]
  ]
};

export default config;

Reporter options:

| Name | Type | Default | Description | |------|------|---------|-------------| | domain | string | Required | TestRail domain URL | | username | string | Required | TestRail email | | password | string | Required | TestRail password or API key | | includeAllCases | boolean | false | Whether to include all cases of the TestRail suite to the test run | | includeAttachments | boolean | false | Whether to upload attachments for the test run.Note: May result in longer execution time as each attachment requires a separate API call | | closeRuns | boolean | false | Whether to close test runs in the end.Note: Ensure user has permissions to close runs in TestRail | | createEmptyRuns | boolean | false | Whether to create TestRail runs even when all tests in a suite are skipped or have no results | | apiChunkSize | number | 10 | The number of requests to send in parallel to TestRail API | | runNameTemplate | string | Playwright Run #{date} | Template for the test run name.Supports variables:• #{date}: Current date/time in YYYY/MM/DD HH:MM:SS UTC format• #{timestamp}: Current timestamp in milliseconds• #{suite}: Test suite name (increases execution time as it requires additional API call per each test run) |

Usage

Tagging Tests

Tag your tests with TestRail case IDs in one of the following formats:

• @<project_id>-<suite_id>-<case_id> (for projects with test suites)
• @<project_id>-<case_id> (for projects with a single repository structure)

Where:

• project_id: TestRail project ID
• suite_id: TestRail test suite ID
• case_id: TestRail test case ID (might include prefix)

Example:

import { test } from '@playwright/test';

test('simple test matching one case', { tag: ['@101-204-R3453'] }, async ({ page }) => {
  // Your test code
});

// Multiple test cases
test('complex feature with multiple cases from multiple projects', { tag: ['@101-204-3453', '@203-305-4567'] }, async ({ page }) => {
  // Your test code
});

test('simple test matching one case from single repository structure', { tag: '@110-R999' }, async ({ page }) => {
  // Your test code
});

Tagging Test Steps

Tagging test steps is optional, but is recommended for long E2E tests.

Tag your test step titles with TestRail case ID (might include prefix) with @. Test step might contain multiple case IDs.

Example

import { test } from '@playwright/test';

test('simple test matching three cases', { tag: ['@101-204-555', '@101-204-556', '@101-204-557'] }, async ({ page }) => {
  await test.step('Step 1 @555', async () => {
    // Your step code
  });

  await test.step('Step 2 @556 @R557', async () => {
    // Your step code
  });
});

Rules Regarding Tagged Steps

The main benefit of using tagged steps is for longer E2E tests to correctly mark passed steps in TestRail if the test fails on a later stage in Playwright.

• If a test contains some valid tags, tagged steps should use the same case IDs. Otherwise, the reporter will log an error and ignore those steps
• If a step does not contain a valid TestRail case ID, it will be ignored by reporter
• If a step contains a valid TestRail case ID and it passes, the corresponding TestRail case will be marked as passed regardless of the result of the whole test execution
• If a step contains a valid TestRail case ID and it fails, the corresponding TestRail case will get the same status and comment as if steps was not tagged

Avoid situation where all test tags are matched to test steps, but some steps are not tagged. If such situation occurs and the test fails, the reporter might miss the failed step and mark the test as passed in TestRail. Consider either adding tags to all steps or make sure that some tags are not matched to test steps.

❌ Incorrect usage:

import { test } from '@playwright/test';

test('simple test matching one case', { tag: ['@101-204-555'] }, async ({ page }) => {
  await test.step('Step 1 @555', async () => {
    // This step will be marked as passed in TestRail
  });

  await test.step('Step 2', async () => {
    throw new Error('TestRail will not know about this failure')
  });
});

Running Tests

Run your Playwright tests as usual:

npx playwright test

Your test results will be automatically sent to TestRail.

Logging

If you want to have more detailed logs, consider setting TESTRAIL_REPORTER_DEBUG_MODE environment variable to true.

Additional Information

Graceful Failure Logic

The reporter logs errors to the console if it is not configured correctly or if any API calls fail. It does not throw errors and allows test execution to continue.

Creating and Updating Test Runs

TestRail test runs are created after all tests finish their execution. If Playwright run results in a timeout or interrupted statuses, test runs are not created.

Empty Runs in TestRail

By default, the reporter skips creating TestRail runs for suites where all tests are skipped or have no results. If you need those runs to still appear in TestRail (for example, to keep a consistent list of runs per suite), set createEmptyRuns to true. In that case, the reporter will create runs even when all tests in a suite are skipped.

API Retry Logic

The reporter will retry the API calls up to 3 times if the API call fails with a 5xx status code.

Multiple Playwright Tests Matching One TestRail Case

If you have multiple Playwright tests that match the same TestRail case, the reporter will abide by the following rules:

1. If all Playwright tests finish with the same status, the TestRail case will be marked with that status, and the comment (and error in case of fail) will be generated from the first test that finished.
2. If any Playwright tests finish with different statuses, the reporter will prioritize the following statuses in order: passed, failed, blocked, untested.

Known Issues

When several Playwright tests match the same TestRail case, each test will upload its attachments to the TestRail case. This may result in duplicate attachments.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details