Playwright Azure Reporter

A Playwright reporter that publishes test results to Azure DevOps Test Plans.

Table of Contents

• Quick Start
• Defining Test Case IDs
  • In Test Title
  • Using Tags
  • Using Annotations
• Configuration
  • Required Options
  • General Options
  • Authentication
  • Publishing Modes
  • Retry Results
  • Attachments & Logs
  • Test Point Mapping
  • Test Case ID Matching
  • Test Case Summary
  • Auto-Mark as Automated
• CI/CD Integration
• Releases

Quick Start

Install the package:

npm install @alex_neo/playwright-azure-reporter

Add the reporter to your playwright.config.ts:

import { defineConfig } from '@playwright/test';
import type { AzureReporterOptions } from '@alex_neo/playwright-azure-reporter';

export default defineConfig({
  reporter: [
    ['list'],
    [
      '@alex_neo/playwright-azure-reporter',
      {
        orgUrl: 'https://dev.azure.com/your-organization-name',
        token: process.env.AZURE_TOKEN,
        planId: 44,
        projectName: 'SampleProject',
        publishTestResultsMode: 'testRun',
      } as AzureReporterOptions,
    ],
  ],
});

Each test must reference an existing Azure DevOps test case ID. See Defining Test Case IDs for all supported formats.

Defining Test Case IDs

You must link each test to an existing test case in Azure DevOps before running tests. There are three ways to do this.

In Test Title

Wrap test case IDs in square brackets inside the test name:

• [1] Test name - single test case
• [1,2,3] Test name - multiple test cases
• [1, 2, 3] Test name [4] another [5][6] combined - mixed formats

test.describe('Test suite', () => {
  test('[1] First Test', () => {
    expect(true).toBe(true);
  });

  test('[10,11,12] Multiple test cases', () => {
    expect(true).toBe(true);
  });
});

Using Tags

Use Playwright's tag syntax with the format @[ID]:

test.describe('Test suite', () => {
  test('Test name', {
    tag: ['@[1]', '@smoke', '@slow'],
  }, () => {
    expect(true).toBe(true);
  });
});

The @ prefix is required by Playwright. Non-ID tags (like @smoke) are ignored by the reporter.

Using Annotations

Use Playwright's annotation API with testCaseIdZone: 'annotation':

test('Test case', {
  annotation: { type: 'TestCase', description: '12345' },
}, () => {
  expect(true).toBe(true);
});

See Test Case ID Matching for advanced matching patterns with annotations.

Configuration

Required Options

| Option | Type | Description | |--------|------|-------------| | orgUrl | string | Full URL for your organization. Example: https://dev.azure.com/your-organization-name | | projectName | string | Name of your Azure DevOps project | | planId | number | ID of the test plan (from the test plan URL: planId=4) |

Note: Some APIs (e.g. ProfileApi) must be hit at the deployment level: https://vssps.dev.azure.com/{yourorgname}

General Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | isDisabled | boolean | false | Disable the reporter entirely | | environment | string | undefined | Prefix for test run names (e.g. 'QA') | | logging | boolean | false | Enable debug logging | | testRunTitle | string | 'Playwright Test Run' | Title for newly created test runs | | testRunConfig | object | undefined | Extra data passed when creating a test run. See API docs |

Authentication

The reporter supports three authentication methods:

Personal Access Token (PAT) - Default

{
  token: 'your-personal-access-token',
  authType: 'pat' // Optional, this is the default
}

Access Token

{
  token: 'your-access-token',
  authType: 'accessToken'
}

Azure Managed Identity

{
  token: 'not-used', // Token field is required but ignored for managedIdentity
  authType: 'managedIdentity',
  applicationIdURI: '499b84ac-1321-427f-aa17-267ca6975798/.default'
}

When to use each:

• PAT: Personal Access Token from Azure DevOps. Suitable for most use cases.
• Access Token: OAuth access token. Suitable for CI/CD environments with token-based authentication.
• Managed Identity: Azure environments with managed identity. Uses DefaultAzureCredential supporting Azure CLI, managed identity, environment variables, etc.

For more examples, see Authentication Examples.

Required Token Scopes

| Scope | Access | Purpose | |-------|--------|---------| | Test Management | Read & Write | Creating/updating test runs, publishing results, uploading attachments | | Work Items | Read & Write | Marking test cases as automated (only if autoMarkTestCasesAsAutomated is enabled) | | Project and Team | Read | Accessing project information |

Alternatively, use a token with Full access scope.

Note: If your token lacks Work Items write permissions but autoMarkTestCasesAsAutomated is enabled, the reporter will log errors for automation updates but continue publishing test results normally.

Publishing Modes

Control how and when test results are sent to Azure DevOps with publishTestResultsMode:

| Mode | Default | Description | |------|---------|-------------| | 'testResult' | Yes | Results published after each test completes, in parallel with test execution | | 'testRun' | | Results batched and published at the end of the test run | | 'testRunADO' | | Updates existing results in an Azure DevOps test run (requires isExistingTestRun: true and testRunId) |

Note: In testRun mode, if the same test case ID appears in multiple tests, only the last result is kept.

Existing Test Runs

To publish results to an existing test run instead of creating a new one:

{
  publishTestResultsMode: 'testRunADO',
  isExistingTestRun: true,
  testRunId: 12345, // or set AZURE_PW_TEST_RUN_ID env variable
}

• isExistingTestRun [boolean] - Publish to an existing test run without creating/completing it. Default: false.
• testRunId [number] - ID of the existing test run. Can also be set via AZURE_PW_TEST_RUN_ID env variable. Reporter options take precedence over the env variable.

Note: When using isExistingTestRun, the test run is not completed automatically. You must complete it manually.

Retry Results

Control how retry attempts are published with publishRetryResults:

| Mode | Description | |------|-------------| | 'all' | Every retry attempt is published as a separate test result (default) | | 'last' | Only the final attempt is published. Intermediate failures are skipped |

{
  publishRetryResults: 'last',
  // works with Playwright's retries setting
}

Attachments & Logs

| Option | Type | Default | Description | |--------|------|---------|-------------| | uploadAttachments | boolean | false | Upload attachments (screenshots/video) after each test | | attachmentsType | (string\|RegExp)[] | ['screenshot'] | Filter which attachments to upload by type or name pattern | | uploadLogs | boolean | false | Upload stdout/stderr logs (independent of uploadAttachments) |

Test Point Mapping

testPointMapper

A callback to map tests to specific test configurations (e.g. by browser):

import { TestCase } from '@playwright/test/reporter';
import { TestPoint } from 'azure-devops-node-api/interfaces/TestInterfaces';

{
  testPointMapper: async (testCase: TestCase, testPoints: TestPoint[]) => {
    switch (testCase.parent.project()?.use.browserName) {
      case 'chromium':
        return testPoints.filter((tp) => tp.configuration.id === '3');
      case 'firefox':
        return testPoints.filter((tp) => tp.configuration.id === '4');
      case 'webkit':
        return testPoints.filter((tp) => tp.configuration.id === '5');
      default:
        throw new Error('invalid test configuration!');
    }
  },
  testRunConfig: {
    // Get configuration IDs from:
    // https://dev.azure.com/{organization}/{project}/_apis/test/configurations
    configurationIds: [3, 4, 5],
  },
}

Important: Define testRunConfig.configurationIds and/or testPointMapper, otherwise results are published for all configurations.

rootSuiteId

Restricts test point resolution to a specific suite. Useful when different suites (e.g. smoke, integration) contain the same test case IDs. Can also be set via AZURE_PW_ROOT_SUITE_ID env variable. Reporter options take precedence.

Automation Tests
  - Smoke Tests (suiteId: 5)
    - Test 1 (caseId: 1)
    - Test 2 (caseId: 2)
  - Integration Tests (suiteId: 6)
    - Test 1 (caseId: 1)
    - Test 2 (caseId: 2)

Without rootSuiteId, running smoke tests would also match test points in Integration Tests. Set rootSuiteId: 5 to scope results to Smoke Tests only.

Test Case ID Matching

testCaseIdMatcher

A string, RegExp, or array of either, used to extract test case IDs. Default: /\[([\d,\s]+)\]/

Examples with test titles:

| Test Title | Matcher | Extracted IDs | |-----------|---------|---------------| | Test case @tag1=123 | /@tag1=(\d+)/ | ['123'] | | Test case @TC=123 [@TC=456] | /@TC=(\d+)/ | ['123', '456'] | | Test case test123 TEST456 | [/[a-z]+(\d+)/, /[A-Z]+(\d+)/] | ['123', '456'] |

If an invalid matcher is provided (e.g. a number), an error is thrown: "Invalid testCaseIdMatcher. Must be a string or RegExp.".

testCaseIdZone

Specifies where to look for test case IDs: 'title' (default) or 'annotation'.

• 'title' - Extracts IDs from the test title and tags
• 'annotation' - Extracts IDs from test annotations only

Important: When using testCaseIdZone: 'annotation', you must also define testCaseIdMatcher.

When using annotation zone, the matcher works in two steps:

1. First matcher matches the annotation type (e.g. /(TestCase)/)
2. Subsequent matchers extract IDs from the annotation description

Annotation examples:

Simple description:

// testCaseIdZone: 'annotation', testCaseIdMatcher: /(TestCase)/
test('Test case', {
  annotation: { type: 'TestCase', description: '12345' },
}, () => { /* ... */ });
// Extracted: ['12345']

Azure DevOps URLs:

// testCaseIdZone: 'annotation', testCaseIdMatcher: [/(Test Case)/, /\/(\d+)/]
test('Test case', {
  annotation: {
    type: 'Test Case',
    description: 'https://dev.azure.com/org/project/_workitems/edit/12345, https://dev.azure.com/org/project/_workitems/edit/54321',
  },
}, () => { /* ... */ });
// Extracted: ['12345', '54321']

Bracketed format:

// testCaseIdZone: 'annotation', testCaseIdMatcher: [/(Test Case)/, /\[([\d,\s]+)\]/]
test('Test case', {
  annotation: { type: 'Test Case', description: '[12345, 67890]' },
}, () => { /* ... */ });
// Extracted: ['12345', '67890']

Test Case Summary

Generate a report of test cases that don't match the test plan. Helps identify missing, misconfigured, or unassigned test cases.

{
  testCaseSummary: {
    enabled: true,
    outputPath: './test-case-summary.md',
    consoleOutput: true,
    publishToRun: true,
  },
}

| Sub-option | Type | Default | Description | |------------|------|---------|-------------| | enabled | boolean | false | Enable summary generation | | outputPath | string | undefined | File path for the summary report | | consoleOutput | boolean | true | Print summary to console | | publishToRun | boolean | false | Upload summary as a Markdown attachment to the ADO test run |

Output channels work independently - enable any combination:

| consoleOutput | outputPath | publishToRun | Result | |:-:|:-:|:-:|--------| | true | - | - | Console only | | true | set | - | Console + file | | - | - | true | ADO attachment only | | true | set | true | All three channels |

When all test cases match, a success message is logged instead.

Note: When publishToRun: true, the attachment is only uploaded if unmatched test points exist. Key summary lines are always logged regardless of the logging setting to ensure CI visibility.

Example output:

# Test Case Summary

Found 1 test(s) with test case IDs that don't match the test plan:

## Tests with No Matching Test Points (1)

- **[777] Test with file output**
  - File: tests/example.spec.ts:3
  - Test Case IDs: [777]

## Recommendations

- Verify test case IDs exist in Azure DevOps test plan 4
- Check that test cases are assigned to configurations: [10, 20] (Firefox, Safari)
- Ensure test cases are included in the test plan suite structure

Auto-Mark as Automated

Automatically update Azure DevOps test case work items to reflect their automation status when tests are executed.

{
  autoMarkTestCasesAsAutomated: {
    enabled: true,
    updateAutomatedTestName: true,
    updateAutomatedTestStorage: true,
  },
}

| Sub-option | Type | Default | Description | |------------|------|---------|-------------| | enabled | boolean | false | Enable automatic marking | | updateAutomatedTestName | boolean | true | Set AutomatedTestName field | | updateAutomatedTestStorage | boolean | true | Set AutomatedTestStorage field | | automatedTestNameFormat | string | 'title' | 'title' or 'titleWithParent' (includes parent suite name) | | automatedTestStoragePath | boolean \| function | false | false: filename only, true: full path, function: custom callback | | automatedTestType | function | undefined | Callback to set AutomatedTestType (e.g. 'Unit Test', 'E2E Test') |

What it does:

1. Checks each test case's automation status in Azure DevOps
2. If "Not Automated": sets AutomationStatus to "Automated", updates name/storage/type fields, generates a new AutomatedTestId GUID
3. If already "Automated": optionally updates name and storage fields if they differ

Advanced example:

{
  autoMarkTestCasesAsAutomated: {
    enabled: true,
    updateAutomatedTestName: true,
    updateAutomatedTestStorage: true,
    automatedTestNameFormat: 'titleWithParent',
    automatedTestStoragePath: (testCase) => {
      // Use relative path from project root
      const parts = testCase.location.file.split('/');
      const idx = parts.indexOf('my-project');
      return idx >= 0 ? parts.slice(idx + 1).join('/') : testCase.location.file;
    },
    automatedTestType: (testCase) => {
      if (testCase.location.file.includes('/e2e/')) return 'End-to-End Test';
      if (testCase.location.file.includes('/integration/')) return 'Integration Test';
      return 'Functional Test';
    },
  },
}

Tip: Use automatedTestNameFormat: 'titleWithParent' to prevent name collisions across suites. Use automatedTestStoragePath callback to avoid exposing absolute paths.

Note: Requires Work Items write permissions. Works in all publishing modes (testResult, testRun, testRunADO).

CI/CD Integration

Environment Variables

| Variable | Description | |----------|-------------| | AZURE_PW_TEST_RUN_ID | Set automatically after the test run is created. Available as process.env.AZURE_PW_TEST_RUN_ID. In testResult mode it's set at the start; in testRun mode at the end; in testRunADO mode it contains the existing run ID. | | AZURE_PW_ROOT_SUITE_ID | Alternative to the rootSuiteId option | | AZUREPWDEBUG | Enable debug logging: '1' = enabled, '0' = disabled (default) |

Azure DevOps Pipeline Example

- script: npx playwright test
  displayName: 'Run Playwright tests'
  name: 'playwright'
  env:
    CI: 'true'
    AZUREPWDEBUG: '1'

# Access the test run ID from subsequent steps (since v1.10.0)
- script: echo $(playwright.AZURE_PW_TEST_RUN_ID)
  displayName: 'Print test run id'

Releases

For detailed information about stable releases, beta releases, and version management, see RELEASES.md.