@testchimp/playwright

A Playwright / Mobilewright reporter that sends test execution results to the TestChimp platform. It powers QA intelligence insights, surfaces AI-native steps (e.g. ai.act, ai.verify) from the test runner into TestChimp for CI, and augments RUM events from @testchimp/rum-js so test runs align with real user events for TrueCoverage.

Purpose

1. Report execution results to TestChimp for QA intelligence

The reporter collects test runs (pass/fail, steps, errors, timing) and sends them to the TestChimp backend. TestChimp uses this data to:

• Track which tests ran, when, and their outcome
• Drive dashboards, trends, and QA intelligence
• Correlate failures with steps and screenshots for faster debugging
• Support traceability between tests, scenarios, and coverage

You run tests with the normal Playwright CLI or via your CI (GitHub Actions etc.); the reporter runs in process and posts results to TestChimp without changing how you execute tests.

2. Pipe AI-native steps through TestChimp so they work wherever you run tests

The reporter plugin pipes AI-native step calls (ai.act, ai.verify, etc.) via TestChimp backends, so that those steps work seamlessly—wherever you run your tests (local, CI, or any environment).

3. Augment RUM events for TrueCoverage (test ↔ real user alignment)

@testchimp/rum-js (web), TestChimpRum (iOS — Swift package testchimp-rum-ios), and TestChimpRum (Android — testchimp-rum-android, typically via JitPack: com.github.testchimphq:testchimp-rum-android:<tag> per JitPack) emit real user events to TestChimp. When the same app is exercised in CI (Playwright or Mobilewright), you want those events tagged with which test produced them so TestChimp can:

• Align test runs with real user sessions (TrueCoverage)
• See which tests generated which events
• Compare test coverage to production usage to drive better QA strategy.

Read more about TestChimps' TrueCoverage feature here.

Installation

npm install @testchimp/playwright

Peer dependency: @playwright/test (e.g. >=1.40.0) for web projects, and mobilewright for mobile projects.

Runtime switching is controlled by TESTCHIMP_PROJECT_TYPE:

• web or unset: uses Playwright fixture page (default, backward-compatible).
• ios / android (case-insensitive): uses Mobilewright fixture screen.

Quick start

1. Playwright config

Configure the reporter in your playwright.config.js like below:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

// Optional: import runtime so CI test info is injected for @testchimp/rum-js (TrueCoverage)


export default defineConfig({
  reporter: [
    ['list'],
    ['@testchimp/playwright/reporter', {
      verbose: true,
      reportOnlyFinalAttempt: true,
      captureScreenshots: true,
    }],
  ],
});

For TrueCoverage, wrap and re-export test from your fixtures entry (recommended):

import { test as base } from '@playwright/test'; // or '@mobilewright/test'
import { installTestChimp } from '@testchimp/playwright/runtime';
export const test = installTestChimp(base);

A side-effect-only import '@testchimp/playwright/runtime' does not apply extended fixtures or mobile hooks; the returned TestType from installTestChimp must be what your specs import.

Web: the runtime injects __TC_CI_TEST_INFO on the page fixture for @testchimp/rum-js.

Mobile (iOS/Android): set TESTCHIMP_PROJECT_TYPE=ios or android. The runtime always registers beforeEach/afterEach hooks that call device.openUrl to push CI metadata into the app. Integrate TestChimpRum for that platform (see testchimp-rum-ios / testchimp-rum-android READMEs): URL scheme / intent filter for testchimp-rum://truecoverage/....

export TESTCHIMP_PROJECT_TYPE=ios  # or android

Optional URL overrides: TESTCHIMP_RUM_AUTOMATION_SET_PREFIX, TESTCHIMP_RUM_AUTOMATION_CLEAR_URL (defaults match the native SDKs).

2. Environment variables

Set these so the reporter can talk to TestChimp (env vars override programmatic options):

| Variable | Required | Description | |----------|----------|-------------| | TESTCHIMP_API_KEY | Yes | API key for TestChimp. | | TESTCHIMP_PROJECT_ID | No | Legacy/optional; the backend resolves the project from the API key when omitted. | | TESTCHIMP_TESTS_FOLDER | No | Base folder for relative paths (default: tests). | | TESTCHIMP_RELEASE | No | Release/version identifier. | | TESTCHIMP_ENV | No | Environment (e.g. staging, prod). | | TESTCHIMP_PROJECT_TYPE | No | Fixture/runtime switch: web (default) or mobile (ios/android). Mobile mode always pushes CI JSON via device.openUrl (TrueCoverage). | | TESTCHIMP_RUM_AUTOMATION_SET_PREFIX | No | Override set-URL prefix (default testchimp-rum://truecoverage/v1/set?p= + base64url JSON). | | TESTCHIMP_RUM_AUTOMATION_CLEAR_URL | No | Override clear URL (default testchimp-rum://truecoverage/v1/clear). |

If TESTCHIMP_API_KEY is missing, the reporter logs a warning and disables reporting (no request is sent).

3. Run tests

export TESTCHIMP_API_KEY=your-api-key
npx playwright test

Results are reported to TestChimp after each test (or after the final attempt when using retries and reportOnlyFinalAttempt: true).

Reporter options

You can pass options in playwright.config.ts:

['@testchimp/playwright/reporter', {
  apiKey: '...',           // override env (not recommended in CI)
  backendUrl: '...',       // override TESTCHIMP_BACKEND_URL
  projectId: '...',       // optional override env
  testsFolder: 'tests',   // base dir for relative path calculation
  release: '1.0.0',
  environment: 'staging',
  reportOnlyFinalAttempt: true,  // only send report for last retry (default: true)
  captureScreenshots: true,      // attach screenshots to failing steps (default: true)
  verbose: false,               // extra logging (default: false)
}]

Environment variables take precedence over these options.

What gets reported

For each test (or its final attempt when using retries), the reporter sends a Smart Test Execution Report that includes:

• Identity: folderPath, fileName, suitePath, testName (derived from test file and describe blocks).
• Run context: batchInvocationId, branchName (from env when available), release, environment.
• Job detail:
  • Steps: Every Playwright step with category test.step, expect, or pw:api (including AI-native steps).
  • Status: Completed, Failed, or Unknown (mapped from Playwright status).
  • Error: Top-level test error message if failed.
  • Screenshots: For failing steps, when captureScreenshots is true and Playwright has attached screenshots (e.g. screenshot: 'only-on-failure').

Retries are tracked; with reportOnlyFinalAttempt: true only the last attempt is reported.

Exports

• Subpath: @testchimp/playwright/reporter — explicit reporter entry for Playwright reporter config.
• Named: TestChimpReporter, TestChimpApiClient, and types/utilities from ./types and ./utils.
• Subpath: @testchimp/playwright/runtime — use installTestChimp(test) on your runner’s test object (see Quick start).
  • Web: extends page to set __TC_CI_TEST_INFO for @testchimp/rum-js.
  • Mobile: when TESTCHIMP_PROJECT_TYPE is ios/android, registers hooks that call device.openUrl for the iOS Swift SDK and Android Kotlin SDK (same URL contract).
• Named: buildCiTestInfoJson, attachMobileRumAutomationHooks, etc., for advanced wiring.

Troubleshooting

• “Reporting disabled”
  Set TESTCHIMP_API_KEY (or pass it in reporter options). TESTCHIMP_PROJECT_ID should be present to enable TrueCoverage.

• No steps or only some steps
  Only steps with category test.step, expect, or pw:api are reported. Internal/hook steps are excluded.

• No screenshots on failure
  Enable screenshot capture in Playwright (e.g. use: { screenshot: 'only-on-failure' }). The reporter only attaches existing attachments to failing steps.

• RUM events not linked to tests
  Use export const test = installTestChimp(...) from a fixtures barrel and import test from there. On mobile, set TESTCHIMP_PROJECT_TYPE and handle testchimp-rum:// URLs in the app (TestChimpRum.handleAutomationURL on iOS, TestChimpRum.handleAutomationIntent on Android).

• Verbose logging
  Set verbose: true in reporter options or use it during setup to see which steps are captured and when reports are sent.

License

MIT.