@blaze-cms/plugin-testing-e2e

Portable Playwright e2e framework + common spec library for Blaze ecommerce sites.

The split: each repo owns its site profiles (locator registry, routes, feature flags, test data). This package owns everything generic — the locator engine, page objects, fixtures, global setup, base Playwright config, and the shared specs (smoke, search, auth, registration, cart, checkout, account, subscription signup). Specs gate themselves on profile feature flags and skip with a visible reason where a site lacks a capability.

Consuming suite layout (per repo)

tests-pw/e2e/
├── playwright.config.ts      # blazeConfig(e2e) + repo overrides
├── global-setup.ts           # createGlobalSetup(e2e)
├── .env / example.env
└── src/
    ├── profiles/             # THIS REPO'S locator registry (site profiles)
    │   └── mysite.ts
    ├── e2e.ts                # createBlazeE2E({ profiles, defaultSite })
    └── tests/
        ├── smoke/smoke.spec.ts        # registerSmokeSpecs(e2e)  — one-liners
        ├── cart/cart.spec.ts          # registerCartSpecs(e2e)
        └── mysite/special.spec.ts     # site-specific specs, using e2e.test + page objects

Wiring

// src/e2e.ts
import { createBlazeE2E } from '@blaze-cms/plugin-testing-e2e';
import { mysiteProfile } from './profiles/mysite';

export const e2e = createBlazeE2E({ profiles: [mysiteProfile], defaultSite: 'mysite' });
export const { test, expect, requireFeature } = e2e;

// playwright.config.ts
import { defineConfig } from '@playwright/test';
import { blazeConfig } from '@blaze-cms/plugin-testing-e2e';
import { e2e } from './src/e2e';

export default defineConfig(
  blazeConfig(e2e, { globalSetup: require.resolve('./global-setup') }),
);

// global-setup.ts
import { createGlobalSetup } from '@blaze-cms/plugin-testing-e2e';
import { e2e } from './src/e2e';
export default createGlobalSetup(e2e);

// src/tests/smoke/smoke.spec.ts
import { registerSmokeSpecs } from '@blaze-cms/plugin-testing-e2e';
import { e2e } from '../../e2e';
registerSmokeSpecs(e2e);

Profiles

A profile deep-merges overrides onto the shared .blaze-ecommerce-* baseline:

import { blazeBaseLocators, deepMerge, type SiteProfile } from '@blaze-cms/plugin-testing-e2e';

export const mysiteProfile: SiteProfile = {
  key: 'mysite',
  name: 'My Site',
  defaultBaseUrl: 'https://frontend-mysite.stage.blazecms.app',
  locators: deepMerge(blazeBaseLocators, {
    auth: { email: { css: '#customer-email', first: true } },
  }),
  routes: { home: '/', login: '/login', basket: '/basket', checkout: '/checkout' },
  features: { search: false, addToCart: true, flyoutBasket: true, cookieBanner: false,
              registration: false, subscriptionSignup: false, checkout: 'none' },
  data: { products: [{ path: '/p/thing', title: 'Thing' }], registrationValues: {} },
};

Runtime knobs (same across all consumers): BLAZE_TEST_PLAYWRIGHT_SITE picks the profile, BLAZE_TEST_PLAYWRIGHT_LOCATOR_OVERRIDES deep-merges a JSON file over the active profile's locators (strings like "/accept/i" become regexes), plus the usual BLAZE_TEST_PLAYWRIGHT_BASE_URL, BLAZE_TEST_HTTP_CREDENTIALS_*, BLAZE_X_CLIENT, BLAZE_TEST_PLAYWRIGHT_USER_EMAIL/_PASSWORD, _WORKERS, _RETRIES, _PLATFORM_COVERAGE, _ALLOW_REGISTRATION.

Platform coverage

BLAZE_TEST_PLAYWRIGHT_PLATFORM_COVERAGE picks the matrix. Any other value is an error, so a mistyped CI input can't quietly fall back to one browser.

| Coverage | Local browsers | BrowserStack | |---|---|---| | chromium (default) | Desktop Chrome | Chrome / Windows 11 | | core | Chrome, WebKit, Pixel 7 and iPhone 14 emulation | Chrome / Windows 11, Safari 26 / macOS, real iPhone 17, real Galaxy S25 | | desktop | Chrome, Firefox, WebKit | Chrome, Edge, Firefox / Windows 11; Chrome / macOS; Safari 18 and 26 / macOS | | mobile | Pixel 7, iPhone 14 emulation | Real iPhone 14 Pro, 15, 16, 17; Galaxy S25, Pixel 9; iPad Air 13, Galaxy Tab S9 | | all | Everything above | Everything above |

BrowserStack

BrowserStack is on by default. blazeConfig runs on real browsers and devices whenever BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY are set, and on local browsers when they aren't. BLAZE_TEST_BROWSERSTACK overrides that:

• unset or auto: BrowserStack if credentials are present, local otherwise.
• true: BrowserStack, and missing credentials are an error. CI uses this.
• false: local browsers.

Specs, profiles and storage states don't change between modes. What it handles:

• Desktop browsers connect through Playwright's connectOptions, one BrowserStack session per worker.
• Real Android devices connect through Playwright's Android protocol, one session per worker, with a fresh browser on the device for each test.
• Real iOS devices get one session per test, because a session allows only one browser context. Connecting takes about 30s, so iOS tests are the slowest part of a run.
• Sessions open only when a test uses a page, so API-only tests (route checks using request) never connect to BrowserStack or use its minutes.
• Every session is marked passed or failed in the dashboard, with the failing test titles as the reason.
• Real iOS devices run without request routes, because any route leaves the page blank. So the WAF header (BLAZE_X_CLIENT) isn't sent and the profile's blockHosts don't apply there. A target that requires the WAF header will block them; the run warns about this.
• Failure screenshots are off on BrowserStack, because the teardown screenshot hangs WebKit when a route is active. Failure traces still carry screenshots.
• Workers follow BLAZE_TEST_BROWSERSTACK_PARALLELS (default 2), since the plan's parallel session limit binds before the target does. A numeric workers in the repo's config still caps it. The test timeout rises to at least 90s.
• Real devices keep their hardware viewport, so a test that calls page.setViewportSize() on one skips itself with that reason. To skip earlier, gate with test.skip(({}, testInfo) => isRealDeviceProject(testInfo.project.name), '…').

Suites that don't use createBlazeE2E, such as visual regression, can import all of this from @blaze-cms/plugin-testing-e2e/browserstack. It loads no .env and no site profiles: readBrowserStackEnv(), browserStackProjects() and the platform list, plus a test carrying the BrowserStack fixtures. @blaze-cms/plugin-testing-base uses it.

Other settings:

| Variable | Purpose | |---|---| | BLAZE_TEST_BROWSERSTACK_PLATFORMS | Comma-separated platform keys from BROWSERSTACK_PLATFORMS; overrides coverage | | BLAZE_TEST_BROWSERSTACK_PROJECT / _BUILD | Dashboard names; project defaults to the GitHub repo or npm package name | | BLAZE_TEST_BROWSERSTACK_LOCAL / BROWSERSTACK_LOCAL_IDENTIFIER | For targets not reachable from the internet; needs the BrowserStack Local binary running | | BLAZE_TEST_BROWSERSTACK_DEBUG | Session video, console and network logs | | CUSTOM_TAG_<N> | Build tags |

Global setup still builds its storage states in a local Chromium, so npx playwright install chromium is needed in both modes. Visual-regression baselines depend on OS fonts, so baselines recorded locally won't match a BrowserStack run.

Lighthouse

registerLighthouseSpecs(e2e) audits the profile's key routes under Lighthouse's mobile and desktop settings. It needs lighthouse (13 or later) installed in the suite; without it the tests skip. It also needs Node 22.19 or later, so set the e2e job's node-version to 22 or later. On older Node the tests fail with a message saying so, since some audits break part-way there (on Node 20, canonical errors and leaves SEO unscored).

npm install -D lighthouse

// src/tests/lighthouse/lighthouse.spec.ts
import { registerLighthouseSpecs } from '@blaze-cms/plugin-testing-e2e';
import { e2e } from '../../e2e';

registerLighthouseSpecs(e2e);

The tests are tagged @lighthouse. blazeConfig runs them only in a lighthouse project, in a local Chromium whatever the platform coverage or BrowserStack setting, and leaves them out of every other project. Run them alone with --grep Lighthouse.

Each audit gets its own browser, which Playwright launches and Lighthouse drives, so basic auth, the WAF header, the anonymous storage state and blockHosts all apply. The full Lighthouse report is attached to each test result.

| Category | Default minimum | On failure | |---|---|---| | Accessibility | 90 | fails the test | | Best practices | 90 | fails the test | | SEO | 90 | fails the test | | Performance | 50 | warning only: dev and stage swing too much to gate on |

When one of a category's audits errors, Lighthouse leaves the category unscored. That counts as under the minimum, and the message names the audit and its error.

is-crawlable and robots-txt are skipped, since dev and stage are kept out of search engines on purpose. A profile tunes the rest under data.lighthouse:

data: {
  lighthouse: {
    routes: ['/', '/books', '/books/xj40-engineering'], // default: home, search, first product, first extra route
    devices: ['mobile'],                                  // default: mobile and desktop
    thresholds: { 'best-practices': 0.75 },
    warnOnly: ['performance', 'best-practices'],
    skipAudits: ['third-party-cookies'],
  },
},

Failure triage

blazeConfig adds a reporter that writes test-results/e2e-results.json. The file holds per-platform counts and, for each failed or flaky test:

• the error messages
• the failing test's source around the failing line
• for tests that opened a page, what the page was at the failure: its URL and title, the last navigation's HTTP status, recent script and console errors, and an accessibility snapshot. createBlazeE2E's page fixture records this with listeners only, no routes.

Secret values from the environment (the WAF header, basic auth, test users, BrowserStack, API keys) are redacted before the file is written.

npx blaze-e2e-triage then sends the failures to Claude (claude-opus-5-5) in one call. Claude puts each failure in one category (product bug, test bug, flaky, environment, BrowserStack, unknown), with a confidence level, its evidence and a next step. The CLI:

• appends the triage to the GitHub run summary, or prints it outside Actions
• posts to Slack when something failed: a few lines in the channel (headline, verdict, a count per category, links) with the per-failure details in a thread reply. Threading needs a Slack app's bot token (SLACK_BOT_TOKEN, chat:write scope, app invited to the channel) and SLACK_CHANNEL_ID. With only SLACK_WEBHOOK_URL, which can't reply in threads, it posts just the summary.
• writes test-results/e2e-triage.json
• when something failed, writes test-results/e2e-triage.html: a self-contained page with the triage and every failure's errors, page at failure and snapshot

Without ANTHROPIC_API_KEY it still writes a plain list of failures. It never fails the job.

Upload the HTML report as a short-lived artifact so it deletes itself. With BLAZE_TRIAGE_REPORT_ARTIFACT set to the artifact's name, the run summary and the Slack post say where to find it.

- name: Triage results
  if: always()
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    ANTHROPIC_WORKSPACE_ID: ${{ vars.ANTHROPIC_WORKSPACE_ID }}
    SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
    SLACK_CHANNEL_ID: ${{ vars.SLACK_CHANNEL_ID }}
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_BLAZE_TESTING_QA }}
    BLAZE_TRIAGE_REPORT_ARTIFACT: e2e-triage-report
  run: |
    cd packages/frontend/tests-pw/e2e
    npx --no-install blaze-e2e-triage

- name: Upload triage report
  uses: actions/upload-artifact@v4
  if: always()
  with:
    name: e2e-triage-report
    path: packages/frontend/tests-pw/e2e/test-results/e2e-triage.html
    if-no-files-found: ignore
    retention-days: 14

--dry-run prints the report, the Slack payload and the size of what would be sent, and calls neither Claude nor Slack. --no-slack skips the post. --results <file> reads a different results file. BLAZE_TRIAGE_MODEL overrides the model. With an API key that isn't scoped to a workspace, set ANTHROPIC_WORKSPACE_ID to the workspace to bill.

Local (pre-publish) consumption

Until this package is published, consumers can alias it in tsconfig.json — Playwright resolves tsconfig paths and transpiles the TS source directly:

{ "compilerOptions": { "paths": {
  "@blaze-cms/plugin-testing-e2e": ["../../../../../blaze-testing-tools/packages/e2e/src/index.ts"]
} } }

Once published (yarn build emits dist/ CJS + typings), swap the alias for a regular dependency: "@blaze-cms/plugin-testing-e2e": "^0.1.0".