@lockweb/qa-workflow

Playwright-powered QA workflow tooling for recording tests, running targeted test sets, capturing visual baselines, comparing snapshots, and generating shareable reports.

This package is built around a reusable CLI-first workflow exposed through qaw.

Author: Chris Lock Owner: Lock Web Development LLC

Status

This package is public, but it is not open source at this time. It currently uses the UNLICENSED license while the public API and reuse terms are still being finalized.

Goals

• Make Playwright setup easier to adopt.
• Make test recording and publishing easier to manage.
• Support targeted test execution across suites and targets.
• Provide visual regression capture and compare workflows.
• Generate review-friendly reports that are easier to share with stakeholders.

CLI

The primary CLI entrypoints are:

qaw
qaw init
qaw validate

qaw opens the interactive menu by default.

Direct commands are also available:

qaw record
qaw run
qaw publish
qaw clean
qaw auth
qaw tests
qaw targets
qaw visual
qaw visual capture
qaw visual compare
qaw update
qaw status

All commands also accept:

--config ./path/to/qa-workflow.config.json

When a command supports --ci, the behavior is:

• passed values are used directly
• missing optional values fall back to the normal default
• missing required values fail instead of prompting
• invalid passed values fail instead of prompting

Boolean flags accept true|false, yes|no, 1|0, on|off.

Command Reference

qaw

Interactive main menu.

qaw --config ./qa-workflow.config.json

qaw init

Initial project scaffolding.

qaw init [--root-dir qa-workflow] [--sample-csv true|false] [--config true|false] [--playwright-config true|false] [--script true|false] [--script-name qaw] [--overwrite] [--ci]

Flags:

• --root-dir Workflow root folder. Default: qa-workflow
• --sample-csv Create the sample visual regression CSV
• --config Create qa-workflow.config.json
• --playwright-config Create playwright.config.js
• --script Add a local npm script to package.json
• --script-name npm script name to add. Default: qaw
• --overwrite Overwrite existing generated files when supported
• --ci Run without prompts

qaw validate

Config validation.

qaw validate [--config ./qa-workflow.config.json]

qaw record

Interactive or scripted raw recording flow.

qaw record [--suite <suite>] [--target <target>] [--auth continue|open|cancel] [--host <hostKey>] [--name <test-name>] [--subfolder folder/path] [--path /relative/path] [--ci]

Flags:

• --suite Suite name
• --target Record-enabled target for that suite
• --auth Auth preflight action: continue, open, or cancel
• --host Start host key. Alias: --start-host
• --name Recording name. Default: new-test
• --subfolder Optional subfolder under test/recorded/<suite>/
• --path Relative path to open on the selected host
• --ci Run without prompts

qaw record warns when the generated raw spec contains no Playwright assertions and prevents silent overwrite of existing recorded files.

qaw publish

Promotes a recorded spec into a runnable published spec.

qaw publish [--file recorded/spec.raw.spec.ts] [--name <test-name>] [--subfolder folder/path] [--smoke true|false] [--regression true|false] [--writes-content true|false] [--tags foo,bar] [--personas admin:pass,editor:fail] [--remove-recorded true|false] [--ci]

Flags:

• --file Recorded spec path relative to test/recorded
• --name Final published test name
• --subfolder Optional subfolder under test/published/<suite>/
• --smoke Add or omit @smoke
• --regression Add or omit @regression
• --writes-content Mark the generated test as content-writing. Alias: --writesContent
• --tags Extra comma-separated tags
• --personas Comma-separated persona declarations in user[:pass|fail] form
• --remove-recorded Remove the raw recorded spec after publish. Default: true
• --ci Run without prompts

qaw publish derives the suite and target from metadata written into the raw recording by qaw record. It does not prompt for or override the recorded target.

For interactive publish, the selected recorded file is the source of defaults for name, subfolder, smoke, regression, writes-content, and extra tags.

qaw publish prevents silent overwrite of existing published test files.

qaw clean

Removes recorded specs under the configured recorded directory.

qaw clean [--yes]

Flags:

• --yes Skip the confirmation prompt

qaw run

Runs published tests for a suite folder, with optional tag filtering.

qaw run [--suite <suite>] [--target <target>] [--run-type all|tags|path] [--mode headless|headed|ui|watch] [--tags smoke,regression] [--path folder/or/spec.spec.ts] [--auth continue|open|cancel] [--serial] [--ci] [Playwright options]

Flags:

• --suite Suite folder under test/published
• --target Run-enabled target for that suite
• --run-type all, tags, or path
• --mode headless, headed, ui, or watch
• --tags Tag filter used when --run-type tags is selected
• --path File or folder path relative to the selected suite root. Used when --run-type path is selected
• --auth Auth preflight action: continue, open, or cancel
• --serial Run the entire selected test set as one normal Playwright phase with one worker. @serial and @personaserial tags remain available for filtering but do not create separate phases.
• --ci Run without prompts

Other Playwright CLI options are passed through to each applicable Playwright phase. For example, to retry failed tests once:

qaw run --suite example --target stage --run-type path --path ckeditor --retries=1

QAW-owned options for suite selection, paths, tag filtering, phase control, configuration, and reporting remain managed by QAW.

watch mode runs tests headed with one worker and sets PLAYWRIGHT_WATCH=true for the generated Playwright config template. That template disables the overall test timeout in watch mode and applies launchOptions.slowMo using PLAYWRIGHT_SLOW_MO, which defaults to 250.

Setup files

qaw run runs setup files before the main Playwright run. There are two kinds:

setup.ts           directory setup: applies to every spec in its folder and below
view.setup.ts      matched setup: applies to view.spec.ts only

Either kind may use .ts, .js, .mjs, or .cjs, and a matched setup pairs with its spec whatever the extension. A folder may hold at most one setup file per name, so setup.ts next to setup.js (or view.setup.ts next to view.setup.cjs) is an error that stops the run before anything starts.

Directory setup files cascade from the suite root down. Given:

suite/
  setup.ts
  catalog/
    setup.ts
    listing.spec.ts
    orders/
      setup.ts
      returns/
        setup.ts
        refund.setup.ts
        refund.spec.ts

running --path catalog/orders/returns/refund.spec.ts runs:

setup.ts -> catalog/setup.ts -> catalog/orders/setup.ts -> catalog/orders/returns/setup.ts -> catalog/orders/returns/refund.setup.ts

A spec always gets the same setup chain however it is selected:

• Single spec path: the directory setup files from the suite root down to the spec's folder, then its matched setup.
• Folder path, or --run-type all: the directory setup files above the folder, then every setup file inside it.
• --run-type tags: the chains of every spec the tags select, without duplicates.

Parents run before children, and a folder's setup.ts runs before its matched setup files. QAW loads the setup files in that order through one Playwright invocation with one worker, in serial mode, so the first failure skips the remaining setup files and the main test run does not start. The main run then uses the configured worker count.

Setup files run on every selected run, including single-spec runs, so they should only create what is missing and be safe to run repeatedly. Put shared logic in a helper without the .setup. suffix (for example fixtures/catalog.ts) and call it from each setup file that needs it. Specs that depend on setup output should still check for it and fail with a clear message naming the setup file to run.

When setup files are present, QAW keeps a single setup/teardown lifecycle across the test phases:

globalSetup -> setup files -> regular tests -> serial tests -> persona serial tests -> globalTeardown

The setup invocation keeps the project globalSetup enabled and disables globalTeardown. The regular test invocation disables both lifecycle hooks when a serial phase is needed, and the serial invocation disables globalSetup while keeping globalTeardown enabled. This prevents project fixtures from being rebuilt between setup, regular tests, and serial tests.

This is a QAW runner convention, not raw Playwright behavior: running npx playwright test directly does not run setup files.

Serial tests

Mark tests that must run after regular tests and with one worker using the Playwright tag convention:

test('updates shared content', { tag: ['@serial'] }, async ({ page }) => {
  // ...
});

When the selected run contains @serial tests, QAW automatically runs regular tests first using the worker count from the consuming project's Playwright configuration. It then runs the @serial tests sequentially with one worker and merges both phases into the same suite report. Existing --tags and --path filters apply to both phases.

For persona-aware serialization, mark a test or enclosing test.describe block with @personaserial:

personaTest('updates shared content', {
  tag: ['@personaserial'],
}, async ({ page, persona }) => {
  // ...
});

QAW runs the selected @personaserial tests in one worker per configured persona. Different personas run concurrently, while tests for the same persona run sequentially. Persona keys come from the suite's personas configuration. Child results are merged into the same suite report, and setup/teardown still run once.

Phased serial execution is intended for headless and headed runs. ui and watch modes reject a selected run containing @serial or @personaserial tests because those modes are persistent interactive sessions rather than finite phases.

To run the complete selected set sequentially as one normal Playwright phase, use --serial. This disables QAW's special @serial and @personaserial phase splitting and passes --workers 1 to the main test invocation. Setup files still run in their own step before the main invocation.

qaw run writes structured suite run data under reports/runs and renders a suite dashboard under reports/html when a suite report renderer is available. The default renderer is @lockweb/qa-report, and consumers can override it with reporting.suite.renderer. If no renderer is installed, qaw still saves the report data bundle and prints a message telling you how to install a renderer.

Suite reports are generated for one selected run at a time. Use qaw report list --suite <suite> [--target <target>] to inspect saved runs, qaw report render --suite <suite> --run-id <run-id> to regenerate the dashboard from a saved run, qaw report package --suite <suite> --run-id <run-id> to create an email-ready .tar.gz report archive, or qaw report package --suite <suite> --run-id <run-id> --output-dir <directory> to create the same self-contained report site without an archive. Use qaw report clean --suite <suite> [--target <target>] --keep <count> --delete to remove old run directories after reviewing the dry-run output. Saved run directories contain the test data and copied screenshots/videos needed to regenerate their report.

The list, clean, and remove commands work without @lockweb/qa-report. Without a renderer, render still writes the report data bundle, while package/archive reports that @lockweb/qa-report must be installed to create the HTML archive.

qaw auth

Host-level auth management for a suite and target.

qaw auth [--suite <suite>] [--target <target>] [--host <hostKey>] [--action set|clear|back] [--ci]

Flags:

• --suite Suite name
• --target Target name
• --host Host key to operate on directly
• --action set, clear, or back
• --ci Run without prompts. In --ci mode, --host and --action are required

Auth status behavior:

• shared-auth hosts show the last saved timestamp for that host
• persona-auth hosts show persona progress as saved X/Y persona states
• qaw auth uses a stable host menu with Set, Clear, and Back
• when perDomainAuth is enabled for a host, Set can create all personas or one persona at a time
• persona-auth flows can optionally start each persona on a custom path by setting personas.<personaKey>.authStartPath
• shared-auth hosts keep the existing combined auth flow and do not need persona state files
• suites that do not opt into persona auth continue to use the current behavior

qaw tests

Lists published tests for a suite, grouped by folder and annotated with tags.

qaw tests [--suite <suite>] [--folder folder/path] [--tags smoke,checkout] [--sort folder|name|tags] [--ci]

Flags:

• --suite Suite folder under test/published
• --folder Optional subfolder filter under the suite
• --tags Optional comma-separated tag filter. All listed tags must be present
• --sort Sort within the suite by folder, name, or tags
• --ci Run without prompts

qaw targets

Lists configured targets.

qaw targets [--suite <suite>] [--target <target>]

Flags:

• --suite Restrict output to one suite
• --target Restrict output to one target

qaw visual

Interactive visual regression menu.

qaw visual
qaw visual [--action capture|compare|delete|cancel] [--suite <suite>] [--target <target>] [--set <name>] [--prefix run-name] [--before snapshot] [--after snapshot] [--zip true|false] [--tolerance percent] [--host hostKey] [--persona persona] [--viewports names] [--concurrency n] [--reload-per-viewport true|false] [--scroll-before-capture true|false] [--path-delay ms] [--navigation-timeout ms] [--load-event-timeout ms] [--network-idle-timeout ms] [--asset-load-timeout ms] [--screenshot-delay ms] [--post-scroll-delay ms] [--failed-load-retries n] [--yes] [--ci]

Flags:

• --action capture, compare, delete, or cancel
• --suite Suite name
• --target Visual-enabled target for capture
• --set Visual set from visual.sets to capture or compare. Without it, the menu prompts for a set (the first set in --ci mode). Compare lists only snapshots captured with that set, so sets that share a CSV (for example one set per screen size) never mix.
• --prefix Optional capture name prefix. Snapshot names are <prefix>-<target>-<suite>-<set>-<timestamp>.
• --before Before snapshot name for compare
• --after After snapshot name for compare
• --zip Include the share zip when comparing
• --tolerance Mismatch tolerance in percent for compare. When omitted, the interactive menu prompts for it with the suite's resolved tolerance as the default; in --ci mode compare resolves it from env and config.
• --host, --persona, --viewports, --concurrency, --reload-per-viewport, --scroll-before-capture, --path-delay, --navigation-timeout, --load-event-timeout, --network-idle-timeout, --asset-load-timeout, --screenshot-delay, --post-scroll-delay, --failed-load-retries Capture overrides, forwarded to qaw visual capture (see below); not prompted for.
• --yes Required to allow delete in --ci mode
• --ci Run without prompts

qaw visual capture

Direct visual capture entrypoint.

qaw visual capture --target <target> --suite <suite> --set <name> [--snapshot snapshot-name]
  [--host <hostKey>] [--persona <persona>] [--viewports <name,name>] [--concurrency <n>]
  [--reload-per-viewport true|false] [--scroll-before-capture true|false] [--path-delay <ms>]
  [--navigation-timeout <ms>] [--load-event-timeout <ms>] [--network-idle-timeout <ms>] [--asset-load-timeout <ms>]
  [--screenshot-delay <ms>] [--post-scroll-delay <ms>] [--failed-load-retries <n>]

Older positional form is still supported:

qaw visual capture [snapshot] <target> <suite> <set>

--set names the visual set to capture: its CSV, host, persona, and settings. Every capture uses a set, so a suite needs at least one entry in visual.sets. Without --snapshot, the snapshot is named <target>-<suite>-<set>-<timestamp>. --csv was removed; passing it stops with an error.

Every other flag overrides the matching setting of the set for this capture only (see Config Reference for defaults). Settings named ...Timeout are limits a step may end before; settings named ...Delay always wait their full time.

• --host Host key to capture instead of the set's host.
• --persona Persona to capture as instead of the set's persona.
• --viewports Comma-separated names from visual.viewports, captured in the order given, for example --viewports desktop or --viewports phone,desktop. Any defined viewport can be named, not only the set's.
• --concurrency Number of CSV paths captured in parallel. Integer of at least 1.
• --reload-per-viewport true gives every viewport its own page load; false resizes the page loaded for the first viewport.
• --scroll-before-capture true scrolls through each page before screenshots; false skips it.
• --path-delay Fixed pause in milliseconds between paths, for pacing requests. Integer of at least 0.
• --navigation-timeout Maximum time in milliseconds for a page to reach DOMContentLoaded; past it the page view fails to load. Integer of at least 1. The wait for body to become visible is capped at the smaller of this value and 15 seconds.
• --load-event-timeout Maximum wait in milliseconds for the page's load event. Integer of at least 0; 0 skips the wait.
• --network-idle-timeout Maximum wait in milliseconds for the network to go idle after each page load. Integer of at least 0; 0 skips the wait.
• --asset-load-timeout Maximum wait in milliseconds for stylesheets, images, web fonts, and CSS background images. Integer of at least 0; 0 skips the wait.
• --screenshot-delay Fixed pause in milliseconds after the asset wait, before scrolling or screenshots. Integer of at least 0.
• --post-scroll-delay Fixed pause in milliseconds after scrolling back to the top, before screenshots. Integer of at least 0.
• --failed-load-retries Number of end-of-run retry passes for page views that failed to load. Integer of at least 0; 0 disables retries.

An invalid override stops the capture with an error before the browser starts. The earlier flag names (--timeout, --idle-timeout, --asset-timeout, --settle, --scroll-settle, --retries, --reload, --scroll, --wait) stop the capture with an error naming the new flag.

Visual sets and which host is captured:

• Each visual set (see visual.sets) chooses its own CSV, capture host, persona, targets, regions, viewports, and settings, so one suite can check public pages on one host with every region and logged-in pages on another with fewer targets, regions, or viewports, or longer waits.
• A set's host must exist in the hosts of every target in the set's targets. Without host, capture uses each target's default host. qaw validate reports each target that is missing a set's host, and capture stops with the same error before loading any page.
• Without a persona, capture loads the saved base login of every login-managed host on the target (the combined auth state), or no login when no host needs one.
• With a persona, capture loads that persona's saved login for the capture host plus the base logins for the target's other hosts, the same combination persona tests use. The capture host must have perDomainAuth: true (persona logins exist only there), and the persona's login must already be saved with qaw auth; otherwise capture stops before loading any page.
• --host and --persona override the set's values for one capture.

Capture order and viewports:

• Capture walks the CSV once. For each path it captures every viewport in the set's viewports order before moving to the next path, so all viewports of a page are captured within seconds of each other.
• The first viewport is always a fresh page load. With reloadPerViewport on (the default), every later viewport also gets its own page load, so scripts that initialize for the screen size run as they would on a real device. With reloadPerViewport off, later viewports resize the already loaded page instead: faster and fewer requests, but responsive behavior that only runs at page load (menus, sliders, "is this mobile?" checks) may not match a real device. List the viewport you care about most first when reloadPerViewport is off.
• One browser context is shared by the whole capture, so cached assets (CSS, JS, fonts, images) are reused across paths and viewports.
• concurrency paths run in parallel, each on its own page. The default of 1 captures one path at a time.

Waiting for each page. Each step has its own limit; the limits add up, and navigationTimeout is not an overall cap:

1. page.goto waits for domcontentloaded, up to navigationTimeout. Exceeding it is a page-load failure.
2. body must become visible, up to the smaller of navigationTimeout and 15 seconds. Exceeding it is a page-load failure.
3. The load event, up to loadEventTimeout, then network idle, up to networkIdleTimeout. Capture moves on when either runs out.
4. The asset wait, up to assetLoadTimeout: every stylesheet (including ones added by scripts after load) and every image that is loading must finish or fail, then web fonts must be ready and every CSS background image used by a rendered element must load. Lazy images far outside the viewport are skipped here and caught after scrolling. Assets that fail to load count as done, so a broken URL does not hold up capture.
5. A fixed screenshotDelay pause (default 0).
6. With scrollBeforeCapture on (the default), capture scrolls to the bottom of the page in steps and back to the top, waits two animation frames, runs the asset wait again for lazy-loaded and scroll-triggered content, then a fixed postScrollDelay pause (default 0).
7. Screenshots, one per region.
8. A fixed pathDelay pause before that worker starts its next path (default 0).

• A resized viewport (reloadPerViewport off) waits two animation frames for layout, then runs the asset wait and screenshotDelay instead of reloading.
• When the asset wait runs out, capture still takes the screenshots, marks the page view assetsTimedOut: true in the manifest, and prints (assets still loading) on its progress line. Those screenshots may show unstyled or partly loaded content; raise assetLoadTimeout or check the page.
• The delays always wait their full time, on every page view. The asset wait makes screenshotDelay and postScrollDelay unnecessary for CSS and images; use them only for content the asset wait cannot see, such as animations, transitions, or script-rendered content.
• pathDelay paces requests between paths; it does not delay screenshots.
• Network idle means no requests for 500 ms. Pages with analytics beacons, chat widgets, or polling never go idle, so they always use the full idle wait. The 100 ms default relies on the load event and the asset wait instead. Raise it for pages that render content from late XHR/fetch responses.
• Faster capture means more requests per second to the target. If the target sits behind a WAF or rate limiter, keep concurrency at 1, raise pathDelay to slow capture down, and check the capture summary for 403/429 responses.

Regions:

• Each page view is screenshotted once per region in the set's regions (every region in visual.regions when the set does not list them), in that order.
• A region whose selector is @page is a full-page screenshot of the whole scrollable page, not just the part inside the viewport. Any other value is a CSS selector, and the first matching element is captured in full.

Page-load failures and retries:

• A page view that fails to load (navigation timeout, connection error, or body never becoming visible) does not stop the capture. It is logged and capture continues.
• With reloadPerViewport off, a failed first load fails every viewport of that path, since there is no loaded page to resize.
• After all paths finish, capture runs up to failedLoadRetries retry passes over only the failed page views, grouped by path. The first viewport retried for a path is a fresh page load. Retrying at the end gives slow servers, cold caches, and network blips time to clear, and keeps one slow page from holding up the rest.
• Page views that still fail after the last pass are written to the manifest with status: null, ok: null, an error message, and no files. qaw visual compare then reports every region of that page view as missing ("Unavailable" in the report).
• Regions that are absent on a page that did load (selector matched nothing, element not visible, or screenshot error) are also left out of files and are reported as missing by compare. They are not retried.

Capture prints progress as it runs:

• A header with the snapshot name, base URL, set, and host key, path and viewport counts, regions, and the resolved capture settings.
• One line per page view: [desktop 12/245] 200 1.4s /about-us, with the HTTP status (--- when there was no response), time spent on that viewport, and path. Viewports captured by resizing are marked (resized); regions that produced no screenshot are appended as (missing: main); page views whose asset wait ran out are marked (assets still loading).
• A failed page view prints ERR in place of the status, followed by the error: [desktop 13/245] ERR 20.0s /slow-page: page.goto: Timeout 20000ms exceeded.
• Retry pass 1/1: 4 failed page views... before each retry pass, followed by the same per-page lines for the retried page views.
• A final summary: Captured 732/735 page views in 27m40s: 3 failed to load, 1 non-OK responses, 2 with missing regions, 4 with assets still loading., then the page views that failed after retry, those with non-OK HTTP responses, and those with assets still loading (the first 20 of each; the rest are in the manifest).

Counters show each path's position in the CSV, including during retry passes. With concurrency above 1, lines from different paths interleave.

The capture manifest (visual-regression/runs/<snapshot>/manifest.json) records the snapshot, target, suite, set, hostKey, persona (null when none), base URL, CSV path, regions, viewports, and captureSettings (concurrency, reloadPerViewport, scrollBeforeCapture, pathDelay, navigationTimeout, loadEventTimeout, networkIdleTimeout, assetLoadTimeout, screenshotDelay, postScrollDelay, failedLoadRetries). Each entry records the path, viewport and size, freshLoad (false for resized viewports), HTTP status and ok (a resized viewport reuses the values from its path's page load), finalUrl, loadDurationMs, CDN cache headers, the region files, assetsTimedOut: true when the asset wait ran out, and error when the page view failed to load.

qaw visual compare

Direct visual compare entrypoint.

qaw visual compare --before <snapshot> --after <snapshot> --suite <suite> [--zip true] [--tolerance <percent>]

Older positional form is still supported:

qaw visual compare <beforeSnapshot> <afterSnapshot> <suite> [--zip] [--tolerance <percent>]

--tolerance (alias --mismatch-tolerance) sets the mismatch tolerance for this compare only. It must be a non-negative number; anything else stops the compare with an error before any images are read.

Compare always saves the report data bundle. If a visual renderer is installed, qaw also renders HTML and opens the report automatically. If no renderer is available, qaw prints a message telling you to install @lockweb/qa-report or configure another visual report package.

Both snapshots must use the same visual set, the same CSV path, the same regions, and the same persona (compare stops with an error when they differ). The persona is saved in report.json and shown in the rendered report header. Page views are paired by path and viewport name; rows follow the after snapshot's viewport order. A page view present in the after snapshot but not the before snapshot is reported as missing.

Compare prints progress as it runs:

• A header with both snapshots and their targets, the page view count, regions, tolerance, and persona.
• One line per page view, in the after snapshot's order: [desktop 12/735] changed 3.21% /about-us (main 3.21%, footer 0.40%). The status is matched, changed, or missing; the percentage is the row's headline mismatch (-- when nothing could be compared); changed regions are listed with their mismatch and missing regions by name.
• A final summary: Compared 735 page views in 1m12s: 700 matched, 30 changed, 5 unavailable., followed by the ten largest changes.

Grading:

• Each region is graded on its own. A region is fail only when its pixel mismatch percent is strictly greater than the mismatch tolerance, otherwise pass.
• The tolerance resolves in this order: the --tolerance flag (or the value entered at the menu prompt), then the VISUAL_MISMATCH_TOLERANCE environment variable, then the mismatchTolerance of the set the snapshots were captured with, then 0 (any differing pixel fails). An invalid --tolerance is an error; an env value that is not a non-negative number is ignored.
• A region with no capture on either side is missing. A row is fail when any region fails, otherwise missing when any region is missing, otherwise pass.
• A row's headline mismatchPercent is the highest mismatch among its compared regions. Missing regions keep a sentinel mismatchPercent of 100 in the data but never count toward the headline, so a dropped capture shows as missing instead of a 100% mismatch.
• report.json records the visual set the snapshots were captured with.
• The tolerance used is saved as mismatchTolerance in report.json. The rendered report opens at that tolerance and has a Tolerance control for regrading it live in the browser, so the right value can be found while reviewing. report.json also lists the viewports compared.

Tolerance is only applied at compare time, so the same two snapshots can be compared repeatedly with different tolerances without re-capturing:

qaw visual compare --before <snapshot> --after <snapshot> --suite <suite> --tolerance 0.05

Each compare of the same snapshot pair writes to the same report directory and replaces the previous report, so copy it first if you want to keep both.

qaw update

Updates @lockweb/qa-workflow in the consuming project and refreshes Playwright browser binaries for the installed version.

qaw update [--version 0.2.2]

Flags:

• --version Install a specific version instead of latest

qaw status

Shows package and Playwright runtime state.

qaw status [--json]

Flags:

• --json Emit JSON instead of plain text

Project Setup

For a one-off setup command:

npx qaw init

qaw init can:

• ask where workflow files should live and set paths.rootDir
• create qa-workflow.config.json
• create a sample visual regression CSV at <rootDir>/visual-regression/sample.csv
• create playwright.config.js
• add a local npm script such as npm run qaw

playwright.config.js is required for running tests through qa-workflow. It scopes Playwright to the package-managed test directory so Playwright does not auto-discover unrelated *.spec.* files elsewhere in the repo.

It should also set outputDir inside the workflow root so Playwright writes .last-run.json and other run artifacts under the workflow workspace instead of the repo root.

After the package is installed in a project, the same commands can be run through the local binary or a repo script.

The package also includes a config validation command:

npx qaw validate --config ./qa-workflow.config.json

qaw validate checks:

• config file presence and JSON parsing
• consumer playwright.config.js presence
• suites structure
• target capability fields and host definitions
• referenced CSV file paths
• persona support module paths when personas are configured
• selector and boolean field types
• basic path configuration sanity

Configuration

The tool is structured around project-level configuration rather than hardcoded project logic.

Example:

{
  "paths": {
    "rootDir": "."
  },
  "suites": {
    "public-suite": {
      "personas": {
        "admin": {
          "label": "Administrator",
          "username": "alice",
          "password": "example-password",
          "authStartPath": "/admin/people?user=alice"
        },
        "editor": {
          "label": "Editor",
          "accountId": "editor-123",
          "authStartPath": "/admin/people?user=editor-123"
        }
      },
      "personaSupport": {
        "module": "qa-workflow/support/persona-switch.cjs"
      },
      "tests": {
        "targets": ["stage", "prod"],
        "recordTargets": ["stage"],
        "personaTestMode": "parallel"
      },
      "visual": {
        "regions": {
          "full": "@page",
          "header": "header",
          "main": "main",
          "footer": "footer"
        },
        "viewports": {
          "desktop": { "width": 1440, "height": 2200 },
          "tablet": { "width": 1024, "height": 1800 },
          "phone": { "width": 430, "height": 2200 }
        },
        "sets": {
          "public-pages": {
            "csv": "visual-regression/public-suite/public-pages.csv",
            "host": "site.origin",
            "persona": null,
            "targets": ["stage", "prod"],
            "regions": ["full", "header", "main", "footer"],
            "viewports": ["desktop", "tablet", "phone"],
            "reloadPerViewport": true,
            "scrollBeforeCapture": true,
            "concurrency": 1,
            "pathDelay": 200,
            "navigationTimeout": 20000,
            "loadEventTimeout": 15000,
            "networkIdleTimeout": 100,
            "assetLoadTimeout": 5000,
            "screenshotDelay": 0,
            "postScrollDelay": 0,
            "failedLoadRetries": 1,
            "mismatchTolerance": 0.01
          },
          "editor-pages": {
            "csv": "visual-regression/public-suite/editor-pages.csv",
            "host": "content",
            "persona": "editor",
            "targets": ["stage"],
            "regions": ["main"],
            "viewports": ["desktop"],
            "screenshotDelay": 500
          }
        }
      },
      "targets": {
        "stage": {
          "writesContent": true,
          "hosts": {
            "content": {
              "baseUrl": "https://content.stage.example.com",
              "requiresLogin": true,
              "perDomainAuth": true,
              "default": true
            },
            "site.origin": {
              "baseUrl": "https://origin.stage.example.com",
              "requiresLogin": false
            }
          }
        },
        "prod": {
          "writesContent": false,
          "hosts": {
            "content": {
              "baseUrl": "https://content.example.com",
              "requiresLogin": true,
              "perDomainAuth": true,
              "default": true
            },
            "review": {
              "baseUrl": "https://review.example.com",
              "requiresLogin": true,
              "perDomainAuth": false
            },
            "site.public": {
              "baseUrl": "https://www.example.com",
              "requiresLogin": false,
              "perDomainAuth": false
            },
            "site.origin": {
              "baseUrl": "https://origin.example.com",
              "requiresLogin": false
            }
          }
        }
      }
    }
  }
}

Each suite groups its settings by purpose:

• targets, personas, and personaSupport are shared by test runs, recording, auth, and visual regression.
• tests holds settings used only by test runs.
• visual holds everything used only by visual regression capture and compare.

Config Reference

Configuration fields:

• paths.rootDir Base workflow folder inside the consuming repo. Standard subpaths are derived from this automatically.
• paths.testsDir Optional override for the tests root. Default: <rootDir>/test
• paths.recordedDir Optional override for raw recorded specs. Default: <testsDir>/recorded
• paths.specsDir Optional override for published specs. Default: <testsDir>/published
• paths.supportDir Optional override for consumer-owned support files such as persona switch modules. Default: <testsDir>/support
• paths.authDir Optional override for saved auth state. Default: <rootDir>/auth/.auth
• paths.visualDir Optional override for visual regression assets. Default: <rootDir>/visual-regression
• paths.visualRunsDir Optional override for captured visual runs. Default: <visualDir>/runs
• paths.visualReportsDir Optional override for generated visual reports. Default: <visualDir>/reports
• reporting.visual.renderer Optional report renderer package or path. Default: @lockweb/qa-report
• suites Ordered collection of logical test buckets. The first suite listed is treated as the default suite.
• suites.<suite>.personas Optional ordered map of persona keys to persona metadata. Extra fields are allowed and are passed through to the consumer persona support module unchanged.
• suites.<suite>.personas.<personaKey>.authStartPath Optional path to open when qaw auth sets a persona-auth state for that persona. This is resolved against the selected host base URL. Default: /
• suites.<suite>.personaSupport.module Optional project-relative path to the consumer-provided persona switch module. This module should export switchPersona(page, personaKey, persona, context) and may export resetPersona(page, context).
• suites.<suite>.targets Ordered map of the suite's environments. Which features can use each target is set by the feature target lists (tests.targets, tests.recordTargets, and each visual set's targets); when a list is not set, the feature can use every target, in this order.
• suites.<suite>.targets.<target>.writesContent Optional boolean. Default: true. Controls whether tests marked as content-writing are allowed to run on that target.
• suites.<suite>.targets.<target>.hosts Required non-empty object of host definitions for that target.
• suites.<suite>.targets.<target>.hosts.<hostKey>.baseUrl Required base URL for that host.
• suites.<suite>.targets.<target>.hosts.<hostKey>.aliases Optional array of additional absolute origins that should normalize to the same host key during publish. Use this for canonical-domain pairs such as https://example.test and https://www.example.test.
• suites.<suite>.targets.<target>.hosts.<hostKey>.requiresLogin Required login mode for that host. Use true for login required, "optional" for login available but not required, or false for no login handling. This is the only login setting: a target needs login (for test runs, the smoke run, and visual capture) when any of its hosts sets true or "optional". Each login-managed host has its own saved auth state (<authDir>/<target>--<hostKey>.json). Test runs, recording, and visual capture load a combined auth state that merges the saved auth of every login-managed host on the target. Shared login, where one auth cookie is accepted by several hosts (single sign-on, or a session cookie on a shared parent domain), is not currently supported. It can be achieved by setting requiresLogin on each host that shares the login, saving auth once with qaw auth for one of those hosts, and copying that auth state file to the file name of each other host, for example cp auth/.auth/stage--site.admin.json auth/.auth/stage--site.review.json. For hosts with perDomainAuth, copy each persona file (<target>--<hostKey>--<persona>.json) the same way. Repeat the copy whenever the saved auth is refreshed.
• suites.<suite>.targets.<target>.hosts.<hostKey>.default Required on exactly one host per target. Used as the default start host and Playwright base URL for that target, and as the visual capture host for a visual set without host.
• suites.<suite>.targets.<target>.hosts.<hostKey>.perDomainAuth Optional boolean. When true, that host uses persona-specific auth state files instead of the shared host auth state. Default: false. Only enable it on hosts that need persona-backed auth.

Test run settings (suites.<suite>.tests):

• tests.targets Optional ordered array of target names that test runs can use. The first is the default target in the run menu. An empty array turns test runs off for the suite. Default: every target, in targets order.
• tests.recordTargets Optional ordered array of target names that recording can use. The first is the default target in the record menu. Default: every target, in targets order.
• tests.personaTestMode Optional mode for persona-generated tests: serial or parallel. Default: serial

Visual regression settings (suites.<suite>.visual). Regions and viewports are defined once under visual; each set refers to them by name, the same way it refers to targets, hosts, and personas.

• visual.regions Optional ordered map of region names to what each region captures. "@page" is a full-page screenshot of the whole scrollable page; any other value is a CSS selector, and the first matching element is captured in full. Default: { "full": "@page" }
• visual.viewports Optional ordered map of viewport names to { "width", "height" }, positive integers. Default: desktop 1440x2200, tablet 1024x1800, phone 430x2200.
• visual.sets Ordered map of visual set names to sets. The first set listed is the default set for the suite. Each set is an object with:
  • csv: required project-relative CSV file path.
  • host: optional host key to capture, for example an origin host that bypasses a CDN or a CMS host, without changing the base URL used by test runs. It must exist in the hosts of every target in the set's targets. Default: each target's default host.
  • persona: optional persona key from personas to capture as, or null for none. The set's host must have perDomainAuth: true on every target in the set's targets. Default: none (the host's base login, or no login when the host does not require one).
  • any of the settings below.

Each setting resolves in this order: the matching capture flag, then the set, then the built-in default. There are no shared defaults between sets; spell a setting out on each set that needs a non-default value.

Set settings. Settings named ...Timeout are limits a step may end before; settings named ...Delay always wait their full time.

• targets Optional ordered array of target names the set can capture. The first is the default target in the visual menu. An empty array turns the set off. Default: every target, in targets order.
• regions Optional ordered array of names from visual.regions to capture, in this order. Each region becomes its own report section. Default: every defined region, in config order.
• viewports Optional ordered array of names from visual.viewports to capture, in this order. The first viewport is always a fresh page load. Default: every defined viewport, in config order.
• reloadPerViewport Optional boolean. true gives every viewport its own page load; false resizes the page loaded for the first viewport. Default: true
• scrollBeforeCapture Optional boolean. Scroll through each page before screenshots so lazy-loaded and scroll-triggered content renders. Default: true
• concurrency Optional positive integer. Number of CSV paths captured in parallel. Default: 1
• pathDelay Optional non-negative integer. Fixed pause in milliseconds between paths, for pacing requests. Default: 0
• navigationTimeout Optional positive integer. Maximum time in milliseconds for a page to reach DOMContentLoaded; past it the page view fails to load. Default: 20000
• loadEventTimeout Optional non-negative integer. Maximum wait in milliseconds for the page's load event; 0 skips the wait. Default: 15000
• networkIdleTimeout Optional non-negative integer. Maximum wait in milliseconds for the network to go idle after each page load; 0 skips the wait. Default: 100
• assetLoadTimeout Optional non-negative integer. Maximum wait in milliseconds for stylesheets, images, web fonts, and CSS background images, after each page load or resize and again after scrolling; 0 skips the wait. Default: 5000
• screenshotDelay Optional non-negative integer. Fixed pause in milliseconds after the asset wait, before scrolling or screenshots. Default: 0
• postScrollDelay Optional non-negative integer. Fixed pause in milliseconds after scrolling back to the top, before screenshots. Default: 0
• failedLoadRetries Optional non-negative integer. End-of-run retry passes for page views that failed to load; 0 disables retries. Default: 1
• mismatchTolerance Optional non-negative number, in percent, used by qaw visual compare. A region fails only when its mismatch percent is strictly greater than this value, which absorbs sub-pixel antialiasing and timing noise. The VISUAL_MISMATCH_TOLERANCE environment variable or qaw visual compare --tolerance overrides it. It sets compare's grading and summary output and the rendered report's starting Tolerance, which reviewers can change in the report. Default: 0

Moved and renamed settings: qaw validate rejects older keys and names their replacement.

• Suite level: csvSets moved to visual.sets (each entry's path is now csv); selectors, regions, and full became visual.regions (use "@page" for the full page); visualCaptureMode, visualCaptureWait, visualCaptureTimeout, visualCaptureIdleTimeout, visualCaptureRetries, and visualMismatchTolerance became the set settings concurrency, pathDelay, navigationTimeout, networkIdleTimeout, failedLoadRetries, and mismatchTolerance; personaTestMode moved to tests.
• Targets: the per-target run, record, and visual flags became the tests.targets and tests.recordTargets lists and each visual set's targets. defaultRequiresLogin was removed: each host's requiresLogin decides whether login is needed. The host-level visualDefault flag is replaced by host on each visual set.
• Visual block: visual.csvSets became visual.sets; visual.defaults was removed (put its settings on each set); visual.host was removed (set host on each set); other settings directly under visual, such as visual.targets or visual.concurrency, moved to each set; visual.viewports changed from an array of { "name", "width", "height" } to a map of name to { "width", "height" }; a set's regions and viewports are now lists of names.
• Set settings renamed: wait → pathDelay, timeout → navigationTimeout, idleTimeout → networkIdleTimeout, assetTimeout → assetLoadTimeout, settle → screenshotDelay, scrollSettle → postScrollDelay, retries → failedLoadRetries, reload → reloadPerViewport, scroll → scrollBeforeCapture.

A suite may only contain targets, personas, personaSupport, tests, and visual; any other suite-level key, and any unknown key inside visual, a visual set, or tests, is rejected so a misplaced setting is never silently ignored.

Default behavior:

• first suite listed = default suite
• first target in a feature's target list (tests.targets, tests.recordTargets, a visual set's targets) = that feature's default target; without a list, the first target under targets
• first set listed in visual.sets = default visual set
• a visual set without regions or viewports captures every defined region or viewport; without visual.regions, capture takes one full-page screenshot per page view
• a feature without a target list can use every target
• writesContent defaults to true unless a target sets it to false

This lets a project keep the config minimal while still allowing any path or suite-specific behavior to be overridden explicitly.

Persona behavior:

• if a suite does not define personas and persona support, publish does not ask about personas
• if a suite defines personas and either a persona support module or a host with perDomainAuth enabled, publish offers those personas
• if a test references a persona that no longer exists in suite config, runtime fails with a clear message
• if a test does not set a persona:
  • no auth required = anonymous user
  • auth required = base logged-in user
• hosts that do not opt into perDomainAuth continue to use the shared host auth state

Auth status behavior:

• shared host auth shows the last saved timestamp for that host
• persona-auth hosts show how many persona auth states are saved for that host

Persona test API

Published specs can use createPersonaTest from @lockweb/qa-workflow/support/persona-test.

Persona-generated tests preserve the original spec callsite in Playwright UI and reports, so shared helpers do not shift those tests under the package's own node_modules path.

Shared-loop pattern:

import { test } from '@playwright/test';
import { createPersonaTest } from '@lockweb/qa-workflow/support/persona-test';

const personaTest = createPersonaTest(test, [
  { user: 'group_creator_osc' },
  { user: 'group_publisher_osc' },
]);

personaTest('shared flow', async ({ page, persona }) => {
  if (persona.user === 'group_creator_osc') {
    // creator branch
  }

  if (persona.user === 'group_publisher_osc') {
    // publisher branch
  }
});

Single-persona pattern:

import { test } from '@playwright/test';
import { createPersonaTest } from '@lockweb/qa-workflow/support/persona-test';

createPersonaTest(test, [{ user: 'group_creator_osc' }])(
  'creator-only flow',
  async ({ page }) => {
    // creator-only steps
  }
);

When personas are active, the callback fixtures include persona with:

• persona.key Persona key used for lookup in suite config
• persona.user Persona user value from the test declaration
• persona.record Full persona record from suite config
• persona.label Resolved display label used in the generated test title
• persona.expectation Optional pass or fail expectation from the test declaration

Auth host behavior:

• hosts with requiresLogin: true appear in Auth and are treated as login required
• hosts with requiresLogin: "optional" appear in Auth and are treated as login optional
• hosts with requiresLogin: false are excluded from Auth and other auth status surfaces

Repo Boundary

qa-workflow is the shared engine. Consuming repositories own their local setup and assets.

Package-owned:

• CLI entrypoints
• config loading and default resolution
• auth orchestration
• run / record / publish logic
• visual capture, compare, and reporting
• generic persona orchestration
• sanitized scaffolding in templates/

Consumer-owned:

• qa-workflow.config.json
• playwright.config.js
• suite definitions and target URLs
• visual regression CSV files
• published specs under test/published
• recorded specs under test/recorded
• saved auth state
• persona switch modules and other project-specific support code

If playwright.config.js is missing, qaw run may cause Playwright to scan unrelated tests in the consuming repo. The simplest fix is to run npx qaw init and create it.

This package should not contain real consumer target URLs, real consumer CSVs, published specs, recorded specs, auth state, or project-specific persona switch logic.

Maintainer Notes

The sections below are for package maintenance, not for consuming projects.

Package Layout

Current package structure:

• bin/ Thin CLI entrypoints such as qaw
• src/cli/ Command parsing, menu flow, setup, validation, and maintenance commands
• src/config/ Config loading, defaults, and target resolution
• src/auth/ Auth state and login helpers
• src/run/ Record, publish, run, and listing flows
• src/visual/ Visual capture, compare, report generation, and visual menu flow
• src/support/ Package-owned support helpers such as persona orchestration
• templates/ Sanitized files used by qaw init

Local Development

Common local development commands in this repo:

npm run qaw
npm run smoke:cli
npm test
npm run record
npm run run:interactive
npm run visual:interactive

The short CLI name exposed by the package is:

qaw

The package repo also includes a smoke check for CLI and config loading:

npm run smoke:cli

Maintainer-facing package tests run with:

npm test

Published Package Contents

Published package contents are intentionally limited to:

• bin/
• src/
• templates/
• README.md
• LICENSE