playwright-visual-diff-manifest

Manifest-driven pixel visual regression for web apps. Three small pieces:

1. A page + breakpoint manifest — one typed list of pages to capture, each at a canonical 5-tier responsive grid (base 375 / sm 640 / md 768 / lg 1024 / xl 1280).
2. A pure PNG comparator — pixelmatch over pngjs, no fs, no browser. Two buffers in, mismatch count + ratio + rendered diff PNG out.
3. A baseline orchestrator — reconciles a captured frame against a committed baseline directory: update mode writes baselines, compare mode diffs and dumps .actual.png / .diff.png artifacts on failure.

The screenshotting itself is not in this package. You bring the browser — typically a Playwright spec that iterates the manifest. This keeps the library pure and deterministic, and keeps browser concerns (viewports, waits, masking, animation kill) in your test code where you can see them.

The breakpoint-manifest concept

Most visual-regression setups accumulate screenshots ad hoc, one toHaveScreenshot() per test. The manifest inverts that: a single data structure declares what is captured and at which widths, and one generic spec iterates it. Adding a page to coverage is one entry, and it is automatically exercised at all five breakpoints — so a reflow regression at any tier is a red frame, not a silent layout break.

Each target declares:

• id — stable, becomes the baseline filename (<id>-<breakpoint>.png)
• path — the route to capture
• waitFor — a data-testid that must be visible before capture (proves the hydrated body rendered; never screenshot a half-painted frame)
• mask (optional) — data-testid substrings whose elements are blanked before capture, so non-deterministic regions (timestamps, generated references) can never flip a frame red
• breakpoints (optional) — a per-target subset when a page does not need the full grid

Install

npm install --save-dev github:bayraak/playwright-visual-diff-manifest   # not on npm yet; install from GitHub

pixelmatch and pngjs come with it. Playwright (or whatever takes your screenshots) is yours to install.

Wiring it into a Playwright spec

Define your manifest once:

// e2e/visual/targets.ts
import { defineVisualTargets } from 'playwright-visual-diff-manifest'

export const VISUAL_TARGETS = defineVisualTargets([
  { id: 'home', path: '/', waitFor: 'home-body' },
  { id: 'pricing', path: '/pricing', waitFor: 'pricing-page' },
  { id: 'checkout', path: '/checkout', waitFor: 'checkout-view' },
  // Mask a region that renders a generated reference string:
  { id: 'confirmation', path: '/confirmation', waitFor: 'confirmation-view', mask: ['order-ref'] },
])

Then one spec iterates the grid:

// e2e/visual/visual-regression.spec.ts
import { fileURLToPath } from 'node:url'
import { dirname, join } from 'node:path'

import { expect, test } from '@playwright/test'
import {
  type BaselineDirs,
  baselineName,
  breakpointsFor,
  describeOutcome,
  reconcileFrame,
} from 'playwright-visual-diff-manifest'

import { VISUAL_TARGETS } from './targets'

const HERE = dirname(fileURLToPath(import.meta.url))
const DIRS: BaselineDirs = {
  baselineDir: join(HERE, 'baselines'),
  artifactsDir: join(HERE, '..', '..', 'test-results', 'visual-artifacts'),
}
const UPDATE = process.env.VISUAL_UPDATE === '1'

// Kill animation/transitions/caret so no frame catches a mid-animation pixel.
const DETERMINISM_CSS = `
  *, *::before, *::after {
    animation-duration: 0s !important;
    animation-delay: 0s !important;
    transition-duration: 0s !important;
    transition-delay: 0s !important;
    caret-color: transparent !important;
    scroll-behavior: auto !important;
  }
`

for (const target of VISUAL_TARGETS) {
  for (const bp of breakpointsFor(target)) {
    const frame = baselineName(target.id, bp.name)

    test(`visual: ${target.id} @ ${bp.name} (${bp.width})`, async ({ page }) => {
      // Freeze randomness/time so client-derived strings never flip a frame.
      await page.addInitScript(() => {
        const FIXED = 1_700_000_000_000
        Date.now = () => FIXED
        Math.random = () => 0.42
      })

      await page.setViewportSize({ width: bp.width, height: bp.height })
      await page.goto(target.path, { waitUntil: 'networkidle' })

      // Prove the hydrated body rendered before capturing.
      await expect(page.getByTestId(target.waitFor)).toBeVisible({ timeout: 15_000 })

      // Init scripts run pre-navigation; re-inject the animation kill into the live tree.
      await page.addStyleTag({ content: DETERMINISM_CSS }).catch(() => {})

      // Blank the manifest's volatile regions so they can't redden a frame.
      await Promise.all(
        (target.mask ?? []).map((m) =>
          page
            .locator(`[data-testid*="${m}"]`)
            .evaluateAll((els) => {
              for (const el of els) (el as HTMLElement).style.visibility = 'hidden'
            })
            .catch(() => {}),
        ),
      )

      await page.waitForLoadState('networkidle')
      const actual = await page.screenshot({ fullPage: true, animations: 'disabled' })

      const outcome = reconcileFrame(frame, actual, DIRS, { update: UPDATE })

      if (UPDATE) {
        expect(['created', 'updated']).toContain(outcome.status)
        return
      }

      // A missing baseline is a real failure in gate mode — fail loudly with the
      // exact reason and the regeneration hint.
      expect(outcome.status, describeOutcome(outcome)).toBe('pass')
    })
  }
}

Workflow:

VISUAL_UPDATE=1 npx playwright test e2e/visual   # (re)generate committed baselines
npx playwright test e2e/visual                   # gate against them

Commit the baselines/ PNGs. On a failing frame, the actual and rendered diff land in your artifacts directory for inspection (upload it as a CI artifact).

Determinism notes, learned the hard way:

• Capture a built, statically served site where possible, not a dev server — dev overlays and streaming render differently run-to-run.
• Freeze Date.now / Math.random in an init script, kill animations with CSS, and mask anything derived from a clock. The frame should be a pure function of layout + theme.
• A footer copyright year that changes annually is usually far below the 0.5% mismatch budget — measure before masking.

API

Manifest

• BREAKPOINTS: readonly Breakpoint[] — the canonical grid: base 375, sm 640, md 768, lg 1024, xl 1280, all at height 1600 (full-page screenshots ignore height; a fixed value keeps above-the-fold captures deterministic).
• interface Breakpoint { name, width, height }
• interface VisualTarget { id, path, waitFor, mask?, breakpoints? }
• defineVisualTargets(targets): readonly VisualTarget[] — validates and returns the manifest. Throws on a duplicate/empty id, a path without a leading /, or an empty waitFor.
• breakpointsFor(target): readonly Breakpoint[] — the target's own subset, or the full grid.
• baselineName(targetId, breakpointName): string — <id>-<breakpoint>.png.

Comparator

• comparePng(baseline: Buffer, actual: Buffer, options?): CompareResult — decodes and pixel-diffs two PNG buffers. A dimension mismatch is a hard fail (ratio: 1, diff: null); it is never silently passed.
  • options.pixelThreshold — pixelmatch per-pixel threshold, default 0.1 (DEFAULT_PIXEL_THRESHOLD).
• interface CompareResult { dimensionsMatch, mismatch, ratio, total, width, height, diff } — diff is a rendered diff PNG buffer (mismatched pixels painted red).
• isWithinBudget(result, mismatchRatio?): boolean — pass/fail against the mismatch-ratio budget, default 0.005 (DEFAULT_MISMATCH_RATIO — 0.5% absorbs font-hinting/AA jitter while catching real layout or color drift).

Baseline orchestrator

• reconcileFrame(name, actual: Buffer, dirs: BaselineDirs, options?): FrameOutcome
  • options.update: true → (over)write the baseline; status created | updated.
  • options.update: false (default) → diff; status pass | fail | missing-baseline. On fail, writes <name>.actual.png and <name>.diff.png under dirs.artifactsDir.
  • options.mismatchRatio — per-call budget override.
• interface BaselineDirs { baselineDir, artifactsDir }
• interface FrameOutcome { name, status, baselinePath, result }
• describeOutcome(outcome): string — one-line human-readable summary for test failure messages (includes the mismatch percentage, or the regeneration hint for a missing baseline).

The optional Playwright runner

The core package never imports Playwright. If your suite uses Playwright, the playwright-visual-diff-manifest/playwright entry runs the whole discipline for you against a page your test already owns:

import { test, expect } from '@playwright/test'
import { runVisualTarget } from 'playwright-visual-diff-manifest/playwright'
import { targets } from './targets'

for (const target of targets) {
  test(`visual: ${target.id}`, async ({ page }) => {
    const outcomes = await runVisualTarget(page, target, 
      { baselineDir: 'e2e/visual/baseline', artifactsDir: 'test-results/visual' },
      { baseURL: 'http://localhost:4173', update: process.env.VISUAL_UPDATE === '1' })
    for (const o of outcomes) expect.soft(o.status, o.name).toMatch(/pass|created|updated/)
  })
}

Per frame it sets the breakpoint viewport, navigates, waits for the target's waitFor test-id, masks the target's mask test-ids via Playwright's native screenshot masking, captures full-page with animations disabled, and reconciles against the baseline. The page parameter is typed structurally, so playwright, playwright-core, and @playwright/test all work. @playwright/test is declared as an optional peer dependency — installing it is only needed for this entry, never for the core.

Agent skill

The repo ships a SKILL.md for AI agents: manifest authoring, the spec skeleton, the baseline update flow, masking and determinism rules, and how to read a failing diff. Point an agent at the repo and it picks the skill up, or copy the repo folder into your agent's skills directory (e.g. ~/.claude/skills/).

License

MIT