found-pixel

Storybook build + visual regression workflow for Buildkite, in one CLI.

Each consumer repo drops in a .storybook/found-pixel.config.json and one pipeline step instead of copying shell + Node scripts between repos.

What it does

On every build:

1. Runs the configured storybook build command
2. Uploads the static site, then the archive, to S3 under <repoName>/storybook/<branch-slug>/ (the archive marks a complete upload)
3. Annotates the Buildkite build with a link to the hosted storybook, and posts a storybookStatus commit status linking to it, e.g. "3722 stories published" (with a GitHub token)
4. If the branch is the default branch, also uploads under the commit SHA for future base resolution, and updates the shared baseline screenshots for PRs (see Baselines)

On pull requests (additional to the above):

5. Resolves a base storybook from S3, waiting up to ~5 minutes for the merge-base SHA (PRs into the default branch) or the base branch slug (stacked PRs). If it never arrives, builds it locally from the merge-base, unless a lockfile changed since then (the PR's installed dependencies would hide changes they cause); in that case, or if the build fails, it falls back to the default branch. A rebuilt base isn't uploaded. The PR comment flags either case as a partial failure.
6. Runs storybook-diff between base and head to rule out stories the change can't affect, writing the JSON to artifacts/; if storybook-diff fails, every story is checked instead
7. Screenshots the changed stories with Playwright (see Screenshots), compares them against the base, and re-shoots differences in isolation before reporting them; uploads screenshots to S3 and constructs PR comment image URLs via screenshots.baseUrl (if configured)
8. Upserts a PR comment with added / modified / deleted sections, thumbnail previews for added, visually changed and deleted stories, and a footer counting candidate stories and differences, stories skipped because only ignoreModules changed them (listed in results.json as ignored, with a hint to re-run with the first fullReviewLabels label) and flaky differences; or a caution when visual regression fails, and posts a UI Review commit status (see UI Review status)
9. Annotates the Buildkite build with a diff summary and a link to the PR comment
10. Rebuilds index.json at the root of screenshots.uploadPrefix from every repo's <repoName>/results/*.json modified in the last 31 days in the screenshots bucket, so diff viewers can list all available comparisons (best effort: failures only warn)

Install

In the consumer repo:

npm install --save-dev found-pixel
# or
yarn add --dev found-pixel

Runtime requirements on the Buildkite agent:

• Node 20+
• aws CLI v2, released 2025 or later (for S3 reads/writes, including conditional writes)
• git (for merge-base resolution)
• buildkite-agent (for annotations and artifact upload)

Chromium is installed on each review and baseline run via playwright-core install --with-deps --only-shell chromium. playwright-core is pinned, so every build uses the same Chromium; bumping it makes the next default-branch build re-shoot the whole baseline.

Configuration

Create found-pixel.config.json in one of the following locations (searched in order, first match wins):

1. ./found-pixel.config.json (repo root)
2. ./.storybook/found-pixel.config.json
3. ./.github/found-pixel.config.json
4. ./tools/storybook/found-pixel.config.json

Example:

{
  "defaultBranch": "master",
  "buildCommand": "npm run build-storybook",
  "outputDir": "storybook-static",
  "artifactsDir": "artifacts",
  "s3": {
    "bucket": "my-org-storybook",
    "region": "ap-southeast-2"
  },
  "screenshots": {
    "baseUrl": "https://d1234abcd.cloudfront.net",
    "s3": {
      "bucket": "my-org-screenshots",
      "region": "ap-southeast-2"
    },
    "uploadPrefix": "public"
  },
  "github": {
    "repo": "my-org/my-app"
  },
  "storybookDiff": {
    "ignoreModules": [
      "*/generated/*"
    ]
  },
  "visualRegression": {
    "timeoutMinutes": 120,
    "maxChangedPixels": 2,
    "pixelThreshold": 0.01,
    "includeAA": false,
    "reshootBelow": 1,
    "reshootMaxStories": 100,
    "shotConcurrency": 5,
    "baselines": true
  },
  "frozenResources": {
    "enabled": true,
    "branches": ["main", "master", "develop"],
    "match": ["^.{0,512}$", "!^https://www\\.google\\.com/"]
  },
  "comment": {
    "heading": "📘 UI Review",
    "marker": "<!-- found-pixel-comment -->",
    "viewerUrl": "https://my-org.github.io/diffy",
    "maxChars": 62000
  }
}

Required: s3.bucket, s3.region, github.repo.

Notable defaults (all optional):

• buildCommand: "npm run build-storybook"
• outputDir: "storybook-static"
• artifactsDir: "artifacts"
• defaultBranch: "master"
• visualRegression.timeoutMinutes: 120 — time budget for all screenshots in one run; stories still unshot when it runs out are reported as failed
• visualRegression.shotConcurrency: 5 — stories shot in parallel, each in its own browser context
• visualRegression.maxChangedPixels: 2 — changed pixels a shot may have and still pass. Below 1, a fraction of the shot's pixels (0.0005 = 0.05%); 1 or more, an absolute pixel count. The default absorbs the odd stray edge pixel; 0 lets nothing through. A shot whose size changed is always reported, whatever this is set to.
• visualRegression.pixelThreshold: 0.01 — per-pixel colour tolerance passed to pixelmatch, as a fraction of the maximum YIQ distance (pixelmatch's own default is 0.1). The default ignores ±1 colour rounding between renders (e.g. after a dependency bump) while counting any visible change; 0 counts every colour change. External images are frozen, so CDN re-encodes don't vary between runs, and pixelmatch already skips antialiased pixels
• visualRegression.includeAA: false — passed to pixelmatch; true counts antialiased pixels as differences instead of skipping them
• visualRegression.reshootBelow: 1 — differences up to this fraction of a shot's pixels are re-shot in isolation before they're reported; larger ones are reported straight away. 1 re-shoots every difference
• visualRegression.reshootMaxStories: 100 — when more stories than this differ, skip the re-shoot and report them all; a change that broad is real, and re-shooting it takes minutes
• visualRegression.baselines: true — keep shared baseline screenshots of the default branch and compare PRs against them (requires screenshots)
• skipLabels: [ "skip-ui-review" ] — GitHub labels (from BUILDKITE_PULL_REQUEST_LABELS) that skip the review stage (storybook is still built and uploaded)
• fullReviewLabels: [ "full-ui-review" ] — GitHub labels that force a full review: every story in the storybook is screenshotted and compared, bypassing the storybook-diff filter
• requireReview: "*[bot]" — post the UI Review status as pending instead of success when there are story changes: true for every PR, false for none, or a glob pattern (or list of them) matched case-insensitively against the PR author's login, e.g. ["*[bot]", "dusty*"] (see UI Review status)
• storybookStatus: "Storybook" — name of the commit status linking to the hosted storybook once it's uploaded; null to skip it
• codeowners: true — list the owners of changed stories in the PR comment, so a review app can invite them and require their PR approval; false to opt out (see Story owners)
• frozenResources.enabled: true — freeze resources from other origins, images by default (requires screenshots.baseUrl, see Screenshots)
• frozenResources.branches: ["main", "master", "develop"] — case-sensitive glob pattern(s) for the branches, besides defaultBranch, whose runs upload newly fetched resources; "*" lets any branch upload. Runs on every branch replay what's been uploaded
• frozenResources.contentTypes: the common image types (see frozenResources options) — content types that are frozen, exactly (image/png) or by main type (image/*). They also pick which requests are looked up: image types cover images, font types fonts, text/css stylesheets, JavaScript types scripts and audio/video types media; any other type only covers requests the browser labels other, and a type/* covers every kind its subtypes can. Add "font/*" to freeze fonts too, plus "application/octet-stream" if a CDN serves them that way
• frozenResources.match: "^.{0,512}$" — a regex, or list of them, matched against each resource's URL. A resource is frozen when its URL matches a pattern and no !-prefixed one; with only ! patterns, anything they don't match is frozen. The default skips URLs over 512 characters, which are usually tracking pixels carrying per-run IDs. Setting it replaces the default, so keep "^.{0,512}$" in the list when adding exclusions, e.g. ["^.{0,512}$", "!^https://www\\.google\\.com/"]
• comment.heading: "📘 UI Review"
• comment.marker: "<!-- found-pixel-comment -->"
• comment.maxChars: 62000
• screenshots.uploadPrefix: "public"

Options a version doesn't support, or with invalid values, are ignored with a warning, so a config written for an older found-pixel keeps working. Required options still fail.

screenshots is optional. When provided, visual regression screenshots are uploaded to screenshots.s3 under <repoName>/shots/<slug>/. Set screenshots.baseUrl to the domain serving that bucket (e.g. a CloudFront distribution) — image URLs in the PR comment are then constructed as ${baseUrl}/${key}. Set screenshots.uploadPrefix if your bucket serves content from a subdirectory (e.g. "public" uploads to public/<key> but the CDN serves it at baseUrl/<key>). Without a baseUrl, screenshots are still uploaded but no thumbnail URLs are generated. When screenshots is omitted entirely, screenshots are not uploaded.

comment.viewerUrl is optional. When set, story names and thumbnails in the PR comment link to the viewer with ?id=<storyId>&r=<repo>&a=<baseSlug>&b=<compareSlug> (plus type=added or type=deleted for those sections), with a small storybook ↗ link beside each added or modified story for the live build. When unset, names link to the hosted storybook and thumbnails to the screenshot URL.

UI Review status

With comment.viewerUrl set, every PR build posts a UI Review commit status on the head commit linking to the viewer, or to the build when it fails:

| Story changes | review not required | review required | |---|---|---| | none | success "No changes detected" | success "No changes detected" | | some | success "N added, M modified" | pending "N added, M modified" | | visual regression failed, or a story could not be screenshotted | error "Visual regression failed" | error "Visual regression failed" |

Review is required when requireReview is true, or when the PR author's login matches one of its patterns (the default "*[bot]" covers dependabot, renovate and the like); if the author can't be read it is not required. When it is, the PR comment carries a hidden <!-- ui-review fp=… --> line and the status URL &fp=<fingerprint> whenever there is something to review: an 8-character checksum of the changed stories and their current screenshots, which pushes that leave the shots unchanged keep. A story without a screenshot to hash (capture failed) makes the checksum commit-specific instead, so an approval never carries over something unseen. That is what lets a reviewer's approval of a pending status be carried forward by whatever flips it to success, and a hidden <!-- ui-approved … --> stamp such a tool adds to the PR comment is kept when found-pixel rewrites the comment. results.json always has the checksum as fingerprint.

Story owners

Each story in results.json's diff carries owners: the CODEOWNERS owners of the file it's defined in (its importPath, relative to where storybook was built), read from .github/, the repo root or docs/ like GitHub does. The key is left out when a story has no owners. results.json also lists every distinct entry as a top-level owners (left out when none), and index.json carries it for each result ([] when none). Both also record the PR author and pullTitle when GitHub returns them, and the tool that produced them as found-pixel@<version>. Each found-pixel version pins exactly one storybook-diff version.

With codeowners: true, the PR comment also carries a hidden <!-- ui-owners @org/team @org/a,@org/b … --> line listing the owners of changed stories, one CODEOWNERS entry per space (comma-separated owners share an entry, and any one of them can approve it). The line is there even when no changed story has owners. found-pixel invites no one itself: a separate GitHub App reads the line, invites the owners once the PR is ready for review and holds UI Review pending until each owner outside the author's teams has approved the PR.

storybookDiff options

All flags accepted by the storybook-diff CLI:

| Key | Type | Default | Maps to | |---|---|---|---| | ignoreModules | string[] | [] | --ignore-modules (repeatable) |

frozenResources options

By default, GET request URLs over 512 characters aren't frozen, as they're generally tracking URLs or contain values that change every run.

Defaults:

{
  "frozenResources": {
    "enabled": true,
    "branches": ["main", "master", "develop"],
    "contentTypes": [
      "image/avif",
      "image/bmp",
      "image/gif",
      "image/jpeg",
      "image/png",
      "image/svg+xml",
      "image/webp",
      "image/x-icon",
      "image/vnd.microsoft.icon"
    ],
    "match": "^.{0,512}$"
  }
}

See src/config.ts for the full zod schema.

Screenshots

Stories are rendered in Chromium from the built storybook, served over a local HTTP server. Each stories file is loaded once and its stories are rendered in turn through Storybook's preview channel. A story is shot once Storybook reports it rendered and the page has settled: no network requests in flight, no pending timers of 3s or less, no DOM changes for 150ms, fonts loaded and images decoded. CSS animations are disabled and prefers-reduced-motion is set. Requests to other origins are fetched once per run and replayed to every page, so a CDN that varies its responses can't make the two sides differ. When screenshots.baseUrl is set, resources from other origins that frozenResources allows (images, by default) are also frozen: the first run on defaultBranch or one of frozenResources.branches to fetch one saves it to <repoName>/frozen/<sha256 of its URL> in the screenshots bucket, and every later run, baseline or review, fetches that copy from screenshots.baseUrl as each page requests it, so a CDN that re-encodes an image can't make a cached baseline differ either.

The viewport is 1280×720, resized to the page's content before the shot. These story parameters are honoured:

| Parameter | Effect | |---|---| | viewport | width/height, a named viewport from globals.viewport, or viewport.defaultViewport | | visualRegression.mask | [{ selector }] regions blanked out before comparing | | visualRegression.threshold | Per-story threshold | | visualRegression.waitBeforeScreenshot | Extra wait (ms) once the story has settled | | visualRegression.disable | Skip the story |

parameters.lostpixel is deprecated, but still read when visualRegression is absent.

Stories rendered one after another in the same page can leak state into each other (module-level caches, randomness seeded once per page). So every difference is re-shot on both sides in a fresh page per story, and only those that reproduce are reported; a difference that can't be re-shot is still reported. Seed any randomness per story (e.g. faker.seed() in a decorator) to keep those re-shoots rare. Stories whose difference didn't reproduce are tallied in <repo>/flaky.json next to the results: a count, first and last seen, and the last 10 runs (PR, branch, commit, build) for each story.

Baselines

The default branch publishes baseline screenshots to screenshots.s3, one immutable manifest per commit:

| Key | Contents | |---|---| | <repoName>/baseline/manifests/<sha>.json | {sha, engine, createdAt, shots, failed}: every shot ID mapped to the sha256 of its PNG, plus stories that failed to shoot | | <repoName>/baseline/blobs/<sha256>.png | Each distinct image, written once and never overwritten | | <repoName>/baseline.json | The latest published commit and Chromium, plus stories that failed to shoot |

Each default-branch build runs storybook-diff against the latest manifest's commit, re-shoots the stories it reports as changed, those that manifest records as failed and any whose image is missing, uploads images that manifest doesn't already use and writes a manifest for its own commit. The first build, or one with a different Chromium, shoots every story. A commit that already has a manifest is skipped, and a manifest is written only if none exists yet, so it never changes once PRs can read it. Every other write lands on a new key, so concurrent builds can't corrupt each other. Only moving baseline.json takes <repoName>/baseline.lock, an S3 conditional write recording the build's ref and commit; a build leaves baseline.json alone if another holds the lock or has moved it since. A lock left by a build that died expires after 10 minutes.

PRs into the default branch fetch the manifest for their merge-base and download its images instead of screenshotting the base storybook. If the merge-base has no manifest, they fall back to the latest one and shoot the base storybook for stories that changed in between. Stories without a baseline image are shot from the base storybook too. A difference from a baseline shot is re-shot from the base storybook before it counts.

After publishing, each default-branch build keeps the newest 100 manifests, plus the one baseline.json points to, and deletes the rest. It then deletes images that no kept manifest uses once they are a day old, so it never races a concurrent build that has uploaded images but not yet written its manifest. A PR whose merge-base manifest has gone falls back to the latest one.

Environment variables

Read from the Buildkite environment automatically:

| Variable | Purpose | |---|---| | BUILDKITE_BRANCH | Current branch (slugified with / → - unless BRANCH_SLUG is set) | | BUILDKITE_COMMIT | Commit SHA; used as the upload key on default-branch builds | | BUILDKITE_BUILD_URL | Linked from the PR comment and annotations | | BUILDKITE_JOB_ID | Linked from the PR comment and annotations | | BUILDKITE_PULL_REQUEST | PR number, or "false" for non-PR builds | | BUILDKITE_PULL_REQUEST_BASE_BRANCH | Target branch; drives base-storybook resolution | | BUILDKITE_PULL_REQUEST_LABELS | Comma-separated PR labels; if any match skipLabels, the review stage is skipped (the build still runs). If any match fullReviewLabels, every story is reviewed. Live labels are read from GitHub when a token is available | | BRANCH_SLUG | Optional override for the derived slug | | GITHUB_PR_TOKEN or GITHUB_TOKEN | Optional GitHub Pull Request integration |

Pipeline usage

Buildkite pipeline step:

steps:
  - label: ":storybook: Visual Regression"
    command: npx found-pixel run
    plugins:
      - docker#v5:
          image: node:20

Or split across steps if you want to parallelise other work:

steps:
  - label: ":storybook: Build"
    command: npx found-pixel build
    key: storybook-build

  - label: ":mag: Review"
    command: npx found-pixel review
    depends_on: storybook-build
    if: build.pull_request.id != null

CLI reference

found-pixel build     # build storybook, upload to S3, annotate
found-pixel baseline  # (default branch only) update the shared baseline screenshots
found-pixel review    # (PR only) download base, diff, regress, upsert comment
found-pixel run       # build, then baseline or review

review is a no-op when the build isn't a pull request, and baseline is a no-op off the default branch.

Library usage

The comment renderer is a pure function, exported for reuse in custom pipelines:

import { renderComment } from "found-pixel/comment";

const body = renderComment({
  diff,                            // storybook-diff JSON output
  visualIds,                       // string[]
  totalStories,                    // stories in the head storybook
  baseSlug,
  compareSlug,
  repoName,
  buildUrl,
  storybookUrl: (slug) => `https://.../storybook/${slug}/admin/index.html`,
  shotUrls: (storyId, kind) => urls[kind][storyId] ?? [],
  comment: {
    heading: "📘 UI Review",
    marker: "<!-- found-pixel-comment -->",
    maxChars: 62000,
    viewerUrl: "https://diffy.example.com",
  },
});

Development

npm install
npx playwright-core install --only-shell chromium   # once, for the screenshot engine tests
npm test           # runs vitest against all fixtures
npm run build      # tsc → dist/
npm run typecheck  # tsc --noEmit

Publishing

Both paths below are handled by .github/workflows/publish.yml and publish via npm trusted publishing (OIDC through the npm GitHub environment) — there is no npm token stored in the repo.

Alpha prerelease, from a PR. Put [publish] anywhere in the PR's head commit message:

git commit --allow-empty -m "[publish] trigger alpha"
git push

The workflow publishes <package.json version>-alpha.<run_number> under the dist-tag alpha-<branch-slug>, then upserts a comment on the PR with the exact version and a ready-to-run npx line. The branch slug is head_ref lowercased with every run of non-alphanumeric characters collapsed to -, so feat/Big_Thing becomes alpha-feat-big-thing. Consume it downstream with the tag rather than the version, so each new push is picked up:

npm install --save-dev found-pixel@alpha-<branch-slug>

Two gates apply, and both bite silently when they fail:

1. Path filter — the PR must change at least one file matching the workflow's paths: list (src/**, package.json, package-lock.json). A docs- or test-only PR never starts the workflow at all, so there is no skipped job to notice.
2. Head commit only — only HEAD is checked, not the whole PR. [publish] in an earlier commit does nothing; amend or add a new commit to re-publish. Each push increments run_number, so every alpha gets a distinct version.

Release, on merge to main. Bump version in package.json as part of the PR. Once it lands on main the workflow compares the version against HEAD~1 — if it changed, it publishes to the latest tag, pushes a v<version> tag, and creates a GitHub release with generated notes. If the version is untouched, the run succeeds and publishes nothing, which is what makes dependency-only and refactor-only merges safe.

Comment format changes

The comment generator has snapshot tests that write rendered markdown to test/snapshots/*.md. The .md files are committed so diffs are reviewable inline when the format changes.

Workflow:

# 1. Change src/comment.ts
# 2. Regenerate snapshots
npm run test:update
# 3. Inspect the diff in test/snapshots/*.md before committing

To preview a single fixture without running the test suite:

npm run preview-comment modified-with-visuals

Adding a fixture

1. Create test/fixtures/<name>/input.json matching the shape in test/helpers.ts (FixtureInput)
2. Run npm run test:update to create the snapshot
3. Review the generated test/snapshots/<name>.md

Size-driven cases (large-diff collapsing, oversized truncation) are programmatic in test/comment.test.ts rather than on-disk fixtures, since their scale would make the input JSON unreviewable.