Comparoscope SDK

Capture screenshots, upload them to Comparoscope in Pydantic Platform, review what changed.

Published to npm as @pydantic/comparoscope-sdk. Install it where the screenshots are captured:

pnpm add -D @pydantic/comparoscope-sdk

The protocol and the service still ship together -- this SDK tracks routes/v1/visual.py in Pydantic Platform -- so treat the version as part of that contract and upgrade it when the route changes rather than letting a consumer drift.

import { ComparoscopeClient, uploadRun } from '@pydantic/comparoscope-sdk'

const client = new ComparoscopeClient({
  baseUrl: process.env.LOGFIRE_API_URL!,
  token: process.env.COMPAROSCOPE_TOKEN!,
})

await uploadRun({ client, screenshots, build: { name: 'stories' } })

Commit, branch, base branch and pull request number are read from the CI environment when not passed. Getting those wrong is the difference between "three screenshots changed" and "the whole suite is new", so on a pull_request event the SDK prefers the head sha and head ref over GITHUB_SHA and GITHUB_REF_NAME, which Actions sets to a synthesized merge commit and <number>/merge.

How an upload goes

1. Hash everything. A screenshot's content hash is its identity throughout the service.
2. Register the manifest. The service answers with the hashes it has never seen. On a typical pull request that is a handful out of thousands.
3. Upload only those, batched to stay under the request ceiling.
4. Finalize. Only on success: a run that crashed halfway has captured part of its suite, and finalizing it would report everything it never reached as deleted.

A rerun that changes nothing transfers nothing.

Sharding

Every shard uploads to the same build. They share a parallel_nonce, and the service plans the comparisons once the last shard finalizes, so a 3000-screenshot suite can run on eight machines and still be reviewed as one thing.

import { shardNonce, storiesForShard } from '@pydantic/comparoscope-sdk'

const shard = { index: Number(process.env.SHARD_INDEX), total: Number(process.env.SHARD_TOTAL) }
const mine = storiesForShard(allStories, { baseUrl, shard })

await uploadRun({
  client,
  screenshots,
  build: {
    name: 'stories',
    parallel_nonce: shardNonce('stories'),
    parallel_total: shard.total,
  },
})

The split is a stable hash of each story id, not a slice of the discovered order: discovery order is not stable across machines, and a runner that captured a different slice than it registered would have the difference reported as deletions. The nonce includes the run attempt, so re-running a failed job starts a clean build instead of joining the abandoned one.

Complete builds

A complete build speaks for the whole suite while rendering only part of it. It declares its roster, every screenshot name the suite has, and which of them this run renders; each name it does not render is carried from the build it descends from. main can then render only the stories a merge affected and still be a full baseline, and a pull request reports the stories it deleted.

CI creates the build once, in the job that already knows the selection, and hands its id to every shard. Re-running a failed shard then uploads into the same build rather than starting a new one.

# Planning job: Ladle's meta.json gives the roster, the selection says what to render.
comparoscope create --name stories --roster roster.json --selected selected.json \
  --shards 8 --ancestors auto
# {"build_id": "...", "number": 1234}

# Each shard: upload what it rendered, even if that is nothing.
comparoscope --dir story-captures --build-id "$BUILD_ID" --build-number "$BUILD_NUMBER" --shard 3/8

--ancestors auto sends the first-parent history of HEAD^1: on a pull request, the base GitHub merged it with; on a push, the commit's parent. The service compares against the build of the nearest one, so it needs fetch-depth: 0. A pull request whose roster would remove more than a tenth of the suite is refused unless --allow-mass-removal says the deletion is deliberate.

Ladle

ladle build writes a meta.json listing every story it compiled, so a new story is covered the moment it exists rather than when somebody remembers to write a test for it.

import { captureStoryViewports, readStories, storiesForShard } from '@pydantic/comparoscope-sdk/ladle'

const options = {
  baseUrl,
  shard,
  viewports: [
    { name: 'desktop', width: 1280, height: 800 },
    { name: 'mobile', width: 390, height: 844 },
  ],
}

for (const story of storiesForShard(readStories('build/meta.json'), options)) {
  screenshots.push(...(await captureStoryViewports(page, story, options)))
}

Each viewport is separately baselined, because a change at mobile width is a change even when the desktop rendering is untouched. The default viewport's screenshots are not suffixed, so adding a second viewport later does not rename every existing baseline and lose it.

Stabilization

Almost every false positive comes from the page, not the differ: a caret blinking, a font swapping in late, a GIF on frame three, an image that had not decoded. capture() removes each before the shutter rather than compensating with a looser diff threshold, which would hide real changes too.

The in-page script is a string, not a function, on purpose. Playwright serialises an evaluated function with toString(), and bundlers with esbuild's keepNames on (tsx, among others) inject a __name helper that does not exist in the browser.

When it goes wrong

Errors name the likely cause and the next step, because they are read in a CI log by somebody who did not write this code and cannot attach a debugger.

comparoscope: Could not reach http://127.0.0.1:9
  fetch failed
  Check the API URL is reachable from CI and includes the scheme, e.g. https://logfire.pydantic.dev

comparoscope: no such directory: e2e/screenshot

A 401 names the environment variable to check; a 403 explains that a read token cannot upload; a 413 owns the problem as an SDK batching bug rather than asking you to tune something.

Seeing what a run would do

planRun (and --dry-run) answers "why is this uploading 900 screenshots?" without spending the upload, and checks a shard split before a real run trusts it.

$ comparoscope --dir shots --dry-run
would upload 2 of 3 screenshots in 1 request(s), 148 B
  home
  settings/roles
  settings/tokens

What a run prints

Comparoscope build #12
  captured    3000
  uploaded    20 (1.0 MB) in 1 request(s)
  already had 2980
  status      changes_requested
  comparisons 20
  review      https://logfire.pydantic.dev/acme/web/visual-tests/b-1

The interesting number is how little travelled. Pass --app-url, --org and --project for the review link.

CLI

For suites that already write PNGs to disk:

comparoscope --dir e2e/screenshots --api-url "$LOGFIRE_API_URL" --name stories --shard 3/8

A Playwright suite is better served capturing in-process and calling uploadRun directly.

Development

pnpm install
pnpm check          # typecheck + lint + tests
pnpm test:coverage  # the same tests, with the coverage thresholds enforced
pnpm mutate         # mutation testing: does the suite notice when the code is wrong?
pnpm build          # emit dist/, which is what actually ships

How this is tested

test/emulator.ts is a stand-in for the upload API that keeps the state the real one keeps: which hashes are registered, which bytes have arrived, whether a shard has closed. It refuses what the service refuses -- bytes for a hash that was never registered, a second finalize of an unsharded build, a sharded finalize with no index -- so a passing test means the exchange was legal and not merely well-formed. It answers routes/v1/visual.py in Platform; when that route changes, change this with it.

Coverage is required at 100 percent on every metric, so a line nobody exercised fails the build rather than being noticed later. Where a branch is genuinely unreachable -- a guard that exists for noUncheckedIndexedAccess and nothing else -- the line carries a v8 ignore naming the reason, rather than the threshold being lowered to hide it.

The mutation threshold is a ratchet set just under what the suite achieves. Raise it when the number goes up. Lowering it to make a run pass is the thing it exists to stop.

src/ imports carry explicit .ts extensions, and tsconfig.build.json turns them into .js on the way out with rewriteRelativeImportExtensions. That is why the build is plain tsc and the package has no bundler.

dist/ is what consumers get, and it has to be: Node refuses to strip types under node_modules, so a package exporting src/*.ts installs fine and then throws ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING the first time anything imports it. CI packs the tarball and uses it from a clean install so that cannot reach a release again.

Releasing

Bump version in package.json, merge it, then cut a GitHub Release whose tag is v<version>. The publish workflow refuses anything that is not a tag matching the manifest, runs the checks, and publishes through npm Trusted Publishing -- there is no NPM_TOKEN in this repository. It does not attest provenance: npm does not support that from a private source repository.