vlm-diff
v0.10.0
Published
The visual verifier for agentic coding workflows — deterministic-first UI screenshot diffing (DOM+pixel), tiered descriptions, CLI, MCP server, GitHub Action
Maintainers
Readme
👁 VLM-Diff
The visual verifier for agentic coding workflows
Did my UI change, and what exactly? — answered with DOM ground truth, deterministic-first (zero tokens where the DOM explains the change), severity tiers, and evidence you can check.
Quick Start · How it works · Results · GitHub Action · Agent recipes · Paper · Changelog
Detection pipeline on five cases: changed regions are localized, typed, and described — mostly without any VLM call.
✨ Why vlm-diff
⚡ Deterministic-first
The DOM diff already knows most of the answer — rgb(37,99,235) → rgb(220,38,38) becomes "background color changed from blue to red" from a template. 97% of regions never touch a model.
🤖 Agent-native
An MCP server returns typed structuredContent + the annotated frame as an image block — coding agents verify their own UI edits and literally see the numbered regions.
🛡 CI gate, semantics included
CLI exit codes 0/1/2/3 and a composite GitHub Action that comments a severity table on the PR and uploads annotated HTML reports.
🧾 Evidence, not vibes
Every description carries the DOM values it was derived from (backgroundColor: rgb(37,99,235) → rgb(220,38,38)). You can check any claim against the raw evidence.
💰 Cost is a feature
Per-call cost log (fn, model, tokens, latency, $), budget gate (VLM_DIFF_BUDGET_USD), output caps, opt-in cheap-tier routing, content-hash cache.
♿ Accessibility layer
Deterministic WCAG contrast math flags drops below AA (5.2:1 → 2.9:1), and alt/aria removals are caught even with zero pixel change. Unique among visual-diff tools.
🚀 30-second quick start
npm i -g vlm-diff
# capture the page before and after your change
vlm-diff snapshot http://localhost:3000 --out-dom before.dom.json --out-png before.png
# …make the change…
vlm-diff snapshot http://localhost:3000 --out-dom after.dom.json --out-png after.png
# diff: deterministic descriptions at 0 tokens, VLM only for pixel-only deltas
vlm-diff diff before.png after.png --dom-before before.dom.json --dom-after after.dom.json \
--report report.html⚠️ CHANGE DETECTED
summary: Element background color changed from blue (rgb(37, 99, 235)) to red (rgb(220, 38, 38))
type: color-change severity: cosmetic
evidence: DOM backgroundColor: rgb(37, 99, 235) → rgb(220, 38, 38)No API key needed for DOM-observable changes — they are described deterministically. Pixel-only deltas (canvas repaints, image swaps) escalate to a VLM when a key is configured (.env: DASHSCOPE_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY, MOONSHOT_API_KEY, OPENCODE_API_KEY).
🧠 How it works
| Stage | What it does | Cost |
|---|---|---|
| 1 · Detect | DOM diff (structure, styles, text, geometry, attributes) fused with perceptual pixel diff. DOM-unchanged + sub-noise-floor pixels ⇒ unchanged. | 0 tokens |
| 2 · Describe | Regions the DOM fully explains get template descriptions — colors named, geometry signed ("moved 28px right"), lifecycle explicit, reflow followers attributed to their root cause. Nested moves merge into one logical change. | 0 tokens |
| 3 · Escalate | Only pixel-only deltas (canvas repaints, SVG fills, image swaps) crop + classify via VLM — and a pair whose escalations all come back none flips back to unchanged, end-to-end honest. | cropped regions only |
The --report output: before/after with numbered, severity-colored region boxes, and every region card carrying the DOM evidence its description was derived from.
graph TB
Input["📸 Input: Screenshot Pair<br/>+ DOM snapshots (JSON)"]
Input --> Stage1["⚙️ Stage 1: Deterministic Detection"]
Stage1 --> DOMDiff["🔍 DOM Diff<br/>• Structure / styles / attrs<br/>• Geometry with signed deltas<br/>• Fuzzy matching (reorder-safe)"]
Stage1 --> PixelDiff["🎨 Perceptual Diff<br/>• pixelmatch (AA-aware)<br/>• Flood-fill grouping"]
DOMDiff --> Fusion["🔗 Fusion<br/>DOM=0 & sub-floor → unchanged<br/>DOM=0 & over-floor → visual-only"]
PixelDiff --> Fusion
Fusion --> Decision{"Changed?"}
Decision -->|No| Done["✅ DONE (0 tokens)"]
Decision -->|Yes| Tier["🤖 Tiered Description<br/>DOM-explained → template (0 tok)<br/>pixel-only → crop + VLM"]
Tier --> Result["📋 changeType + description + evidence + severity"]Why DOM diff as ground truth? Pixel diff alone flags 15–25% of unchanged pairs (anti-aliasing, focus rings, render timing). The DOM answers whether the page model changed. Thresholded suppression keeps the honest middle: above-floor pixel deltas with no DOM signal are visual-only and escalate — a pair flips back to unchanged only if the VLM calls every escalated region none.
v0.3 structural upgrades. Fuzzy DOM matching pairs nodes by stable key (unique id, else unique tag|className|text) so list reorders/head-inserts no longer fan out into phantom regions; semantic merging collapses a card and its children moving together into one logical change; deterministic severity tiers (breaking for element lifecycle + numeric text, moderate for large geometry/copy, cosmetic for color/style).
Why crop-then-classify? Feeding VLMs near-identical full images is a known weakness (VLM-SubtleBench: models lag humans on subtle differences across 10 categories, spatial worst). Our own ablation showed plain crops lose the reference frame (56.7% conditional accuracy on Kimi without the DOM-field hint). The fix: pass the detector's changed fields as a ~50-token text prior — 83–90% recovered, and the tiered describer makes most regions free.
Remaining DOM-layer limits: CSS animations mid-frame, cross-origin iframes, and deltas below pixelmatch's own sensitivity (~0.15 YIQ) are invisible by construction.
📊 Results at a glance
193-pair benchmark (7 fixtures × 37 mutation ids, incl. pixel-only canvas/SVG pairs and 42 no-change pairs). Detection layer — offline replay of the deterministic pipeline over all 193 pairs, 0 API calls (results/router-replay.json): 151/151 changed pairs, 0/42 false positives (95% CI on FP: [0%, 8.4%]). Four-arm live evals with real API calls: 36 pairs (2026-08-28, Kimi K3, $0.18) and 51 pairs (2026-09-02, qwen3.8-max, $1.36) — tables below.
| Model | tiered: recall / FP / type-acc | tokens/pair | escalation | hint: type-acc | no-hint: type-acc | raw: recall / precision | |---|---|---|---|---|---|---| | kimi-k3 (36 pairs) | 100% / 0% / 100% | 11+4 | 1.3% | 93.3% | 56.7% | 76.7% / 82.6% | | qwen3.8-max (36 pairs) | 100% / 0% / 100% | 9+17 | 1.3% | 90.0% | 90.0% | 83.3% / 24.0% | | qwen3.8-max (51 pairs, v0.2 subset) | 100% / 0% / 97.1% [84.7%, 99.9%] | 112+131 | 18.7% | 91.2% | 85.3% | 85.3% / 37.9% | | qwen3.7-flash (51 pairs, weak model) | 100% / 0% / 91.2% [76.3%, 98.1%] | 112+127 | 23.6% | 35.3% | 26.5% | 0% / — |
Full-dataset offline replay (193 pairs, 0 API calls, results/router-replay.json): 662 detected regions, 97.1% answered deterministically (643/662, zero tokens), region escalation rate 2.9% (19 regions across 7 pairs). The 1.3% escalation in the table above is the 36-pair minimal arm, not the full set.
The weak-model row is the point: with a model whose raw full-image arm misses every changed pair (0% recall), the tiered pipeline still lands 100% recall and 91% type accuracy. The robustness is architectural, not model-dependent. A blind cross-vendor judge scores the zero-token templates on par with VLM descriptions (14/14/2, all bootstrap CIs straddling zero), and human calibration on all 30 judged pairs backs the aggregate read (66.7–80% per-pair agreement — judge is an aggregate instrument, not a per-pair one).
MVP Validation (real API calls, 2026-08-19)
15-pair representative subset (5 subtle/boundary, 7 clear changes, 3 no-change), three arms: rawPairToVlm vs fullPipeline (no hint) vs fullPipeline + DOM-field hint — the detector's changedFields passed to the classifier as a ~50-token text prior. Kimi K3 via Alibaba DashScope, two runs.
| Metric | rawPairToVlm | fullPipeline (no hint) | fullPipeline + DOM hint | |--------|--------------|------------------------|------------------------------| | Recall (12 changed pairs) | 66.7% (8/12) | 100% (12/12) | 100% (12/12) | | FP rate (3 no-change pairs) | 33.3% (1/3) | 0% | 0% | | Classification accuracy (among detected) | 100% (8/8) | 66.7% (8/12) | 83.3% (10/12) | | End-to-end type accuracy (of 12) | 66.7% (8/12) | 66.7% (8/12) | 83.3% (10/12) | | Avg input tokens/pair | ~1505 | ~583 | ~692 |
Findings: detection is reproducibly won by the pipeline (+33pp recall, 0% FP, byte-identical across runs while the raw arm fluctuated and produced a false positive); plain crop-then-classify is refuted (58.3–66.7% vs raw's 100% conditional); the DOM-field hint recovers the gap (+25pp/+16.7pp across runs) and wins end-to-end outright.
Four-Arm MVP (real API calls, 2026-08-28)
36 pairs (30 changed + 6 no-change), Kimi K3, all four arms in one run. Total cost: $0.18.
| Metric | tieredPipeline | fullPipeline + hint | fullPipeline (no hint) | rawPairToVlm | |--------|--------------------|---------------------|------------------------|--------------| | Recall (30 changed) | 100% [88.4%, 100%] | 100% | 100% | 76.7% [57.7%, 90.1%] | | FP rate (6 no-change) | 0% [0%, 45.9%] | 0% | 0% | 0% | | End-to-end type accuracy | 100% [88.4%, 100%] (30/30) | 93.3% (28/30) | 56.7% | 95.7% of detected (7 pairs missed) | | Avg tokens/pair | 11 + 4 | 910 + 388 | 782 + 611 | 1505 + 196 |
The tiered arm wins on every axis — 74/75 regions deterministic (one pixel-only repaint band escalated), 98.9% fewer tokens than the hinted pipeline, fixing both failure modes the hinted VLM arm made (a container's reflow growth out-voting an element-add; a border-color change mis-typed as style-change). McNemar (paired, same pairs): tiered vs raw recall significant (7/0, p=0.016); tiered vs hint type accuracy not significant (2/0, p=0.5); hint vs no-hint significant (11/0, p=0.001).
v0.2 Live Re-Run (51 pairs, qwen3.8-max, 2026-09-02)
Four-arm protocol on the expanded 51-pair subset (34 changed — including all 4 pixel-only media pairs — and 17 no-change), per-pair error isolation + pair-level checkpointing. Total cost $1.36.
| Metric (95% CI) | tieredPipeline | fullPipeline + hint | fullPipeline (no hint) | rawPairToVlm | |---|---|---|---|---| | Recall (34 changed) | 100% [89.7%, 100%] | 100% | 100% | 85.3% [68.9%, 95.0%] | | FP rate (17 no-change) | 0% [0%, 19.5%] | 0% | 0% | 0% | | End-to-end type accuracy | 97.1% [84.7%, 99.9%] | 91.2% | 85.3% | 79.4% | | Avg tokens/pair | 112+131 | 737+1734 | 646+1320 | 1116+742 |
All 4 pixel-only media pairs classified correctly by the VLM tier; subset escalation 18.7% with 90.2% token savings vs the hinted arm. The tiered arm's only type "miss" (media-image-src-swap) is a label-granularity artifact — the VLM wrote "the icon background changed from blue to red while the glyph and layout remained identical", factually correct, typed color-change vs ground truth other. McNemar: tiered vs raw recall 5/0 (p=0.063, same direction as v0.1's 7/0 at lower power).
Note (v0.9 docs sync): the
media-image-src-swapnarrative above records the v0.2 run. Since v0.6,srcis a captured DOM attribute, so this pair is answered deterministically and no longer reaches the VLM tier at all (results/router-replay.json: kindother, 1 region, 0 VLM calls).
Blind Judge: template vs VLM description quality
Blind judge (qwen3.8-max judging Kimi K3's descriptions — different vendor, labels randomized, ground truth provided), 30 pairs:
| Dimension | tiered (template) | VLM | |---|---|---| | accuracy | 3.90 | 4.27 | | specificity | 3.90 | 3.73 | | readability | 4.60 | 4.50 | | winner | 14 | 14 (+2 ties) |
Bootstrap CIs on the gaps (10k seeded resamples): accuracy Δ −0.37 [−0.80, +0.07], specificity Δ +0.17 [−0.47, +0.83], readability Δ +0.10 [−0.20, +0.40] — every dimension straddles zero. Human calibration on all 30 pairs (data/judge-human-labels.json): both descriptions correct on 27/30; the judge's vlm lean concentrates on the 3 factually wrong descriptions. Comparable quality at ~1% of the tokens.
Real-World Validation (git-diff ground truth)
scripts/validate-real-repo.ts — two real open-source repos (mdn/beginner-html-site-styled, mdn/beginner-html-site-scripted), 6 commit-pair scenarios, zero API calls: 0 false positives, 0 misses. An icon-file swap (same <img> path, new pixels) escalated exactly as designed; a CSS refactor stayed 83% deterministic; lang attribute, font-protocol swap, and JS-only fixes were correctly reported as visually silent (the harness separates "source changed, pixels didn't" from true misses).
Original predictions (for reference)
| Metric | rawPairToVlm | pixelDiffOnly | fullPipeline | |--------|--------------|---------------|------------------| | Recall | 70-75% | 92% ✓ | 92% | | Precision (IoU>0.3) | 30-40% | 85-90% | 88-92% | | FP rate (no-change) | 15-25% | 0% ✓ | 0% ✓ | | Type classification accuracy | 55-65% | N/A | 75-82% |
✓ = confirmed on the real dataset. Hypothesis criteria: recall ≥+10pp (+25pp ✅), FP <20% (0% ✅), type accuracy ≥+15pp (crop-only refuted; with DOM hint met).
🧰 Usage
CLI — diff any two screenshots
See the quick start. Reference highlights:
- Without
--dom-*the CLI degrades to pixel-only mode (every significant delta escalates to the VLM). --jsonfor machine-readable output; exit codes 0 no change / 1 changed / 2 error / 3 changed but needs a VLM key.--no-vlmruns detection + routing with zero API calls (escalated regions listed as pending).--report <file.html>writes the shareable annotated report shown above.- API keys come from the environment or a local
.env;--provider/--modeloverride the preset.
Audit a page with no baseline
Every other command compares two frames. audit judges one frame, so it needs no baseline, no previous run and no API key — the check an agent can call before any history exists:
vlm-diff audit --url http://localhost:3000/ # capture + check in one command
vlm-diff audit page.dom.json --png page.png --diag page.diag.json # offline⚠️ AUDIT — 5 findings (1 breaking, 4 moderate) in http://localhost:3000/
breaking button.far-right interactive element extends 176px past the right edge of the viewport — unreachable
moderate button.tiny interactive target is 80×18px, below the 24×24 minimum (WCAG 2.5.8)
moderate div.card text clipped by 326px (ellipsis)
moderate div.clipped content overflows vertically by 78px and is clipped (overflow-y: hidden)
moderate div.collapsed element collapsed to 1248×0px — its content renders in zero space
Page diagnostics:
console errors: 2
boom: failed to load pricing module ×2
Fetch API cannot load /api/itemsRules are deliberately narrow (a false positive in an audit costs more than a missed edge case): collapsed, text-truncated (ellipsis configured and the text actually overflows), content-overflow (clipped, not scrollable), offscreen-interactive, and small-target — prose links and checkbox/radio inputs are exempt. Console errors, uncaught exceptions, failed requests and HTTP ≥400 are captured by snapshot --out-diag and reported alongside; a page that threw exits 1 even when the layout is structurally clean.
Why did this style change? (vlm-diff why)
diff tells you the background went blue → red. why tells you which CSS rule now wins and what else could have set it — the difference between a report and a diagnosis, and the piece that lets an agent go fix the code instead of guessing:
vlm-diff why http://localhost:3000 --selector "#save" --property background-color#save · background-color = rgb(0, 128, 0)
winning rule: caused by: #save ((1,0,0)) → background-color: green · overrode: .btn--danger (background-color: red), .btn (background-color: blue)
contender: .btn--danger (0,1,0) background-color: red — loses on specificity
contender: .btn (0,1,0) background-color: blue — loses on specificityPure CSSOM read-back — zero tokens, no VLM, runs against the live page. diff answers what; why answers which rule.
Baseline workflow (watch a project over time)
vlm-diff init http://localhost:3000/ # .vlm-diff/config.json (pages to watch)
vlm-diff baseline # golden snapshots → .vlm-diff/baseline/
# …later, after changes…
vlm-diff check # diff every page vs baseline, CI exit codes
vlm-diff review # approve changed pages (interactive; --approve-all in CI)
vlm-diff trends # change frequency + flaky-page detection from history
vlm-diff trends --suggest-ignores # mask candidates for flaky pages (review before adding)
# or refresh everything explicitly (confirmation required):
vlm-diff baseline --yesConfig knobs in .vlm-diff/config.json:
{
"viewport": { "width": 960, "height": 500 },
"viewports": [{ "width": 1280, "height": 800 }, { "width": 390, "height": 844 }], // responsive matrix
"ignore": ["#ad-slot", "[data-timestamp]"], // dynamic-content masks — never flagged
"pages": [
{ "name": "home", "url": "http://localhost:3000/" },
{ "name": "menu-open", "url": "http://localhost:3000/", "actions": [{ "click": "#menu-btn" }] }
]
}Playwright SDK (two lines inside your existing tests)
import { capturePage, checkAgainstBaseline } from "vlm-diff/playwright";
const before = await capturePage(page); // screenshot + DOM snapshot in one call
// …apply your change…
const verdict = await checkAgainstBaseline(page, "home"); // diff vs the golden baseline
if (verdict.changed) console.log(verdict.severity, verdict.summary);checkAgainstBaseline returns the full DiffVerdict (changeType, severity, evidence, a11y impact) and baselineMissing: true when no golden exists yet (seed it with vlm-diff baseline).
MCP server (agents verify their own UI edits)
claude mcp add vlm-diff -- vlm-diff-mcpSix tools over stdio:
| Tool | What it does |
|---|---|
| snapshot_url | capture a URL's DOM snapshot + screenshot |
| diff_screenshots | the full tiered diff (typed verdict + annotated image block) |
| baseline_seed | capture current pages as the golden baseline |
| baseline_check | diff pages against the baseline (pages: [...] to check only what you changed) |
| audit_url | zero-baseline layout + a11y audit (no baseline, no key needed) |
| why_url | CSS cause attribution — which rule wins, what to edit |
baseline_check fails soft: no baseline yet → it tells the agent to call baseline_seed first, so the loop closes itself. Integration recipes for Claude Code / Cursor / ZCode: docs/agent-recipes.md. npm run mcp:smoke walks the full handshake offline.
GitHub Action (gate PRs on visual severity)
- uses: shidesheng0218/[email protected]
with:
fail-on: breaking # breaking | any | never
# provider: dashscope # optional — only needed for pixel-only escalationThe composite action installs, builds, runs vlm-diff check --json --report, uploads annotated per-page HTML reports as an artifact, and comments a severity table on the PR. Same exit-code contract as the CLI: errors always fail, changes fail per fail-on, "needs a VLM key" warns without failing. Verified live on a real PR (comment + artifact + correct gate).
Demos & evaluation workflow (for contributors)
npm run demo:quick # real algorithm on synthetic in-memory inputs (no keys)
npm run demo:generate # render 3 demo pairs with Playwright
npm run demo:detect # deterministic detection on them
npm run dataset:gen # regenerate the 193-pair dataset
npm test # 310 unit tests, zero API calls
npm run eval:mvp # four-arm live eval on the 51-pair subset (needs a key)
npm run stats:report # CIs + McNemar + bootstrap over any report
npm run compare:models # cross-model table from all reports💰 Cost & trust
Every provider is wrapped in an instrumented layer:
- Cost log — every VLM call appends one JSON line to
.cache/cost-log.jsonl:{ts, fn, provider, model, inputTokens, outputTokens, latencyMs, costUsd, ok}. Per-feature spend is answerable withjq. - Output caps — per-call
maxTokens(classification 512, judges/raw 1024). - Budget gate —
VLM_DIFF_BUDGET_USD=0.50fails fast past the cap. - Model routing (strictly opt-in) —
VLM_DIFF_CLASSIFY_MODELsends high-volume classification to a cheaper tier; nothing routes unless set. - Timeouts + retry — 120s per call, inside the retry layer, so a hang is a retryable failure, not a stalled run.
- Cache — content-hash over crop pixels + prompt + model identity (switching models can never return another model's verdict).
Data handling & privacy. Only cropped before/after regions (never full pages) plus a short DOM-field hint leave your machine, sent to the model API you configured. Keys, full screenshots, DOM snapshots, and baselines never leave it. State lives in .vlm-diff/ and .cache/ (both gitignored); there is no telemetry, no accounts, no server of ours.
🗂 Dataset
193 screenshot pairs — 7 fixtures (card grid, form, navbar, table, modal, dashboard, and a media page with canvas/SVG/img widgets) × 37 mutation ids. Detection (offline replay, 0 API calls): 151/151 changed pairs, 0/42 false positives (FP CI [0%, 8.4%]).
Each pair: before/after PNGs (960×500), DOM snapshots (path/tag/id/class/text/rect + 7 computed styles + whitelisted attributes), ground-truth rect, kind, and natural-language ground truth.
| Mutation category | Count | Example |
|---|---|---|
| Spatial shift (3–28px) | 31 | margin/transform moves; includes 2 list-reorder pairs (fuzzy matching) |
| Color change (bg/text/border/contrast) | 31 | #2563eb → #dc2626; 5 contrast-degrade pairs (a11y) |
| Size change (scale 0.9×–1.35×) | 25 | CSS transform scaling |
| Text change (similar/different/shorten/numeric/alt) | 23 | "Project Falcon" → "Project Falcan"; alt removal (zero-pixel change) |
| Element add | 6 | clone a card/row/button |
| Element remove | 12 | delete a card/row/button |
| Style change (weight/radius/shadow/opacity) | 22 | bold → normal, rounded → square |
| Other | 1 | image src swap (deterministic since v0.6 — src is a captured attribute) |
| No-change | 42 | re-render / reload / settle-time variants |
Counts are by kind and sum to 193. "Pixel-only" is a cross-kind construction property (zero DOM signal: canvas ×2, SVG fill, image swap, shadow styles), not a kind — the canvas/SVG pairs still exercise the VLM escalation path; the image-swap pair left it in v0.6 when attribute capture landed.
The v0.1 dataset was DOM-observable by construction (escalation circularity); the pixel-only pairs and the 42 no-change pairs fix the two biggest holes (the FP CI upper bound dropped from ≈46% at n=6 to ≈8% at n=42). Since v0.6, attribute changes (src/href/alt/aria) are captured, which moved the image-swap pair from the escalation path into the deterministic tier.
📉 Limitations & roadmap
Resolved recently: pixel-only blind spot (thresholded suppression + escalation, v0.2) · tiny FP denominator (42 no-change pairs, v0.2) · escalation circularity (real-repo validation, v0.2) · point estimates without CIs (v0.2) · uncalibrated judge (human calibration, v0.2) · reorder explosions (fuzzy matching, v0.3) · invisible attributes (attribute capture, v0.6).
Still open: third-party widgets (ads/chat/maps) whose pixels change independently of your code · synthetic fixtures dominate (6 real-repo scenarios are a start, not a benchmark) · Chromium only · judge is an aggregate instrument only · sub-sensitivity changes below pixelmatch's threshold are invisible · Claude/GPT-class replication pending an API key (Kimi/Qwen evidence stands).
- Phase 1 ✅ research prototype validates the hypothesis
- Phase 2 ✅ (v0.2) usable tool — CLI, MCP, expanded dataset, statistical rigor
- Phase 3 ✅ (v0.3–v0.5) structural understanding, trust & cost layer, agent verifier + GitHub Action
- Phase 4 ✅ (v0.6) attribute awareness, deterministic a11y, review/trends loop
- Next: specialist model fine-tuned on the deterministic tier's free labels (652 exported — see
docs/specialist-model.md); GitHub Marketplace listing; Claude/GPT-class replication.
🔬 For researchers
Relationship to WUICC-bench (the closest prior work)
WUICC-bench (2026) defines Web UI Image Change Captioning — describing UI changes in words — and is the first benchmark for the task. Its headline finding is that all 11 image-difference-captioning methods it evaluated, plus 2 zero-shot general LLMs, struggle on web UIs because of layout diversity, dense text, and fine-grained changes.
Every one of those methods is purely visual — none uses DOM structure. That is exactly the gap this project fills: the DOM diff knows which computed property changed and its before/after values, so 97.1% of regions get a provably-correct answer with zero model involvement, and the VLM is reserved for the 2.9% the DOM can't see (canvas, SVG, image pixels). Where WUICC-bench documents that the visual-only approach is hard, this project shows the structural approach makes most of it free.
Why not just run on their benchmark? Their dataset is image pairs + human captions; it has no DOM snapshots, so our deterministic tier can't run on it directly. The honest comparison is the other direction: our 193-pair benchmark with DOM ground truth is open, and we report how much of it never needs a model.
Paper draft: paper/vlm-diff.tex · writeups: Dev.to · 掘金 · competitive analysis: docs/marketing/COMPETITIVE_ANALYSIS.md
@software{vlm_diff_2026,
title={VLM-Diff: Visual Regression Detection with Structural Ground Truth},
author={shidesheng},
year={2026},
month={October},
version={0.10.0},
url={https://github.com/shidesheng0218/vlm-diff},
note={Deterministic-first UI diffing with tiered VLM escalation: CLI, MCP server, dataset, and evaluation framework}
}- VLM-SubtleBench (2026): arXiv:2603.07888 — VLMs lag humans on subtle image differences across 10 categories (spatial, temporal, …); non-UI domains (industrial/aerial/medical)
- OmniDiff (2025): arXiv:2503.11093 — image-difference captioning dataset (324 scenarios, 12 change types) + a fine-tuned specialist model
- WUICC-bench (2026): arXiv:2607.01728 — the closest prior work: first benchmark for describing web UI changes in words. Evaluated 11 image-difference-captioning methods + 2 zero-shot LLMs and found they struggle on web UIs (layout diversity, dense text, fine-grained changes). None of them uses DOM structure — see below.
- Applitools MCP (2026): blog post
- Chain-of-Focus (May 2026): arXiv:2505.15436 — VLM iterative attention
- SPARC (February 2026): arXiv:2602.06566 — two-stage visual search + reasoning
- v0.9 — trustworthy numbers: corrected detection precision + region-level type scoring, prompt-versioned cache, cost parity and budget-gate fixes, docs-number gate in CI
- v0.7 — Playwright SDK (
vlm-diff/playwright), ignore regions for dynamic content, scenario actions + viewport matrix, flaky→ignore suggestions - v0.6 — attribute-aware capture (image swaps/link targets now deterministic), deterministic a11y (WCAG contrast + alt/aria),
review+trendsworkflow - v0.5 — agent verifier: MCP image blocks + structuredContent, GitHub Action,
check --json/--report - v0.4 — trust & cost: evidence everywhere, baseline overwrite confirmation, cost log, budget gate, timeouts, output caps
- v0.3 — fuzzy DOM matching, region merging, severity tiers, annotated HTML report, baseline workflow
- v0.2 — CLI + MCP, thresholded suppression, 42 no-change pairs, statistics layer, real-repo validation
Full details in CHANGELOG.md.
