@freely-labs/agent-eye 👁️

🌐 简体中文

Agent-Eye (@freely-labs/agent-eye) is a cognitive-science-inspired, intelligent visual discrepancy and layout feedback engine designed specifically for AI Coding Agents (e.g. Claude Code, Cursor, Windsurf, Devin), design-to-code workflows, and automated UI self-correction loops.

Traditional pixel-by-pixel diffing tools are highly fragile when dealing with minor layout shifts, subpixel rendering noise, or viewport dimension variations, causing "100% red visual diffs" despite being visually correct to the human eye. Agent-Eye resolves this by combining Structural Similarity Index Measure (SSIM) with morphological component clustering, segmented vertical layout alignment, and Universal View Tree (UVT) DOM mapping. It translates raw pixel mismatches into precise, code-level actionable JSON metadata for AI consumption.

✨ Key Features

• 🧠 Visual Attention Focus Modes: Multi-stage cognitive preprocessing to reduce LLM token/attention entropy.
  • structure: Blurs away high-frequency text. Focuses strictly on macro layouts (grid columns, cards, container wraps).
  • details: Laplacian high-pass filter neutralizes flat backgrounds. Focuses strictly on fine characters, icon contours, typos, and 1px padding alignments.
  • style: Analyzes HSV color theme changes while ignoring fine boundary shifts.
  • full: Full baseline SSIM discrepancy comparison.
• 📐 Layout-Aware Segmented Alignment (--auto-layout): Slices the design mockup into component bands via Horizontal Projection Profiles (HPP) and vertically slide-aligns each band on the screenshot via 1D template matching. This completely neutralizes "cascading red diff errors" caused by a single wrapping shift at the top.
• 🤖 Universal View Tree DOM Mapping (--uvt): Recursively traverses standard viewport JSON trees to map detected visual diff boxes to the deepest target leaf element selector (e.g. div.tab-item, svg > rect).
• 🔍 Auto DPI Normalization: Detects retina high-DPI screenshots (e.g. @3x) relative to logique mockups and scales them down via area decimation (cv2.INTER_AREA), eliminating subpixel anti-aliasing noise.
• 🎨 Technical Visualization Dashboard:
  • Draws bounding boxes labeled with severity ratings (Critical / Warning / Info) and error ratios.
  • Exports a dark composite dashboard: [Design Mockup] | [Annotated Screenshot] | [Jet Colormap Heatmap].
• 🔒 Zero-Conflict Python Sandbox: The npm package automatically builds an isolated virtual environment (.venv/) inside the package folder during installation, keeping your global Python environment perfectly clean!

🛠️ Pipeline Overview

[Design Mockup] ──► [DPI Downscale] ──► [Segmented Alignment] ──┐
                                                                 ├──► [Attention Mode Filter] ──► [SSIM Engine] ──► [UVT Selector Map] ──► [JSON & Dashboard]
[Screenshot]    ──► [DPI Downscale] ──► [Segmented Alignment] ──┘

📦 Installation & Quick Start

A. Node.js / Front-end Ecosystem (npm)

If you are using JavaScript/TypeScript, visual testing frameworks (Playwright, Puppeteer, Cypress), or Node-based AI tools:

1. Global Installation

npm install -g @freely-labs/agent-eye

2. Run Globally in your Terminal

# Run the built-in synthetic test suite
agent-eye --test

# Compare design mockup vs screenshot with layout alignment and semantic mapping
agent-eye --mockup mockup.png --screenshot screenshot.png --auto-layout --uvt path/to/uvt.json

3. Zero-Install Execution (npx)

npx @freely-labs/agent-eye --test

B. Python / AI Ecosystem (pip)

If you are building Python-based AI Agents (e.g. LangChain, CrewAI, AutoGPT) or custom backend services:

1. Installation

git clone https://github.com/steveleeh/agent-eye.git
cd agent-eye
pip install .

2. Run Globally in your Terminal

agent-eye --test

🖥️ Command Line Interface (CLI)

agent-eye --mockup <path_to_mockup> --screenshot <path_to_screenshot> [options...]

Options:

| Parameter | Type | Default | Description | | :--- | :---: | :---: | :--- | | --mockup | str | - | Path to the original design mockup/draft. | | --screenshot| str | - | Path to the screenshot generated from the code. | | --align | str | pad | Canvas alignment method: pad (pad with white boundaries) or resize (scale/stretch). | | --threshold | int | 30 | SSIM difference threshold (0-255). Lower is more sensitive. Default: 30 (~12% discrepancy). | | --focus | str | full | Visual Attention Mode: full (default), structure (ignores text), details (ignores flat colors), style. | | --auto-layout| bool| False | Enable HPP-based vertical segmented local layout alignment. | | --uvt | str | None | Path to Unified View Tree (UVT) JSON file for semantic DOM selector mapping. | | --dilation | int | None | Morphological dilation kernel size (pixels). If omitted, dynamically computed based on width. | | --min-area | int | None | Minimum pixel area to filter out tiny noise. If omitted, dynamically computed. | | --merge-dist| int | None | Maximum pixel distance to merge nearby boxes. If omitted, dynamically computed. | | --out-panel | str | diff_panel.png| Output path for the side-by-side dashboard panel. | | --out-json | str | diff_report.json| Output path for the JSON discrepancy metadata. |

🐍 Python SDK Programmatic API

You can easily integrate Agent-Eye into your own AI Agent loops or visual testing suites:

from agent_eye import ImageCompareEngine, ImageCompareVisualizer
import cv2
import json

# 1. Initialize the engine with scale-independent calculations and layout alignment
engine = ImageCompareEngine(
    threshold=30,
    auto_layout=True  # Enables segmented layout-shift compensation
)

# 2. Load UVT JSON data
with open("real_uvt.json", "r") as f:
    uvt_data = json.load(f)

# 3. Run visual comparison under "details" attention mode
results = engine.compare(
    "design.png", 
    "screenshot.png", 
    align_method="pad",
    uvt=uvt_data,
    focus_mode="details"  # Targets text spelling, typos and fine padding mismatches
)

# 4. Read overall metrics
print(f"Overall SSIM Index: {results['overall_similarity'] * 100:.2f}%")

# 5. Traverse the semantic mappings
for reg in results["regions"]:
    print(f"\nRegion ID: #{reg['id']} | Severity: {reg['severity']}")
    print(f"Coordinates [x, y, w, h]: {reg['box']}")
    print(f"Mapped Selector: {reg.get('selector', 'None')}")
    print(f"Element Type: {reg.get('element_type', 'None')}")
    print(f"Mean BGR Color absolute difference (MAE): {reg['mean_color_diff']:.2f}")

📊 Exported JSON Schema (diff_report.json)

The generated JSON report contains high-fidelity selector mapping and severity categories:

{
    "overall_similarity_score": 0.7700,
    "overall_mismatch_percentage": 30.79,
    "regions_detected_count": 16,
    "focus_mode": "details",
    "auto_layout_aligned": true,
    "mismatched_regions": [
        {
            "id": 1,
            "box": [13, 95, 365, 78],
            "mismatch_ratio": 0.5218,
            "mean_color_diff": 20.24,
            "severity": "Critical",
            "selector": "div.phone-screen > div > div > div.tabs-scrollable > div.tab-item",
            "element_type": "container"
        },
        {
            "id": 2,
            "box": [10, 14, 95, 24],
            "mismatch_ratio": 0.6561,
            "mean_color_diff": 40.04,
            "severity": "Critical",
            "selector": "div.phone-screen > div > div > span",
            "element_type": "text"
        }
    ]
}

• box: [x, y, width, height] coordinates of the visual mismatch area on the logical viewport.
• selector: Surgical DOM/UI element path recursively matched from the Unified View Tree (UVT).
• severity: Discrepancy severity classification (Critical / Warning / Info).

📄 License

This project is licensed under the MIT License. Feel free to fork, distribute, or open Pull Requests!