deep-semantic-scanner

A framework-neutral, robust semantic website scanner that extracts a compact, rich semantic graph from a rendered DOM structure. Perfect for accessibility analysis, automated testing, screen-reader-like parsing, and building intelligent autonomous agents that interact with web interfaces.

Features

• 🌳 Semantic Graph Extraction: Converts a complex DOM tree into a lightweight graph of semantic nodes containing roles, accessible names, descriptions, and state.
• 👀 Visibility & Layout Tracking: Accurately computes whether elements are visible and captures their bounding rectangles and screen coordinates.
• ♿ Accessibility (a11y) First: Computes accessible names (via aria-label, aria-labelledby, text content) and resolves standard ARIA attributes and roles.
• 🤖 WebAgent Action Layer: A full agent runtime on top of the scanner — free-text intent matching, synthetic clicks / typing / key presses / selects / drag-and-drop, multi-step plans, wait-for-element retries, typed errors, and a lifecycle event stream that drives virtual-cursor visualizations. See the WebAgent Guide, including the LLM-driven planning loop.
• 🔗 Relationship Mapping: Automatically links <form> elements to their controls and <label> elements to their associated inputs.
• ⚡ Mutation Observation: Includes an optimized DOM observer that incrementally updates the semantic tree on DOM changes without requiring a full rescan.

Installation

npm install deep-semantic-scanner

CDN (no build step)

<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/browser/deep-semantic-scanner.min.js"></script>
<script>
  const { scan, WebAgent } = DeepSemanticScanner;
  console.log(scan({ root: document.body }));
</script>

Or as an ES module:

<script type="module">
  import { scan } from "https://cdn.jsdelivr.net/npm/[email protected]/dist/browser/deep-semantic-scanner.esm.min.js";
</script>

See the CDN Guide for unpkg, versioning, and file details.

Quick Start (Agent Workflow)

The fastest path is the high-level WebAgent, which wraps the scanner with intent matching and real action execution:

import { WebAgent } from "deep-semantic-scanner";

const agent = new WebAgent({ root: document.getElementById("app") });

agent.scan();                                      // perceive the page
const matches = agent.query("subscribe button");   // free text → ranked candidates

await agent.click("subscribe button");             // or act on a string directly
await agent.type("work email", "[email protected]"); // char-by-char, framework-safe
await agent.execute([                              // multi-step plans
  { action: "select", target: "country", value: "Germany" },
  { action: "check",  target: "agree terms" },
  { action: "click",  target: "create account" },
]);

// Drive a virtual cursor / highlight overlay from the event stream:
agent.events.on("move",  ({ to, duration }) => flyCursor(to, duration));
agent.events.on("click", ({ point }) => ripple(point));

Full documentation — options, all actions, the complete event table, error codes, a copy-paste virtual cursor, and the ai-agent-dom-demo.html wiring — is in docs/AGENT-GUIDE.md.

Low-level workflow (scanner only)

You can use SemanticScanner (which aliases to PerceptionEngine) directly to bridge the gap between an LLM's intent and physical DOM actions.

import { SemanticScanner } from "deep-semantic-scanner";

// 1. Initialize the scanner. Defaults to using `document` as the root.
const scanner = new SemanticScanner();

// 2. Perform the scan to extract the current semantic tree
scanner.scan();
const tree = scanner.getSemanticTree();
console.log(`Found ${tree.length} interactive semantic nodes.`);

// 3. Find a specific node based on an LLM query (e.g. "submit button")
const submitNode = scanner.findByRole("button").find(n => n.name?.includes("Submit"));

if (submitNode) {
  // 4. Safely retrieve the live DOM element using the node's internal reference ID
  const el = scanner.getElement(submitNode.ref);
  
  if (el) {
    // 5. Perform the physical action
    el.click();
  }
}

Configuration Options

When instantiating the SemanticScanner, you can customize its behavior:

const scanner = new SemanticScanner({
  // Limit scanning to a specific container (default is global document)
  root: document.getElementById('my-app'), 

  // Whether to include visually hidden elements (default: true)
  includeHidden: false,  
  
  // Whether to include structural/layout-only elements like divs (default: true)
  includeLayout: false,  
  
  // Extra data attributes to scrape into the node's `attributes` dictionary
  attributes: ['data-testid', 'data-cy'],

  // Max shadow-root depth to descend into (default: 32)
  maxShadowDepth: 32,

  // Max iframe nesting depth to descend into, same-origin only (default: 5)
  maxFrameDepth: 5
});

API Reference

Lifecycle

• scanner.scan(): Performs a synchronous, one-shot scan of the DOM and returns a ScanResult.
• scanner.refresh(): Alias for .scan(), useful for forcing a manual refresh.
• scanner.observe(callback, options): Starts a MutationObserver. The callback receives a fresh ScanResult efficiently patched after DOM changes.
• scanner.disconnect(): Stops observing DOM mutations.

Semantic Queries

These helpers allow you to query the internal state after a scan:

• scanner.findByRole(role: string): Returns all SemanticNodes matching a specific ARIA or HTML role (e.g. "button", "navigation").
• scanner.findByAccessibleName(name: string): Performs a case-insensitive search matching the accessible name.
• scanner.findByLabel(label: string): Same as findByAccessibleName, meant for form controls.
• scanner.findByPlaceholder(placeholder: string): Returns nodes matching the given placeholder string.

Agent Action Binding

• scanner.find(ref: number): Retrieves a specific SemanticNode by its unique reference ID.
• scanner.getElement(ref: number): Returns the physical, live DOM Element associated with the reference ID. Use this right before performing clicks or keyboard events to avoid retaining stale DOM nodes.

Tree & Component Retrieval

• scanner.getSemanticTree(): Returns the array of parsed SemanticNode objects.
• scanner.getDocumentTree(): Returns the hierarchical document structure.
• scanner.getComponents(): Returns composite/complex components identified during the scan.
• scanner.getForms(): Returns parsed forms with linked form controls.
• scanner.getLandmarks(): Returns structural landmarks (e.g., main, nav, banner).
• scanner.getOrders(): Returns navigation orders (readingOrder, tabOrder, visualOrder, focusOrder, accessibilityOrder).
• scanner.getStatistics(): Returns metadata like total nodes, scan time, and depth.

Serialization

• scanner.serialize(): Converts the current semantic tree to a compact string format.
• scanner.serializePretty(): Serializes the tree in a highly readable, indented format.
• scanner.serializeOutline(): Serializes the hierarchical document tree as an indented outline.
• scanner.serializeDebug(): Verbose dump including state, geometry, and relationships — for debugging.

SemanticNode Structure

Every element discovered in the tree is represented as a SemanticNode containing rich context:

interface SemanticNode {
  ref: number;                // Unique internal reference ID (stable across rescans)
  role: string;               // Semantic role (e.g. 'button', 'textbox')
  name?: string;              // Accessible name
  description?: string;       // Accessible description
  tag: string;                // HTML tag name

  // Visibility flags
  visible: boolean;           // In render tree, not css-hidden, has area
  painted: boolean;           // Actually painted inside the current viewport
  interactable: boolean;      // Visible, enabled, receives pointer events
  clickable: boolean;
  focusable: boolean;

  state: SemanticState;       // disabled, focused, checked, invalid, expanded, …
  geometry?: Geometry;        // viewport/document rects, center point, visual order

  relationships: {
    parent?: number;
    children: number[];
    labels: number[];         // Connected <label> elements
    labelFor?: number;        // If this node IS a label, the control it names
    form?: number;            // Enclosing <form> element
    controls: number[];       // aria-controls targets
    describedBy: number[];    // aria-describedby targets
  };

  depth: number;              // Depth within the semantic tree
  order: number;              // Document order index
  inShadow: boolean;          // True when inside a shadow root
  inFrame?: boolean;          // True when inside a same-origin iframe document
  frameRef?: number;          // Ref of the enclosing <iframe> host node
  crossOrigin?: boolean;      // True on an opaque cross-origin <iframe> node
  accessible?: boolean;       // False on a cross-origin iframe (cannot descend)
  landmark?: string;          // Landmark kind, when this node is one

  value?: string;             // Input value / leaf text (passwords never exposed)
  href?: string;
  src?: string;
  attributes?: Record<string, string>; // Captured attributes of semantic interest
}

Iframes

The scanner descends into same-origin iframes automatically (bounded by maxFrameDepth, default 5). Nodes discovered inside a frame carry inFrame: true and a frameRef pointing at the enclosing <iframe> node, and their geometry is translated into the top-level document's viewport so WebAgent clicks/types land on the right pixel. Shadow DOM inside a frame composes normally.

Cross-origin iframes are a browser security boundary: their content document is unreachable by any script. The scanner catches the SecurityError, never aborts the scan, and represents the frame as a single opaque node with crossOrigin: true and accessible: false. This is a documented limitation of the web platform, not a bug — there is no way to perceive or drive elements inside a cross-origin frame from the parent page.

WebAgent API (summary)

| Member | Purpose | |---|---| | new WebAgent(options) | Scanner options + pacing (moveDuration, hoverDelay, typeDelay, stepDelay), resolveTimeout (retry resolution on async-rendering SPAs), requireVisible, logger | | agent.scan() / agent.query(text) / agent.locate(text) | Perceive the page; rank nodes against free text | | agent.waitFor(text, { timeout }) | Poll until a query matches (late-rendering elements) | | agent.click / doubleClick / hover / focus / scrollTo (target) | Pointer actions with full synthetic event sequences | | agent.type(target, text) / select(target, value) / setChecked(target, bool) | Form actions (labels auto-resolve to their controls) | | agent.press(target, key) | Keyboard key (Enter, Escape, Tab, arrows…); Enter submits the enclosing form | | agent.drag(source, destination) | HTML5 drag-and-drop with interpolated pointer travel | | agent.execute(steps) | Sequential multi-step plans; stops on first failure | | agent.events.on(event, fn) | Lifecycle stream: move, hover, click, type:char, scroll:start/end, risk:flagged, action:end, error, … | | agent.cancel() / agent.isBusy / agent.dispose() | Control & cleanup | | rankByIntent(nodes, query, opts) | Pure scoring function, usable without a WebAgent | | AgentError | Typed failures: TARGET_NOT_FOUND, TARGET_DISABLED, INVALID_ARGUMENT, CANCELLED, CONFIRMATION_REQUIRED, CONFIRMATION_DENIED, … | | classifyRisk(node, keywords?) | Heuristic "low"/"high" risk tier; gate destructive actions via confirmHighRisk / onConfirm / riskClassifier |

Targets can be a free-text string, a semantic ref number, a SemanticNode, or a raw Element. Actions return ActionResult and never throw.

Destructive-action safety: by default (confirmHighRisk: true) actions whose target looks destructive (delete, pay, submit, transfer…) don't execute unsupervised — supply onConfirm to approve/decline, or they fail with CONFIRMATION_REQUIRED so a human can be asked. See docs/AGENT-GUIDE.md §8–9.

See docs/AGENT-GUIDE.md for the full reference and the demo integration walkthrough (ai-agent-dom-demo.html).

Demo

ai-agent-dom-demo.html is a live playground: an agent console co-browsing a mock SaaS page, with a virtual cursor rendered purely from agent.events. Build the SDK first, then serve the folder (ES modules don't load from file://):

npm run build
npx serve .          # or: python3 -m http.server
# open http://localhost:3000/ai-agent-dom-demo.html

License

ISC