@instruments/visual-grounding

One streaming contract for evidence-bearing observations of visual surfaces.

import { analyzeVisualSurfaces } from "@instruments/visual-grounding";
import { createOpenRouterDetector } from "@instruments/visual-grounding/openrouter";

const pipeline = {
  detector: createOpenRouterDetector({ model: "openai/gpt-5.6-luna" }),
};

for await (const event of analyzeVisualSurfaces(pipeline, {
  image,
  instruction: "Detect every distinct interior material surface.",
  capabilities: ["geometry", "semantics", "segmentation"],
  sessionId,
})) {
  if (event.type === "surface.detected") {
    drawBox(event.surface);
  }
  if (event.type === "surface.refined") {
    inspect(event.capability, event.observation);
  }
}

Capabilities

Consumers request outcomes rather than pipeline stages:

• geometry detects surface occurrences and normalized boxes;
• semantics adds descriptions and controlled vocabulary fields;
• segmentation produces a surface mask;
• ownership identifies the eroded, source-sized mask interior whose pixels are eligible for color measurement;
• color returns the observed palette, spot, ColorVec-compatible embedding, exact pixel counts, ownership mask, and source dimensions.

Requesting ownership or color plans segmentation automatically. Requesting color also plans ownership. Ownership completes only when a color profiler returns its exact eroded data-url mask; the raw segmentation mask is not reported as measured pixels.

Events and outcomes

surface.detected streams usable geometry immediately. Every surface.refined event identifies the capability that settled and carries a complete immutable observation snapshot. complete contains both the legacy surfaces collection and the evidence-bearing observations collection.

Each requested capability settles as one readonly outcome:

• completed carries a value and a receipt naming the method;
• failed carries a technical error message;
• abstained carries the evidence-based reason the package could not make the observation.

A failed or abstained refinement does not discard successful geometry or other completed capabilities. Segmentation failure causally abstains ownership and color, while a profiler exception fails ownership and color without discarding segmentation.

Legacy detail presets

detail remains as a compatibility preset for existing callers: "boxes" maps to geometry and "masks" maps to geometry plus segmentation. When both fields are supplied, capabilities takes precedence. Detail callers continue to receive surface.enriched, surface.segmented, and surface.failed; capability callers receive surface.detected, surface.refined, and complete.

Provider and model choices remain behind adapters. Catalogue retrieval, ranking, and product truth remain downstream.