@getpipher/vision

Capability-aware vision + paste extension for the pi coding agent.

@getpipher/vision adds a describe_image tool to pi that is aware of the active primary model's input modalities — so image analysis is never wasteful.

Why

Two community pi extensions — pi-vision-tool and pi-paster — are load-bearing but leave real workflow friction. The biggest: pi-vision-tool always delegates image analysis to a second vision model, even when the active primary model is multimodal and can see images natively. That's a wasteful double-call: extra API roundtrip, extra tokens, and a hardcoded description that's worse than the model's own native image understanding.

@getpipher/vision fixes this with capability-aware delegation:

• Multimodal primary model (e.g. minimax-m3:cloud, qwen3.5:cloud) → the image passes through to the model natively. Zero delegation calls, zero extra tokens, zero extra latency. The describe_image tool is hidden from the model entirely, so the wasted call is structurally impossible — not just discouraged.
• Text-only primary model → describe_image delegates to a configured vision model and returns the text description (the existing, working pattern).

The capability check is automatic and silent — no config, no opt-in, no flag. It just works under the hood.

Install

pi install npm:@getpipher/vision

@getpipher/vision replaces pi-vision-tool and pi-paster. Uninstall those first to avoid duplicate tools and double-attachment:

pi uninstall npm:pi-vision-tool
pi uninstall npm:pi-paster

Configure

The vision tool needs a configured vision model for DELEGATE mode (text-only primaries). Run /vision to open an interactive settings panel (pi's native SettingsList — the same engine /settings uses): arrow keys navigate, Enter cycles a value or opens the vision-model picker, Escape exits. Changes apply live.

/vision

Or use the typed subcommands (power users / scripts):

/vision provider ollama
/vision model minimax-m3:cloud

The provider and model must be defined in ~/.pi/agent/models.json and the model should have "input": ["text", "image"]. Other /vision subcommands:

| Subcommand | Purpose | |---|---| | /vision (no arg) | Open the interactive settings panel (like /settings) | | /vision show | Display the current config as text | | /vision model (no arg) | Open a picker of vision-capable authed models | | /vision on / /vision off | Enable / disable the tool | | /vision provider <name> | Set the vision provider | | /vision model (no arg) | Open a picker of vision-capable authed models | | /vision model <id> | Set the vision model id (typed fallback) | | /vision max-dim <px> | Max image dimension for compression (1–8000) | | /vision quality <1-100> | JPEG re-encode quality | | /vision reasoning-effort <off\|minimal\|low\|medium\|high\|xhigh> | Default reasoning effort for delegation | | /vision system-prompt [<text>\|clear] | Set/clear a custom system prompt for the vision model (no arg → multi-line editor) | | /vision cache <clear\|show> | Clear the cache or show stats (memory + disk entries) | | /vision fallback <provider/model>\|clear | Set/clear a fallback vision model | | /vision clear | Reset config to defaults | | /vision-use [provider/model] | Switch the DELEGATE vision model inline (no arg → picker). Hotkey: ctrl+shift+i (rebindable via keybindings.json; on Mac, alt-based combos need macos-option-as-alt=true) | | /vision paste-mode [hint\|auto\|off] | Set how pasted images are handled on a text-only primary (no arg → cycle). | | /vision marker-style [code\|bold\|plain] | Set the markdown style for [Image-#N] markers (no arg → show current). | | /vision auto-prompt [<text>\|clear] | Set/clear the generic auto-delegation prompt (no arg → multi-line editor). | | /vision preview <path> | Open a full-screen TUI preview of an image (Kitty/iTerm2 graphics, text fallback on tmux). | | /vision batch-concurrency [<1-20>] | Max parallel image delegations in a batch (describe_image image_paths + paste auto mode). 1 = serial; 20 = aggressive. Default 5. | | /vision local-only [on\|off] | Toggle local-only mode (no arg → show current). When on, image bytes never leave the machine — cache hits still work, a cache miss refuses with a clear error instead of a network call. | | /vision audit <clear\|show\|path\|on\|off> | Audit log management. clear truncates; show tails the last 10 entries + count; path prints the resolved path; on/off toggle logging. Default on. | | /vision audit-path [<path>\|clear] | Set/clear a custom audit log path (power users; no arg → show current). Default ~/.pi/agent/vision-audit.log. |

Config is stored at ~/.pi/agent/vision.json (not vision-tool.json, so it doesn't collide with the community package during transition).

Resilience (v0.2.0)

DELEGATE mode (text-only primary) is resilient + cheap:

• Caching. Successful delegation results are cached by a content-addressed key (image hash + compression params + prompt + vision model + reasoning). A second call on the same image costs zero vision-model API calls. In-memory by default; opt into cross-session persistence with Persist cache to disk (LRU-evicted at the configured max entries). Only successes are cached — failures never are. /vision cache clear wipes both layers.
• Retry + fallback. On a retryable failure (HTTP 5xx, 429, network), the primary vision model is retried with exponential backoff (abort-aware — a cancelled turn stops retrying immediately). On a non-retryable error or exhausted retries, a configured fallback vision model is tried once. Configure both via the /vision panel or /vision fallback <provider/model>.
• Custom system prompt. A per-workflow framing prepended to the vision-model request (/vision system-prompt <text>, or the panel row).
• Inline model switch. ctrl+shift+i (or /vision-use) switches the DELEGATE vision model mid-session without opening the full panel.

Paste UX (v0.3.0)

When you reference an image file path in your message, the paste hook makes it visible and actionable — capability-aware for both multimodal and text-only primaries.

• [Image-#N] markers. Image file paths in your message are replaced with [Image-#N] markers (sequential, 1-indexed). The marker style is configurable: code (inline code — default, visually distinct), bold, or plain. On a multimodal primary, the image is also attached as a native attachment (zero delegation). On a text-only primary, the marker is informational — the image is NOT attached (text-only models can't process images).
• Text-only paste modes. When the primary model is text-only, pasted images can't be processed natively. Three configurable modes:
  • hint (default): markers + a hint line nudging the model to call describe_image. Zero tokens — the model decides whether to delegate.
  • auto (opt-in): auto-delegates each image via the v0.2.x pipeline (cache/retry/fallback) and appends the descriptions to your message. Timeout-protected (own AbortController, default 30s) — falls back to hint on timeout or failure.
  • off: markers only — no attachment, no hint, no delegation.

Image Preview (v0.3.3)

@getpipher/vision can render images directly in the terminal — no external viewer needed.

Compose-time auto-preview. As you type a message referencing an image path, the image renders above the editor (WhatsApp/Telegram compose-box style) ~300ms after you stop typing. Clears when you remove the path or submit the message. Enable/disable via the /vision panel (Compose preview row) or the config field composePreview.

On-demand preview. /vision preview <path> opens a full panel showing the image + metadata (filename, dimensions, MIME, file size, detected protocol). Useful for checking an image before deciding to analyze it.

Terminal support:

| Terminal | Image rendering | |---|---| | Kitty, Ghostty, WezTerm, Warp (standalone) | ✅ Real graphics (Kitty protocol) | | iTerm2 (standalone) | ✅ Real graphics (iTerm2 protocol) | | tmux (any terminal) | Text fallback ([Image: filename image/png WxH]) | | VSCode, Alacritty, other | Text fallback |

The text fallback still shows useful metadata (filename, dimensions, format, file size) and confirms the image was found. Real graphics require running pi outside tmux on a graphics-capable terminal.

Batch + scale (v0.4.0)

describe_image accepts multiple images in a single call via image_paths (alongside the single image_path for back-compat). For text-only primaries (where the tool is visible), this lets the model analyze, compare, or cross-reference several images in one tool call instead of N serial round-trips:

describe_image(
  image_paths: ["/tmp/before.png", "/tmp/after.png", "/tmp/diff.png"],
  prompt: "Compare these three screenshots. What changed?"
)

Delegations run in parallel, bounded by batchConcurrency (default 5, configurable 1–20 via /vision batch-concurrency). 1 = serial escape hatch; 20 = aggressive (rate-limit risk is yours). Each image reuses the v0.2.x resilience pipeline (cache/retry/fallback) independently — a cache hit returns 0 vision-model calls, a failed image becomes an [error: …] section rather than failing the whole batch, and isError is set only if every image failed. The result is one structured, order-stable text block:

[Batch: 3 image(s)]

[Image 1] /tmp/before.png
<vision model description>

[Image 2] (cached) /tmp/after.png
<vision model description>

[Image 3] /tmp/diff.png
[error: not_found — image not found at /tmp/diff.png]

A hard cap of 50 images per call (MAX_BATCH_IMAGES) defends against an over-eager model passing an absurd array; split across calls if you need more.

Parallel auto-delegation (paste auto mode). When textOnlyPasteMode is "auto" and you paste multiple image paths, delegations now run in parallel (one batch-level timeout = the total budget, bounded by batchConcurrency) instead of serially — so a 5-screenshot paste completes in ~ceil(N/c) × per-call instead of N × per-call.

Hint mode now exposes paths. In text-only + "hint" mode (the default), the hint line now lists the image paths and names the image_paths batch affordance, so the model can actually invoke describe_image (previously the hint named the tool but the path markers erased the paths, leaving the model unable to call it). Paste 2+ images and the model learns it can pass them all to image_paths for batch analysis.

Clipboard paste just works. Pi binds ctrl+v (alt+v on Windows) to paste the system clipboard image: it reads the clipboard, writes the bytes to /tmp/pi-clipboard-<uuid>.<ext>, and inserts that path at the cursor. Our existing path-token pipeline detects it, renders a [Image-#N] marker, and attaches (multimodal) or delegates (text-only) — no separate clipboard code path needed. Multi-image clipboard = N ctrl+v presses = N paths = handled as a batch.

Config + security (v0.5.0)

v0.5.0 makes the tool trustable — audit where your images go, opt out of network delegation entirely, and get workflow-fit defaults out of the box.

Audit log

Every vision-model delegation (success, cache hit, fallback, or failure) is recorded in a persisted, append-only JSONL log at ~/.pi/agent/vision-audit.log (one line per delegation). The log answers "where did each image go?" without storing the image bytes or the full prompt.

Each line is a JSON object:

| Field | Meaning | |---|---| | ts | ISO 8601 timestamp of the delegation event | | provider | The configured primary provider (the attempted route) | | model | The model that actually responded (the fallback on a fallback success) | | image_path | The path the user passed (file paths full; data:URL/base64 truncated) | | source_hash | SHA-256 of the original image bytes (a content fingerprint — not the bytes) | | cached | true if served from the cache (0 network calls) | | fallback | true if the result came from the fallback vision model | | fallback_model | The fallback model id, if fallback was used | | ok | true if the delegation succeeded | | error_code | Error code on failure (local_only, vision_call_error, aborted, …) | | latency_ms | Round-trip latency (0 for a cache hit / local-only refusal) | | local_only | true if local-only mode was active |

Privacy stance: the log records routing (where bytes went), never content. Image bytes are never logged (only the source_hash fingerprint); the full prompt is never logged (the conversation log already has it). A data: URL or raw base64 image_path is truncated to the first 64 chars + a size suffix. The audit log is on by default (opt-out, not opt-in — the security posture is traceability without you remembering to enable it):

/vision audit off        # disable logging
/vision audit show       # tail the last 10 entries + total count
/vision audit clear      # truncate the log
/vision audit path       # print the resolved log path
/vision audit-path /tmp/my-vision.log  # custom location (power users)

Local-only mode

When local-only mode is on, image bytes never leave the machine. This is a structural guarantee, not a polite request — the network-call code path is never entered. Cache hits still work (the cache is local — memory + disk under ~/.pi/agent), so local-only mode is effectively "cache-only mode": previously- seen images get their cached descriptions, new images refuse with a clear error.

/vision local-only on    # refuse all network delegation (cache hits OK)
/vision local-only off   # allow delegation

In paste auto mode + local-only, the hook skips delegation entirely and falls straight to the hint line (no autoDelegateTimeoutMs burned waiting for a refused call to abort). The describe_image tool stays visible so the model can still retrieve cached descriptions and report the local-only error on a cache miss. Every local-only event is audited with local_only: true (greppable proof: grep '"local_only":true' ~/.pi/agent/vision-audit.log).

Workflow-fit defaults (auto-detect)

On a fresh install (or after /vision clear), the extension auto-detects the vision model at session start from ~/.pi/agent/models.json — aligned to the AGENTS.md LLM-backend policy of "Ollama Cloud primary + frontier escalation":

• Primary: prefers the Ollama provider's vision-capable models (the :cloud ones — Ollama Cloud, flat-rate + private). On RECTOR's setup this picks Ollama/minimax-m3:cloud (first by sorted id).
• Fallback: NOT auto-populated (v0.5.1). The frontier-escalation mechanism (the fallback field + the retry→fallback pipeline) is still there — set it explicitly via /vision fallback <provider/model> if you want a secondary vision model tried when the primary fails.
• The auto-detected primary is persisted once to vision.json (you see the choice + can override it). /vision clear resets to defaults and re-triggers detection on the next session start.
• Escape hatch: /vision auto-detect off (set autoDetectVisionModel: false) disables auto-detection entirely — a fresh config stays unconfigured.

Auto-detect only fires when both provider and model are unset (a truly fresh state). A partial config (one set, one blank) is you mid-configuration — it's not overwritten.

No config migration from pi-paster

pi-paster is a paste-format extension — it has no configurable vision model (it doesn't delegate; it just marks + attaches). @getpipher/vision's markerStyle (code/bold/plain) is a superset of pi-paster's fixed format, so there's nothing to migrate. Uninstall pi-paster, run /vision, done.

How it works

Two mechanisms combine to guarantee the behavior:

1. Conditional tool availability (mechanism A). describe_image is always registered, but its visibility to the LLM is toggled with pi.setActiveTools() based on the active model's input capability. On a multimodal primary, the tool is removed from the active set — the LLM never sees it, so it can never waste a call. On a text-only primary, the tool is added back so the LLM can delegate. The model_select event re-syncs on mid-session /model switches.

2. Capability-aware paste hook. When the primary model is multimodal, the input hook detects image file paths in your message, attaches them as native image content, and renders [Image-#N] markers (mechanism B-lite, graduated in v0.3.0). On a text-only primary, the hook renders markers but does NOT attach images — instead it hints, auto-delegates, or stays silent based on textOnlyPasteMode. The hook dedups by data hash so it coexists cleanly with other paste extensions during transition.

Using describe_image

The tool accepts a file path, data URL, or raw base64. For a single image:

describe_image(image_path: "/tmp/screenshot.png", prompt: "What's in this image?")

For multiple images (batch — parallel delegation, one structured result):

describe_image(
  image_paths: ["/tmp/a.png", "/tmp/b.png", "/tmp/c.png"],
  prompt: "Compare these screenshots. What changed between them?"
)

Parameters:

| Param | Type | Description | |---|---|---| | image_path | string? | Path to a single image, data: URL, or raw base64. Use for one image. | | image_paths | string[]? | Multiple paths to analyze together (comparison/cross-reference). Up to 50. | | prompt | string | What to analyze or answer about the image(s). For a batch, applies to each; describe what to compare. | | compress | boolean? | Optimize the image(s) before delegation (default true) | | reasoning | enum? | Reasoning effort for the delegation (off…xhigh) |

When caching or fallback is active, the tool result details include cached: true (cache hit) and fallback: true (result from the fallback model) for traceability. For a batch, details.batch is an array of per-image results (index, path, ok, cached, fallback, errorCode) in input order.

For multimodal primaries you don't call describe_image — just reference the image path in your message and the model sees it natively.

Credits

Inspired by pi-vision-tool (xezpeleta) and pi-paster (beowulf11) — both filled this gap first and motivated this clean-room reimplementation. Superseded by @getpipher/vision.

This package contains no code copied from either project. The behavior is specified independently and implemented fresh.

License

MIT