pi-vision-bridge-tomoyo

Give your text-only model (e.g. DeepSeek V4 Flash) real image understanding: images pasted into the chat and screenshots are described by a cheap vision model (default MiniMax M3 via opencode.go) and fed to the main model as text.

Zero-blocking · zero-clutter · no redundant reads.

Why

DeepSeek V4 Flash is officially text-only — no API path accepts image input. The industry-standard workaround (and the one DeepSeek's own Copilot integration recommends) is vision-agent bridging: route the image to a vision model, get a text description back, and hand the description to the text model.

This extension automates that bridge inside pi, tuned for good UX: no frozen UI on send, no giant transcripts polluting the chat, no double-reads by the model.

How it works

Three cooperating layers, all sharing one vision engine and one content-hash cache:

1. input hook — non-blocking prefetch. When a message contains an image path, the extension immediately returns the message untouched (zero wait, zero injection) and kicks off a background vision description (fire-and-forget) that fills the cache.
2. before_agent_start — inject or guide, zero block. After your message is displayed and before the agent starts thinking, the extension synchronously reads the cache:
   • cache hit → the description is appended to this turn's system prompt (invisible in the chat UI), with an explicit note telling the model "already described — do NOT call describe_image again";
   • cache miss → a one-line guide is appended instead ("call describe_image to view this image"), while prefetch keeps running so the tool call will likely hit the cache.
3. Tool layer — describe_image(path, question?). The model can call it on demand for fine-grained reading (error dialogs, UI, charts), optionally with a specific question. Cache hits return instantly.

capability-aware: if the main model already supports images (model.input includes image), every layer is bypassed — zero overhead, zero interference.

Why not just "let the model call the tool itself"?

The model calling describe_image on its own is the primary path (and the guided path above is exactly that). But making it the only path would gamble the core feature on the model's initiative:

• Some models treat a pasted image path as a "suggestion" rather than an instruction and silently hallucinate the image content (pi issue #6373, #3429/#3110).
• The prefetch + injection layer guarantees the description is available to the model even when it never decides to call the tool — at no extra cost, because the cache makes a later tool call free.

So: model autonomy first, deterministic fallback second, zero redundant cost.

Install

pi install npm:pi-vision-bridge-tomoyo

Requires an opencode.go subscription (opencode-go.key in ~/.pi/agent/auth.json, or set PI_VISION_API_KEY).

Configuration (environment variables)

| Variable | Default | Description | |---|---|---| | PI_VISION_MODEL | minimax-m3 | Vision model id (any image-capable model on opencode.go, e.g. mimo-v2.5, gpt-5.6-luna) | | PI_VISION_BASE_URL | https://opencode.ai/zen/go/v1 | OpenAI-compatible endpoint | | PI_VISION_API_KEY | read from auth.json | API key | | PI_VISION_AUTO_DESCRIBE | 1 | Set 0 to disable the automatic layers (keep only the describe_image tool) |

Model selection

Screened by cost × quality experiments against opencode.go pricing:

• Default minimax-m3 ($0.30/$1.20 per 1M tokens): highest measured quality, low latency (~8s).
• Budget mimo-v2.5 ($0.14/$0.28): nearly as good at half the price.
• Avoid grok-4.5, kimi-k3, qwen3.8-max (too expensive) and glm-5.2 (accepts images but is actually text-only — a "fake accept").

Verified

• 18 unit tests (cache hit, API failure fallback, timeout abort, oversized rejection, question passthrough, path extraction incl. Chinese full-width colon).
• Real end-to-end: a pasted screenshot of a build error is transcribed by MiniMax M3 and DeepSeek V4 Flash answers the error, file and line correctly.
• System-prompt-tail injection is actually read: with a ~1.1k-token system prompt and the description at the very end, DeepSeek V4 Flash reproduced all key facts from the description; without it, it honestly admitted it cannot see the image.
• Coexists with pi-task-orientation: in a live run the model first completes skill assessment, then calls describe_image — no deadlock, no interference.

Development

# unit tests
node --experimental-strip-types --test test/*.test.ts

# type check
npx -y typescript tsc --noEmit -p tsconfig.check.json

License

MIT

中文说明

为纯文本主模型（如 DeepSeek V4 Flash）补图像理解能力：粘贴进对话框的图片、以及模型需要查看的截图，都由一个廉价视觉模型（默认 MiniMax M3，走 opencode.go）转成文本描述后交给主模型。

零阻塞 · 零污染 · 无重复读取：

• 提交消息时后台预取图片描述（不卡界面、不往消息里塞内容）；
• agent 开始前把已就绪的描述注入系统提示（对话里不可见），未就绪则引导模型调用 describe_image 工具；
• 同一图片只描述一次（内容哈希缓存，自动层与工具层共享）。

主模型本身支持图像时自动旁路。视觉引擎默认 MiniMax M3，可用 PI_VISION_MODEL 切换（mimo-v2.5 更省钱）。