pi-image-vision

将粘贴的图片转成文字，为纯文本模型提供视觉兜底。 Paste an image into pi and get its content transcribed as text — a vision fallback for text-only LLMs.

把截图、照片、图表粘贴到 pi 时，本插件调用你已配置的视觉模型（如 MiMo），将图内内容转录为文字并注入对话，让纯文本模型（如 DeepSeek）也能"看到"图片。 When you paste a screenshot, photo, or diagram, this extension calls a vision model from your configured pi providers (e.g. MiMo), converts the image content into text, and feeds that text to your main model — so text-only models like DeepSeek can "see" images too.

功能特性 / Why

很多 pi 模型是纯文本的，没有本插件时粘贴图片要么失败、要么被静默丢弃。本插件： Many pi models are text-only. Without this extension, pasting an image either fails or is silently dropped. This extension:

• 在图片到达模型之前拦截 Intercepts pasted images before they reach the model
• 通过 pi 内置的 Photon WASM 压缩管线压缩超大图片，减小视觉 API 请求体（GIF 动画跳过压缩、原样直发） Compresses oversized images (via pi's built-in Photon WASM pipeline) to keep the vision API request small (animated GIFs are sent as-is)
• 交给视觉模型做纯转录（全部文字、数字、表格，保持排版顺序），不做任何解读或回答 Sends them to a vision model for pure transcription (all text, numbers, tables, layout order) — no interpretation, no preliminary answers
• 将转录注入对话并移除原图，纯文本模型据此正常响应 Injects the transcription into your conversation and removes the raw image, so the text-only model can actually respond to it
• 主模型支持视觉时自动跳过：当前模型 input 含 image（如 GPT-4o、Claude、MiMo）则原图直发，模型原生看图、细节无损 Vision-capable main models are skipped automatically: if your current pi model supports image input, the raw image passes through untouched. Set PI_VISION_FORCE_ENABLE=1 (or "forceEnable": true) to intercept anyway.

安装 / Install

pi install npm:pi-image-vision

安装后需 /reload 或重启会话生效。 Requires a running pi session with extension hot-reload (/reload) or a restart.

配置 / Setup

两种配置方式，优先级：环境变量 > 配置文件 > 默认值。 Two ways to configure; precedence: environment variable > config file > default.

配置文件（与插件同名 pi-image-vision.json，在 ~/.pi/agent/ 下，可用 PI_VISION_CONFIG_PATH 覆盖）： Config file (named after the plugin, pi-image-vision.json under ~/.pi/agent/, override with PI_VISION_CONFIG_PATH):

{
  "lang": "zh",
  "forceEnable": false,
  "pasteEnabled": true,
  "toolEnabled": true,
  "provider": "opencode-go",
  "modelName": "mimo-v2.5",
  "baseUrl": "https://opencode.ai/zen/go/v1",
  "timeoutMs": 30000,
  "maxDimension": 1600,
  "maxBytes": 1572864,
  "jpegQuality": 80
}

对应环境变量（临时覆盖文件，如 PI_VISION_TOOL_ENABLED=0 关闭工具注入）： Each file key maps to a VISION_* environment variable for one-off overrides (e.g. PI_VISION_TOOL_ENABLED=0 to skip tool injection):

插件只使用 pi 中已配置的视觉模型（复用供应商的 key/baseUrl/headers，无需单独管理密钥）。 The extension works only with vision models from your pi providers — the provider's own API key, baseUrl and headers are reused automatically.

在 pi 中配置好视觉供应商（如 xiaomi/MiMo、OpenAI、智谱…），然后选择模型： Configure a vision-capable provider in pi, then pick a model:

# 交互选择（推荐，持久化保存）：/vision-model
# interactive picker, saved for reuse:

# 或环境变量指定（优先于已保存的选择）
# or pin non-interactively via env (overrides the saved choice):
PI_VISION_PROVIDER=xiaomi
PI_VISION_PROVIDER=openai PI_VISION_MODEL_NAME=gpt-4o

选择顺序：PI_VISION_MODEL_NAME/PI_VISION_PROVIDER 环境变量 → /vision-model 保存的选择。从未配置过模型时插件不生效——粘贴图片原样通过。 Selection order: explicit PI_VISION_MODEL_NAME/PI_VISION_PROVIDER env → config-file provider/modelName (written by /vision-model) → defaults. If you never configured a model, the extension does nothing — pasted images pass through untouched. The /vision-model choice is written straight into ~/.pi/agent/pi-image-vision.json, so the config file always shows the active model.

| 环境变量 / Env var | 默认 / Default | 说明 / Description | | --- | --- | --- | | PI_VISION_LANG | zh | 注入标签/提示词/通知的界面语言：zh 或 en — UI language for injected labels, prompts and notifications | | PI_VISION_TOOL_ENABLED | true | 设 0 不注册 read_image 工具（避免与其他识图工具冲突）；粘贴拦截不受影响 — set 0 to skip registering the read_image tool; paste interception is unaffected | | PI_VISION_PASTE_ENABLED | true | 设 0 关闭粘贴拦截（只留 read_image 工具）— set 0 to disable paste interception (tool-only mode) | | PI_VISION_FORCE_ENABLE | — | 设 1 主模型支持视觉也强制拦截 — intercept even when the main model supports vision | | PI_VISION_PROVIDER | — | 限定供应商 id（如 xiaomi）— restrict selection to one pi provider id | | PI_VISION_MODEL_NAME | — | 指定模型 id（如 mimo-v2.5、gpt-4o）— pin a specific model id | | PI_VISION_BASE_URL | — | 高级：覆盖供应商解析出的 baseUrl — override the provider's resolved base URL | | PI_VISION_MAX_DIMENSION | 1600 | 压缩前边长上限（px）— max image side length before compression | | PI_VISION_MAX_BYTES | 1572864 | 压缩前 base64 大小上限（字节）— max base64 size before compression | | PI_VISION_JPEG_QUALITY | 80 | 重编码 JPEG 质量 1-100 — JPEG quality when re-encoding | | PI_VISION_TIMEOUT_MS | 30000 | 单张图片视觉 API 请求超时（毫秒）— per-image vision API request timeout |

使用 / Usage

正常粘贴图片（Ctrl+V / Cmd+V）即可。扩展转录后注入对话，来源一目了然： Just paste an image as usual. The transcription is injected with clear provenance:

# 中文 / zh
[用户说明]：这个报错什么意思        ← 用户原话，主模型区分"用户说的"
[图片描述 · 经视觉模型识别转换]      ← 扩展注入的标签
【图内原始内容】
- TypeError: Cannot read property 'x' of undefined
- 位置：src/main.js:42

# English / en（PI_VISION_LANG=en）
[User note]: what does this error mean
[Image description · transcribed by vision model]
[Image content]
- TypeError: Cannot read property 'x' of undefined
- at src/main.js:42

处理深度完全由主模型结合用户说明与转录内容自行判断——没有多余的判断层，token 最少。 The main model decides how deep to go based on the user note plus the transcription — no extra judgment layers, minimal tokens.

识图工具 / read_image tool

纯文本主模型还可主动调用 read_image 工具读取本地图片文件——当对话中提到图片/截图路径时（如 ~/screenshots/error.png），模型可自行调用转录，无需用户粘贴： Text-only main models can also actively call the read_image tool to transcribe a local image file — when the conversation references an image path (e.g. ~/screenshots/error.png), the model invokes it itself, no pasting needed:

read_image(path: "~/screenshots/error.png")
→ Error: connection refused
  - at client.js:12

工具与粘贴拦截共用同一管线（压缩 + 纯转录），同样需要已配置视觉模型（/vision-model）。注册条件：已配置视觉模型 + toolEnabled + （主模型不支持视觉 或 forceEnable）。未配置模型、或主模型本身支持视觉时工具不注册——模型工具列表不出现无用项。会话启动时评估；切换模型后 /reload 重新评估。 The tool shares the same pipeline as paste interception (compress + pure transcription) and also requires a configured vision model (/vision-model). Registration conditions: a configured vision model + toolEnabled + (main model lacks vision OR forceEnable). Not registered when no model is configured or when the main model is vision-capable — no useless entries in the tool list. Evaluated at session start; /reload re-evaluates after switching models.

原理 / How it works

1. pi 的 input 事件在 transformMessages 把图片替换成占位符之前触发，event.images 仍是完整 base64 pi's input event fires before transformMessages replaces images with placeholders — event.images still holds the full base64.
2. 超阈值的图片用 resizeImage 压缩（pi 内置 Photon WASM，工作线程执行不阻塞 TUI） Images over threshold are compressed with resizeImage (pi's Photon WASM, runs in a worker thread).
3. 每张图发给视觉模型做纯转录（固定 [CONTENT] 协议，语言无关解析），结果按图标注后拼接 Each image is sent with a pure-transcription prompt (fixed [CONTENT] protocol, language-independent parsing); results are joined with per-image labels.
4. 处理器返回 { action: "transform", text, images: [] } —— 注意 images: []（而非 undefined）才能真正移除图片 The handler returns { action: "transform", text, images: [] } — images: [] (not undefined) is required to actually strip the images.

开发 / Develop

npm install
npm run typecheck
npm test

许可证 / License

MIT