👁️ @bismawy/pi-vision-watcher

Give text-only pi models vision capabilities.

Seamlessly inspect, describe, and convert visual inputs (screenshots, mockups, terminal errors, clipboard pastes) into structured descriptions using your preferred vision model, and hand them off to text-only coding models without interrupting your workflow.

⚡ Quick Start

1. Installation

pi install npm:@bismawy/pi-vision-watcher

(Or install directly from Git: pi install git:github.com/bismawy/pi-vision-watcher)

2. Select Vision Model

Open the interactive TUI selector to choose your vision describer model from your connected providers:

/vision-watcher

(You can also set it directly: /vision-watcher model openai/gpt-4o)

3. Work Seamlessly

Switch to any text-only model in Pi (e.g. DeepSeek, Claude text-only, local models). Whenever you paste an image, attach a file, or the agent runs read on an image, pi-vision-watcher describes it automatically in the background.

🚀 Key Capabilities

• 🎯 Connected-Only Interactive Picker: Shows only vision-capable models from providers where you actually have active credentials (/login, models.json, or environment variables).
• ⚡ DataLoader Batching & SHA-256 Cache: Automatically groups multiple images across parallel tool calls or multi-file prompts into a single batched describer request. Cached images are never re-described.
• 🛡️ Proactive False-Vision Healing: Aggregator providers often mistakenly flag models (like DeepSeek V4) as multimodal, causing HTTP 400 errors (This model does not support image). pi-vision-watcher proactively forces handoff for these models and auto-heals models.json modelOverrides in-process.
• 🧠 Thinking & Reasoning Controls: Adjust reasoning levels (off, minimal, low, medium, high, xhigh, max) for reasoning-capable vision models (o-series, Claude, DeepSeek).
• 🔄 Multi-Model Fallback Chains: Automatically falls back to backup vision models if your primary provider is rate-limited or unavailable.

🕹️ Command Reference

| Command | Action | |---|---| | /vision-watcher | Open interactive TUI picker for connected vision models | | /vision-watcher model <provider/id> | Set primary vision describer directly | | /vision-watcher status | View current configuration and active model status | | /vision-watcher auto <on\|off> | Toggle automatic handoff for non-vision models (default: on) | | /vision-watcher add <provider/id> | Force handoff on a specific model | | /vision-watcher remove <provider/id> | Remove model from forced handoff list | | /vision-watcher thinking <level> | Configure reasoning effort for vision models | | /vision-watcher enable / disable | Toggle extension active state | | /vision-watcher help | Show in-CLI command documentation |

📖 Deep Dive & Advanced Configuration

Configuration is stored at ~/.pi/agent/extensions/pi-vision-watcher.json:

{
  "enabled": true,
  "visionModel": "openai/gpt-4o",
  "fallbackModels": [],
  "autoHandoff": true,
  "handoffModels": [],
  "thinking": false,
  "thinkingLevel": "medium",
  "describeTimeoutMs": 45000,
  "prewarmPastedImages": false,
  "asyncClipboardHandoff": false,
  "maxTokens": null,
  "cacheMax": 50,
  "maxDescriptionLines": 0
}

| Field | Type | Default | Description | |---|---|---|---| | enabled | boolean | true | Master switch for handoff processing. | | visionModel | string \| null | null | Primary describer model ref (provider/id). | | fallbackModels | string[] | [] | Ordered backup models if the primary model fails. | | autoHandoff | boolean | true | Automatically describe images for models lacking native vision. | | handoffModels | string[] | [] | Specific model IDs forced to receive descriptions. | | thinking | boolean | false | Enable reasoning tokens for vision model. | | thinkingLevel | string | "medium" | Reasoning effort (minimal, low, medium, high, xhigh, max). | | describeTimeoutMs | number | 45000 | Per-batch timeout before aborting or triggering fallbacks. | | prewarmPastedImages | boolean | false | Start describing clipboard images immediately upon pasting in prompt. | | asyncClipboardHandoff | boolean | false | Async clipboard injection fallback mechanism. | | maxTokens | number \| null | null | Max output tokens for descriptions (null = model default). | | cacheMax | number | 50 | Maximum cached image hashes per session. | | maxDescriptionLines | number | 0 | Truncate lines in description block (0 = full description). |

Structured Error Logging

If a vision call fails, errors are appended with stack traces and request metadata to:

~/.pi/agent/logs/pi-vision-watcher/errors.log

Failures degrade gracefully to [Image: description unavailable] without breaking the agent turn.

False-Vision Auto-Recovery

When a model falsely advertises image capability and returns an HTTP 400 rejection:

1. pi-vision-watcher captures the error in the message_end event.
2. It automatically updates ~/.pi/agent/models.json under providers.<name>.modelOverrides.<model>.input = ["text"].
3. It triggers an in-process registry refresh so subsequent turns use handoff naturally.

bun install
bun run test          # Run Vitest test suite (240+ unit tests)
bun run typecheck     # Run TypeScript compiler check
bun run lint:dead     # Scan for unused exports with Knip

📜 License & Acknowledgments

• Built for the pi coding agent ecosystem.
• Evolved from concepts in pi-vision-handoff by Tom X Nguyen and pi-umans-provider.
• Distributed under the MIT License.