👁️ dsh-vision-handoff

Give text-only DeepSeek Harness models vision

Describe images with a vision model you pick, then hand the text to models that can't see.

dsh-vision-handoff adapts pi-vision-handoff into a native DeepSeek Harness plugin and bundle. It hooks the llm/stream waterfall — one layer before the runtime's own text-only image projection — and swaps each image block in the outbound request for an exhaustive description produced by a vision model you pick in Settings → Vision Handoff. The durable session log keeps its images; the text-only model reads vivid text instead of a bare placeholder.

The implementation composes with Harness rather than building a parallel pipeline: DSH keeps provider/model identity, request lifecycle, attachments, and settings persistence. Vision Handoff adds one request transform, a content-addressed description cache, and its Settings section.

Why vision handoff?

| | Capability | What it unlocks | | :-: | --- | --- | | ⚡ | Request-time swap | Image blocks become description text on a cloned request, ahead of adapter dispatch — every text-only model can suddenly see. | | 🖼️ | Batched describes | All un-described images in a request ride ONE vision call with <<<IMAGE k>>> sections; unparseable batches fall back to parallel single-image calls. | | 🗂️ | Content-addressed cache | Attachment ids are sha256, so a described image never costs a second vision call in the process's lifetime. | | 🧠 | Automatic targets | Applied to every model whose adapter omits image from inputModalities; explicit extraTargets force it even for vision-capable routes. | | 🔁 | Retry + failover | A totally failed batch retries once, then walks your fallbackModels list — a transient blip never costs the agent its images. | | 🛡️ | Graceful degradation | Unresolvable route, provider error, empty answer? The image stays put and the runtime's native [image omitted …] projection takes over. Failures are never cached. | | 🗣️ | Aware prompt (opt-in) | A system note tells text-only targets they CAN answer image questions, for models that otherwise refuse from self-knowledge. | | 📊 | Live status | Settings page shows the resolved vision route, cache size, in-flight describes, the last failure, and last call's token usage. | | 🧩 | Native composition | Cordis lifecycle, DSH settings, and the Web host remain the only runtime authorities. |

How it fits

flowchart LR
  Loop[Agent loop request] --> WF[llm/stream waterfall]
  WF --> VH{vision handoff}
  VH -->|text-only target, images present| Cache[(description cache)]
  Cache -->|miss| Batch[ONE batched vision call]
  Batch --> Vision[Vision route adapter]
  Cache -->|hit + miss| Clone[cloned request: image → text]
  Clone --> Adapter[Target adapter dispatch]
  VH -->|vision target / disabled / no images| Adapter

Provider and model identity stay unchanged. Descriptions live outside the session log: the stored messages keep their image blocks, and only the LLM-bound clone carries text.

What it provides

• A request transform on the llm/stream waterfall, applied per model call (including prepareCall-dispatched loop requests) without mutating the frozen, log-derived original
• One batched describer call per request image set, delimited <<<IMAGE k>>> … <<<END>>> and split back per image
• An LRU description cache keyed by content-addressed attachment id
• Same-route retry plus ordered fallbackModels failover for totally failed batches
• Parallel single-image fallback for sections a batched response could not split
• Optional aware-prompt system note, reasoning-effort forwarding, output-cap clamping to the describer's context window, and a visible [... description truncated …] marker
• A dedicated Vision Handoff section in DSH's Settings modal with a 👁-marked vision-model picker
• Same-origin, loopback-only status and catalog endpoints for that page (/plugins/dsh-vision-handoff/status, /plugins/dsh-vision-handoff/models)

Install

Requirements: Node.js ^22.19.0 || >=24, pnpm 11 for this checkout, and DeepSeek Harness ^0.1.6.

pnpm dlx @monotykamary/dsh@latest plugin --profile web add dsh-vision-handoff

For local development, build this checkout and add its absolute path:

pnpm install --frozen-lockfile
pnpm run check
pnpm dsh plugin --profile web add link:/absolute/path/to/dsh-vision-handoff

The bundle patch installs the vision-handoff plugin. Open Settings → Vision Handoff, pick the vision provider and model (models declaring image input are marked 👁), and keep Enabled on. Text-only routes now receive image descriptions inline.

Settings UI

The browser plugin contributes a standalone settings.section named Vision Handoff. Every field lives in the vision-handoff settings namespace (writable from the page) with defaults from the plugin's composition config:

| Field | Default | Effect | | --- | --- | --- | | enabled | true | Master switch. When false, requests pass through untouched. | | visionModel | "" | The describer as provider/model; empty means not configured (handoff inactive). | | fallbackModels | [] | provider/model routes tried in order when the primary describer call fails entirely. | | autoHandoff | true | Apply the handoff to every model whose adapter declares no image input. | | extraTargets | [] | Extra provider/model routes forced through the handoff even when vision-capable. | | maxTokens | 0 | Per-call output cap for the describer, clamped to contextWindow − 8192 when the window is known; 0 = the route's adapter default. | | cacheMax | 50 | Described images kept in the in-memory cache. | | maxDescriptionLines | 0 | Cap on description lines (0 = unbounded); over the cap keeps the first lines with an ... (N more lines) footer. | | reasoningEffort | "" | Adapter effort id sent to the describer; ignored when the route doesn't offer it. | | awarePrompt | false | Append a system note telling handoff targets they can answer image questions. | | prompt | "" | Override the describer system prompt ("" = the built-in exhaustive prompt). | | userPromptPrefix | "" | Override the prefix prepended to your original request text ("" = built-in). |

Secrets stay out of every response: the status endpoint reports route state, cache counters, and token usage only, and its catalog endpoint describes model ids and declared modalities.

How it works

The harness serializes images as durable attachment references and projects them out for text-only routes at adapter dispatch. This plugin hooks the waterfall one layer earlier:

→ llm/stream listener (per model call)
    • pass-through when: no image blocks, disabled, no vision model,
      target declares image input (and isn't in extraTargets), or the
      vision route can't be resolved as vision-capable
    • collect the request's unique image attachment refs
    • memoize per attachment id; misses join ONE batched describer call
      (streamed through the vision route like any model call — adapters
      re-read the attachment bytes themselves)
    • parse `<<<IMAGE k>>>` sections back to per-image text (parallel
      single-image fallback for unparsed ones)
    • clone the request; swap each described image block for its
      `[Image: …]` text, recursively through tool-result content
    • re-dispatch the marked clone through ctx.llm.stream; the runtime's
      own text-only projection then finds nothing left to strip
→ failed descriptions stay out of the cache: the next request retries,
  and a placeholder never masquerades as a description

Because attachment ids are content-addressed, a long conversation re-describes nothing — only genuinely new images cost a vision call.

Product notes vs the pi extension

The pi version additionally pre-warms pasted clipboard images at paste time, races an async clipboard injection, persists descriptions into the session file for /resume, and strips pi's [Current model does not support images…] read-tool note. Those attack pi-specific surfaces (editor wrapping, read-tool internals, session-file markers) with no harness equivalent; the DSH port deliberately excludes them. Descriptions here are recomputed after a host restart only for images present in the next request.

Development and release

pnpm install --frozen-lockfile
pnpm run check
pnpm pack --dry-run

pnpm run check type-checks the host and browser faces, runs the complete Vitest suite, and builds the ESM service plus browser client bundle. prepublishOnly repeats that check before npm creates a release payload.

Relationship to the other projects

• DeepSeek Harness owns provider/model identity, request and session lifecycle, attachments, settings persistence, the Web host, and client composition.
• pi-vision-handoff is the pi-coding-agent original this plugin ports.
• dsh-codex, dsh-multiprovider, dsh-fabric, dsh-fovea, and dsh-tool-repair are the other external bundles for the same runtime.

License

MIT © Tom Nguyen.