dsh-vision-bridge

npm: dsh-vision-bridge-dsh · source/GitHub: AbdullahElTiby/dsh-vision-bridge (the npm package is named dsh-vision-bridge-dsh because the shorter name was already taken on npm).

Host-plane plugin that gives text-only models (DeepSeek and any provider route that does not declare image input) the ability to "see" images, using a pluggable vision provider as the eyes: Gemini, Groq, or any OpenAI-compatible endpoint (OpenAI, OpenRouter, Ollama, LM Studio, …).

What it does

1. Dispatch interception — the two LLM dispatch entry points are wrapped: llm.prepareCall (the agent loop's prepared-call path, used for main turns and subagents) and llm.stream (session titles, compaction, unprepared loops). Before the adapter stream is built, every image block in the conversation is described by the configured vision provider and replaced with a [Image (mediaType, WxH): …] text block. The session history and UI keep the real image; only the model request is rewritten. This works for every text-only route (deepseek-official, pi-ai providers such as opencode-go, …). Routes that genuinely declare image input are passed through untouched.

   Why method patching instead of the llm/stream waterfall: in this harness build the waterfall ignores arguments passed to next() (listeners always receive the original args) and dsh-llm's default handler closes over the original options object. A waterfall listener can wrap the chunk stream but can never replace the request the adapter receives — a waterfall-only bridge silently loses its rewrite and the text-only adapter throws UNSUPPORTED_CONTENT ("… does not support image input"). Wrapping the service methods makes the rewritten request reach the adapter.

2. Image admission — llm.resolveModelInfo is patched so bridged routes also report image input. This admits image uploads in chat, model switches with images already in the session, and the built-in read_image tool for text-only routes (its image blocks are described by the bridge on the next model call).

3. describe_image tool — the model can inspect an image file on disk (PNG/JPG/JPEG/WebP/GIF) on demand; useful for screenshot and file analysis.

4. System-prompt section — the model is told images arrive as descriptions.

Descriptions are cached per attachment, so history images are described once per session, not on every model call.

Installation

The plugin runs inside the DeepSeek Harness (DSH) web profile. Both install paths below need Node ≥ 20 + pnpm, the DSH web app run once (so the profile folder exists), and an API key for one vision provider (Gemini, Groq, or an OpenAI-compatible endpoint).

| Path | For | What it is | | --- | --- | --- | | Option A — automated | Anyone with an AI assistant | Paste one small prompt; your AI fetches the repo, installs, configures and verifies everything for you | | Option B — manual | Humans | The same install, explained step by step |

AI agents working in this repo should also read AGENTS.md, which restates the install facts in agent-friendly form.

Option A — Automated install (give your AI a prompt)

Copy the prompt below and paste it into any AI assistant that can run shell commands on this machine (a DSH agent, Claude, Codex, …). The AI will fetch the repo docs, install the package, register the plugin row, set up the credential and verify the result — you only supply the vision-provider API key when it asks for it.

Install the dsh-vision-bridge plugin into the DeepSeek Harness (DSH) web
profile on this machine.

Plugin: npm `dsh-vision-bridge-dsh`, source github.com/AbdullahElTiby/dsh-vision-bridge.
It gives text-only models (e.g. DeepSeek) vision via a Gemini, Groq, or
OpenAI-compatible provider.

Procedure:
1. Fetch the install docs from the repo (README.md → Installation, and
   AGENTS.md → "Installing this plugin into a harness") and follow them.
   If you don't have the repo locally, fetch those files from GitHub.
2. Install the package into the DSH web profile (~/.dsh/profiles/web):
   `pnpm add dsh-vision-bridge-dsh` (or `dsh plugin --profile web add
   dsh-vision-bridge-dsh`).
3. Register the `vision-bridge` row in cordis.patch.yml (default provider:
   Gemini). Skip this if the row already exists.
4. Make sure a GEMINI_API_KEY credential exists (env var or
   ~/.dsh/.credentials.yaml). If no key is set, ASK ME for it — never
   invent or reuse a key without asking.
5. Verify: `dsh --profile web --dump-config` must show the vision-bridge
   row.
6. Report what you did, whether a key is still needed, and how I can check
   it works (attach an image in a chat with a text-only model).

Every step must be idempotent and safe to re-run. If I ask for Groq or an
OpenAI-compatible provider instead of Gemini, use the matching row config
and credential ref from the Providers table in the README.

What the AI will run — the exact idempotent commands per OS — is documented in AGENTS.md § "Installing this plugin into a harness (agent instructions)", so you can review what it did (or run the same steps yourself).

Final functional check: attach an image in a chat with a text-only model (e.g. a DeepSeek route) — it should be described instead of rejected with UNSUPPORTED_CONTENT.

Option B — Manual install (step by step)

1. Install the package

The package is installed into your web profile's node_modules. From the profile directory, add the npm package:

cd ~/.dsh/profiles/web
dsh plugin --profile web add dsh-vision-bridge-dsh

or via pnpm directly:

cd ~/.dsh/profiles/web
pnpm add dsh-vision-bridge-dsh

(As a fallback you can also install straight from GitHub with pnpm add github:AbdullahElTiby/dsh-vision-bridge, or copy the package folder into ~/.dsh/profiles/node_modules/ — the user-owned module fallback.)

2. Register the plugin row

Edit ~/.dsh/profiles/web/cordis.patch.yml and add one of the rows below. Gemini (the default provider):

- insert:
    - id: vision-bridge
      name: 'dsh-vision-bridge'
      config:
        provider: gemini            # optional: gemini is the default
        model: gemini-2.5-flash     # optional
        apiKeyRef: GEMINI_API_KEY   # optional

Groq (free tier, vision-capable models — see the table below):

- insert:
    - id: vision-bridge
      name: 'dsh-vision-bridge'
      config:
        provider: groq
        model: qwen/qwen3.6-27b     # vision-capable on the free tier
        apiKeyRef: GROQ_API_KEY

Any OpenAI-compatible endpoint:

- insert:
    - id: vision-bridge
      name: 'dsh-vision-bridge'
      config:
        provider: openai
        baseURL: https://openrouter.ai/api/v1
        model: <vision model id of that endpoint>
        apiKeyRef: OPENROUTER_API_KEY

3. Set your API key

Add the key to ~/.dsh/.credentials.yaml (or export the env var):

GEMINI_API_KEY: your-gemini-key
GROQ_API_KEY: your-groq-key      # gsk_… from console.groq.com/keys

Auth-free local endpoints (Ollama, LM Studio) still need a credential ref to be present — put any placeholder in it, e.g. OPENAI_API_KEY: ollama; the bridge sends it as a bearer token the local server ignores.

4. Restart

Close and reopen DSH (dsh web). You can confirm the row mounts by dumping the composed config: dsh --profile web --dump-config.

To verify it works, attach an image in a chat with a text-only model (e.g. a DeepSeek route) — it should be described instead of rejected with UNSUPPORTED_CONTENT.

Enable/disable

Remove the vision-bridge row from cordis.patch.yml to disable the feature (hot-reloaded); delete the package folder to remove it permanently. See the Notes section for how edits to the plugin code are — and are not — hot-reloaded.

Providers

| provider | API | default model | default apiKeyRef | default baseURL/endpoint | | --- | --- | --- | --- | --- | | gemini (default) | Google Generative Language generateContent | gemini-2.5-flash | GEMINI_API_KEY | https://generativelanguage.googleapis.com/v1beta | | groq | Groq chat completions (OpenAI-compatible) | qwen/qwen3.6-27b | GROQ_API_KEY | https://api.groq.com/openai/v1 | | openai | any OpenAI-compatible chat completions | gpt-4o-mini | OPENAI_API_KEY | https://api.openai.com/v1 |

model, apiKeyRef and baseURL on the row override the per-provider defaults. endpoint is the legacy name of baseURL and still works for the Gemini transport; for groq/openai, baseURL wins over endpoint.

Groq free tier — which models can see? (live-verified)

Tested against the live API with a real PNG: on the free plan exactly one chat model accepts images:

| model | vision | free limits (RPM / RPD / TPM / TPD) | | --- | --- | --- | | qwen/qwen3.6-27b | ✅ multimodal (vision + text), thinking + non-thinking modes — default for provider: groq | 30 / 1K / 8K / 200K | | openai/gpt-oss-20b / openai/gpt-oss-120b | ❌ served text-only on Groq — the API rejects image_url parts with messages[1].content must be a string | 30 / 1K / 8K / 200K | | llama-3.1-8b-instant, llama-3.3-70b-versatile | ❌ text-only (same content must be a string rejection) | — | | groq/compound(-mini), llama-prompt-guard-2-*, openai/gpt-oss-safeguard-20b | ❌ compound/moderation models, no image description | — | | whisper-large-v3(-turbo), canopylabs/orpheus-* | ❌ audio (STT / TTS) | — |

Pointing the bridge at a non-vision model surfaces a description unavailable: groq HTTP 400 … placeholder — only qwen/qwen3.6-27b works for vision on the free tier.

Free-plan budget (measured live): each image call on qwen/qwen3.6-27b charges ~2.7K against the 8K TPM bucket (the reported usage is ~1.3K prompt tokens, but Groq bills an image floor) — so expect roughly 3 image descriptions per minute, 1000 per day (RPD), 30 per minute max (RPM). Rate-limit responses carry Groq's x-ratelimit-* headers; npm run test:groq prints them.

Configuration (row config on the vision-bridge row)

| key | default | meaning | | --- | --- | --- | | provider | gemini | vision transport: gemini, groq, or openai | | model | per provider (see table) | vision model id | | apiKeyRef | per provider (see table) | credential ref (env var / ~/.dsh/.credentials.yaml / .env) | | baseURL | per provider (see table) | API base URL (chat completions for groq/openai; the Gemini endpoint for gemini) | | endpoint | per provider (see table) | legacy alias for baseURL (Gemini rows keep using it) | | maxOutputTokens | 1024 | description length cap | | reasoningEffort | none for groq, unset otherwise | reasoning effort sent to the endpoint (reasoning_effort); none keeps qwen out of thinking mode so descriptions come back without a <think>…</think> block | | temperature | 0.4 | sampling temperature | | timeoutMs | 30000 | per-call timeout | | maxImageBytes | 15728640 | largest image sent to the vision provider | | admitImages | true | patch resolveModelInfo (image admission) | | tool | true | register describe_image | | systemSection | true | contribute the prompt section | | cacheSize | 256 | description cache size |

Set admitImages: false to keep the stock gates (images are then rejected for text-only models and the bridge never fires).

Enabling the key

The bridge reads the credential referenced by apiKeyRef through the harness credential layers (inherited environment wins, then ~/.dsh/.credentials.yaml, then .env). For example, add to ~/.dsh/.credentials.yaml:

GEMINI_API_KEY: sk-…
GROQ_API_KEY: gsk-…

Without a key, attached images are replaced with a short failure placeholder and a warning is logged; the conversation keeps working.

Notes

• The package lives in ~/.dsh/profiles/node_modules/ (the deployment's user-owned module fallback, alongside the auto-created package links) so it survives npx-cache refreshes. Reinstalling the profile via pnpm install may prune that directory; re-create the package afterwards.
• The llm.prepareCall / llm.stream / llm.resolveModelInfo patches are applied per process start by the plugin itself (no shipped package is modified) and are removed when the row stops or reloads.
• Code changes to this package are NOT hot-reloaded: the loader re-imports rows only when their name changes, the HMR watcher ignores **/node_modules, and config-only row updates reuse the already-loaded module. Restart the harness (close and reopen the app) after editing lib/index.js.
• Remove the vision-bridge row from ~/.dsh/profiles/web/cordis.patch.yml to disable the feature (hot-reloaded), or delete the package to remove it permanently.