dsh-image-plugins

Multimodal capability for DeepSeek Harness (dsh) behind a text-only main model (e.g. DeepSeek's official chat route, which cannot carry images). The plugin understands image files and generates images through fully configurable endpoints — bring your own baseUrl / apiKey / model for a vision model and for an image-generation model. Any OpenAI-compatible endpoint works; an optional dashscope adapter speaks the Alibaba Model Studio native API.

Everything is optional: a capability is enabled only when its config block is present, so an unconfigured install is inert and safe. No API keys are shipped in the package — each user configures their own.

What it provides

| Capability | Kind | Behavior | |---|---|---| | understand_image | model tool | Reads a workspace image file, sends it to your vision endpoint (chat/completions + base64 image_url), returns the model's text description as the tool result. The description enters the session log, so a text-only main model can reason about the image without ever receiving one. | | generate_image | model tool | Generates an image from a prompt via your endpoint, saves it into the workspace, returns the saved path. With the dashscope provider it also accepts an optional reference_image for image editing (I2I). |

Auto-understand (V2): implemented but disabled

The plugin contains a dormant agent/pre-step rewrite (config flag autoUnderstand, default off): when enabled, images attached to a chat message are described by the vision model and the message is rewritten to carry that text before it enters the session log, so the main model never receives an image block. The code is unit-tested but was never verified end-to-end in a live session and is not part of the supported surface.

Why it is disabled: attaching an image to a chat message requires the routed model to declare input: [text, image] — the host refuses attachments for text-only models — and for a text-only endpoint (like DeepSeek's) that declaration is a workaround: a claim the endpoint never actually honors, neutralized by the rewrite before the wire. We found that inelegant and disabled the feature. The supported flow is the V1 tools above (image files in the workspace, no declaration needed). A future paste-to-chat iteration would use a lighter paste-to-path client approach instead.

Quick start

1. Install (npm; or see Install for other channels):

   dsh plugin --profile web add dsh-image-plugins

2. Configure — override the image-plugins row in your profile's cordis.patch.yml with your own endpoint and key (any OpenAI-compatible provider):

   - id: image-plugins
     name: dsh-image-plugins
     config:
       vision:
         baseUrl: 'https://your-vision-endpoint.example.com/v1'
         apiKey: 'sk-...'
         model: 'your-vision-model'
       image:
         baseUrl: 'https://your-image-endpoint.example.com/v1'
         apiKey: 'sk-...'
         model: 'your-image-model'
         defaultSize: '1024x1024'

3. Restart dsh web, then in the workspace:

   • 看图："Look at images/screenshot.png and tell me what it shows."
   • 生图："Generate an image of a red apple on a wooden table."（保存到 generated/）
   • 图生图："Change the color of images/logo.png to blue."

Install

The plugin is a standard dsh bundle. From npm (recommended):

dsh plugin --profile web add dsh-image-plugins

Other channels:

# GitHub (pin a version; the first install needs allowBuilds, see below)
dsh plugin --profile web add github:alanzhao0128/dsh-image-plugins#v0.3.5

# Tarball (npm pack output, send the file)
dsh plugin --profile web add ./dsh-image-plugins-0.3.5.tgz

# Local checkout
dsh plugin --profile web add /path/to/dsh-image-plugins

Then restart dsh web (or the profile's process). For a GitHub install, pnpm ≥ 10 refuses to run the package's build script until you allow it in the profile's pnpm-workspace.yaml:

allowBuilds:
  dsh-image-plugins: true

then re-run the add command. npm and tarball installs ship built artifacts and need no allowance.

The bundle inserts its row without configuration, so after install nothing is enabled until you configure it. The plugin loads fine either way.

Configure

Since 0.2.0 the plugin ships a settings panel (web: Settings → 图片插件 / Image Plugins): edit the vision and image endpoint blocks, and manage the two capability secrets through the official credential store. Changes apply immediately (no restart) for endpoint edits; host-code changes still need a dsh web restart after an upgrade.

Since 0.3.0 each capability group has an enable switch at the top of the panel: turn it off and the tool is unregistered (the model never sees understand_image / generate_image), the rest of that group is locked (fields and credential row become read-only), and flipping it back on applies immediately — no restart.

Since 0.3.1 the plugin is dual-engine across the dsh settings rewrite: the same build works on dsh ≤ 0.1.5 (host settings.installSection, client settingsScope service) and on dsh ≥ 0.1.7 / 0.2.0 (host SettingsForms + settings.configure({ auto: false }), client configForms) with no configuration change. Schema fields are marked volatile, which host settings forms require before accepting a live edit, and the plugin no longer declares the removed settingsScope client service — declaring it would hang plugin activation on 0.1.7+.

Since 0.3.4 the generate_image tool schema description completely omits 1024x1024 to prevent model boilerplate bias, and the runtime execution automatically intercepts model hallucinated 1024x1024 parameters when a custom defaultSize (e.g. 1920x1080) is configured, unless the prompt explicitly requested a 1024x1024 or 1:1/square image. Effective size is also clearly echoed in the output result.

Since 0.3.5 reference_image (image-to-image / editing) is no longer hardcoded to provider: 'dashscope'. Both openai (such as Qianwen Cloud's OpenAI-compatible endpoint https://maas.qianwenaiapi.com/compatible-mode/v1) and dashscope native endpoints directly support reference images.

Where the panel persists edits depends on the host: on dsh ≤ 0.1.5 they go to the dsh-image-plugins section of ~/.dsh/settings.yaml; on dsh ≥ 0.1.7 that global file is retired (archived as settings.yaml.imported) and edits land in the profile's cordis.patch.yml under the image-plugins entry. Secrets are stored separately: two fixed credential references back the capabilities — the panel writes secret values into ~/.dsh/.credentials.yaml through the official credential seam and never displays a stored key:

| Reference | Used by | |---|---| | UNDERSTAND_IMAGE_KEY | understand_image (vision endpoint key) | | GENERATE_IMAGE_KEY | generate_image (image-generation endpoint key) |

You can also configure manually — override the image-plugins row (same id) in your profile's cordis.patch.yml, or pass a --patch overlay. The patch value acts as the initial (base) layer; once you save from the panel, the settings.yaml value wins:

- id: image-plugins
  name: dsh-image-plugins
  config:
    vision:
      enabled: true                  # 0.3.0: capability switch, default true
      baseUrl: 'https://your-vision-endpoint.example.com/v1'
      apiKey: 'env:VISION_API_KEY'   # literal key, env:NAME, or cred:NAME (see notes)
      model: 'your-vision-model'
      timeoutMs: 60000               # optional
      maxImageBytes: 20971520        # optional, bytes
      systemPrompt: ''               # optional, sent before the image
      defaultPrompt: ''              # optional, used when the model gives no prompt
    image:
      enabled: true                  # 0.3.0: capability switch, default true
      provider: 'openai'             # 'openai' (default) or 'dashscope'
      baseUrl: 'https://your-image-endpoint.example.com/v1'
      apiKey: 'env:IMAGE_API_KEY'
      model: 'your-image-model'
      timeoutMs: 120000              # optional
      defaultSize: '1024x1024'       # optional
      outputDir: 'generated'         # optional, workspace-relative

Notes:

• Each block is independent: configure only vision, only image, or both. A partially filled block (e.g. baseUrl without apiKey) fails the load loudly.
• Since 0.3.0 each block accepts enabled: true|false (default true). When false, the tool is not registered — the model never sees it and cannot call it (an in-flight call that was dispatched just as the switch flipped also fails closed) — and the settings panel locks the rest of that group. Flipping the switch back on in the panel re-enables the tool immediately, no restart.
• apiKey accepts three forms:
  • a literal secret ('sk-...'),
  • env:VARNAME — resolved from the process environment at load,
  • cred:NAME — resolved through the host credential seam (ctx.credentials, e.g. ~/.dsh/.credentials.yaml) at each request (the credential service may start after plugin load, so resolution is deferred to the request path). Requires the host's credentials service (present in the stock dsh profiles).
  • When using the settings panel, the apiKey is automatically the fixed cred:UNDERSTAND_IMAGE_KEY / cred:GENERATE_IMAGE_KEY reference — the panel writes the secret value through the credential service, so you never handle references manually.
  • Keys never enter the session log or tool results.
• The profile patch targets the row by id and replaces its whole config — restate every key you need.
• Endpoints must be OpenAI-compatible: vision = POST {baseUrl}/chat/completions accepting image_url data URLs; image generation = POST {baseUrl}/images/generations returning data[0].b64_json or data[0].url. Anything compatible — OpenAI, 硅基流动, 智谱, 通义兼容模式, Ollama, etc. — works as-is.

DashScope & 千问AI平台 (Qwen-Image)

千问平台同时提供 OpenAI 兼容接口（https://maas.qianwenaiapi.com/compatible-mode/v1）与 DashScope 原生接口（https://maas.qianwenaiapi.com 或 https://dashscope.aliyuncs.com）：

• OpenAI 兼容（推荐）：provider: 'openai'。文生图直接请求 POST {baseUrl}/images/generations；图生图自动携带 image（base64 data URL）参数，千问原生支持此协议。
• DashScope 原生：provider: 'dashscope'。直接调用 POST /api/v1/services/aigc/multimodal-generation/generation 原生协议。

示例配置（OpenAI 兼容模式）：

- id: image-plugins
  name: dsh-image-plugins
  config:
    vision:
      baseUrl: 'https://maas.qianwenaiapi.com/compatible-mode/v1'
      apiKey: 'sk-...'
      model: 'qwen3.7-flash'
    image:
      provider: 'openai'
      baseUrl: 'https://maas.qianwenaiapi.com/compatible-mode/v1'
      apiKey: 'sk-...'
      model: 'qwen-image-3.0-pro'
      defaultSize: '1920x1080'

DeepSeek official (image understanding)

DeepSeek's official endpoint now advertises an image-capable model, deepseek-v4-flash-vision-exp (dsh ≥ 0.1.1-rc.2 advertises it in the deepseek-official catalog). You can point vision at it with your DeepSeek API key — keep the key in ~/.dsh/.credentials.yaml and reference it with cred::

- id: image-plugins
  name: dsh-image-plugins
  config:
    vision:
      baseUrl: 'https://api.deepseek.com'
      apiKey: 'cred:DEEPSEEK_API_KEY'
      model: 'deepseek-v4-flash-vision-exp'

Two things to know before switching:

• The plugin sends images inline as base64 image_url parts. DeepSeek's official adapter normally uploads images through its Files API and references them by file_id to avoid re-sending bytes; this plugin's direct chat/completions path uses inline data URLs instead. That works (verified), but repeated understanding of the same image re-sends the bytes — if that matters, prefer the host's native image input (select deepseek-v4-flash-vision-exp as the routed model) rather than this plugin for that endpoint.
• Third-party DeepSeek-compatible channels may not carry the vision model. For example, Volcano Ark (火山方舟) accepts the model id for text but rejects image input with 400 Model do not support image input (as of 2026-08). Verify image input on your channel before relying on it.

Image editing (I2I) with a reference image

generate_image accepts an optional reference_image path. The reference (PNG/JPEG/WebP/GIF, ≤ 10 MiB, cap configurable via image.maxReferenceBytes) is sent to the model as base64 alongside the prompt:

Change the color of images/logo.png to blue, keep everything else identical.

The model edits the reference image instead of generating from scratch. Both openai (supported by 千问/Qwen) and dashscope providers support reference images.

Use

Understand an image (V1 tool, recommended)

Put the image somewhere in the workspace, then ask the agent:

Look at images/screenshot.png and tell me what it shows.

The agent calls understand_image with the path, optionally passing a specific question as prompt (e.g. "what is the trend of the third row in this chart?").

Generate an image (V1 tool, recommended)

Generate an image of a red apple on a wooden table.

The agent calls generate_image; the file lands in the workspace under generated/ (or your configured outputDir) and the tool result reports the path.

Distribution

| Channel | Install command | Notes | |---|---|---| | npm | dsh plugin --profile web add dsh-image-plugins | Recommended; no build allowance | | GitHub | dsh plugin add github:alanzhao0128/dsh-image-plugins#v0.3.5 | Needs allowBuilds once | | Tarball | dsh plugin add ./dsh-image-plugins-0.3.5.tgz | From npm pack; safe to delete after install (a later pnpm install in the profile may then need the file back) |

How it stays compatible with dsh's architecture

• Tools are registered through the documented ctx.tools seam (@deepseek-ai/dsh-tools defineTool); tool results are durable log entries, which is exactly the channel the "model-visible ⟺ logged" invariant requires.
• The settings surface is version-adaptive rather than version-pinned: the host half duck-types installSection (≤ 0.1.5) vs SettingsForms.configure (≥ 0.1.7), and the browser half resolves configForms (≥ 0.1.7) vs settingsScope (≤ 0.1.5) at runtime. Both client controllers expose the same getSnapshot / subscribe / mutate(ops, revision) contract, so the panel code is shared.
• The plugin depends only on published @deepseek-ai/dsh-tools and @deepseek-ai/schemastery; no internal modules.

Development

npm install
npm test          # unit tests against mock endpoints + real Cordis mount
npm run build     # tsc -> lib/ (also runs on prepare)

Smoke-verify against a scratch profile (does not touch your real profiles):

DSH_HOME=/tmp/dsh-image-test-home dsh plugin --profile test add /path/to/dsh-image-plugins
DSH_HOME=/tmp/dsh-image-test-home dsh --profile test --dump-config   # shows the layer

Known Limitations and Deferred Work

• Binary writes bypass the fs approval events. The fs seam exposes no binary write today, so generate_image resolves the target through ctx.fs (consistent path rules, session-workspace cwd) but writes the bytes with node:fs. The write therefore does not emit fs/write-intent approval events. Switch to a seam write when the fs service grows one.
• Vision responses are text-only. The plugin returns descriptions as text; it never emits image content blocks, because a text-only route cannot carry them into the next request.
• No inline chat preview yet. Generated images are returned as paths with a generic tool card (the path is clickable to open). An inline preview needs a client-side tool.call.toolview registration (V1.5, not shipped).
• No video generation. Planned as a background-job capability (ctx.jobs) once a provider interface is chosen.
• No per-request retry/backoff for endpoint failures; the caller sees the error.
• Version pinning. Verified against @deepseek-ai/* 0.1.5, 0.1.7-rc.2 and 0.2.0-rc.2 (settings API probed per host); peerDependencies declare >=0.1.2-rc.1 so the 0.1.7+ host compatibility pre-check accepts the plugin. dsh is in developer preview and breaking changes are expected between releases — re-run npm test after upgrading the host.

License

MIT