dsh-tool-describe-image — Image Understanding Tool Plugin

English | 中文

Model-facing describe_image tool: gives text-only models (DeepSeek V4 etc.) image understanding. Each call loads one image — a local file path, an http(s) URL, or a session attachment reference — and asks an OpenAI-compatible vision endpoint (Qwen-VL, GLM-4V, GPT-4o, a local Ollama endpoint…) to answer over the Chat Completions or Responses protocol; only the returned text enters the conversation, the image itself never enters the session log.

Ported from deepseek-harness packages/vision/tool-describe-image (mirrored at whitelonng/dsh-plugin-describe-image), adapted to the dsh-web-ui family conventions: official NPM SDK only, host-side plugin with a browser half, live settings, no dsh source changes.

Capabilities

| Capability | Description | | --- | --- | | Inputs | Local absolute path, http(s) URL (redirects refused), or an existing session-attachment reference | | Model-facing tool | The text model calls describe_image; only the returned description enters the session log. The plugin does not add a separate image-picker button or replace DSH's native composer input | | Custom instructions | The prompt argument carries your precise instruction (OCR, chart reading, UI diagnosis, translation…); the defaultPrompt config sets the fallback when the model passes none | | Live config card | Settings → Plugin config → Web UI Plugins → "Image understanding" card edits baseURL / apiStyle / model / API key / default instruction / bounds (through the settings seam); effective immediately, no restart | | Protocol styles | apiStyle: chat-completions (default) posts to baseURL/chat/completions; apiStyle: responses posts to baseURL/responses with input / max_output_tokens and reads output_text | | Raw image route | GET /describe-image/raw/<id> serves the stored bytes (loopback-only, content-addressed id) so the pasted reference renders in the conversation | | Per-call key resolution | Inline apiKey → credential seam (apiKeyEnv, default VISION_API_KEY) → launch environment, tiered fallback | | Safety and bounds | All requests refuse redirects; maxBytes / maxOutputTokens / timeoutMs caps; magic-byte type gate; bounded error excerpts (200 chars); keys never logged | | Canonical return | { text, model, image, mimeType, bytes } — the model only sees text |

Security model

• Vision requests and image downloads both refuse HTTP redirects (redirect: 'error'); bearer credentials and image bytes never reach a source other than the configured deployment.
• The request body carries the base64 image but no key; request headers and resolved credentials are not logged.
• Only http(s) URLs and local paths are accepted; every other URL scheme is rejected.
• The attach route validates base64, magic bytes, and the byte bound before the attachment store persists anything; only the reference JSON (text) crosses into the conversation.
• Response bodies are truncated at the cap (maxOutputTokens * 8 + 64 KiB) before parsing.

Installation

Install the family aggregate @neystan/dsh-web-ui-all (all plugins and skins in one package), or this plugin alone:

# Recommended: install directly from npm
dsh plugin --profile web add @neystan/dsh-tool-describe-image

The aggregate mounts this plugin without configuration: loading is unaffected, and the first call fails with a clear error (describe-image: baseURL must be an absolute http(s) URL) until configured. Fill in the endpoint and model on the "Image understanding" card under Settings → Plugin config to start immediately, no restart needed. (Difference from upstream: upstream validates eagerly at load; the family aggregate has no config entry, so validation is eager only when a composition entry actually configures it and per-call otherwise.)

Configuration

| Key | Default | Meaning | | --- | --- | --- | | baseURL | — (required) | OpenAI-compatible endpoint root (e.g. https://dashscope.aliyuncs.com/compatible-mode/v1); trailing slashes stripped | | apiStyle | chat-completions | Protocol style: chat-completions appends /chat/completions; responses appends /responses (OpenAI Responses API input / max_output_tokens / output_text shapes) | | model | — (required) | Vision model id | | apiKey | — | Inline key for local debugging; prefer !!js process.env.VISION_API_KEY over a hardcoded secret | | apiKeyEnv | VISION_API_KEY | Credential reference (environment-variable name); empty string disables reference resolution | | defaultPrompt | see source | The instruction used when a call omits its prompt — tune it to your workload (OCR, UI review, translation…) | | maxBytes | 10485760 | Image byte bound (local files and downloads alike) | | maxOutputTokens | 1024 | Output-token cap: max_tokens under chat-completions, max_output_tokens under responses | | timeoutMs | 60000 | Per-call vision request timeout |

Configured mount example (profile cordis.patch.yml / composition file):

- id: describe-image
  name: '@neystan/dsh-tool-describe-image'
  config:
    baseURL: https://dashscope.aliyuncs.com/compatible-mode/v1
    model: qwen-vl-max
    apiKey: !!js process.env.VISION_API_KEY

Endpoints exposing only the Responses API set apiStyle: responses:

- id: describe-image
  name: '@neystan/dsh-tool-describe-image'
  config:
    baseURL: https://api.openai.com/v1
    apiStyle: responses
    model: gpt-4o-mini
    apiKey: !!js process.env.VISION_API_KEY

Usage

Custom instructions

The tool takes a prompt argument: tell the vision model exactly what you need — "transcribe all text", "extract the table as CSV", "diagnose the UI layout problems", "translate the text into Chinese". A targeted instruction beats a generic description; the tool description steers the text model toward passing one. Calls without a prompt fall back to defaultPrompt.

Sending images

The plugin is a model-facing tool, not a replacement composer. Give the model a local path, an http(s) URL, or an attachment reference in the conversation; it can then call describe_image. Whether a file can be selected directly in the composer is still controlled by DSH and the selected model's native image-input support.

Known limitations

• Only the magic-byte gate checks the type; the image is not decoded, so a header-valid but corrupt file fails only at the vision endpoint.
• One image per answer: no multi-image input, no follow-up on the previous image, no structured output (coordinates / boxes).
• Extracting text still costs one VLM call: OCR-only deployments can point baseURL at a cheaper OCR model.
• OpenAI-compatible protocol only: Chat Completions (/chat/completions) and Responses (/responses) are supported; vendors with other request/response shapes need separate adapters.

Source and copyright

• Source: ported from whitelonng/dsh-plugin-describe-image (deepseek-harness packages/vision/tool-describe-image), moved in 2026-08; tests ported with the source (pnpm --filter @neystan/dsh-tool-describe-image test).
• Copyright: the original code belongs to its authors (deepseek-ai / whitelonng); this repository only hosts and maintains it and claims no copyright; the ported contribution is licensed by its contributor under the family license.
• License: the family is licensed under Apache-2.0 (repository root LICENSE); this package's license field is Apache-2.0.