dsh-local-vision

Give a text-only DeepSeek Harness agent local "eyes". This plugin registers a model-callable local_vision tool that describes an image using a local vision model through Ollama. The image never leaves your machine — no cloud, no uploads, no API keys.

Requirements

• Ollama running (default http://localhost:11434).

• At least one vision-capable model pulled, e.g.:

  ollama pull qwen2.5vl:3b        # small, fast — text / colors / summary
  ollama pull gemma4:12b-it-qat   # larger — thorough description

Install

# from npm (recommended)
npm install -g @deepseek-ai/dsh
dsh plugin --profile web add dsh-local-vision

# or from this repository (use `file:` so @deepseek-ai/dsh-tools resolves)
dsh plugin --profile web add file:/path/to/deepseek-harness-plugins/dsh-local-vision

Restart the DSH session so the new bundle is composed. The package then appears in Settings → Plugins → Plugin list and the tool local_vision is available.

Usage

local_vision(image: "/abs/path/screenshot.png", mode: "fast")
local_vision(image: "/abs/path/mockup.png", mode: "detailed", prompt: "Describe the navigation and buttons")
local_vision(image: "screen", mode: "detailed")   # capture the current display (macOS screencapture)

Modes

| mode | intent | default model | |---|---|---| | fast | transcribe visible text, name dominant colors, one-line summary | qwen2.5vl:3b | | detailed | full factual description: layout, UI elements, all text, colors, state | gemma4:12b-it-qat |

Parameters

| Param | Type | Notes | |---|---|---| | image | string (required) | Absolute path to a PNG/JPEG/WebP/GIF, or the literal screen. | | mode | "fast" | "detailed" | Defaults to fast. | | prompt | string | Optional focused question. Defaults per mode. | | model | string | Optional override of the Ollama model id for this call. |

Configuration

Optional, at $DSH_HOME/local-vision.json (default ~/.dsh/local-vision.json):

{
  "host": "http://localhost:11434",
  "models": { "fast": "qwen2.5vl:3b", "detailed": "gemma4:12b-it-qat" },
  "temperature": 0.1,
  "timeoutMs": 300000
}

All keys are optional; defaults are shown above. The plugin sends the image to Ollama's OpenAI-compatible /v1/chat/completions endpoint as a base64 data URI.

Notes & limitations

• The first call after the model is cold loads it into memory (a 7 GB model takes a while); subsequent calls are fast.
• image: "screen" requires macOS and Screen Recording permission for the terminal.
• The tool calls Ollama directly (not through the DSH sandbox executor).

License

MIT