vision plugin

Upstream: this is a fork of opencode-vision (MIT) by Nicolás Ríos Herrera. It has been meaningfully changed (in-process plugin, NIM provider chain, config-in-opencode, dual-describe) but the original work is theirs. This code was generated by an AI assistant; verify before trusting.

Adds vision_describe, vision_ocr, vision_analyze to opencode so a text-only model can still read images. Runs in-process: no MCP server, no subprocess, no Python.

Install

Zero config — copy the bundled file into your plugin dir and restart:

cp opencode-vision-plugin.js ~/.config/opencode/plugin/opencode-vision-plugin.js

Keys come from your environment (NVIDIA_API_KEY, GEMINI_API_KEY / GOOGLE_API_KEY). No config block needed.

Optional config block — if you want to override defaults (key auth, endpoints, timeouts, model-of-choice), pin the npm package and drop a block into ~/.config/opencode/opencode.jsonc:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": [
    ["opencode-vision-plugin", {
      "provider": "chain",
      "chain": ["gemini", "nim"],
      "models": {
        "gemini": "gemini-3.6-flash",
        "nim": "meta/llama-3.2-90b-vision-instruct"
      },
      "keys": {
        "gemini": "{env:GEMINI_API_KEY}",
        "nim": "{env:NVIDIA_API_KEY}"
      },
      "baseUrls": {
        "gemini": "https://generativelanguage.googleapis.com/v1beta",
        "nim": "https://integrate.api.nvidia.com/v1"
      },
      "timeout_ms": 60000
    }]
  ]
}

Every string supports {env:VAR}, resolved at load time.

What the config actually controls

The tool itself already knows every model and keeps sensible defaults, so the config block exists to override defaults, not to enable the tools.

| Key | Default | What it does | |---|---|---| | provider | chain | Which provider to hit first | | chain | ["gemini","nim"] | Fallthrough order — if the first fails, try the next | | models.gemini | gemini-3.6-flash | Model used when you call without a model arg | | models.nim | meta/llama-3.2-90b-vision-instruct | Model used when you call without a model arg | | keys.* | env vars | API keys (endpoints reject unauthenticated calls) | | baseUrls.* | standard endpoints | Where each provider lives | | timeout_ms | 60000 | Global budget; a hanging provider gives up so the chain can fall through |

You can still pick any model at call time via the model argument — the config just sets the backstop default. If you're happy with the defaults, skip the block entirely.

Behavior

• Default chain tries gemini then nim — if Gemini's daily quota is out, NIM keeps you working.
• vision_describe with no model runs two NIM VLMs and returns each labelled section (structured + detail) — this is the recommended default call. The tool description steers agents to it; models are flaky, so calls are independent.
• Every tool takes a compulsory blind boolean: true means the reader cannot see the image, so the description must be exhaustive and spatial (the blind-reader clause is appended to the prompt). false = normal description. The flag decides the prompt — no hardcoded per-model rules.
• image_path accepts a local path, an https URL, or a data: URL.

Available models

• meta/llama-3.2-90b-vision-instruct — structured
• nvidia/nemotron-nano-12b-v2-vl — detail
• meta/llama-3.2-11b-vision-instruct — light
• nvidia/llama-3.1-nemotron-nano-vl-8b-v1 — concise
• gemini-3.6-flash

Requirements

Node/bun global fetch + node fs/path. Needs NVIDIA_API_KEY and/or GEMINI_API_KEY.

Restart

Config is read once at startup. After changing the config or the plugin file, quit and restart opencode. If you installed via npm, also clear ~/.cache/opencode/packages/opencode-vision-plugin — opencode won't refetch.