@we64/opencode-vision-multimodal

Native image, PDF and document analysis for opencode — using any OpenAI-compatible multimodal API.

opencode's built-in read tool is text-only. This plugin adds a vision tool that sends any file (images, PDFs, Word/Excel/PowerPoint, text, audio, video, …) natively to a multimodal endpoint. No OCR, no page rendering, no conversion — the file's bytes are base64-encoded and handed to the model, which decides whether it can read the type.

Maintained by WE64 AI.

Features

• Any file type — the plugin does not filter by extension; the AI API decides what it can process.
• Native sending — the file is attached as an image_url data URL, the same shape used by Zcode and other multimodal clients. Nothing is converted locally.
• Works with any provider — anything that speaks the OpenAI chat/completions API and accepts images/documents (OpenAI, Gemini, Claude gateways, GLM, DeepSeek, Qwen, Kimi, MiMo, Nemotron, OpenRouter, vLLM, LiteLLM, Ollama, LM Studio, …).
• Configurable — bring your own key + endpoint via config or environment variables.
• Zero-config fallback — with nothing configured, it uses whichever multimodal model is selected in your opencode terminal.
• Redirects read — the agent is nudged to use vision for binary/rich files instead of the text-only reader.
• No secrets in code — keys are read from your config/environment at runtime.

Installation

Step 1 — add the plugin to opencode

From the CLI:

opencode plugin @we64/opencode-vision-multimodal

Or edit opencode.json yourself:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": ["@we64/opencode-vision-multimodal"]
}

opencode installs npm plugins automatically with Bun at startup — nothing else to run.

Step 2 — configure your endpoint

See Configuration below.

Step 3 — restart opencode

Global install (all projects)

Put the same "plugin": ["@we64/opencode-vision-multimodal"] line in your global config at:

• macOS / Linux: ~/.config/opencode/opencode.json
• Windows: %USERPROFILE%\.config\opencode\opencode.json

It then works in every project.

Configuration

Pick one. The plugin checks them in this order (first value set per field wins).

1. Environment variables (simplest)

macOS / Linux (~/.bashrc or ~/.zshrc):

export OPENCODE_VISION_BASE_URL="https://api.yourprovider.com/v1"
export OPENCODE_VISION_API_KEY="sk-your-key"
export OPENCODE_VISION_MODEL="your-multimodal-model"

Windows PowerShell:

setx OPENCODE_VISION_BASE_URL "https://api.yourprovider.com/v1"
setx OPENCODE_VISION_API_KEY "sk-your-key"
setx OPENCODE_VISION_MODEL "your-multimodal-model"

2. vision block in opencode.json

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": ["@we64/opencode-vision-multimodal"],
  "vision": {
    "baseURL": "https://api.yourprovider.com/v1",
    "apiKey": "sk-your-key",
    "model": "your-multimodal-model"
  }
}

3. Standalone config file

~/.config/opencode/vision.json:

{
  "baseURL": "https://api.yourprovider.com/v1",
  "apiKey": "sk-your-key",
  "model": "your-multimodal-model"
}

4. Nothing configured

If none of the above are set, the plugin uses the currently selected model in your opencode terminal (its provider endpoint and key from your opencode config). Works when that provider/model accepts the file type.

Usage

Just ask naturally — the agent calls vision on its own:

analyze screenshot.png
what does this invoice.pdf say?
summarize report.docx and list the key numbers
read this scan.jpg and transcribe it

Or call it explicitly:

vision({ filePath: "Images/photo.jpg", prompt: "transcribe all text" })
vision({ filePath: "contract.pdf", prompt: "summarize the obligations" })

Every result begins with a header showing which endpoint served it:

[vision via custom/your-multimodal-model]

<model output>

Per-call override:

vision({ filePath: "scan.pdf", baseURL: "https://...", apiKey: "...", model: "..." })

How it works

1. Reads the file from disk.

2. Base64-encodes it into a data:<mime>;base64,... URL.

3. POSTs to {baseURL}/chat/completions:

   {
     "model": "your-multimodal-model",
     "messages": [
       {
         "role": "user",
         "content": [
           { "type": "text", "text": "…prompt… (file: name.pdf)" },
           { "type": "image_url", "image_url": { "url": "data:application/pdf;base64,…" } }
         ]
       }
     ]
   }

4. Returns the model's text response to opencode.

One HTTP request per vision call. Data goes only to the baseURL you configure.

Notes & limitations

• The model must support the file type. A text-only model will reject the call; switch to a multimodal model or configure a different vision endpoint.
• Large files become large base64 payloads. Some providers cap request size — if a file is rejected, try a smaller one or a provider with a higher limit.
• PDFs are sent as a data URL. Most multimodal endpoints accept this; a few only accept page images, in which case split the PDF first.
• Audio/video work only on models that accept those modalities.

Security

• Keys are never hardcoded; they are read from your config/environment at runtime.
• Data is sent only to the baseURL you configure.
• Store keys in environment variables or a git-ignored config file.
• Do not commit vision.json containing a real key.

For maintainers — publishing to npm

The package is published under the @we64 scope.

One-time setup

1. Create a free npm account: https://www.npmjs.com/signup

2. Create the we64 organization: https://www.npmjs.com/org/create (choose the name we64; free plan allows public packages).

3. Log in locally:

   npm login
   npm whoami

Publish

cd opencode-vision-router
npm version patch        # or minor / major
npm publish --access public

publishConfig.access is already set to public, so --access public is belt-and-braces for scoped packages.

Verify

npm view @we64/opencode-vision-multimodal

Then anyone can install with:

opencode plugin @we64/opencode-vision-multimodal

License

MIT © WE64 AI