vision-mcp

An MCP server that gives a text-only coding agent vision, using the z.ai coding plan you already have.

The z.ai coding plan's text/coding models have no vision — they reject image input. But glm-4.6v is a multimodal model on that same plan, reachable at the same coding-plan endpoint (https://api.z.ai/api/coding/paas/v4) with the same API key and same billing. vision-mcp is the bridge: it accepts an image (or a video, or a cropped region) plus a prompt, forwards it to glm-4.6v on your plan, and returns text. It exposes six tools over stdio — including adaptive inspection (inspect_image), video analysis (ffmpeg frame extraction), and region zoom (sharp) for fine detail — to any MCP client. Every request flows through a pipeline that adds bounded retries, opt-in content-addressed caching, and optional request metadata. The default backend is z.ai GLM-4.6V (OpenAI Chat Completions format); an Anthropic-format provider is also supported for other gateways.

The published npm package @keohanoi/vision-mcp runs under Node.js ≥ 18 via npx — no install step. For development / from-source, bun runs the TypeScript directly with no build step.

Why

The z.ai coding plan already bundles a vision-capable model alongside its text/coding models — but the models you actually drive (and most agent defaults) are text-only and can't see images. This server routes image/OCR/video calls to glm-4.6v on your same plan, same API key, same billing — no separate vision subscription and no second account.

Use it whenever your coding agent needs to:

• Read a screenshot or UI mock
• OCR an error message, stack trace, log, or config from an image
• Compare two UI states (before/after, diffs)
• Summarize a video by sampled frames

Works with any MCP client — Claude Code, OpenCode, and anything else that speaks MCP over stdio.

Prerequisites

• Node.js ≥ 18 — required to run the published package via npx
• ffmpeg + ffprobe on PATH — only required for the analyze_video tool

For from-source / development (see From source (development) below): bun, and sharp (installed automatically by bun install; used by analyze_image / extract_text / analyze_region for crop, resize, and format control).

Install

No install step is needed for normal use — npx fetches @keohanoi/vision-mcp on first run. See Quick start (published package) below.

For development / running from source, see From source (development) and run bun install in the repo.

Configure

Copy .env.example to .env and fill in the values for the provider you want to use.

cp .env.example .env

Quick start (published package)

The published npm package @keohanoi/vision-mcp is the primary install/config path. It runs under Node via npx — no clone, no build.

Prerequisites

• Node.js ≥ 18 (npx runs the package under Node)
• ffmpeg/ffprobe on PATH — only required if using the analyze_video tool

Configure

Add the server to your MCP client config (e.g. Claude Code's mcpServers, OpenCode, etc.):

{
  "mcpServers": {
    "vision": {
      "command": "npx",
      "args": ["-y", "@keohanoi/vision-mcp"],
      "env": {
        "VISION_PROVIDER": "openai",
        "VISION_BASE_URL": "https://api.z.ai/api/coding/paas/v4",
        "VISION_API_KEY": "your-z.ai-coding-plan-key",
        "VISION_MODEL": "glm-4.6v"
      }
    }
  }
}

CLI alternative (the VISION_* env vars must be exported in your shell first when using this form):

claude mcp add vision -- npx -y @keohanoi/vision-mcp

Environment variables

| Variable | Required | Default | Description | |---|---|---|---| | VISION_PROVIDER | no | openai | anthropic (→ /v1/messages), openai (→ {base}/chat/completions), or responses (→ {base}/responses, OpenAI Responses API). Defaults to openai (z.ai) | | VISION_BASE_URL | no (openai) / yes (anthropic) | https://api.z.ai/api/coding/paas/v4 (when provider=openai) | Recommended. Collision-safe alias; takes precedence over the provider-specific var below | | VISION_API_KEY | yes | — | Recommended. Collision-safe alias for the API key | | VISION_MODEL | no (openai) / yes (anthropic) | glm-4.6v (when provider=openai) | Recommended. Collision-safe alias; must be a multimodal/vision model. On the z.ai coding plan: glm-4.6v (default), also glm-4.5v and glm-4.6v-flash | | VISION_THINKING | no | disabled | enabled | disabled — controls GLM thinking mode. Default disabled for fast/cheap perception & OCR | | ANTHROPIC_BASE_URL | anthropic | — | Anthropic-format endpoint (no /v1; the SDK appends paths). Used only if VISION_BASE_URL is unset | | ANTHROPIC_API_KEY | anthropic | — | API key. Used only if VISION_API_KEY is unset | | ANTHROPIC_MODEL | anthropic | — | Must be a multimodal/vision model. Used only if VISION_MODEL is unset | | OPENAI_BASE_URL | openai | — | OpenAI-format endpoint. Does not have to end in /v1 (z.ai ends in /v4). Used only if VISION_BASE_URL is unset | | OPENAI_API_KEY | openai | — | API key. Used only if VISION_API_KEY is unset | | OPENAI_MODEL | openai | — | Must be a multimodal/vision model. Used only if VISION_MODEL is unset | | VISION_MAX_TOKENS | no | 1024 | Max output tokens per request | | VISION_TIMEOUT_MS | no | 60000 | Request timeout in ms | | VISION_MAX_RETRIES | no | 3 | Max retries for transient errors (429/timeout/5xx/network). Non-retryable errors (401/403/4xx) run exactly once | | VISION_RETRY_BASE_DELAY_MS | no | 500 | First retry backoff (ms); exponential with jitter, capped at VISION_RETRY_MAX_DELAY_MS. Honors Retry-After | | VISION_RETRY_MAX_DELAY_MS | no | 8000 | Cap on a single retry backoff | | VISION_CACHE_DIR | no | (off) | Directory for content-addressed caching. Opt-in — caching is off unless this and a non-zero TTL are set | | VISION_CACHE_TTL_SECONDS | no | 0 | Cache entry lifetime (seconds). 0 disables caching | | VISION_CACHE_MAX_MB | no | 512 | Soft size cap; oldest entries evicted when exceeded | | VISION_INCLUDE_METADATA | no | disabled | enabled | disabled — append a _meta block (model, timing, dimensions, cache hit, passes) to tool results. Informational only |

Why VISION_* aliases? bun's .env loader does not override variables already present in the process environment. If a host already exports ANTHROPIC_* (e.g. a coding agent's own credentials, as Claude Code does), those inherited values silently shadow your config. The VISION_* names never collide, so prefer them in mcpServers env blocks and in .env.

From source (development)

For development, or if you prefer to run from a clone, bun runs the TypeScript directly with no build step. After cloning:

bun install

Then configure your MCP client to run the server from source.

Run as an MCP server — z.ai GLM-4.6V (default)

{
  "mcpServers": {
    "vision": {
      "command": "bun",
      "args": ["run", "/ABS/PATH/vision-mcp/src/index.ts"],
      "env": {
        "VISION_PROVIDER": "openai",
        "VISION_BASE_URL": "https://api.z.ai/api/coding/paas/v4",
        "VISION_API_KEY": "your-z.ai-coding-plan-key",
        "VISION_MODEL": "glm-4.6v",
        "VISION_THINKING": "disabled",
        "VISION_MAX_TOKENS": "1024",
        "VISION_TIMEOUT_MS": "60000"
      }
    }
  }
}

Alternative: OpenCode Go / Anthropic

{
  "mcpServers": {
    "vision": {
      "command": "bun",
      "args": ["run", "/ABS/PATH/vision-mcp/src/index.ts"],
      "env": {
        "VISION_PROVIDER": "anthropic",
        "VISION_BASE_URL": "https://opencode.ai/zen/go",
        "VISION_API_KEY": "your-key",
        "VISION_MODEL": "minimax-m3",
        "VISION_MAX_TOKENS": "1024",
        "VISION_TIMEOUT_MS": "60000"
      }
    }
  }
}

On OpenCode Go, MiniMax M3 / Qwen3.7 are served at /v1/messages (Anthropic format, minimax-m3); GLM / Kimi / DeepSeek / MiMo are at /v1/chat/completions (OpenAI format). The Go coding models are code-optimized — verify a model is multimodal before relying on it for images. [1m] is not part of any model id — it's a CLI-only suffix and is rejected by the API. The free Zen tier (https://opencode.ai/zen/v1) routes differently from the Go subscription.

Alternative: OpenCode Go / GPT-5.6 Luna (Responses API)

GPT-5.6 Luna on Go is served only at /v1/responses (OpenAI Responses API format) — chat/completions returns 400 for it. Use VISION_PROVIDER=responses with a base URL that includes /v1:

{
  "mcpServers": {
    "vision": {
      "command": "bun",
      "args": ["run", "/ABS/PATH/vision-mcp/src/index.ts"],
      "env": {
        "VISION_PROVIDER": "responses",
        "VISION_BASE_URL": "https://opencode.ai/zen/go/v1",
        "VISION_API_KEY": "your-key",
        "VISION_MODEL": "gpt-5.6-luna",
        "VISION_THINKING": "disabled",
        "VISION_MAX_TOKENS": "1024",
        "VISION_TIMEOUT_MS": "60000"
      }
    }
  }
}

VISION_THINKING=disabled maps to reasoning: {effort: "none"} (fast/cheap perception & OCR); enabled omits the field so the model uses its default reasoning. Note the Responses API requires input_image.image_url as a plain string data URI — the {url: …} object form is rejected with 400 invalid_prompt by strict upstreams.

Standalone live test

Dev / from-source utility — this script is run from a clone with bun.

scripts/test-vision.ts runs a single end-to-end call against your configured provider (bun auto-loads .env from the cwd). It sends your API key to your configured endpoint — that is the point.

# Uses an inline 1x1 PNG:
bun run scripts/test-vision.ts

# Or pass any image (path/URL/data-uri/base64):
bun run scripts/test-vision.ts ./photo.jpg

Tools

• analyze_image — Describe or answer a question about a single image.
  • image (string, required): file path, http(s) URL, data: URI, or base64
  • prompt (string, optional): structured default (summary → subjects / quoted text / colors / layout)
  • response_format ("text"|"evidence", optional, default text): evidence separates directly-visible facts from inferences and returns JSON {visible[], inferences[], unreadable[], confidence} (confidence = visual legibility, not diagnosis correctness)
  • format ("auto"|"png"|"jpeg"|"webp", optional): png is lossless — best for text/diagrams
  • max_dimension (number, optional): downscale longest side ≤ N px (cost control)
  • min_dimension (number, optional): upscale longest side ≥ N px (legibility for small images)
  • include_metadata (boolean, optional): append a _meta block
• inspect_image — Higher-level adaptive inspection: detects hard-to-read regions, zooms/upscales them, optionally OCRs critical text, and returns one consolidated answer with per-region evidence. Best for terminals, DevTools, dense diagrams, receipts, IDE screenshots. Bounded (≤ 3 passes, ≤ 4 regions); never recurses.
  • image (string, required)
  • question (string, optional): what to ask / find out
  • mode ("auto"|"fast"|"thorough", optional, default auto): fast=1 pass, auto≤≤2, thorough≤3 + OCR of critical regions
  • response_format ("text"|"evidence", optional, default text): evidence returns the visible-vs-inference split as JSON
  • format, max_dimension, min_dimension, include_metadata (all optional)
• compare_images — Compare 2–4 images (structured similarity/differences default).
  • images (string[] or [{image, label?}], required, 2–4): each entry is an image, optionally with a label so the model never refers to "image 1/2" ambiguously
  • labels (string[], optional): labels applied in order to plain-string images
  • mode ("semantic"|"ui-regression"|"pixel-layout"|"text-diff", optional, default semantic): ui-regression normalizes both images to a common size and returns a structured JSON diff {missing, added, moved, textChanges, styleChanges, likelyAcceptable}
  • prompt, include_metadata (optional)
• extract_text — OCR a single image. Always re-encodes to lossless PNG first (JPEG artifacts wreck OCR); the prompt enforces reading order and exact quoting.
  • image (string, required)
  • output_format ("plain" | "markdown" | "json" | "blocks", optional, default plain): blocks returns spatial blocks {blocks:[{text, type, region}]} with best-effort normalized 0–1 regions
  • include_metadata (optional)
• analyze_region — Zoom into part of an image for fine detail (small text, dense diagrams, tiny UI). Crops + upscales via sharp.
  • image (string, required)
  • region ({x,y,width,height} in normalized 0–1, optional): zoom exactly this box. Omit for auto-detect of up to 4 regions of interest.
  • max_passes (1–3, optional, default 1): iteratively re-zoom a region until its detail is legible (bounded ≤ 3). Default 1 keeps the current single-pass behavior.
  • prompt, zoom (1–8, default 2), format, max_dimension, include_metadata (all optional)
• analyze_video — Summarize a video by sampling frames with ffmpeg and sending them to the model as images. Each frame carries its timestamp.
  • video (string, required): file path, http(s) URL, data: URI, or base64
  • frames (number, optional, 1–16, default 8): target frame count
  • sampling ("uniform"|"scene-change"|"hybrid", optional, default uniform): scene-change samples at shot transitions (gt(scene,0.3)); hybrid merges even + scene-change frames
  • start_seconds / end_seconds (number, optional): analyze only a window of the video
  • response_format ("text"|"events", optional, default text): events returns JSON {events:[{timestamp, observation}], summary}
  • prompt, include_metadata (optional)

Improving vision quality

These tools exist to make the model actually see better, not just forward pixels:

• analyze_region is the big one. The model normalizes images to ~1568px, so fine text that occupies a small fraction of a large image gets downscaled past readability and garbled (e.g. 1Z→12, q3-vl→q3-v1, MZ9K→M29K). Cropping tightly around the text so it fills more of the frame restores legibility. Verified by A/B: a full-image read of a dense narrow-column receipt misread 5+ tokens; the zoomed analyze_region read them all correctly.
• inspect_image automates the zoom loop. Instead of manually chaining analyze_image → analyze_region → extract_text, inspect_image does a full-image pass, detects regions worth zooming, crops/upscales each, asks whether the detail is now legible, re-zooms if not (bounded ≤ 3 passes), and optionally OCRs critical text — returning one consolidated answer with per-region evidence and a visible-vs-inference split.
• extract_text forces lossless PNG, and analyze_image exposes format / max_dimension / min_dimension, for the same reason — don't let JPEG artifacts or over-downscaling destroy text.

Reliability, caching & metadata

Every model request flows through a single pipeline that layers three behaviors (architecture invariants: only the pipeline retries or caches; tools and providers never do):

• Retries. Transient failures (429, timeout, 5xx, network) are retried with exponential backoff + jitter, honoring any Retry-After header (up to VISION_MAX_RETRIES). Non-retryable errors (401/403/other 4xx) run exactly once. If the provider rejects an image format, the image is transcoded to PNG once and retried once before failing.
• Caching (opt-in). Set VISION_CACHE_DIR + a non-zero VISION_CACHE_TTL_SECONDS to memoize results in a content-addressed on-disk cache (keyed on image bytes + provider + model + prompt + options — never API keys). Preprocessed crops are cached separately too. This is valuable when an agent re-inspects the same screenshot while reasoning. Off by default; entries are TTL-expired and size-capped (VISION_CACHE_MAX_MB).
• Metadata. Set VISION_INCLUDE_METADATA=enabled (or pass include_metadata: true per call) to append a _meta block: model, duration, retry attempts, cache hit, image dimensions, and pass count. _meta is informational only — do not parse it in code.

Caveats

• Default provider is now openai (z.ai). Previously anthropic — if you relied on the old default, set VISION_PROVIDER=anthropic explicitly.
• GLM-4.6V accepts jpg/png only (not webp/gif). The default tool paths emit PNG, so they're safe; an explicit format: "webp"/"gif" on analyze_image / analyze_region is rejected by z.ai — but the pipeline now auto-transcodes to PNG once and retries on a format rejection, so such calls usually still succeed.
• VISION_THINKING defaults to disabled for speed/cost on perception calls; set enabled for harder reasoning.
• Don't rely on ANTHROPIC_* in a host that already exports them. bun's .env does not override inherited process env, so a coding agent's own ANTHROPIC_* will shadow your config. Use the VISION_* aliases in .env and in mcpServers env.
• API keys stay in .env. .env is gitignored — never commit it.
• Video is frame-extraction, not native input. The provider endpoints accept only still images — analyze_video samples N evenly-spaced frames via ffmpeg and sends them as image blocks, so it captures motion but not audio or sub-frame detail; cost/latency scale with frame count (capped at 16). (z.ai also offers native video/file inputs we don't use — out of scope.)

Alternative-provider caveats (OpenCode Go / Anthropic)

These apply only when routing through OpenCode Go instead of the z.ai default:

• [1m] is not part of the model id. opencode's CLI prints minimax-m3[1m] to denote a 1M-context variant, but the API model field takes the bare id minimax-m3. Sending [1m] → Model … is not supported.
• OpenCode Go vs. Zen routing. A Go subscription's base URL is always https://opencode.ai/zen/go. The separate free Zen tier (https://opencode.ai/zen/v1) routes differently.
• minimax-m3 is multimodal and verified working for image analysis on Go — and cheap ($0.30 in / $1.20 out per 1M tokens).

Verified

End-to-end green-path test against z.ai with glm-4.6v (OpenAI provider, config read straight from .env — only VISION_API_KEY required beyond baked defaults). Fetched a real image from a URL and the model read its text — confirming the full pipeline: baked defaults → image fetch → base64 → z.ai Bearer auth → OpenAI Chat Completions response parsing (effectively OCR-grade vision).

$ bun run scripts/test-vision.ts 'https://placehold.co/600x400/png'
Provider: openai | Model: glm-4.6v | Base URL: https://api.z.ai/api/coding/paas/v4
→ 'This is a tiny test image (placeholder image). It displays a plain light gray
   background with the text "600 × 400" centered in a medium gray, sans-serif font…'

Note: z.ai rejects the inline 1×1 transparent PNG used by the no-argument form of scripts/test-vision.ts with HTTP 400 / error code 1210 ("image input format/parse error") — it dislikes degenerate/blank images, not the request shape. Pass a real image (URL/path) to verify the pipeline.

Reproduce with any image (local path, URL, data-URI, or base64):

bun run scripts/test-vision.ts ./photo.jpg