visionsearch-mcp

Multimodal Web Intelligence MCP — a local-first Model Context Protocol server that gives Claude Desktop and Claude Code the ability to research images and PDFs from the web using a locally-running vision model via LM Studio.

No cloud vision APIs. No silent downloads. All AI inference runs locally.

How it works

The agent passes URLs — base64 conversion happens internally inside each tool.

User prompt
    │
    ▼
Chat agent (Claude Desktop / Claude Code)
    │
    ▼  stdio MCP
visionsearch-mcp
    ├── vision_web_search     → SearXNG → classified {images[], documents[], articles[]}
    ├── analyze_image(url)    → fetch + LM Studio vision model
    ├── classify_image(url)   → fetch + LM Studio vision model
    ├── visual_qna(url)       → fetch + LM Studio vision model
    ├── ocr_image(url)        → fetch + Tesseract.js (local OCR, no LM Studio)
    ├── analyze_document(url) → fetch + pdf-parse (PDF only) + LM Studio text model
    ├── fetch_image(url)      → download image, return raw base64
    └── fetch_document(url)   → download document, return raw base64

Requirements

| Requirement | Details | |---|---| | Node.js | 18+ | | LM Studio | Running at localhost:1234 with a vision-capable model loaded | | SearXNG | Local instance — default http://localhost:8080 |

Recommended model: google/gemma-4-e4b — handles both image and text tasks in a single load, avoiding cold-load delays when LM Studio switches between models.

Avoid Qwen3 / thinking models as the text model — they emit tokens into reasoning_content and leave content empty, which causes analyze_document to return blank results.

Installation

npm install -g visionsearch-mcp

Claude Code setup

claude mcp add visionsearch -s user \
  -e SEARXNG_BASE_URL=http://localhost:8080 \
  -e LMSTUDIO_BASE_URL=http://localhost:1234/v1 \
  -e LMSTUDIO_VISION_MODEL=google/gemma-4-e4b \
  -e LMSTUDIO_TEXT_MODEL=google/gemma-4-e4b \
  -- npx -y visionsearch-mcp

The -s user flag registers the server at user scope (available across all projects). The -- separator is required before the command — omitting it causes a "missing required argument" error.

Claude Desktop setup

Locate your config file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

macOS / Linux

{
  "mcpServers": {
    "visionsearch-mcp": {
      "command": "npx",
      "args": ["visionsearch-mcp"],
      "env": {
        "SEARXNG_BASE_URL": "http://localhost:8080",
        "LMSTUDIO_BASE_URL": "http://localhost:1234/v1",
        "LMSTUDIO_VISION_MODEL": "google/gemma-4-e4b",
        "LMSTUDIO_TEXT_MODEL": "google/gemma-4-e4b"
      }
    }
  }
}

Windows

{
  "mcpServers": {
    "visionsearch-mcp": {
      "command": "C:\\Program Files\\nodejs\\node.exe",
      "args": [
        "C:\\Users\\YOUR_USER\\AppData\\Roaming\\npm\\node_modules\\visionsearch-mcp\\dist\\index.js"
      ],
      "env": {
        "SEARXNG_BASE_URL": "http://localhost:8080",
        "LMSTUDIO_BASE_URL": "http://localhost:1234/v1",
        "LMSTUDIO_VISION_MODEL": "google/gemma-4-e4b",
        "LMSTUDIO_TEXT_MODEL": "google/gemma-4-e4b"
      }
    }
  }
}

Restart Claude Desktop after saving.

Environment variables

| Variable | Default | Description | |---|---|---| | SEARXNG_BASE_URL | http://localhost:8080 | SearXNG instance URL | | LMSTUDIO_BASE_URL | http://localhost:1234/v1 | LM Studio OpenAI-compatible endpoint | | LMSTUDIO_VISION_MODEL | llava | Model used for image tools | | LMSTUDIO_TEXT_MODEL | llava | Model used for document summarization |

Available tools

vision_web_search

Queries SearXNG using two parallel requests — one for the images category (returns direct image URLs via the img_src field) and one for general (returns documents and articles). Results are merged and returned as three classified arrays.

Filtered automatically: icon CDNs (cdn.jsdelivr.net, unpkg.com, cdnjs.cloudflare.com), SVGs, GIFs, and ICO files.

{ "query": "modern japanese architecture", "max_results": 10 }

{
  "images":    [{ "url": "https://…/photo.jpg", "title": "…", "thumbnail": "…" }],
  "documents": [{ "url": "https://…/report.pdf", "title": "…", "type": "pdf" }],
  "articles":  [{ "url": "https://…", "title": "…", "snippet": "…" }]
}

analyze_image

Fetches an image from a URL (max 20 MB), sends it to the local LM Studio vision model as a base64 data URI, and returns structured analysis. If the model returns malformed JSON the raw text is returned as description.

{ "url": "https://example.com/photo.jpg", "user_confirmation": true }

{
  "url": "https://example.com/photo.jpg",
  "description": "A coyote walking across open arid ground mid-stride.",
  "objects": ["coyote", "ground"],
  "scene": "Wild canine traversing a natural outdoor landscape.",
  "confidence": 1
}

classify_image

Fetches an image from a URL and asks the local vision model to classify it into one of: screenshot, photo, document, chart, product, ui, meme, unknown. Confidence is fixed at 0.85 for a matched class and 0.3 for unknown — the model does not return logprobs.

{ "url": "https://example.com/photo.jpg", "user_confirmation": true }

{ "url": "https://example.com/photo.jpg", "classification": "photo", "confidence": 0.85 }

ocr_image

Fetches an image from a URL and runs Tesseract.js OCR locally. No LM Studio involved — OCR processing is entirely local. The fetch itself requires a network connection.

Requires a proper raster image (JPEG or PNG). Transparent PNGs and binary non-image content will cause Tesseract to throw an error.

{ "url": "https://example.com/scan.png", "user_confirmation": true, "language": "eng" }

{ "url": "…", "text": "Hello World", "confidence": 0.91, "word_count": 2 }

visual_qna

Fetches an image from a URL and sends it to the local vision model along with a question. Returns the model's answer.

{ "url": "https://example.com/photo.jpg", "question": "What colors are dominant?", "user_confirmation": true }

{ "url": "…", "question": "What colors are dominant?", "answer": "Gray, tan, and brown." }

analyze_document

Fetches a document from a URL (max 50 MB) and analyses it.

PDF: text is extracted page-by-page using pdf-parse, capped at 12 000 characters, then summarized by the local text model. If pdf-parse returns no text (scanned PDF), falls back to Tesseract.js OCR on the raw buffer.

DOC / PPT / XLS: no dedicated parser — falls back to Tesseract.js OCR on the raw binary, which is unlikely to produce useful output. For these formats, consider converting to PDF first.

{ "url": "https://example.com/report.pdf", "user_confirmation": true, "max_pages": 20 }

{
  "url": "…",
  "summary": "…",
  "sections": ["Introduction", "Methods", "Results"],
  "key_points": ["Finding 1", "Finding 2"],
  "page_count": 5,
  "used_ocr_fallback": false
}

fetch_image

Downloads an image from a URL and returns the raw base64 content. Validates the response Content-Type against an allowed list (image/jpeg, image/png, image/webp, image/gif, image/bmp, image/svg+xml, image/avif). Max size: 20 MB. Blocked until user_confirmation: true.

{ "url": "https://example.com/photo.jpg", "user_confirmation": true }

{ "confirmed": true, "url": "…", "mime_type": "image/jpeg", "size_bytes": 35000, "content_base64": "…" }

fetch_document

Downloads a document from a URL and returns the raw base64 content. Validates the Content-Type against allowed MIME types (application/pdf, application/msword, Office Open XML variants, application/vnd.ms-powerpoint). Falls back to URL extension check (.pdf, .doc, .docx, .ppt, .pptx) when servers mis-report MIME. Max size: 50 MB. Blocked until user_confirmation: true.

{ "url": "https://example.com/report.pdf", "user_confirmation": true }

{ "confirmed": true, "url": "…", "file_type": "pdf", "mime_type": "application/pdf", "size_bytes": 20000, "content_base64": "…" }

Example agent pipeline

User: "find images and a PDF about climate change and explain them"

1. vision_web_search({ query: "climate change infographic" })
   → { images: [{ url: "https://…/chart.jpg" }], documents: [{ url: "https://…/report.pdf" }] }

2. analyze_image({ url: "https://…/chart.jpg", user_confirmation: true })
   → { description: "Global temperature anomaly chart…", objects: ["graph", "axis"] }

3. classify_image({ url: "https://…/chart.jpg", user_confirmation: true })
   → { classification: "chart" }

4. analyze_document({ url: "https://…/report.pdf", user_confirmation: true })
   → { summary: "…", key_points: ["…", "…"] }

5. Agent synthesizes results → unified answer

Development

git clone https://github.com/tiagohgms/visionsearch-mcp.git
cd visionsearch-mcp
npm install
npm run build

Run all tests:

node test/runner.mjs

Publish a new version:

npm version patch   # or minor / major
npm publish

License

MIT