Web Search MCP

Multi-source web search MCP server with RRF fusion, 4-layer URL extraction, and provider health tracking.

Status

Phase 2: 5 providers (SearXNG/ddgr/Tavily/Exa/opencli-zh), RRF fusion, snippet cleaning (rules + optional LLM compression), search/extract/doctor tools, provider health tracking with auto-degradation.

Install

Add to Claude Desktop 3P config (~/Library/Application Support/Claude-3p/configLibrary/<uuid>.json):

{
  "name": "web-search",
  "source": "user",
  "transport": "stdio",
  "command": "node",
  "args": ["/Users/hades/projects/web-search-mcp/bin/cli.js"],
  "env": {
    "SEARXNG_URL": "http://localhost:18443",
    "TAVILY_API_KEY": "...",
    "EXA_API_KEY": "...",
    "FIRECRAWL_API_KEY": "...",
    "WSM_LLM_PROVIDER": "haiku",
    "ANTHROPIC_API_KEY": "..."
  },
  "toolPolicy": {
    "search": "allow",
    "extract": "allow",
    "doctor": "allow"
  }
}

Once published to npm, replace command: "node" + absolute path with command: "npx" + args: ["-y", "@wsm/cli@latest"].

Configuration

Environment variables (or .env in cwd or ~/.agents/skills/web-search/.env):

| Var | Default | Required | Description | |---|---|---|---| | SEARXNG_URL | http://localhost:18443 | no | SearXNG base URL | | TAVILY_API_KEY | — | no | Enables Tavily provider | | EXA_API_KEY | — | no | Enables Exa provider | | FIRECRAWL_API_KEY | — | no | Enables Firecrawl in extract pipeline (layer 3) | | WSM_LLM_PROVIDER | none | no | haiku for Anthropic, local for Ollama, none to disable | | ANTHROPIC_API_KEY | — | no | Required when WSM_LLM_PROVIDER=haiku | | WSM_LOCAL_LLM_URL | — | no | Required when WSM_LLM_PROVIDER=local | | WSM_MAX_RESULTS | 10 | no | Default max results | | WSM_TIMEOUT_MS | 25000 | no | Global search timeout |

Providers without keys are automatically disabled. The search tool still works with whatever is available. ddgr, opencli, defuddle are auto-detected via which.

Tools

search

{
  query: string,              // required
  max_results?: number,       // 1-30, default 10
  category?: 'general'|'news'|'science',
  mode?: 'fast'|'quality',    // quality = LLM compression (top-5 results)
  providers?: string[],       // override default list (replace semantics)
}

Returns:

{
  count: number,
  took_ms: number,
  providers_used: string[],     // providers that returned ≥1 result
  providers_failed: string[],   // providers that errored OR returned []
  results: [{
    title, url, snippet,
    providers: string[],        // which sources surfaced this URL (multi-source corroboration)
    rrf_score: number,          // reciprocal rank fusion score
  }]
}

opencli-zh auto-routing: When query contains keywords like 知乎, 小红书, 雪球, B站, 微博, or patterns like 600519.SH, the opencli-zh provider routes to the corresponding adapter. Force a specific site with opencli_sites (passed via providers extension, see orchestrator).

extract

{
  url: string,                 // required
  timeout?: number,            // 5-60 seconds, default 20
}

Tries 4 layers in order until one succeeds:

1. defuddle — fastest, pure-text sites
2. opencli adapter — for known sites (zhihu/xiaohongshu/xueqiu/bilibili/weibo)
3. Firecrawl — JS-rendered pages (requires FIRECRAWL_API_KEY)
4. opencli browser — last resort, drives real Chrome

Returns:

{
  content: string,             // markdown
  source: 'defuddle'|'opencli'|'firecrawl'|'opencli-browser'|'none',
  took_ms: number,
  error?: string,              // only when source === 'none'
}

doctor

{} // no arguments

Returns provider availability + runtime health stats:

{
  providers: {
    searxng:    { available: true, url: "http://localhost:18443" },
    ddgr:       { available: true },
    tavily:     { available: true },
    exa:        { available: true },
    opencli_zh: { available: true },
    firecrawl:  { available: true },
    llm:        { available: true, provider: "haiku" },
  },
  health: {
    tavily: {
      total: 24,
      success: 15,
      fail: 9,
      success_rate: 0.625,
      last_failure: "2026-07-06T...",
      last_failure_reason: "timeout",
      recommendation: "healthy" | "insufficient" | "degraded",
    },
    // ...
  },
  version: "0.2.0",
}

When a provider's success_rate drops below 0.5 with at least 20 samples, it's automatically skipped (recommendation: degraded). Health stats are in-memory and reset on restart.

Development

npm test                  # all tests (unit + integration, 158 total)
npm run test:unit         # unit only
npm run test:integration  # integration only (needs searxng at SEARXNG_URL)
npm start                 # start stdio server

Architecture

MCP client (Claude/Cowork)
  │ JSON-RPC over stdio
  ▼
server.js ───┬── search tool ──→ search.js (orchestrator)
             ├── extract tool ─→ extract/index.js (4-layer pipeline)
             └── doctor tool ──→ health tracker snapshot

search.js:
  Provider layer (parallel, isolated failure)
    searxng | ddgr | tavily | exa | opencli-zh
  ▼
  Merger layer: normalize → dedup(URL) → RRF(k=60)
  ▼
  Cleaner layer: rules (default) | LLM compression (quality mode, top-5)
  ▼
  Health tracker records success/failure per provider

extract/index.js:
  defuddle → opencli adapter → Firecrawl → opencli browser
  (each layer returns null on failure, falls through to next)

Phase history

• Phase 1 (15 tasks): 4 providers (searxng/ddgr/tavily/exa), RRF fusion, rule-based cleaning, stdio MCP, search tool only
• Phase 2 (8 tasks): opencli-zh provider, LLM quality mode, 4-layer extract, doctor + health auto-degradation

License

MIT