SearXNG + Crawl4AI MCP Server

A self-hosted Model Context Protocol (MCP) server that provides powerful web search and scraping capabilities through SearXNG (metasearch engine) and Crawl4AI (advanced web scraper).

Features

• 🔍 Web Search: Aggregate results from 70+ search engines via SearXNG
• 🕷️ Web Scraping: Advanced scraping with Playwright browser automation
• 🚀 Fast Performance: 3x faster than native Claude Code tools
• 🏠 Fully Self-Hosted: No API limits, rate limits, or external dependencies
• 🔄 Flexible Deployment: Run locally or connect to remote server
• 📦 Easy Installation: NPM package with simple configuration

Quick Start

Prerequisites

• Node.js 18+
• Docker and Docker Compose
• Claude Code or any MCP-compatible client

Installation

Option 1: Local Development

# Clone repository
git clone https://github.com/yourusername/searxng-crawl4ai-mcp.git
cd searxng-crawl4ai-mcp

# Install dependencies
npm install

# Start Docker services
docker compose up -d

# Services will be available at:
# - SearXNG: http://localhost:8081
# - Crawl4AI: http://localhost:8001

Option 2: NPM Package (Coming Soon)

# Install globally
npm install -g searxng-crawl4ai-mcp

# Or use with npx
npx searxng-crawl4ai-mcp

Configure Claude Code

Add to your .mcp.json:

{
  "mcpServers": {
    "searxng-crawl4ai": {
      "command": "node",
      "args": ["mcp-server.js"],
      "cwd": "/path/to/searxng-crawl4ai-mcp",
      "env": {
        "SEARXNG_BASE_URL": "http://localhost"
      }
    }
  }
}

For remote server setup, change SEARXNG_BASE_URL:

"env": {
  "SEARXNG_BASE_URL": "http://192.168.1.100"
}

Available Tools

search_web

Search the web using SearXNG metasearch engine.

{
  "query": "latest AI developments",
  "maxResults": 10
}

crawl4ai_scrape

Scrape content from a single URL.

{
  "url": "https://example.com",
  "formats": ["markdown", "html"]
}

search_and_scrape

Search and automatically scrape top results (most powerful).

{
  "query": "Bitcoin price analysis",
  "maxResults": 3
}

Environment Variables

Copy .env.example to .env:

# Base URL for services (default: http://localhost)
# For remote server: http://YOUR_SERVER_IP
SEARXNG_BASE_URL=http://localhost

# Optional: Proxy for scraping
PROXY_URL=http://username:[email protected]:port

Docker Services

# Start services
npm run docker:up

# Stop services
npm run docker:down

# View logs
npm run docker:logs

Performance Comparison

| Metric | Native Claude Tools | SearXNG+Crawl4AI MCP | |--------|-------------------|---------------------| | Search Speed | 2500-3000ms | 935ms | | Scraping Success | 0% (WebFetch hangs) | 100% | | Content Extracted | 0 words | 29,807 words | | API Limits | Yes | None |

Architecture

┌─────────────┐     ┌─────────────┐     ┌──────────────┐
│ Claude Code │────▶│ MCP Server  │────▶│ Docker Stack │
│   (Client)  │     │ (Node.js)   │     │  - SearXNG   │
└─────────────┘     └─────────────┘     │  - Crawl4AI  │
                                        │  - Redis     │
                                        └──────────────┘

Deployment Options

1. Local: Everything runs on your machine
2. Remote Server: Docker services on server, MCP client connects remotely
3. Shared Infrastructure: One server supports multiple users

Testing

# Test MCP functionality
npm test

# Test search
node test-mcp-simple.js

# Test services directly
curl "http://localhost:8081/search?q=test&format=json"

Troubleshooting

MCP server not responding

• Verify SEARXNG_BASE_URL is correct
• Check services: docker compose ps
• For remote: ensure ports 8081 and 8001 are accessible

Crawl4AI browser errors

• Check logs: docker compose logs crawl4ai
• Playwright should be auto-installed in container

Search returning no results

• Verify SearXNG: curl http://localhost:8081/search?q=test&format=json
• Check searxng-settings.yml configuration

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - See LICENSE file for details.

Support

For issues or questions:

• Open an issue on GitHub
• Check Docker logs: docker compose logs [service]
• Review CLAUDE.md for detailed documentation