voltcrawl

Verified web evidence and headless browser rendering for coding agents.

voltcrawl is an open-source Model Context Protocol (MCP) server that connects AI coding agents to real headless Chromium rendering, sitemap navigation, and search candidate discovery. It strips cookie consent banners, paywalls, and navigation chrome from rendered pages, returning clean Markdown, structured JSON schemas, and verbatim quoted evidence.

flowchart TD
    subgraph Clients["MCP Clients"]
        OpenCode["OpenCode"]
        Cursor["Cursor"]
        Claude["Claude Desktop"]
    end

    subgraph Transports["Transports"]
        Stdio["stdio (JSON-RPC)"]
        StreamableHTTP["Streamable HTTP (JSON)"]
    end

    subgraph Core["voltcrawl Engine"]
        Server["MCP Server (server.ts)"]
        NoiseFilter["DOM Noise Filter (research.ts)"]
        ProofExtractor["Proof Extractor (research.ts)"]
    end

    subgraph Upstream["Upstream Services"]
        SearchProxy["Search Grounding Proxy<br/>(Vertex ADC / Custom HTTP)"]
        Firecrawl["Firecrawl Engine<br/>(Headless Chromium + NuQ)"]
    end

    OpenCode --> Stdio
    Cursor --> Stdio
    Claude --> Stdio
    OpenCode --> StreamableHTTP

    Stdio --> Server
    StreamableHTTP --> Server

    Server -->|"Search queries"| SearchProxy
    SearchProxy -->|"Candidate URLs & snippets"| Server
    Server -->|"Render & crawl requests"| Firecrawl

    Firecrawl -->|"Rendered DOM & Markdown"| NoiseFilter
    NoiseFilter --> ProofExtractor
    ProofExtractor -->|"Verified evidence pack"| Server

Why voltcrawl

AI coding agents need current web information to debug code, verify library APIs, and inspect documentation. Standard web search tools fail agents in two ways:

1. Search indexers return snippet summaries. These snippets frequently contain truncated code, obsolete syntax, or hallucinated facts.
2. Basic HTTP scrapers fail on modern single-page applications. They return blank root containers, cookie consent walls, and scriptless fallbacks.

voltcrawl uses Firecrawl as an honest operational engine for headless browser rendering and sitemap parsing. It pairs that engine with search candidate discovery and deterministic DOM noise filtering. It is not an affiliate pitch or a proprietary cloud wrapper. It connects to any self-hosted Firecrawl instance, local Docker container, or cloud endpoint.

Architecture and highlights

• Headless Chromium rendering: Executes client-side JavaScript and single-page applications before extracting text.
• Deterministic noise filtering: Strips OneTrust dialogs, cookie banners, ad-block warnings, and navigation sidebars. Preserves author bylines, timestamps, code blocks, and data tables.
• Truth contract enforcement: Separates search candidates from rendered evidence. Agents cite verified DOM output rather than search index snippets.
• Decoupled backends: Connects to self-hosted Firecrawl instances on localhost or remote hosts. Pairs with Google Search Grounding via Vertex AI ADC, custom search proxies, or offline test doubles.
• Offline determinism: Test suites run completely offline without cloud credentials or network connections.

Research workflow

The volt_crawl_research tool resolves natural-language questions to verified evidence in a single MCP call:

sequenceDiagram
    autonumber
    actor Agent as Coding Agent
    participant VC as voltcrawl (MCP)
    participant SG as Search Grounding (Vertex ADC)
    participant FC as Firecrawl (Chromium)
    participant NF as Noise Filter & Proof Extractor

    Agent->>VC: volt_crawl_research(query, scrapeTop, waitFor)
    VC->>SG: searchGoogle(query, numResults)
    SG-->>VC: Candidate URLs & snippets

    loop Render Top Candidates Sequentially
        VC->>FC: Scrape candidate URL (waitFor)
        FC-->>VC: Rendered page Markdown
        VC->>NF: Clean page chrome & extract proof
        NF-->>VC: Verified proof lines & excerpt
    end

    VC-->>Agent: Evidence pack (candidates + rendered proof)

Quickstart

Run directly using Bun or npx.

Stdio mode (Default)

Connect to a local Firecrawl instance running on port 3002:

export FIRECRAWL_API_URL="http://localhost:3002"

# Run via npx
npx voltcrawl

# Or run via bun
bunx voltcrawl

Streamable HTTP mode

Run as a standalone HTTP MCP service with token authentication:

export MASTER_API_KEY="your-secret-master-token"
export FIRECRAWL_API_URL="http://localhost:3002"
export MCP_HTTP_PORT=8787
export MCP_HTTP_HOST="127.0.0.1"

npx voltcrawl --http

Verify service readiness:

curl http://127.0.0.1:8787/healthz

Client configuration

Add voltcrawl to your MCP client configuration file.

OpenCode

Add to ~/.config/opencode/opencode.json or your project opencode.json:

{
  "mcpServers": {
    "voltcrawl": {
      "type": "local",
      "command": ["npx", "-y", "voltcrawl"],
      "environment": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

For remote HTTP deployments:

{
  "mcpServers": {
    "voltcrawl": {
      "type": "remote",
      "url": "https://mcp.example.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_MASTER_API_KEY"
      },
      "timeout": 180000
    }
  }
}

Cursor

Add to .cursor/mcp.json or ~/.cursor/mcp.json:

{
  "mcpServers": {
    "voltcrawl": {
      "command": "npx",
      "args": ["-y", "voltcrawl"],
      "env": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

Claude Desktop / Claude Code

Add to claude_desktop_config.json or .mcp.json:

{
  "mcpServers": {
    "voltcrawl": {
      "command": "npx",
      "args": ["-y", "voltcrawl"],
      "env": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

Available tools

The server exposes six tools across HTTP and stdio transports:

| Tool | Parameters | Description | |---|---|---| | volt_crawl_scrape | url (req), formats, onlyMainContent, waitFor, jsonPrompt, jsonSchema | Renders a target URL with headless Chromium. Extracts Markdown, HTML, or structured JSON. | | volt_crawl_search | query (req), numResults | Discovers search candidates using Google Search Grounding. Returns ranked URLs, titles, and snippets. | | volt_crawl_research | query (req), numResults, scrapeTop, waitFor, mode, maxExcerptChars | Resolves natural-language questions to verified evidence. Strips DOM noise and pulls verbatim proof lines. | | volt_crawl_map | url (req), search, limit | Traverses domain sitemaps via sitemap.xml. Returns an empty list when no sitemap exists. | | volt_crawl_crawl | url (req), limit, maxDiscoveryDepth | Starts asynchronous recursive crawling following links from rendered pages. | | volt_crawl_status | id (opt), includeDocs, limitDocs | Polls crawl job progress or verifies instance readiness. |

Error handling

Failed operations return { isError: true } with structured error categories:

stateDiagram-v2
    [*] --> RequestReceived
    RequestReceived --> ValidateParams
    ValidateParams --> ExecuteTool: Valid parameters
    ValidateParams --> [*]: Invalid schema (error)

    state ExecuteTool {
        [*] --> CallUpstream
        CallUpstream --> UpstreamSuccess
        CallUpstream --> UpstreamError
    }

    ExecuteTool --> SuccessResponse: UpstreamSuccess
    ExecuteTool --> ClassifyError: UpstreamError

    state ClassifyError {
        [*] --> CheckAntibot
        CheckAntibot --> BLOCKED: 403 / Cloudflare / Turnstile
        CheckAntibot --> CheckRateLimit: Not antibot
        CheckRateLimit --> RATE_LIMITED: 429 status
        CheckRateLimit --> CheckTimeout: Not rate limit
        CheckTimeout --> TIMEOUT: Deadline exceeded
        CheckTimeout --> CheckNetwork: Not timeout
        CheckNetwork --> NETWORK_ERROR: Connection refused
        CheckNetwork --> UPSTREAM_ERROR: Unclassified
    }

    ClassifyError --> ErrorResponse: Structured ToolErrorDetail
    SuccessResponse --> [*]
    ErrorResponse --> [*]

Dependency mapping and packaging

voltcrawl eliminates installation friction for users and agents:

When running via npx or bunx

• Zero dependency installation. The package ships pre-bundled Node.js and Bun ECMAScript Module artifacts in dist/.
• Executing npx voltcrawl or bunx voltcrawl runs immediately on any system with Node >= 20 or Bun installed.

When installed via npm

Installing voltcrawl automatically pulls in two curated production dependencies:

• @modelcontextprotocol/sdk (^1.6.1): The official Model Context Protocol TypeScript SDK implementing JSON-RPC 2.0 framing, StdioServerTransport, and StreamableHTTPServerTransport.
• zod (^3.24.2): Type-safe schema validation engine powering all six tool parameter contracts.

Configuration template

A documented configuration template is included in .env.example:

cp .env.example .env

Environment variables

| Variable | Default | Purpose | |---|---|---| | FIRECRAWL_API_URL | http://localhost:3002 | Upstream Firecrawl v2 REST endpoint | | FIRECRAWL_API_KEY | (empty) | Optional Bearer token for cloud or authenticated Firecrawl | | SEARCH_PROXY_URL | http://localhost:8088 | Upstream search proxy endpoint for Google Search Grounding | | MCP_HTTP_PORT | 8787 | Port for Streamable HTTP transport | | MCP_HTTP_HOST | 127.0.0.1 | Host binding for HTTP transport | | MASTER_API_KEY | (empty) | Enforced on HTTP transport; stdio transport is unauthenticated |

Development and testing

# Install dependencies
bun install

# Run static typecheck
bun run typecheck

# Run 10 offline test suites (144 tests)
bun test

# Run stdio integration test
bun run tests/stdio-test.ts

# Build production bundle for Node.js
bun run build

License

MIT (c) 2026 adi-IL