Table of Contents

• Why CrawlForge?
• CrawlForge vs. alternatives
• Quick Start (2 Minutes)
• Available Tools
• Pricing
• Advanced Configuration
• Usage Examples
• Security & Privacy
• Support
• Contributing

🎯 Why CrawlForge?

• 31 MCP-native tools — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous agent, a unified multi-format scrape, document processing, stealth browsing, stateful browser sessions, and more, callable directly from your AI assistant.
• Generous free tier — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire.
• Local-LLM by default — extract_with_llm runs against a local Ollama model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
• LLM-ready output — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
• Autonomous agent — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.
• Security-hardened — SSRF protection on every request, a fail-closed backend allow-list, a vetted action allowlist for browser automation, and per-tool credit gating.
• Works everywhere MCP does — Claude Desktop, Claude Code, Cursor, and any other MCP-enabled client, configured in one command.

📊 CrawlForge vs. alternatives

| | CrawlForge MCP | Firecrawl | Raw scraping API | |---|:---:|:---:|:---:| | Native MCP server | ✅ 31 tools | ✅ | ❌ | | Free tier | ✅ 1,000 credits, one-time, never expire | Limited | Varies | | Self-hosted / local LLM extraction (Ollama) | ✅ default, $0/token | ❌ | ❌ | | Autonomous agent (no URLs needed) | ✅ agent | ✅ | ❌ | | Deep research with source verification | ✅ deep_research | Partial | ❌ | | Browser automation / actions | ✅ scrape_with_actions | ✅ | Varies | | Stealth / anti-detection engines | ✅ Chromium + Camoufox | ✅ | Add-on | | Pre-built site templates | ✅ 10 sites | ❌ | ❌ | | License | MIT | AGPL-3.0 | Proprietary |

Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants.

🚀 Quick Start (2 Minutes)

1. Install from NPM

npm install -g crawlforge-mcp-server

2. Setup Your API Key (required)

Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start. The recommended path signs you in through the browser, so the key is never pasted into a terminal (a coding agent can run this for you and relay the URL):

crawlforge login

It prints an approval URL; open it, approve, and the key is stored in ~/.crawlforge/config.json. Then run crawlforge init to register the MCP server with your client. Or use the interactive wizard, which also configures your clients:

npx crawlforge-setup

This will:

• Guide you through getting your free API key
• Configure your credentials securely
• Auto-configure Claude Code and Cursor (if installed)
• Verify your setup is working

Don't have an API key? Get one free at https://www.crawlforge.dev/signup

One-step setup (v4.6.0+): crawlforge init detects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Use crawlforge init --all --yes to configure every detected client non-interactively.

3. Configure Your IDE (if not auto-configured)

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server"]
    }
  }
}

Location:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%/Claude/claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

Restart Claude Desktop to activate.

The setup wizard automatically configures Claude Code by adding to ~/.claude.json:

{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}

After setup, restart Claude Code to activate.

The setup wizard automatically configures Cursor by adding to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}

Restart Cursor to activate.

n8n's built-in MCP Client Tool node connects over Streamable HTTP (works on n8n Cloud and self-hosted). Run the server in HTTP mode:

export CRAWLFORGE_API_KEY=your_api_key
npm run start:http    # Streamable HTTP endpoint at http://localhost:10000/mcp

Then point the MCP Client Tool node at http://<host>:10000/mcp with transport HTTP Streamable and a Bearer credential set to the same API key. On self-hosted n8n you can instead use the community n8n-nodes-mcp node over STDIO (npx -y crawlforge-mcp-server).

Full guide: docs/n8n-integration.md

Which launch command? npx -y crawlforge-mcp-server needs no global install and always runs the published version (recommended for Claude Desktop). For a global install (npm i -g crawlforge-mcp-server), use the dedicated crawlforge-mcp bin — it resolves on your PATH, so it survives Node/nvm version switches. The bare crawlforge command still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — run crawlforge mcp to start the server by hand.

📊 Available Tools

CrawlForge requires a CrawlForge API key — every tool is metered and consumes credits. New accounts get 1,000 free trial credits to start. Get a key at crawlforge.dev/signup.

All Tools (API key required)

| Tool | Credits | What it does | |------|---------|--------------| | fetch_url | 1 | Fetch content from any URL | | extract_text | 1 | Extract clean text from web pages | | extract_links | 1 | Get all links from a page | | extract_metadata | 1 | Extract page metadata (title, OG tags, schema.org) | | scrape_template | 1 | Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors | | list_ollama_models | 1 | List the Ollama models installed locally (helps you pick a model for extract_with_llm) | | get_batch_results | 1 | Retrieve paginated results for a batch_scrape job by batchId | | read_result | 1 | Search, slice, read lines or a JSON path from a result a tool returned with truncated: true and a result_handle (kept 1 hour on your own machine) — never fetch the page again | | scrape | 2 | Unified single-fetch, multi-format extraction. Pass a formats array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema, plus {type:"highlights",query} and {type:"question",question} for only the matching sentences, table rows and code blocks, verbatim with offsets into the markdown: +1 credit once per call, mode:"model" +3) plus onlyMainContent; one fetch serves every requested format with per-format partial-success warnings. escalate:true retries a blocked plain fetch once in the stealth browser and returns the page instead of the block (+5 as the projected ceiling; the charge drops back to 2 when the plain fetch succeeded and no escalation ran) | | scrape_structured | 2 | Extract structured data with CSS selectors | | extract_embedded_state | 2 | Read a page's embedded JavaScript state — __NEXT_DATA__, React Server Component payloads, Nuxt, Apollo, Redux, <script type="application/json"> — with a path to scope the result. No LLM in the extraction path | | extract_content | 2 | Enhanced content extraction | | map_site | 2 | Discover and map website structure (optional search= ranks the discovered URLs) | | process_document | 2 | Multi-format document processing | | localization | 2 | Multi-language and geo-location management | | track_changes | 3 | Monitor content changes over time | | analyze_content | 3 | Comprehensive content analysis | | extract_structured | 3 | LLM-powered schema-driven extraction (your own LLM key or local Ollama) | | extract_with_llm | 3 | Natural-language extraction. Defaults to a local Ollama model; pass provider: "openai" \| "anthropic" with the matching key for cloud models (external LLM billed by your provider) | | browser_session | 3 | A browser page that stays open across calls, keeping its cookies and its login in between. open a session on a URL, snapshot it to list the interactive elements as stable refs (@e1, @e2), act on a ref, read the content, close. Priced per operation: open 3, read 2, snapshot/act/screenshot/close/list 1 each. Reach for it when you must see the page before choosing what to click, when the flow spans several calls, or when a login must hold across later reads; one fixed action chain on one page is scrape_with_actions | | summarize_content | 4 | Generate intelligent summaries | | crawl_deep | 4 | Deep crawl entire websites | | search_web | 5 | Search the web using Google Search API | | reddit_search | 5 | Search Reddit posts/comments or read a full thread — reddit.com blocks direct scraping, so this reads the Arctic Shift community archive (free, no Reddit credentials). A Reddit-wide search spends a web search to discover posts, so it is priced with search_web | | serp_rank | 5 | Check where a domain ranks in Google's real organic SERP for a keyword (the position search_web can't give). Powered by DataForSEO (DATAFORSEO_LOGIN/DATAFORSEO_PASSWORD, billed to your own DataForSEO account). Returns { configured:false } and charges 0 credits until configured | | batch_scrape | 5 | Process multiple URLs simultaneously | | scrape_with_actions | 5 | Browser automation chains | | generate_llms_txt | 5 | Generate AI interaction guidelines | | stealth_mode | 5 | Anti-detection browser management | | agent | 8 | Autonomous research/extraction from a natural-language prompt — no URLs required. Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM) | | deep_research | 10 | Multi-stage research with source verification |

Ten tools (scrape, fetch_url, extract_content, crawl_deep, batch_scrape, stealth_mode, scrape_with_actions, process_document, deep_research, extract_embedded_state) accept max_inline_chars (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS): a result over it comes back as a preview plus a result_handle for read_result, with the full result kept for 1 hour under ~/.crawlforge/results/ on your own machine — nothing is uploaded.

For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see SKILL.md.

💳 Pricing

Every tool is metered and requires an API key. New accounts get 1,000 free trial credits — no credit card required to start.

| Plan | Credits | Best For | |------|---------|----------| | Free | 1,000 one-time | Testing & personal projects | | Hobby ($19) | 5,000 / month | Small projects & development | | Professional ($99) | 50,000 / month | Professional use & production | | Business ($399) | 250,000 / month | Large scale operations |

All plans include:

• Access to all 31 tools
• Credits never expire; paid-plan credits roll over month to month
• API access and webhook notifications

View full pricing

🔧 Advanced Configuration

Environment Variables

# Optional: Set API key via environment
export CRAWLFORGE_API_KEY="cf_live_your_api_key_here"

# Optional: Custom API endpoint (for enterprise)
export CRAWLFORGE_API_URL="https://api.crawlforge.dev"
# As of v3.0.18, this variable is validated against an allow-list of CrawlForge backend hosts.

# Optional: Local LLM (Ollama) overrides — extract_with_llm, extract_structured
# and deep_research all use Ollama when no cloud key is set
export OLLAMA_BASE_URL="http://localhost:11434"   # default; set https://ollama.com for Ollama Cloud
export OLLAMA_DEFAULT_MODEL="gemma3:4b"            # optional; unset = pick the best installed model automatically
                                                   # deep_research judges claims with gemma3:12b when it is installed (ollama pull gemma3:12b);
                                                   # conflict detection is on only with that model, or a cloud provider
export OLLAMA_EMBEDDING_MODEL="nomic-embed-text"   # default: OLLAMA_DEFAULT_MODEL; used for semantic ranking in deep_research
export OLLAMA_API_KEY="..."                        # only for authenticated endpoints (required by Ollama Cloud; a local instance needs none)
export DISABLE_OLLAMA="true"                       # skip Ollama entirely and use CSS/keyword fallbacks

# Optional: Cloud LLM keys — only needed when you pass provider: "openai" or "anthropic"
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Optional: limit which tools this client sees — by name, by group, or both (comma-separated)
export CRAWLFORGE_TOOLS="scrape,search_web,extract_content"
export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape"   # unset = all tools; unknown names/groups are ignored with a warning

# Optional: deep_research stealth extraction fallback (v4.6.6) — see below
export RESEARCH_STEALTH_ENGINE="auto"      # auto (default) | camoufox | chromium
export RESEARCH_STEALTH_FALLBACK="true"    # set to "false" to disable entirely
export RESEARCH_MAX_STEALTH_RETRIES="8"    # cap on stealth retries per research run

MCP Spec Features

CrawlForge tracks the current MCP spec (2025-06-18) plus select experimental extensions:

• Structured output — scrape, map_site, serp_rank, reddit_search, search_web, extract_structured, and crawl_deep return machine-parseable structuredContent alongside the usual text, validated against a published outputSchema; legacy clients keep working off the text.
• Self-correctable errors — invalid tool input now comes back as an isError: true result the calling model can read and retry from, instead of a raw JSON-RPC protocol error.
• JSON Schema 2020-12 tool schemas, deterministic tools/list ordering (client prompt-cache friendly), and cacheable-result hints on read-only tools.
• Icons on the server, its tools, and its prompts.
• Async tasks (experimental) on the four long-running tools — crawl_deep, batch_scrape, deep_research, agent — for clients that support polling; synchronous results are still returned for clients that don't.

See docs/mcp-spec-adoption.md for wire-level examples and client-compatibility notes.

Local-LLM quickstart (extract_with_llm with Ollama)

extract_with_llm defaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).

# 1. Install Ollama:  https://ollama.com
# 2. Pull any model from https://ollama.com/library
ollama pull llama3.2

# 3. Discover what's installed (from your MCP client)
#    list_ollama_models()

# 4. Extract — defaults to Ollama with the model from step 2
#    extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })

Stealth extraction for deep_research (Camoufox)

deep_research automatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through a real fingerprinted browser, then re-extracts from the rendered HTML. It's bounded (RESEARCH_MAX_STEALTH_RETRIES, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.

Engine selection (RESEARCH_STEALTH_ENGINE):

• auto (default) — prefer Camoufox (Firefox anti-detect), fall back to Chromium stealth, then plain fetch.
• camoufox — force Camoufox.
• chromium — force the Chromium stealth engine.

Headless Chromium cannot clear modern challenges (Cloudflare Turnstile, DataDome) — Camoufox can. In testing it recovered Quora and Trustpilot pages that were otherwise fully blocked. To enable it, install the optional dependency and run its one-time binary fetch:

# Camoufox is declared as an optional dependency, so a normal install already pulls it.
# If you installed with --no-optional, add it explicitly:
npm install camoufox

# One-time download of the Camoufox Firefox binary (~130 MB):
npx camoufox fetch

Without the Camoufox binary, deep_research silently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback with RESEARCH_STEALTH_FALLBACK=false.

Note: Hard IP-reputation blocks (e.g. Reddit's edge 403) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide. See docs/stealth-engines.md for details.

Manual Configuration

Your configuration is stored at ~/.crawlforge/config.json:

{
  "apiKey": "cf_live_...",
  "userId": "user_...",
  "email": "[email protected]"
}

📖 Usage Examples

Once configured, use these tools in your AI assistant:

"Search for the latest AI news"
"Extract all links from example.com"
"Crawl the documentation site and summarize it"
"Monitor this page for changes"
"Extract product prices from this e-commerce site"

🔒 Security & Privacy

• Secure Authentication: API keys required for all metered tools
• Local Storage: API keys stored securely at ~/.crawlforge/config.json
• HTTPS Only: All connections use encrypted HTTPS
• No Data Retention: We don't store scraped data, only usage logs
• Rate Limiting: Built-in protection against abuse
• Compliance: Respects robots.txt and GDPR requirements

Security & Approvals

• SSRF enforcement: Every scraped URL is validated before the request is sent — http/https only; blocks loopback, RFC1918, IPv6 private/link-local ranges, cloud metadata endpoints (GCP, Azure), and dangerous ports (SSH, SMTP, DNS, MySQL, Postgres, Redis, MongoDB, etc.). Redirects are re-validated each hop, capped at 5.
• Backend endpoint guard (v3.0.18): The server's own calls to CrawlForge.dev use a separate fail-closed allow-list ({crawlforge.dev, www.crawlforge.dev, api.crawlforge.dev}, HTTPS required). Setting CRAWLFORGE_API_URL to an arbitrary host is blocked at parse time.
• Action allowlist: scrape_with_actions accepts only 11 action types (snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript, select, hover, navigate). No download, file-write, or arbitrary cross-page navigation primitives exist — navigate goes through the same SSRF and robots.txt gate as the initial URL.
• JavaScript gate: The executeJavaScript action throws by default. Set ALLOW_JAVASCRIPT_EXECUTION=true at deploy time to enable (not recommended in production).
• MCP Elicitation (v3.6.0): Four tools request user confirmation before executing expensive operations — deep_research (>50 URLs), batch_scrape (sync mode, >25 URLs), crawl_deep (projected >500 pages), extract_structured (schema has >3 required fields with no LLM configured). Credit-low situations also elicit. Confirmation is best-effort: if the MCP client does not support elicitation the tool proceeds (fail-open).
• Per-tool credit gating: Every tool is wrapped with withAuth() and is metered — credits are checked and deducted before execution, and a valid API key is required for every tool (fail-closed since v3.0.18).

See docs/sandboxing-and-approvals.md for the full reference.

Security Updates

v3.0.3 (2025-10-01): Removed authentication bypass vulnerability. All users must authenticate with valid API keys.

For the full security policy and how to report a vulnerability, see SECURITY.md.

🆘 Support

• Documentation: https://www.crawlforge.dev/docs
• Issues: GitHub Issues
• Email: [email protected]
• Discord: Join our community

📄 License

MIT License - see LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide first.

Built with ❤️ by the CrawlForge team

Website | Documentation | API Reference