Why ShantyCrawl MCP

Most full-featured MCP servers register every tool on every session — including the ones you'll rarely touch — with full JSON schemas re-sent on each tool-list fetch. ShantyCrawl loads 6 core tools up front and lazy-loads the rest on demand, which measured out to ~18,000 fewer tokens on a fresh session in our testing:

| | firecrawl-mcp (official) | shantycrawl-mcp | | --- | --- | --- | | Tools on session start | 28 | 6 | | Initial schema footprint | Full schemas for all 28 tools | Minimal schemas for 6 tools | | Advanced tools (research, monitoring, agent, etc.) | Always loaded | Loaded on demand via tool_enable | | Measured token cost, fresh session | Baseline | ~18k tokens lighter | | Runtime dependencies | — | Zero third-party wrappers, native fetch |

Is This For You?

• ✅ Good fit — you use Firecrawl mostly for scrape/crawl/search, run long or multi-tool agent sessions, or are tight on context budget.
• ⚪ Less critical — you're already using tool_enable-style dynamic loading elsewhere, or you live almost entirely in the monitoring/research toolset and rarely call the core 6 — in that case the savings are smaller, since you'll be loading most of the schema anyway.

🚀 Quick Start

Run instantly via npx — no installation required:

FIRECRAWL_API_URL=http://localhost:3002 npx shantycrawl-mcp

Environment variables:

| Variable | Required | Default | Description | | --- | --- | --- | --- | | FIRECRAWL_API_URL | No | http://localhost:3002 | URL of your Firecrawl instance (self-hosted or cloud) | | FIRECRAWL_API_KEY | Only for Firecrawl Cloud | — | API key, required if FIRECRAWL_API_URL points to a hosted/cloud instance |

Requirements: Node.js 18+ and a reachable Firecrawl instance (self-hosted or cloud).

⚙️ Configuration

Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "shantycrawl": {
      "command": "npx",
      "args": ["shantycrawl-mcp"],
      "env": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

OpenCode

Add this to your workspace settings:

{
  "mcp": {
    "shantycrawl": {
      "type": "local",
      "command": ["npx", "shantycrawl-mcp"],
      "environment": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

🛠️ Tool Architecture

Always Available (6 Core Tools)

| Tool | Description | | --- | --- | | scrape | Extract clean markdown from any URL | | crawl | Crawl a target website recursively | | search | Execute web searches via Firecrawl | | tool_enable | Dynamically inject an advanced tool into the active session | | tool_disable | Unload an advanced tool to free up context | | check_crawl_status | Check the status of an async crawl job |

Lazy-Loaded Tools (22 Advanced)

Activate any tool instantly during a session — e.g. tool_enable map unblocks the mapping capability.

| Category | Tools | | --- | --- | | Discovery | map, extract, parse | | AI Agent | agent, agent_status | | Browser Interaction | interact, interact_stop | | Research & Academic | research_search_papers, research_inspect_paper, research_read_paper, research_related_papers, research_search_github | | Monitoring | monitor_create, monitor_check, monitor_checks, monitor_delete, monitor_get, monitor_list, monitor_run, monitor_update | | Feedback | search_feedback, feedback |

🧠 How Dynamic Loading Works

1. The client sees a minimal 6-tool schema on connect.
2. When the AI needs an advanced capability, it calls tool_enable <tool_name>.
3. The server fires a notifications/tools/list_changed event, and the client's available tools update instantly — no reconnect required.
4. Call tool_disable <tool_name> once you're done to free the context back up.

⚠️ Known Issues

Some agents don't call tool_enable on their own. A compelling tool description isn't always enough — if another skill or system prompt has already taught the agent a different path to the same outcome (e.g. calling the Firecrawl API directly via curl/raw HTTP), it can follow that instead of discovering tool_enable. This shows up most with coding agents that have built-in or third-party Firecrawl skills already loaded.

Fix: add an explicit instruction to your agent's system prompt or AGENTS.md/CLAUDE.md:

This project uses the shantycrawl-mcp server. Advanced tools (map, extract, parse,
agent, interact, research_*, monitor_*) are not available until activated with
tool_enable("<tool_name>"). Always use the MCP server's tools directly — never call
the Firecrawl API via curl, raw HTTP, or other bash-based workarounds.

We also ship a ready-made SKILL.md that encodes this protocol — drop it into your skills directory if your agent supports skill loading, and pair it with the snippet above for the strongest guarantee.

🏗️ Local Development

git clone https://github.com/schlemperdev/shantycrawl-mcp.git
cd shantycrawl-mcp
npm install
npm run build
npm start

🤝 Contributing

Issues and PRs are welcome. If you're adding a new lazy-loaded tool, please keep its schema as lean as the core 6 — that's the whole point of this project.

📄 License

MIT © schlemperdev