FreeWeb MCP Server

Give your LLM unlimited web access - no API keys, no rate limits, no restrictions.

A multi-layer MCP (Model Context Protocol) server for web browsing, search, and content extraction. Uses a fetcher chain with 7 fallback layers — from ultra-fast native fetch() (~43ms) to full Playwright browser automation (~3-5s). No API keys required.

Why FreeWeb?

• 🚀 No API Keys Required - Uses real browser automation and native HTTP, not paid APIs
• ⚡ Multi-Layer Fetcher Chain - Fast static pages in ~400ms, only uses heavy browser when needed
• ♾️ No Rate Limits - Browse as much as you need, no quotas or throttling
• 🌐 True Web Access - Your LLM can visit any website, extract content, follow links
• 🔓 100% Free & Open Source - No subscriptions, no hidden costs, no vendor lock-in

Fetcher Chain Architecture

Every URL request goes through 7 fetcher layers, tried in order. First success wins:

| Priority | Fetcher | Speed | Best For | |----------|---------|-------|----------| | 5 | llms.txt + Markdown | ~300ms | Sites with .md variants | | 10 | GitHub Raw | ~43ms | GitHub READMEs and files | | 30 | RSS/Atom Feed | ~450ms | Blogs, news sites | | 40 | fetch() + jsdom | ~400ms | Static HTML pages (~80% of web) | | 80 | Archive.org Cache | ~1.2s | Dead/blocked pages | | 100 | Playwright | ~3-5s | SPA apps, bot-protected sites |

This means most pages load 10x faster than Playwright-only, while still falling back to full browser automation when needed.

Features

• 🔎 General Web Search: Search the public web without API keys (Yahoo, DuckDuckGo, Marginalia, Ask)
• 📚 Search + Browse: Open the best search hits and extract readable page content
• 🔍 LLMS.txt Inspection: Inspect and debug llms.txt files directly
• 🤖 LLMS.txt Aware Browsing: If a site exposes llms.txt, FreeWeb reads it first and includes the guidance in browse output
• 🧭 Best Next Page Routing: With a query, FreeWeb can follow the most relevant same-site llms.txt link before browsing
• 📝 Markdown Fallback: For llms-aware sites, tries .md page variants before falling back to HTML extraction
• 🐙 GitHub Raw Access: READMEs and files fetched directly from raw.githubusercontent.com (~43ms)
• 📡 RSS/Atom Feed Support: Auto-discover and parse feeds from blogs and news sites
• 🧠 Result Quality Ranking: Domain-aware scoring, snippet cleanup, freshness hints, deduping, and optional llms.txt badges in search results
• 🔒 Security: URL validation, download protection, blocked domain filter
• 📅 Freshness Control: Page date detection, stale content warnings
• ⚡ SPA Support: Auto-detection and handling of React/Vue/Next.js apps
• 🔄 Parallel Processing: Browse multiple URLs concurrently
• 🛡️ Anti-Bot Measures: Stealth browser techniques for reliable access

Installation

Using with npx (Recommended)

Add to your MCP client config (Claude Desktop, Cursor, Kilo, Windsurf, etc.):

{
  "mcpServers": {
    "freeweb": {
      "command": "npx",
      "args": ["-y", "freeweb-mcp@latest"]
    }
  }
}

Using with npm

npm install -g freeweb-mcp@latest

Then add to your MCP config:

{
  "mcpServers": {
    "freeweb": {
      "command": "freeweb-mcp"
    }
  }
}

From Source

git clone https://github.com/xenitV1/freeweb.git
cd freeweb
npm install
npm run build

Then add to your MCP config:

{
  "mcpServers": {
    "freeweb": {
      "command": "node",
      "args": ["/path/to/freeweb/dist/index.js"]
    }
  }
}

Available Tools

| Tool | Description | |------|-------------| | inspect_llms_txt | Inspect and parse a site's llms.txt guidance | | web_search | Search the public web without API keys | | search_and_browse | Search the web, open the best hits, route via llms.txt when relevant, and extract content | | browse_page | Visit URL, read llms.txt if present, optionally route to a better page, and extract content | | smart_browse | SPA-aware browsing with date validation, llms.txt guidance, and optional llms routing | | deep_search | Search across GitHub, npm, MDN | | github_search | Search GitHub repos, code, or issues | | github_repo_files | List files in a GitHub repo | | parallel_browse | Visit multiple URLs in parallel (max 5) | | get_page_links | Extract all links from a page | | screenshot | Capture page screenshot (base64 PNG) |

Configuration Options

MCP Client Config

{
  "mcpServers": {
    "freeweb": {
      "command": "npx",
      "args": ["-y", "freeweb-mcp@latest"],
      "env": {
        "PLAYWRIGHT_BROWSERS_PATH": "0"
      }
    }
  }
}

Security Rules

• ✅ Only HTTPS/HTTP URLs are visited
• ✅ Suspicious domains are blocked
• ✅ Downloads require explicit user approval
• ✅ No forms filled, no logins, no payments
• ✅ IP addresses are blocked

Content Freshness

Every page visit includes:

• Automatic page date detection
• Stale content warnings (>12 months old)
• requireFreshContent option to enforce fresh content only

Requirements

• Node.js >= 18
• Playwright (auto-installed on first run)

Development

npm install
npm run dev      # Watch mode
npm run build    # Build
npm start        # Run
npm test         # Run tests

License

MIT