Smart Web Researcher (MCP Server)

An MCP (Model Context Protocol) server that gives your LLM a real web browser via Playwright, exposing tools for live web research with bot evasion, caching, dynamic waits, and screen dimension controls.

Exposed Tools

| Tool | Purpose | Parameters | |---|---|---| | search_web | Query a search engine and return top result links. | query (string), maxResults (number, default: 5) | | fetch_page_markdown | Fetch a webpage and convert its HTML content into clean Markdown. | url (string), waitSelector (string, optional), waitDelay (number, optional), autoScroll (boolean, optional) | | extract_page_links | Gather all unique hyperlinks found on a specific page. | url (string), sameDomainOnly (boolean, optional) | | take_page_screenshot | Capture and save a screenshot of a webpage (returns the image for direct inline rendering in Claude Desktop and saves a local copy). | url (string), fullPage (boolean, default: true), waitSelector (string, optional), waitDelay (number, optional), autoScroll (boolean, optional), viewportWidth (number, optional), viewportHeight (number, optional) | | find_text_on_page | Scan a webpage for a specific keyword and return matching context snippets. | url (string), keyword (string) | | close_browser | Explicitly close the shared browser session to free up memory and system resources. | None |

Built with the @modelcontextprotocol/sdk (TypeScript) + playwright + turndown + zod.

1. Prerequisites

• Node.js: 18+ (tested on Node v24+)
• npm (Node Package Manager)

2. Setup & Installation

Clone the repository and install the dependencies:

# Install NPM dependencies
npm install

# Install Playwright's Chromium browser binaries
npx playwright install chromium

3. How to Build & Run

We use TypeScript to build the project. The build outputs compiled JavaScript files into the dist/ directory.

Build the Project

npm run build

Run in Development Mode

Runs the TypeScript file directly using tsx (great for rapid iteration):

npm run dev

Run in Production Mode

Executes the compiled JavaScript from the dist/ folder:

npm start

4. Testing Standalone (MCP Inspector)

You can test all tools interactively without hooking it up to a main LLM client using the official Model Context Protocol Inspector:

npx @modelcontextprotocol/inspector node dist/index.js

This starts a local web UI (usually at http://localhost:6274 or http://localhost:5173) where you can call each tool with custom arguments and view the raw JSON-RPC requests, responses, and errors.

5. Registering with Claude Desktop / Claude Code

To allow Claude Desktop to use these tools, add the server to your Claude configuration file.

macOS Configuration Path

~/Library/Application Support/Claude/claude_desktop_config.json

Windows Configuration Path

%APPDATA%\Claude\claude_desktop_config.json

Configuration Content

Add the following to your mcpServers object (replace /absolute/path/to with the actual path to the project directory on your machine).

On macOS, GUI applications like Claude Desktop often do not inherit the command line PATH. Therefore, specifying the absolute path to your node binary (e.g., /usr/local/bin/node or /opt/homebrew/bin/node) and providing a "PATH" in "env" is highly recommended.

{
  "mcpServers": {
    "smart-web-researcher": {
      "command": "/usr/local/bin/node", // Use absolute path to node
      "args": ["/absolute/path/to/mcp-web-researcher/dist/index.js"],
      "env": {
        "PATH": "/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin",
        "BRAVE_API_KEY": "your_brave_api_key_here", // Optional
        "GOOGLE_API_KEY": "your_google_api_key_here", // Optional
        "GOOGLE_CX": "your_google_custom_search_engine_id_here" // Optional
      }
    }
  }
}

Note: Restart Claude Desktop after making configuration changes. The tools should appear under the 🔌 connector icon.

6. Advanced Feature Set

🛡️ Bot Stealth Configurations

To prevent simple bot-detection checkers (e.g. Cloudflare, Akamai) from blocking research sessions, the browser context:

• Uses a realistic browser User-Agent (Mozilla/5.0 ... Chrome/122.0.0.0 Safari/537.36).
• Resizes default viewports to a standard laptop display (1280x800).
• Configures consistent locales (en-US) and timezones (America/New_York).
• Injects standard browser headers (Accept-Language, Sec-Ch-Ua, Sec-Ch-Ua-Platform, etc.).

⚡ Search Engine Fallback APIs

Avoids scraper blocks by using official APIs if configured in the environment:

1. Brave Search API (BRAVE_API_KEY)
2. Google Custom Search API (GOOGLE_API_KEY + GOOGLE_CX)
3. DuckDuckGo Scraping: Used as a zero-setup fallback.

⏱️ Dynamic SPA Selector & Delay Waits

• Pass waitSelector (e.g., #main-content) to wait for a specific DOM selector to render before extracting markdown/screenshots.
• Pass waitDelay (in milliseconds) to wait a fixed duration (useful for client-side JavaScript calculations).

📜 Infinite-Scroll Auto-Scrolling

• Pass autoScroll: true to scroll the page incrementally in 100px increments prior to content retrieval. This triggers infinite-scroll loaders and downloads lazy-loaded image media.

📱 Custom Screen Viewports

• Pass viewportWidth and viewportHeight parameters to take_page_screenshot to resize the active viewport window (useful for capturing mobile layouts).

🔗 Domain Link Filters

• Pass sameDomainOnly: true to extract_page_links to strip out external links and social media sharing noise, returning only local site navigation links.

💾 5-Minute In-Memory Session Cache

• Web markdown and extracted links are stored in an in-memory TTL cache for 5 minutes, significantly speeding up subsequent lookups of the same page during active LLM conversations.

7. Implementation Architecture

• src/browser.ts: Shared browser lifecycle runner containing bot stealth overrides and autoScrollPage() helpers.
• src/cache.ts: 5-minute TTL memory cache manager.
• src/index.ts: Stdio transport initiator, schemas definitions, and server router.
• src/tools/: decoupled tool handlers (e.g., searchWeb.ts, fetchPageMarkdown.ts).