webmesh-mcp

A token-economical web browsing, scraping, and crawling suite delivered as a standard MCP server. Any MCP-capable agent (Claude, Cursor, Cline, etc.) can use it as ordinary tools with no bespoke integration.

Why this is cheaper than a full browser loop

| Strategy | Saving | |---|---| | Tiered fetching — static HTTP first, browser only when HTML is thin | Skips Chromium entirely for the majority of pages | | Markdown output, not raw HTML — noise/nav/ads stripped server-side | 300 KB page → ~8 KB of clean markdown | | Schema extraction — field → CSS selector returns only the values you asked for | No markdown conversion, no LLM reasoning | | Boolean verification — web_check / web_diff never send back full page content | pass: true + 200-char evidence snippet | | ARIA snapshot over screenshot — web_interact returns the accessibility tree | Image tokens only when you explicitly ask | | Persistent browser pool — one Chromium process per server lifetime | ~1-2 s launch cost paid once, not per call | | robots.txt + rate limiting — per-host queues and rule caching | Polite crawling without throttling your agent |

Tools

| Tool | What it does | Browser needed? | |---|---|---| | web_scrape | Fetch a URL → clean markdown or schema-based JSON | Only if JS-rendered | | web_check | Assert text/element present or absent → pass: bool + evidence | Only if JS-rendered | | web_diff | Has this page changed since last check? → changed: bool + snippet | Only if JS-rendered | | web_interact | Click / fill / select / press / waitFor sequence → ARIA snapshot | Always | | web_session_close | Delete persisted cookies for a sessionId | No | | web_crawl | BFS crawl from a seed URL → titles, links, excerpts, cached markdown | Only if JS-rendered | | web_crawl_get_page | Retrieve full markdown cached by a prior web_crawl call | No |

Installation

npm install -g webmesh-mcp
# or use directly with npx (no global install needed):
npx webmesh-mcp

Chromium (optional — only needed for JS-rendered pages)

webmesh-mcp uses playwright-core and does not bundle a browser. You have three options:

A) Install Playwright's managed Chromium (simplest):

npx playwright install chromium

B) Use your system Chrome / Edge — set executablePath in your MCP config (see below).

C) Connect to a running browser — Playwright supports CDP attach; pass --cdp-endpoint flags in args if you want to hook into an already-running instance.

Static pages (most blogs, docs, GitHub, npm, etc.) never trigger the browser path at all.

Connecting to your agent

Add to your MCP config (e.g. ~/.claude/claude_desktop_config.json, .cursor/mcp.json, etc.):

{
  "mcpServers": {
    "web-agent": {
      "command": "npx",
      "args": ["webmesh-mcp"]
    }
  }
}

Or, if you prefer to run from source:

{
  "mcpServers": {
    "web-agent": {
      "command": "npx",
      "args": ["tsx", "/absolute/path/to/webmesh-mcp/src/index.ts"]
    }
  }
}

Agent Skill Integration

This repository ships with a pre-configured Agent Skill in skills/web-agent/SKILL.md.

It provides AI coding assistants (Claude Code, Antigravity, Cursor, Windsurf, Gemini CLI, etc.) with a complete decision tree, trigger keywords, and parameter guidance for using webmesh-mcp tools token-efficiently.

How to use the skill in your project

Copy or symlink the skills/ folder into your AI assistant's skills directory:

• Claude / Antigravity / Gemini CLI: Place in .agents/skills/web-agent/SKILL.md or ~/.gemini/config/skills/web-agent/SKILL.md
• Cursor: Copy contents to .cursor/rules/web-agent.mdc
• Windsurf: Copy contents to .windsurfrules

Tool reference

web_scrape

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Page to fetch | | format | "markdown" \| "json" | "markdown" | Output format | | schema | Record<string, string> | — | {field: cssSelector} — required when format="json" | | selector | string | — | Scope extraction to a CSS subtree | | forceBrowser | boolean | false | Skip static tier, always use Chromium | | ignoreRobots | boolean | false | Bypass robots.txt rules |

web_check

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Page to check | | assertion | enum | — | contains, not_contains, selector_exists, selector_not_exists, text_equals | | value | string | — | Text or CSS selector | | selector | string | — | Scope text search to subtree | | forceBrowser | boolean | false | | | ignoreRobots | boolean | false | |

web_diff

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Page to monitor | | selector | string | — | Scope watch to subtree (e.g. .price) | | forceBrowser | boolean | false | | | ignoreRobots | boolean | false | |

Returns changed: null on the first call (nothing to compare against), true/false on subsequent calls.

web_interact

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Starting URL | | actions | Action[] | — | Sequence of browser actions | | resultSelector | string | — | Scope returned ARIA snapshot | | screenshot | boolean | false | Return base64 PNG (costs image tokens) | | sessionId | string | — | Persist cookies/storage across calls | | ignoreRobots | boolean | false | |

Action shape:

{ type: "click" | "fill" | "select" | "press" | "waitFor", selector?: string, value?: string, timeoutMs?: number }

web_session_close

| Parameter | Type | Description | |---|---|---| | sessionId | string | Session to delete |

web_crawl

| Parameter | Type | Default | Description | |---|---|---|---| | startUrl | string (URL) | — | Seed URL | | maxDepth | number | 2 | Max link depth (0 = seed only) | | maxPages | number | 30 | Hard cap on pages visited | | sameHostOnly | boolean | true | Restrict to same hostname | | includePatterns | string[] | — | Pathname globs URLs must match, e.g. "/docs/*" | | excludePatterns | string[] | — | Pathname globs URLs must NOT match | | contentDepth | "none" \| "summary" \| "full" | "summary" | Output verbosity | | ignoreRobots | boolean | false | |

Full markdown for every visited page is cached in SQLite for web_crawl_get_page.

web_crawl_get_page

| Parameter | Type | Description | |---|---|---| | url | string (URL) | Previously crawled URL |

Environment variables

| Variable | Default | Description | |---|---|---| | MCP_WEB_AGENT_DATA_DIR | <cwd>/.webmesh-mcp | Root directory for cache DB and session files |

Both the SQLite watch/crawl cache (cache/watch.sqlite) and session state files (sessions/) live under MCP_WEB_AGENT_DATA_DIR. Override it to control where runtime data is stored.

Security

SSRF Protection

All outbound fetches — static HTTP, browser navigation, intermediate HTTP redirects, and crawl link-following — are strictly validated against an SSRF blocklist before network connections are established.

• Protocol Restriction: Rejects all non-HTTP/HTTPS schemes.
• Loopback Addresses: 127.x.x.x, IPv6 ::1, localhost, IPv6 unspecified :: / 0:0:0:0:0:0:0:0.
• Private RFC-1918 & Unique Local IPv6: 10.x, 172.16–31.x, 192.168.x, fc00::/7, fd00::/7.
• Link-Local & Cloud Metadata: 169.254.x.x (AWS/GCP/Azure IMDS), fe80::/10.
• Special & Reserved Ranges: Multicast (224.x), Site-Local (fec0::/10), Documentation (2001:db8::/32), and IPv4-mapped IPv6 formats (::ffff:x.x.x.x).
• HTTP Redirect Hardening: Static fetch enforces manual redirect validation loops up to 5 hops, checking ssrfGuard.assertPublicUrl() on every intermediate Location header before following.
• Browser Route Interception: Chromium contexts attach route interceptors (page.route("**/*")) to block subresource requests or redirects targeting private IP space.

Architecture

For a detailed technical architecture and end-to-end data flow specification, see Architecture.md.

index.ts (MCP server, stdio transport)
├── tools/scrape.ts        — web_scrape
├── tools/check.ts         — web_check
├── tools/diff.ts          — web_diff
├── tools/interact.ts      — web_interact
├── tools/crawl.ts         — web_crawl
├── tools/crawlGetPage.ts  — web_crawl_get_page
├── tieredFetch.ts         — static HTTP → browser escalation (manual redirect validation)
├── browserPool.ts         — singleton Chromium process + route interceptor (playwright-core)
├── ssrfGuard.ts           — SSRF protection (blocks private/reserved IPv4 & IPv6 addresses)
├── extract.ts             — HTML → clean markdown / plain text / schema JSON
├── hostGate.ts            — robots.txt parser (regex escaped) + per-host rate-limiting queue
├── sessions.ts            — disk-backed storageState persistence
├── cache.ts               — SQLite: watch hashes + crawled page markdown
└── constants.ts           — shared USER_AGENT, DATA_DIR

Development

git clone https://github.com/creatorpiyush/webmesh-mcp
cd webmesh-mcp
npm install
npx playwright install chromium    # optional, for browser-tier testing

npm run dev        # run MCP server in dev mode (tsx, no compile step)
npm run demo -- https://example.com
npm test           # run the full integration test suite
npm run typecheck  # tsc --noEmit, no output files
npm run format     # format code with prettier
npm run build      # compile to dist/

License

MIT — see LICENSE.