🕷️ MCP Web Scrape

Clean, cached web content for agents—Markdown + citations, robots-aware, ETag/304 caching.

📦 Version

Current Version: 1.0.7

🎬 Live Demos

See MCP Web Scrape in action! These demos show real-time extraction and processing:

📄 Content Extraction

Transform messy HTML into clean, agent-ready Markdown with automatic citations

🔗 Link Extraction

Extract and categorize all links from any webpage with filtering options

📊 Metadata Extraction

Get comprehensive page metadata including title, description, author, and keywords

📝 Content Summarization

AI-powered content summarization for quick insights and key points

🚀 Quick Start Demo

# Extract content from any webpage
npx [email protected]

# Example: Extract from a news article
> extract_content https://news.ycombinator.com
✅ Extracted 1,247 words with 5 citations
📄 Clean Markdown ready for your AI agent

🎯 Tool Examples

# Extract all forms from a webpage
> extract_forms https://example.com/contact
✅ Found 3 forms with 12 input fields

# Parse tables into structured data
> extract_tables https://example.com/data --format json
✅ Extracted 5 tables with 247 rows

# Find social media profiles
> extract_social_media https://company.com
✅ Found Twitter, LinkedIn, Facebook profiles

# Analyze sentiment of content
> sentiment_analysis https://blog.example.com/article
✅ Sentiment: Positive (0.85), Emotional tone: Optimistic

# Extract named entities
> extract_entities https://news.example.com/article
✅ Found 12 people, 8 organizations, 5 locations

# Check for security vulnerabilities
> scan_vulnerabilities https://mysite.com
✅ No XSS vulnerabilities found, 2 header improvements suggested

# Analyze competitor SEO
> analyze_competitors ["https://competitor1.com", "https://competitor2.com"]
✅ Competitor analysis complete: keyword gaps identified

# Monitor uptime and performance
> monitor_uptime https://mysite.com --interval 300
✅ Uptime: 99.9%, Average response: 245ms

# Generate comprehensive report
> generate_reports https://website.com --metrics ["seo", "performance", "security"]
✅ Generated 15-page analysis report

⚡ Quick Start

# Install globally
npm install -g [email protected]

# Try it instantly (latest version)
npx mcp-web-scrape@latest

# Try specific version
npx [email protected]

# Or start HTTP server
node dist/http.js

ChatGPT Desktop Setup

Add to your ~/Library/Application Support/ChatGPT/config.json:

{
  "mcpServers": {
    "web-scrape": {
      "command": "npx",
      "args": ["[email protected]"]
    }
  }
}

Claude Desktop Setup

Add to your ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "web-scrape": {
      "command": "npx",
      "args": ["[email protected]"]
    }
  }
}

🛠️ Available Tools

Core Extraction Tools

| Tool | Description | |------|-------------| | extract_content | Convert HTML to clean Markdown with citations | | summarize_content | AI-powered content summarization | | get_page_metadata | Extract title, description, author, keywords | | extract_links | Get all links with filtering options | | extract_images | Extract images with alt text and dimensions | | search_content | Search within page content | | check_url_status | Verify URL accessibility | | validate_robots | Check robots.txt compliance | | extract_structured_data | Parse JSON-LD, microdata, RDFa | | compare_content | Compare two pages for changes | | batch_extract | Process multiple URLs efficiently | | get_cache_stats | View cache performance metrics | | clear_cache | Manage cached content |

Advanced Extraction Tools

| Tool | Description | |------|-------------| | extract_forms | Extract form elements, fields, and validation rules | | extract_tables | Parse HTML tables with headers and structured data | | extract_social_media | Find social media links and profiles | | extract_contact_info | Discover emails, phone numbers, and addresses | | extract_headings | Analyze heading structure (H1-H6) for content hierarchy | | extract_feeds | Discover and parse RSS/Atom feeds |

Content Transformation Tools

| Tool | Description | |------|-------------| | convert_to_pdf | Convert web pages to PDF format with customizable settings | | extract_text_only | Extract plain text content without formatting or HTML | | generate_word_cloud | Generate word frequency analysis and word cloud data | | translate_content | Translate web page content to different languages | | extract_keywords | Extract important keywords and phrases from content |

Advanced Analysis Tools

| Tool | Description | |------|-------------| | analyze_readability | Analyze text readability using various metrics (Flesch, Gunning-Fog, etc.) | | detect_language | Detect the primary language of web page content | | extract_entities | Extract named entities (people, places, organizations) | | sentiment_analysis | Analyze sentiment and emotional tone of content | | classify_content | Classify content into categories and topics |

SEO & Marketing Tools

| Tool | Description | |------|-------------| | analyze_competitors | Analyze competitor websites for SEO and content insights | | extract_schema_markup | Extract and validate schema.org structured data | | check_broken_links | Check for broken links and redirects on pages | | analyze_page_speed | Analyze page loading speed and performance metrics | | generate_meta_tags | Generate optimized meta tags for SEO |

Security & Privacy Tools

| Tool | Description | |------|-------------| | scan_vulnerabilities | Scan pages for common security vulnerabilities | | check_ssl_certificate | Check SSL certificate validity and security details | | analyze_cookies | Analyze cookies and tracking mechanisms | | detect_tracking | Detect tracking scripts and privacy concerns | | check_privacy_policy | Analyze privacy policy compliance and coverage |

Advanced Monitoring Tools

| Tool | Description | |------|-------------| | monitor_uptime | Monitor website uptime and availability | | track_changes_detailed | Advanced change tracking with similarity analysis | | analyze_traffic_patterns | Analyze website traffic patterns and trends | | benchmark_performance | Benchmark performance against competitors | | generate_reports | Generate comprehensive analysis reports |

Analysis & Monitoring Tools

| Tool | Description | |------|-------------| | monitor_changes | Track content changes over time with similarity analysis | | analyze_performance | Measure page performance, SEO, and accessibility metrics | | generate_sitemap | Crawl websites to generate comprehensive sitemaps | | validate_html | Validate HTML structure, accessibility, and SEO compliance |

🤔 Why Not Just Use Built-in Browsing?

Deterministic Results → Same URL always returns identical content
Smart Citations → Every fact links back to its source
Robots Compliant → Respects robots.txt and rate limits
Lightning Fast → ETag/304 caching + persistent storage
Agent-Optimized → Clean Markdown instead of messy HTML

🔒 Safety First

• ✅ Respects robots.txt by default
• ✅ Rate limiting prevents server overload
• ✅ No paywall bypass - ethical scraping only
• ✅ User-Agent identification for transparency

📦 Installation

# Install specific version
npm install -g [email protected]

# Or use directly (latest)
npx mcp-web-scrape@latest

# Or use specific version
npx [email protected]

🔧 Configuration

# Environment variables
export MCP_WEB_SCRAPE_CACHE_DIR="./cache"
export MCP_WEB_SCRAPE_USER_AGENT="MyBot/1.0"
export MCP_WEB_SCRAPE_RATE_LIMIT="1000"

🌐 Transports

STDIO (default)

mcp-web-scrape

HTTP/SSE

node dist/http.js --port 3000

📚 Resources

Access cached content as MCP resources:

cache://news.ycombinator.com/path → Cached page content
cache://stats → Cache statistics
cache://robots/news.ycombinator.com → Robots.txt status

🤝 Contributing

We love contributions! See CONTRIBUTING.md for guidelines.

Good First Issues:

• Add new content extractors
• Improve error handling
• Write more tests
• Enhance documentation

📄 License

MIT © Mahipal

🌟 Star History

Built with ❤️ for the Model Context Protocol ecosystem