@diogenesgateway/diogenes-web-scraping-engine-mcp

v1.4.1

Published

3 days ago

0High
0Medium
0Low

mcp mcp-server web-scraping html-to-markdown readability metadata-extractor seo rss crawler streamable-http glama tensorblock l402 x402

Diogenes Web Extraction, Readability & Markdown Engine MCP (`diogenes-web-scraping-engine-mcp`)

Diogenes Web Extraction, Readability & Markdown Engine MCP is a focused, zero-bloat Model Context Protocol (streamable-http) package powered by the Diogenes Microservice Gateway.

Instead of loading all 121 gateway endpoints into your LLM's context window, installing this package exposes only the 12 specialized web-scraping-engine tools needed for Web Extraction & Scraping:

12 high-performance web extraction, readability & markdown tools: clean LLM markdown compiler, readable article extractor, OpenGraph & JSON-LD metadata parser, HTML table to JSON/Markdown converter, CSS selector isolator, paywall & overlay bypass auditor, XML sitemap crawler, hyperlink extractor, HTML XSS sanitizer, contact & lead info parser, SEO heading diagnostics, and RSS/Atom feed compiler.

⚡ 1-Click Install for Cursor & Claude Desktop

No API keys, user accounts, or local dependencies required. Zero setup — paste directly into your client config:

🖱️ Cursor IDE (`.cursor/mcp.json`)

Add to .cursor/mcp.json in your project root or global Cursor settings:

{
  "mcpServers": {
    "diogenes-web-scraping-engine": {
      "transport": "streamable-http",
      "url": "https://diogenesgateway.com/mcp/web-scraping-engine"
    }
  }
}

🧠 Claude Desktop (`claude_desktop_config.json`)

Add to claude_desktop_config.json (Mac: ~/Library/Application Support/Claude/claude_desktop_config.json | Windows: %APPDATA%\Claude\claude_desktop_config.json):

{
  "mcpServers": {
    "diogenes-web-scraping-engine": {
      "transport": "streamable-http",
      "url": "https://diogenesgateway.com/mcp/web-scraping-engine"
    }
  }
}

Prefer local stdio? Run via npx:
npx -y @stand-out-of-my-sun/diogenes-web-scraping-engine-mcp

Included Tools in `web-scraping-engine` (12 Tools)

| MCP Tool Name | Underlying REST Route | Settlement | Capability | | :--- | :--- | :--- | :--- | | extract_clean_markdown | POST /api/v1/extract/clean-markdown | Free Developer Tier (0 sats) | Converts raw HTML pages into clean, boilerplate-free, LLM-optimized Markdown with preserved hyperlinks and headings. Use when ingesting raw web documents or blog posts into agent context windows; for isolating tabular data into structured datasets, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('conversion_mode' defaults to 'clean_semantic_article'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | scrape_readable_article | POST /api/v1/extract/readable-article | Free Developer Tier (0 sats) | Extracts primary article body, author bylines, publication timestamps, and estimated reading duration from web pages. Use when stripping navigation bars, advertisement containers, and cookie banners to isolate core editorial content; for converting full HTML to markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('readability_profile' defaults to 'article_body_strict'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | extract_structured_metadata | POST /api/v1/extract/structured-metadata | 10 sats (L402) | Extracts OpenGraph tags, Twitter Card metadata, canonical URLs, and Schema.org JSON-LD microdata scripts from HTML. Use when indexing social share cards, author identity, breadcrumbs, and rich snippet schemas for AI search agents; for scanning page hyperlinks and outbound references, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('metadata_targets' defaults to 'all_protocols'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | parse_html_tables | POST /api/v1/extract/html-tables | 10 sats (L402) | Detects and parses HTML table elements into structured JSON record arrays and GitHub-Flavored Markdown tabular grids. Use when converting financial statements, product comparison grids, or sports statistics into structured data tables; for extracting raw clean markdown prose, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('output_format' defaults to 'markdown_and_json'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | isolate_css_selectors | POST /api/v1/extract/css-selectors | 10 sats (L402) | Evaluates CSS selector queries against HTML DOM trees to extract targeted inner text, outer HTML, or element attributes. Use when targeting specific elements like pricing badges, product titles, review cards, or navigation breadcrumbs; for parsing unstructured contact information, use extract_contact_info instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' and 'selector_query' ('extraction_mode' defaults to 'inner_text'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | detect_paywall_and_bypass | POST /api/v1/extract/paywall-detect | 15 sats (L402) | Audits web page DOM hierarchies to detect soft paywalls, blur overlays, modal gates, and recovers concealed body text. Use when diagnosing paywall obstructions, diagnosing subscriber gating scripts, or auditing content access barriers; for assessing general SEO hygiene, use evaluate_seo_diagnostics instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('detection_profile' defaults to 'heuristic_dom_overlay'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid. | | crawl_sitemap_urls | POST /api/v1/extract/sitemap-crawler | 15 sats (L402) | Parses XML sitemaps and sitemap index files into structured URL trees with change frequency and last-modified filters. Use when discovering domain architectures, mapping crawl frontiers, or indexing blog/product catalogues; for extracting on-page hyperlinks from raw HTML, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'sitemap_xml' ('filter_criteria' defaults to 'all_entries'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid. | | extract_page_hyperlinks | POST /api/v1/extract/page-hyperlinks | 10 sats (L402) | Extracts, normalizes, and filters internal and external hyperlinks, anchor texts, and downloadable asset targets from HTML. Use when building site crawlers, auditing broken links, or extracting reference citations; for isolating primary readable article narrative text, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('link_scope' defaults to 'all_links'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | sanitize_html_content | POST /api/v1/extract/sanitize-html | 10 sats (L402) | Sanitizes raw HTML by removing malicious script tags, iframes, inline event handlers, and data URIs into safe markup. Use when preparing untrusted third-party web content for rendering in agent dashboards or email clients; for converting clean HTML into markdown, use extract_clean_markdown instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('sanitization_strictness' defaults to 'strip_scripts_and_iframes'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | extract_contact_info | POST /api/v1/extract/contact-info | 10 sats (L402) | Extracts email addresses, telephone numbers, physical addresses, and social network handles from web text and DOM footers. Use when researching sales leads, aggregating business contact directories, or extracting customer service info; for parsing structured tabular data, use parse_html_tables instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'content' ('extraction_focus' defaults to 'all_contacts'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | evaluate_seo_diagnostics | POST /api/v1/extract/seo-diagnostics | 10 sats (L402) | Audits HTML heading structures, title lengths, meta descriptions, image alt tags, and OpenGraph compliance into a health score. Use when auditing landing pages, verifying SEO compliance before publishing, or evaluating crawl friendliness; for extracting article bodies, use scrape_readable_article instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'html_content' ('audit_level' defaults to 'comprehensive_audit'), supports '?demo=true' or L402 (10 sats), and returns HTTP 400 if invalid. | | compile_rss_feed | POST /api/v1/extract/rss-feed-compiler | 15 sats (L402) | Transforms JSON item collections or article card arrays into standardized RSS 2.0, Atom 1.0, or JSON Feed 1.1 syndication feeds. Use when syndicating blog articles, product announcements, or changelog cards to RSS readers and AI agents; for compiling structured sitemap listings, use crawl_sitemap_urls instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'items_json' and 'channel_title' ('feed_standard' defaults to 'rss_2_0'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid. |

Free Account-Less Demo Mode: Append ?demo=true to any REST endpoint above for an instant HTTP 200 schema preview at zero cost.

🚀 Upgrade to Frontier AI Models & Autonomous Agent Pipelines

Looking for multi-model reasoning or live web retrieval? Pair web-scraping-engine with our paid Gemini Suite & Frontier Reasoning packages (45 models including Gemini 2.5 Pro, Antigravity Agent, Deep Research Pro, and Web Scraping) starting at only 10 sats / 1 GRLC / $0.01 with account-less pay-per-call micropayments: 👉 Explore Full Gateway & Frontier AI Suite

Other Modular Diogenes MCP Packages

Install only the domain packages relevant to your workflow:

| Package | Scoped MCP Endpoint | Spoke Repository | | :--- | :--- | :--- | | 1. EVE: Frontier Sui Move Suite (14 Tools) | https://diogenesgateway.com/mcp/eve-frontier | stand-out-of-my-sun/diogenes-eve-frontier-mcp | | 2. Agent Guardrails & Sandbox (12 Tools) | https://diogenesgateway.com/mcp/agent-guardrails | stand-out-of-my-sun/diogenes-agent-guardrails-mcp | | 3. Web3, DeFi & 402 Security (15 Tools) | https://diogenesgateway.com/mcp/web3-security | stand-out-of-my-sun/diogenes-web3-security-mcp | | 4. Gemini Suite & AI Agents (15 Tools) | https://diogenesgateway.com/mcp/gemini-suite | stand-out-of-my-sun/diogenes-gemini-suite-mcp | | 5. Context, Code & Solvers (15 Tools) | https://diogenesgateway.com/mcp/context-code-solver | stand-out-of-my-sun/diogenes-context-code-solver-mcp | | 6. Dev, Data & Workflow Utilities (14 Tools) | https://diogenesgateway.com/mcp/dev-utilities | stand-out-of-my-sun/diogenes-dev-utilities-mcp | | 7. Web Extraction, Readability & Markdown Engine (12 Tools) | https://diogenesgateway.com/mcp/web-scraping-engine | stand-out-of-my-sun/diogenes-web-scraping-engine-mcp | | 8. Neural Search, Grounded Research & Deep Intelligence (12 Tools) | https://diogenesgateway.com/mcp/neural-search-engine | stand-out-of-my-sun/diogenes-neural-search-engine-mcp | | 9. Database Intelligence, SQL Engine & Schema Guard (12 Tools) | https://diogenesgateway.com/mcp/database-intelligence | stand-out-of-my-sun/diogenes-database-intelligence-mcp | | 10. Full Master Gateway (All 121 Services) | https://diogenesgateway.com/mcp | stand-out-of-my-sun/diogenes-micro-gateway |

TensorBlock & MCP Index Profile Metadata

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

Diogenes Web Extraction, Readability & Markdown Engine MCP (diogenes-web-scraping-engine-mcp)