⚡ StormScrap: Intelligent Data Platform & Evasion Scraper

StormScrap is an ultra-lightweight, connection-resilient, and WAF-secured Node.js (Express) scraping and automation microservice built using TypeScript, Playwright, and Cheerio.

🧱 Architecture & Request Workflow

The engine is designed with a defense-in-depth security model and an adaptive runtime queue that guarantees high availability and server stability.

graph TD
    Client[Client Request] --> WAF{WebShield WAF}
    WAF -- Threat Detected --> BlockWAF[403 Blocked / Logged]
    WAF -- Clean Payload --> Zod{Zod Validator}
    
    Zod -- SSRF/Invalid URL --> BlockSSRF[400 Blocked / Logged]
    Zod -- Valid URL --> Controller[Scrape Controller]
    
    Controller --> SyncRoute{Is Sync or Async?}
    
    SyncRoute -- Sync Route --> Scraper[Scraper Service]
    SyncRoute -- Async Route --> QueueAdapter{Queue Adapter}
    
    %% Sync Path
    Scraper --> BrowserMgr{Shared Browser}
    BrowserMgr --> Context[Spawn BrowserContext]
    Context --> Interceptor{Needs Screenshots?}
    Interceptor -- No --> BlockAssets[Block Styles/Images/Fonts]
    Interceptor -- Yes --> LoadAssets[Load Full Assets]
    BlockAssets --> FetchDOM[Render Page DOM]
    LoadAssets --> FetchDOM
    FetchDOM --> Cheerio[Cheerio Fast Parse]
    Cheerio --> SuccessResult[Return Extracted Data]
    
    %% Async Path
    QueueAdapter --> MemQueue[In-Memory Queue]
    MemQueue --> Worker[Scrape Promise Worker]
    Worker --> Scraper

🚀 Key Features

🛡️ Defense-in-Depth Security

• WebShield WAF & DDoS Shield: Dynamic heuristics engine filtering out Cross-Site Scripting (XSS), SQL Injection (SQLi), and NoSQL payloads. Built-in sliding-window rate limiting prevents brute-force exploits.
• SSRF Network Protection: Zod schemas strictly parse URLs, instantly blocking attempts to scan internal subnets (10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12), loopbacks (localhost, 127.0.0.1), or cloud provider metadata endpoints (169.254.169.254).
• Crawler Honeypots: Traps standard bot directories (e.g., /.env, /admin, /wp-admin) to immediately isolate and ban scanning IPs.

⚡ Resource & Memory Optimizations

• Singleton Browser Reuse: Spawns a single, global Chromium instance at startup and multiplexes requests via isolated BrowserContext tabs. Spawning contexts uses <10MB of RAM compared to ~150MB for a brand new browser process, eliminating CPU/RAM server hangs.
• Dynamic Asset Interceptor: If the user does not request screenshots or visual renderings, the engine intercepts network requests and blocks heavy assets (images, fonts, stylesheets, and videos). This increases parsing speeds by 3x and decreases bandwidth by 90%.
• Lightweight Async Queue: Features a built-in in-memory job scheduler to queue, buffer, and process async scraping tasks locally, keeping server footprint minimal.
• Premium Anti-Bot Stealth Engine: Erases Playwright's automation footprints by overriding the navigator.webdriver flag, spoofing WebGL rendering vendor properties (mimicking real graphics cards), and emulating human typing delays and coordinates jitter.

🛠️ Installation & Setup

Prerequisites

• Node.js (v18+)

Setup Commands

# Clone the repository and install dependencies
npm install

# Compile TypeScript to JavaScript (dist/)
npm run build

# Start the server in Development mode (watches TS files)
npm run dev

# Start the server in Production mode
npm start

📡 API Usage Guide

1. Synchronous Scrape (POST /api/scrape/sync)

Performs a direct scraping task and returns the scraped data immediately in the HTTP response.

Payload Options:

• url (String, Optional): Target page address. Either url or urls must be provided.
• urls (Array of Strings, Optional): List of target page addresses for bulk scraping.
• selectors (Object, Optional): Selector key-value mappings to query specific DOM elements.
• respectRobots (Boolean, Optional): Set to true to check the target's robots.txt rules first.
• infiniteScroll (Boolean, Optional): Set to true to scroll dynamic pages iteratively to the bottom.
• infiniteScrollLimit (Number, Optional): Max scroll loops. Default is 10.
• waitUntil (String, Optional): Load strategy: "domcontentloaded", "load", or "networkidle".
• captureScreenshot (Boolean, Optional): Capture full-page snapshot (returned as base64 string).
• format (String, Optional): Set to "csv" to get direct tabulated CSV file attachments instead of JSON.
• parentSelector (String, Optional): Parse parent elements (cards) and map child selectors inside them into structured JSON arrays.
• proxy (String, Optional): Proxy server URL to route all traffic through.
• capturePdf (Boolean, Optional): Set to true to export a base64 PDF document of the page.
• seoAnalyze (Boolean, Optional): Set to true to run a structural SEO audit on the page.
• maxRetries (Number, Optional): Max attempts to retry failed requests using exponential backoff.

JSON Example:

{
  "url": "https://news.ycombinator.com/",
  "selectors": {
    "titles": ".titleline > a",
    "scores": ".score"
  },
  "waitUntil": "networkidle",
  "infiniteScroll": false
}

CSV Format Request:

{
  "url": "https://news.ycombinator.com/",
  "selectors": {
    "titles": ".titleline > a"
  },
  "format": "csv"
}

2. Queued Asynchronous Scrape (POST /api/scrape/async)

Enqueues a scraping task and returns a job tracking ID immediately.

// POST /api/scrape/async
{
  "url": "https://news.ycombinator.com/",
  "selectors": {
    "titles": ".titleline > a"
  }
}

Response:

{
  "message": "Scraping job queued successfully",
  "jobId": "mem-1"
}

3. Check Job Status (GET /api/scrape/jobs/:jobId)

Retrieve the state, progress, output results, or failure logs of a queued background job. Response:

{
  "jobId": "mem-1",
  "status": "completed",
  "progress": 0,
  "result": {
    "url": "https://news.ycombinator.com/",
    "success": true,
    "title": "Hacker News",
    "data": {
      "titles": ["Show HN: Scraper Engine", "Ask HN: WAF Security"]
    }
  },
  "failedReason": null
}

4. List Scraped Files (GET /api/scrape/files)

List all locally archived scraped output files on the host server. Response:

[
  {
    "fileName": "1720000000000-scrape.json",
    "sizeBytes": 4120,
    "createdAt": "2026-07-25T06:20:00.000Z"
  }
]

5. Download Scraped File (GET /api/scrape/download/:fileName)

Securely stream the download of an archived file. Rejects path traversal (../../) automatically.

6. Visual Preview Debugger (POST /api/scrape/preview)

Renders the target URL and returns a visual base64 screenshot, page title, and parsed selector preview. Response:

{
  "url": "https://example.com",
  "title": "Example Domain",
  "screenshot": "iVBORw0KGgoAAAANS...",
  "dataPreview": []
}

🧪 Verification & Audit Simulation

Ensure the application is running locally (npm run dev), then execute verification tests:

# Test direct Playwright & Cheerio selectors extraction
npm run test

# Test SSRF block rules and WAF script injection blocks
npm run test:security

🔒 Security & License

• Detailed security disclosures, vulnerabilities protocols, and SSRF lists are documented in SECURITY.md.
• Released under the open-source MIT License.