edgecrawl

Local AI-powered web scraper. Extract structured JSON from any website using on-device ONNX LLMs. No API keys, no cloud, no Python.

Features

• 100% Local AI — Runs Qwen3 ONNX models on your machine via Transformers.js v4 (WebGPU/WASM)
• Zero API Keys — No OpenAI, no Anthropic, no cloud bills. Everything runs on-device
• Structured JSON Output — Define a schema, get clean JSON back
• CLI + Library — Use from the command line or import into your Node.js app

Architecture

Playwright (headless browser)
  |
  v
JSDOM + node-html-markdown -> Clean Markdown
  |
  v
Qwen3 ONNX (Transformers.js v4) -> Structured JSON

Quick Start

Install from npm

npm install edgecrawl
npx playwright install chromium

Install from source

git clone https://github.com/couzip/edgecrawl.git
cd edgecrawl
npm install
npx playwright install chromium

Models are downloaded automatically on first run:

• LLM: Qwen3 ONNX (0.4-2.5 GB depending on preset)

CLI

# Extract structured data from a URL
edgecrawl extract https://example.com

# With custom schema
edgecrawl extract https://example.com -s schemas/product.json -o result.json

# Light model on WASM
edgecrawl extract https://example.com -p light -d wasm

# Convert to Markdown only (no LLM)
edgecrawl md https://example.com

Library

import { scrapeAndExtract, cleanup } from "edgecrawl";

const result = await scrapeAndExtract("https://example.com", {
  preset: "balanced",
});

console.log(result.extracted);
await cleanup();

CLI Commands

extract <url> — Structured extraction

# Default (balanced model, WebGPU)
edgecrawl extract https://example.com

# Light model on WASM
edgecrawl extract https://example.com -p light -d wasm

# Custom schema + output file
edgecrawl extract https://example.com -s schemas/product.json -o result.json

# Target a specific section
edgecrawl extract https://example.com --selector "main article"

batch <file> — Batch processing

# Process URL list (one URL per line)
edgecrawl batch urls.txt -o results.json

# With concurrency control
edgecrawl batch urls.txt -c 5

query <url> <prompt> — Custom question

# Ask a question about page content
edgecrawl query https://example.com "What are the main products?"

md <url> — Markdown conversion only (no LLM)

edgecrawl md https://example.com
edgecrawl md https://example.com -o page.md --scroll

CLI Options

Common Options (extract, batch, query)

| Option | Description | Default | |--------|-------------|---------| | -p, --preset <preset> | Model preset: light / balanced / quality | balanced | | -d, --device <device> | Inference device: webgpu / wasm | webgpu | | -s, --schema <file> | Custom schema JSON file | built-in default | | -o, --output <file> | Output file path | stdout | | -t, --max-tokens <n> | Max input tokens for LLM | 2048 | | --selector <selector> | CSS selector to narrow target content | - |

Batch Options

| Option | Description | Default | |--------|-------------|---------| | -c, --concurrency <n> | Concurrent scraping limit | 3 |

Browser Options (all commands)

| Option | Description | Default | |--------|-------------|---------| | --headful | Show browser window (for debugging) | false | | --user-agent <ua> | Custom User-Agent string | - | | --timeout <ms> | Page load timeout in milliseconds | 30000 | | --proxy <url> | Proxy server URL | - | | --cookie <cookie> | Cookie in name=value format (repeatable) | - | | --extra-header <header> | HTTP header in Key:Value format (repeatable) | - | | --viewport <WxH> | Viewport size | 1280x800 | | --wait-until <event> | Navigation wait condition: load / domcontentloaded / networkidle | load | | --no-block-media | Disable blocking of images/fonts/media | false | | --scroll | Scroll to bottom (for lazy-loaded content) | false | | --wait <selector> | Wait for CSS selector to appear | - |

Library Usage

High-level Pipeline

import {
  scrapeAndExtract,
  batchScrapeAndExtract,
  scrapeAndQuery,
  cleanup,
} from "edgecrawl";

// Basic extraction
const result = await scrapeAndExtract("https://example.com");

// Custom schema
const product = await scrapeAndExtract("https://shop.example.com/item", {
  schema: {
    type: "object",
    properties: {
      name: { type: "string", description: "Product name" },
      price: { type: "number", description: "Price (numeric)" },
      features: {
        type: "array",
        items: { type: "string" },
        description: "Key features or specs",
      },
    },
    required: ["name", "price"],
  },
});

// Batch processing
const results = await batchScrapeAndExtract(
  ["https://example.com/1", "https://example.com/2"],
  { concurrency: 3 }
);

// Custom query
const answer = await scrapeAndQuery(
  "https://example.com",
  "What are the main products?",
  { preset: "quality" }
);

await cleanup();

Low-level APIs

// Use individual modules
import { htmlToMarkdown, cleanMarkdown } from "edgecrawl/html2md";
import { launchBrowser, fetchPage, closeBrowser } from "edgecrawl/scraper";
import { initLLM, extractStructured } from "edgecrawl/llm";

// HTML to Markdown only
await launchBrowser();
const { html } = await fetchPage("https://example.com");
const { markdown, title } = htmlToMarkdown(html, "https://example.com");
const cleaned = cleanMarkdown(markdown);
await closeBrowser();

// Or use the root export
import { htmlToMarkdown, cleanMarkdown, fetchPage } from "edgecrawl";

Custom Schemas

Define what data to extract by providing a JSON schema file:

{
  "type": "object",
  "properties": {
    "name": { "type": "string", "description": "Product name" },
    "price": { "type": "number", "description": "Price (numeric)" },
    "currency": { "type": "string", "description": "Currency code (e.g. USD, EUR, JPY)" },
    "description": { "type": "string", "description": "Product description (1-3 sentences)" },
    "features": {
      "type": "array",
      "items": { "type": "string" },
      "description": "Key features or specs"
    },
    "availability": { "type": "string", "description": "Stock status (in stock, out of stock, etc.)" }
  },
  "required": ["name", "price", "currency"]
}

edgecrawl extract https://shop.example.com/product -s schema.json

See the schemas/ directory for more examples.

Model Presets

| Preset | Model | Size | Speed | Quality | |--------|-------|------|-------|---------| | light | Qwen3-0.6B | ~0.4 GB | Fast | Good for simple pages | | balanced | Qwen3-1.7B | ~1.2 GB | Medium | Best balance (default) | | quality | Qwen3-4B | ~2.5 GB | Slower | Best accuracy |

All models run locally via ONNX Runtime. First run downloads the model to .model-cache/.

Tech Stack

| Component | Library | Role | |-----------|---------|------| | Browser | Playwright | Headless scraping | | HTML -> Markdown | JSDOM + node-html-markdown | Content cleaning + Markdown conversion | | LLM | Transformers.js v4 + Qwen3 ONNX | Local structured extraction | | CLI | Commander.js | Command-line interface |

AI Agent Skill

A skill file is included for AI coding agents. Install it to let your agent use edgecrawl directly:

npx skills add couzip/edgecrawl

Once installed, your AI agent can scrape websites and extract structured data using edgecrawl.

Requirements

• Node.js >= 20.0.0
• Chromium (installed via npx playwright install chromium)
• ~1-3 GB disk space for models (downloaded on first run)
• GPU recommended for WebGPU mode (falls back to WASM/CPU)

License

MIT