@link-assistant/web-capture

v1.10.10

Published

21 days ago

CLI and microservice to render web pages as HTML, Markdown, or PNG

0High
0Medium
0Low

konard

render markdown screenshot microservice

web-capture (JavaScript/Node.js)

A CLI and microservice to fetch URLs and render them as:

Markdown: Converted from HTML with image extraction (default)
HTML: Rendered page content
Plain text: Raw text downloads for paste-like URLs such as xpaste.pro
PNG/JPEG screenshot: Viewport or full-page capture with theme support
ZIP archive: Markdown + locally downloaded images
PDF: Print-quality document with embedded images
DOCX: Word document with embedded images
Shared AI dialog transcript: ChatGPT share captures with structured diagnostics for unsupported providers

Installation

From npm

npm install -g @link-assistant/web-capture

From Source

cd js
npm install

Quick Start

CLI Usage

# Capture a URL as Markdown (default format)
# Output auto-derived to ./data/web-capture/<host>/<path>/document.md
web-capture https://example.com

# Capture as Markdown to a specific file
web-capture https://example.com -o page.md

# Write to stdout explicitly
web-capture https://example.com -o -

# Capture as HTML
web-capture https://example.com --format html -o page.html

# Capture raw paste text
web-capture https://xpaste.pro/p/t4q0Lsp0 --format txt -o paste.txt

# Capture a GitHub repository as compact text or Markdown
web-capture https://github.com/link-assistant/web-capture --format txt -o repository.txt
web-capture https://github.com/link-assistant/web-capture --format markdown -o repository.md

# Take a PNG screenshot
web-capture https://example.com --format png -o screenshot.png

# Take a JPEG screenshot with custom quality
web-capture https://example.com --format jpeg --quality 60 -o screenshot.jpg

# Dark theme full-page screenshot
web-capture https://example.com --format png --theme dark --fullPage -o dark.png

# Custom viewport size
web-capture https://example.com --format png --width 1920 --height 1080 -o wide.png

# Download as PDF
web-capture https://example.com --format pdf -o page.pdf

# Download as DOCX
web-capture https://example.com --format docx -o page.docx

# Create a ZIP archive (default: zip format)
web-capture https://example.com --archive

# Create a ZIP archive with specific format
web-capture https://example.com --archive zip -o site.zip
web-capture https://example.com --archive tar.gz -o site.tar.gz

# Default markdown keeps remote image URLs as direct links
web-capture https://example.com -o page.md

# Keep images inline as base64 (self-contained file, opt-in)
web-capture https://example.com --embed-images -o page.md

# Extract images (inline base64 + remote) to a local images/ folder
web-capture https://example.com --extract-images -o page.md

# Custom images directory
web-capture https://example.com --extract-images --images-dir assets -o page.md

# Disable specific features
web-capture https://example.com --no-extract-latex --no-post-process -o page.md

# Structured search-provider capture (JSON by default)
web-capture search "formal methods" --provider wikipedia

# Shared AI dialog capture (JSON by default)
web-capture shared-dialog https://chatgpt.com/share/SHARE_ID
web-capture shared-dialog https://chatgpt.com/share/SHARE_ID --format demo-memory -o -

# Start as API server
web-capture --serve

# Start server on custom port
web-capture --serve --port 8080

Google Docs

# Capture the live editor model as Markdown (default --capture browser)
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture browser

# Capture via the public export endpoint
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture api

# Capture via the Google Docs REST API
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture api --apiToken YOUR_TOKEN

Browser-model capture waits until DOCS_modelChunk data stops changing before parsing. For very large or slow documents, tune the wait with WEB_CAPTURE_GDOCS_STABILITY_MS (default 1500) and WEB_CAPTURE_GDOCS_MAX_WAIT_MS (default 30000).

API Endpoints (Server Mode)

Start the server with web-capture --serve and use the endpoints below. FormalAI integration should use the stable HTTP/CLI contract documented in ../docs/formalai-contract.md.

HTML Endpoint

GET /html?url=<URL>&engine=<ENGINE>

Returns the raw HTML content of the specified URL.

| Parameter | Required | Description | Default | | --------- | -------- | ------------------------------------ | --------- | | url | Yes | URL to fetch | - | | engine | No | Browser engine: puppeteer/playwright | puppeteer |

Markdown Endpoint

GET /markdown?url=<URL>

Converts the HTML content of the specified URL to Markdown format. By default, original remote image URLs are preserved and base64 data URIs are stripped (clean single-file output). Use keepOriginalLinks=false to strip all images, or embedImages=true to keep base64 images inline.

For xpaste.pro paste URLs, /markdown fetches the visual paste page, keeps visible header, language, metadata, paste body, and footer text in visual order, and appends the raw paste text as a fenced block named xpaste-pro-<id>.txt when the final Markdown stays under 1500 lines. Larger paste pages return a ZIP containing index.md, xpaste-pro-<id>.md, and xpaste-pro-<id>.txt. Canonical /p/<id>, /p/<id>/raw, /ru/p/<id>, and /en/p/<id> URLs are normalized before capture.

For plain GitHub repository URLs such as https://github.com/owner/repo, /markdown returns a compact repository snapshot with repository metadata, the root file tree, and README content. GitHub subpages continue through the regular HTML-to-Markdown conversion path.

| Parameter | Required | Description | Default | | ------------------- | -------- | ------------------------------------------------------------------------- | -------- | | url | Yes | URL to fetch | - | | converter | No | Converter: turndown or kreuzberg | turndown | | format | No | Response format: text or json (json requires converter=kreuzberg) | text | | embedImages | No | Keep base64 images inline (true/false) | false | | keepOriginalLinks | No | Keep original remote URLs, strip base64 | true |

Text Endpoint

GET /txt?url=<URL>

Returns raw text content as a .txt attachment. xpaste.pro paste URLs are normalized to their /raw endpoint, including localized /ru/p/<id> and /en/p/<id> URLs.

Plain GitHub repository URLs return a compact .txt snapshot with repository metadata, the root file tree, and README content.

| Parameter | Required | Description | Default | | --------- | -------- | ------------ | ------- | | url | Yes | URL to fetch | - |

Image Endpoint

GET /image?url=<URL>&format=png&theme=dark&width=1920&height=1080&fullPage=true

Returns a screenshot of the specified URL.

| Parameter | Required | Description | Default | | --------------- | -------- | ---------------------------------------- | --------------- | | url | Yes | URL to capture | - | | engine | No | Browser engine: puppeteer/playwright | puppeteer | | format | No | Image format: png (lossless) or jpeg | png | | quality | No | JPEG quality 0-100 (ignored for PNG) | 80 | | width | No | Viewport width in pixels | 1280 | | height | No | Viewport height in pixels | 800 | | fullPage | No | Capture full scrollable page | false | | theme | No | Color scheme: light, dark | browser default | | dismissPopups | No | Auto-close cookie/consent popups | true |

Archive Endpoint

GET /archive?url=<URL>&localImages=true&documentFormat=markdown

Returns a ZIP archive containing either document.md or document.html and asset directories (images/, css/).

| Parameter | Required | Description | Default | | ------------------- | -------- | -------------------------------------------- | ---------- | | url | Yes | URL to archive | - | | localImages | No | Download images locally into the archive | true | | documentFormat | No | Document format: markdown or html | markdown | | embedImages | No | Keep base64 images inline in the document | false | | keepOriginalLinks | No | Keep original remote URLs (skip downloading) | false |

Archive structure (with localImages=true):

archive.zip
├── document.md       # or document.html when documentFormat=html
├── images/
│   ├── image-1.png
│   ├── image-2.jpg
│   └── ...
└── css/              # only when documentFormat=html
    ├── style-1.css
    └── ...

PDF Endpoint

GET /pdf?url=<URL>&theme=light

Returns a PDF document rendered by the browser engine (all images embedded).

| Parameter | Required | Description | Default | | --------------- | -------- | ------------------------------- | --------------- | | url | Yes | URL to convert | - | | engine | No | Browser engine | puppeteer | | width | No | Viewport width in pixels | 1280 | | height | No | Viewport height in pixels | 800 | | theme | No | Color scheme: light, dark | browser default | | dismissPopups | No | Auto-close popups before export | true |

DOCX Endpoint

GET /docx?url=<URL>

Returns a DOCX document with embedded images.

| Parameter | Required | Description | Default | | --------- | -------- | -------------- | ------- | | url | Yes | URL to convert | - |

Fetch/Stream Endpoints

GET /fetch?url=<URL>
GET /stream?url=<URL>

Proxy fetch and streaming proxy endpoints.

Search Endpoint

GET /search?q=<QUERY>&provider=<PROVIDER>&format=json|markdown

Captures structured results from a search provider and returns them in a normalized, machine-readable shape. wikipedia (the default) uses the CORS-friendly REST API; the HTML engines (duckduckgo, google, bing, brave) are parsed server-side. Blocked or CAPTCHA-gated pages are reported through diagnostics instead of failing.

| Parameter | Required | Description | Default | | ---------- | -------- | ---------------------------------------------------- | ----------- | | q | Yes | Search query (query is accepted as an alias) | - | | provider | No | wikipedia, duckduckgo, google, bing, brave | wikipedia | | limit | No | Maximum number of results | 10 | | format | No | Response format: json or markdown | json |

Example response (format=json):

{
  "query": "formal methods",
  "provider": "wikipedia",
  "captureMode": "fetch",
  "capturedAt": "2026-05-18T20:30:00.000Z",
  "results": [
    {
      "rank": 1,
      "title": "Formal methods",
      "url": "https://en.wikipedia.org/wiki/Formal_methods",
      "snippet": "mathematically rigorous techniques for the specification..."
    }
  ],
  "diagnostics": {
    "status": 200,
    "blockedByCors": false,
    "blockedByCaptcha": false,
    "sourceUrl": "https://en.wikipedia.org/w/rest.php/v1/search/page?q=formal%20methods&limit=10"
  }
}

Shared Dialog Endpoint

GET /shared-dialog?url=<URL>&format=json|meta-language|demo-memory|markdown|txt

Captures replayable shared AI dialog transcripts. ChatGPT share links are decoded from their embedded linear_conversation data. Providers that cannot currently expose transcript data, such as Google AI Mode interstitial pages, return a structured unsupported diagnostic.

CLI Reference

Server Mode

web-capture --serve [--port <port>]

| Option | Short | Description | Default | | --------- | ----- | ------------------------ | ------------------ | | --serve | -s | Start as HTTP API server | - | | --port | -p | Port to listen on | 3000 (or PORT env) |

Capture Mode

web-capture <url> [options]

| Option | --------------------------- | --format | --output | --data-dir | --engine | --theme | --width | --height | --quality | --fullPage | --embed-images | --no-extract-images | --extract-images[=DIR] | --keep-original-links | --images-dir | --archive | --document-format | --localImages | --extract-latex | --no-extract-latex | --extract-metadata | --no-extract-metadata | --post-process | --no-post-process | --detect-code-language | --no-detect-code-language | Short | Description | Default | | ----- | ------------------------------------------------- | ----------------------------------- | | -f | Output format (see below) | markdown | | -o | Output file path. Use -o - for stdout | auto-derived from URL | | | Base directory for auto-derived output paths | ./data/web-capture | | -e | Browser engine: puppeteer, playwright | puppeteer (or BROWSER_ENGINE env) | | -t | Color scheme: light, dark, no-preference | browser default | | | Viewport width in pixels | 1280 | | | Viewport height in pixels | 800 | | | JPEG quality 0-100 | 80 | | | Capture full scrollable page | false | | | Keep images inline as base64 (self-contained) | false | | | Alias for --embed-images | false | | | Extract images to DIR/images/ + download remote | - | | | Keep remote URLs as direct links (the default) | false | | | Subdirectory name for extracted images | images | | | Create archive: zip, 7z, tar.gz, tar | - | | | Document format in archive: markdown, html | markdown | | | Download images locally in archive mode | true | | | Extract LaTeX formulas | true | | | Disable LaTeX extraction | - | | | Extract article metadata | true | | | Disable metadata extraction | - | | | Apply post-processing | true | | | Disable post-processing | - | | | Detect code block languages | true | | | Disable code language detection | - |

Supported formats:

| Format | Description | | ----------------- | ---------------------------------------- | | markdown / md | Markdown conversion (default) | | html | Rendered HTML | | txt / text | Raw text download | | image / png | PNG screenshot (lossless) | | jpeg | JPEG screenshot (configurable quality) | | pdf | PDF with embedded images | | docx | Word document with embedded images | | archive / zip | ZIP archive with markdown + local images |

Search Mode

web-capture search "<query>" [options]

Captures structured search-provider results. Output defaults to JSON; pass --format markdown for a human-readable document.

# Search Wikipedia (default provider), JSON output
web-capture search "formal methods"

# Search DuckDuckGo, limit to 5 results
web-capture search "formal methods" --provider duckduckgo --limit 5

# Render the results as Markdown
web-capture search "formal methods" --format markdown

| Option | Short | Description | Default | | ------------ | ----- | ---------------------------------------------------- | ----------- | | --provider | | wikipedia, duckduckgo, google, bing, brave | wikipedia | | --limit | | Maximum number of results | 10 | | --format | -f | Output format: json or markdown | json |

The library API exposes the same contract:

import { search } from '@link-assistant/web-capture';

const result = await search({
  query: 'formal methods',
  provider: 'wikipedia',
  limit: 5,
});
console.log(result.results[0].title);

Configuration

Configuration values are resolved with the following priority (highest to lowest):

CLI arguments: --port 8080
Environment variables: PORT=8080
Default .lenv file: .lenv in the project root
Built-in defaults

Environment Variables

| Variable | Description | Default | | ---------------------------------- | ----------------------------------- | -------------------- | | PORT | Server port | 3000 | | BROWSER_ENGINE | Browser engine | puppeteer | | API_TOKEN | API token for authenticated capture | - | | WEB_CAPTURE_DATA_DIR | Base directory for output | ./data/web-capture | | WEB_CAPTURE_EMBED_IMAGES | 0/1 — keep images inline | 0 | | WEB_CAPTURE_KEEP_ORIGINAL_LINKS | 0/1 — keep original remote URLs | 0 | | WEB_CAPTURE_IMAGES_DIR | Subdirectory for extracted images | images | | WEB_CAPTURE_EXTRACT_LATEX | 0/1 — extract LaTeX | 1 | | WEB_CAPTURE_EXTRACT_METADATA | 0/1 — extract metadata | 1 | | WEB_CAPTURE_POST_PROCESS | 0/1 — post-processing | 1 | | WEB_CAPTURE_DETECT_CODE_LANGUAGE | 0/1 — detect code langs | 1 |

Browser Engine Support

The service supports both Puppeteer and Playwright browser engines:

Puppeteer: Default engine, mature and well-tested
Playwright: Alternative engine with similar capabilities

Supported engine values:

puppeteer or pptr - Use Puppeteer
playwright or pw - Use Playwright

Popup/Modal Dismissal

By default, the image, PDF, and archive endpoints automatically dismiss common popups before capture:

Google Funding Choices consent dialogs
Cookie consent banners
Generic modals and overlays
Fixed-position overlays covering content

Set dismissPopups=false to disable this behavior.

Docker

# Build and run using Docker Compose
docker compose up -d

# Or manually
docker build -t web-capture-js .
docker run -p 3000:3000 web-capture-js

Development

Available Commands

npm run dev - Start the development server with hot reloading
npm run start - Start the service using Docker Compose
npm test - Run all unit tests
npm run lint - Check code with ESLint
npm run format - Format code with Prettier

Testing

npm test                    # Run unit and integration tests
npm run test:e2e            # Run end-to-end tests
npm run test:all            # Run all tests including build

Some integration suites hit live servers and are skipped by default. Enable them with environment variables:

# Download the Wikipedia page (markdown + image) in every supported engine
WIKIPEDIA_INTEGRATION=true npm test -- --testPathPattern="wikipedia-download"

# Download a GitHub repository page as compact txt/markdown, original HTML, and screenshots
GITHUB_REPOSITORY_INTEGRATION=true npm test -- --testPathPattern="github-readme"

# Habr articles and public Google Docs live suites
HABR_INTEGRATION=true npm test -- --testPathPattern="habr-article"
GDOCS_INTEGRATION=true npm test -- --testPathPattern="gdocs-public-doc"

Built With

Express.js for the web server
Puppeteer and Playwright for headless browser automation
Turndown for HTML to Markdown conversion
archiver for ZIP creation
docx for DOCX generation
Jest for testing

Library Usage

You can also use web-capture as a Node.js library:

import {
  fetchHtml,
  convertHtmlToMarkdown,
} from '@link-assistant/web-capture/src/lib.js';
import { createBrowser } from '@link-assistant/web-capture/src/browser.js';
import { dismissPopups } from '@link-assistant/web-capture/src/popups.js';

// Fetch and convert to markdown
const html = await fetchHtml('https://habr.com/en/articles/895896/');
const markdown = convertHtmlToMarkdown(
  html,
  'https://habr.com/en/articles/895896/'
);

// Take a themed screenshot
const browser = await createBrowser('playwright', { colorScheme: 'dark' });
const page = await browser.newPage();
await page.setViewport({ width: 1920, height: 1080 });
await page.goto('https://habr.com/en/articles/895896/', {
  waitUntil: 'networkidle0',
});
await dismissPopups(page);
const buffer = await page.screenshot({ type: 'png', fullPage: true });
await browser.close();

License

Unlicense — This is free and unencumbered software released into the public domain. You are free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means. See https://unlicense.org for details.