web-fetch

Fetch web content — auto-convert, chunk, cache with progressive disclosure.

web-fetch <url>

HTML → Markdown, JSON → formatted, PDF → text sections, CSV → table. Image/Video/Audio → metadata extraction + original file. Large content → auto-chunked temp cache. Original binaries preserved.

When to use web-fetch (not a curl replacement)

curl is the right tool when you need raw HTTP control (headers, methods, cookies, full response). web-fetch is a higher-level tool for specific scenarios where you want the content, not the bytes:

• HTML page → Markdown: saves ~60% tokens vs raw HTML for LLM consumption
• Large documents: auto-chunked — first chunk previewed, rest indexed. No terminal flood
• PDF papers: extract text by sections, original preserved for reference
• JSON APIs: auto-formatted and split by top-level keys, no jq needed
• CSV data: rendered as Markdown table, auto-paginated at 100 rows
• Media file checks: inline metadata for images (size/format) and video/audio (duration/codec/resolution)
• Network-restricted envs: auto proxy discovery, or explicit --proxy
• Repeated fetches: built-in cache with 1d/30d expiry policy

Bottom line: if you'd pipe curl into jq, html2text, pdftotext, or ffprobe, try web-fetch instead — one command, progressive disclosure, no pipe chains.

For everything else (binary blobs, custom headers, uploads, full HTTP debugging), stick with curl.

Install

pnpm add -g @lenml/web-fetch
# or
npx @lenml/web-fetch <url>

Image metadata via image-size (built-in, zero native deps) For video/audio metadata: install ffmpeg (provides ffprobe)

Usage

# Small content: inline output
web-fetch https://example.com

# Large content: auto-chunked, first chunk previewed
web-fetch https://example.com/large-doc

# PDF: original preserved + text extracted by sections
web-fetch https://example.com/paper.pdf

# CSV: formatted as markdown table, chunks of 100 rows
web-fetch https://example.com/data.csv

# ZIP: file listing + extracted text files content
web-fetch https://example.com/archive.zip

# Image: saved as original + metadata (image-size)
web-fetch https://example.com/photo.jpg

# Video: saved as original + metadata (ffprobe)
web-fetch https://example.com/video.mp4

# Show cached index
web-fetch --cache <id>

# Show specific chunk
web-fetch --cache <id> --chunk <key>

# Through proxy
web-fetch --proxy http://127.0.0.1:10808 https://example.com

# Raw output (no conversion)
web-fetch -r https://example.com

# Allow downloading files larger than 50MB
web-fetch --dl-big-file https://example.com/large-file.bin

# Custom size limit (200 MB)
web-fetch --max-size 200 https://example.com/file.bin

Options

| Flag | Description | |------|-------------| | --proxy <url> | HTTP/SOCKS proxy (skip auto-discovery) | | --never-proxy | Skip proxy auto-discovery, connect directly | | --cache <id> | View cached fetch index | | --chunk <key> | View chunk from cache (use with --cache <id>) | | -r, --raw | Raw binary output, no conversion | | -i, --inline | Force inline output, no chunking | | -g, --global-cache | Use OS temp dir instead of local .fetch-cache/ | | --dl-big-file | Allow downloading files larger than --max-size | | --max-size <mb> | Big file threshold in MB (default 50) | | --disable-banner | Suppress the conversion source banner | | -h, --help | Show help | | --version | Show version |

Design

Progressive Disclosure

Content auto-saved to .fetch-cache/ (local CWD) or $TMPDIR/web-fetch-cache/ (global):

• <50KB — inline output
• >50KB — split into chunks, first chunk previewed, remaining chunks referenced by key
• >1 day — cache marked stale (metadata preserved, content needs re-fetch)
• >30 days — cache auto-deleted on next cleanup

Content Handling

| Type | Conversion | Chunking | |------|-----------|----------| | HTML | Turndown → Markdown | By h1/h2/h3 headings | | JSON | Pretty-print (2-space) | By top-level keys / array elements | | PDF | pdf-parse v2 text extraction | By detected headings, min 1000 chars per chunk | | CSV | Markdown table | 100 rows per chunk (header repeated) | | ZIP | File listing + text file extraction | Per-file chunks | | Image | Original saved + metadata via image-size | Metadata as index chunk | | Video | Original saved + metadata via ffprobe | Metadata as index chunk | | Audio | Original saved + metadata via ffprobe | Metadata as index chunk | | Binary (unknown) | Saved as-is | Single file | | Unknown text | Raw | Paragraph split fallback |

Large File Protection

• HEAD pre-check sends a HEAD request first to check Content-Length
• If file > --max-size (default 50MB), the download is blocked with a warning
• Add --dl-big-file to bypass the check
• Streaming check also enforces the limit during download
• HEAD failure gracefully falls back to direct GET

Media Metadata (Optional Dependencies)

• Images: image-size (built-in) → width, height, format
• Video/Audio: ffprobe from ffmpeg (system PATH) → duration, codec, resolution, bitrate, sample rate, channels
• Missing dependencies → graceful degradation (only file size and type returned)

Proxy Auto-Discovery

1. Explicit --proxy arg → skip auto-discovery
2. Direct connection attempted first
3. --never-proxy → throw original error
4. os-proxy-config (reads Windows registry / macOS prefs / env vars)
5. Env vars HTTP_PROXY / HTTPS_PROXY / all_proxy
6. Fallback: http://127.0.0.1:10808, http://127.0.0.1:7890

Caching

Default cache is project-local .fetch-cache/. Use --global-cache for OS temp directory. Each fetch creates a subdirectory with index.json (metadata) and chunk files. Stale records (1d+) show metadata but mark content as expired. Very old records (30d+) auto-deleted.

Dev

pnpm install
pnpm test
pnpm lint
pnpm typecheck
pnpm build

Built with:

• vite / vitest
• undici (HTTP fetch)
• turndown (HTML to Markdown)
• jsdom (HTML parsing)
• pdf-parse v2 (PDF text extraction)
• adm-zip (ZIP extraction)
• image-size (image metadata, pure JS)
• commander (CLI framework)
• picocolors (terminal colors)
• eslint + typescript-eslint (strict)
• TypeScript 6 + strict mode