@seanmozeik/magic-fetch

URL fetch with impit (browser-like TLS), SSRF checks, and optional Readability → markdown — as a fetch CLI and as a magicFetch / fetchUrl library.

CLI

Install / run with Bun (see package.json bin and dev script).

bun run dev -- --help
fetch --agent "https://example.com"
fetch --skill   # print agent-oriented SKILL.md

Agent-oriented usage is documented in skill/magic-fetch/SKILL.md (also via fetch --skill).

Flags (summary)

• --agent, --json, --html, --raw, -o / --output, --max-bytes, --max-chars, --allow-private-network, --allow-host, --skill

With -o / --output, the full body is written to the path; --max-* limits are not applied.

Output modes:

• default → markdown via Readability; in human terminal mode rendered with ANSI styling.
• --html → stripped HTML (no Readability, no markdown conversion).
• --raw → markdown text without ANSI styling (human mode only; agent/json output is JSON regardless).

Library

Layers

| API | Use when | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | magicFetch | You need an HTTP Response (metadata, streaming body, SSRF checks, impit TLS). | | fetchUrl | You want one string after markdown/raw routing, byte/char limits, and Readability (same pipeline as the CLI). | | createMagicFetch | You want a typeof fetch wrapper with embed-friendly defaults (defaults-win-accept merge, optional timeoutMs, injectable transport for tests). |

magicFetch

Returns ImpitResponse (fetch-shaped: text(), json(), headers, …).

import { magicFetch } from '@seanmozeik/magic-fetch';

const res = await magicFetch('https://example.com', { ssrf: { allowPrivateNetwork: true } });
console.log(res.status, await res.text());

Negotiation

• acceptProfile: markdown-first (default when omitting both profile and legacy contentMode), html-first, or none (library does not set Accept).
• Legacy contentMode: markdown → markdown-first; html → html-first. Prefer acceptProfile for new code.

Header merge

• headerMerge: 'caller-overrides' (default): defaults first, then caller headers overwrite — caller Accept overrides negotiation (historical behavior).
• headerMerge: 'defaults-win-accept': caller headers apply first, then the profile Accept wins unless dangerouslyAllowCallerAccept: true (lets JSON/API Accept overrides when explicitly opted in). Missing Accept-Language / User-Agent are filled.

HTTP/3 (built-in impit only)

• Default is http3: true on the shared Impit client so HTTP/3 can be negotiated when the server supports it. impit falls back to HTTP/2 / HTTP/1 when QUIC is unavailable — use negotiation, not impit's per-request forceHttp3 (that fails hard when H3 cannot be used).
• Pass http3: false on magicFetch / createMagicFetch to build an impit client with QUIC disabled (e.g. future proxyUrl use — impit documents proxies as incompatible with HTTP/3).

Other options

• timeoutMs: wall-clock timeout; rejects with MagicFetchTimeoutError (exported from the package).
• fetchImplementation: swap impit for mocks or another HTTP stack (honour signal for timeouts).
• ssrf: same policy object as validateUrl (allowPrivateNetwork, hostnameAllowlist, …).

createMagicFetch

Factory returning fetch(input, init). Defaults headerMerge to defaults-win-accept so apps can pass cache/proxy headers without accidentally overriding markdown negotiation.

import { createMagicFetch } from '@seanmozeik/magic-fetch';

const mf = createMagicFetch({
  ssrf: { hostnameAllowlist: ['api.example.com'] },
  timeoutMs: 15_000,
});

await mf('https://api.example.com/v1/doc');

Per-call init can override factory options; fetchImplementation on a single call overrides the factory transport (useful in tests).

fetchUrl

Higher-level helper used by the CLI: one string content, markdown vs html pipeline (mode: 'markdown' | 'html'), optional byte/char limits or noTruncate: true.

import { fetchUrl } from '@seanmozeik/magic-fetch';

const out = await fetchUrl({ url: 'https://example.com', mode: 'markdown', noTruncate: true });
console.log(out.content, out.finalUrl, out.truncated);

Internally fetchUrl uses magicFetch with defaults-win-accept and the matching acceptProfile.

Errors

• SsrfBlockedError — blocked host/IP per SSRF policy.
• MagicFetchTimeoutError — timeoutMs elapsed.

Exports

See src/index.ts: magicFetch, createMagicFetch, fetchUrl, validateUrl, SsrfBlockedError, MagicFetchTimeoutError, negotiation constants ACCEPT_MARKDOWN_FIRST / ACCEPT_HTML_FIRST, DEFAULT_USER_AGENT, and types MagicFetchInit, CreateMagicFetchOptions, AcceptProfile, MagicHeaderMerge, MagicFetchTransport, WebFetchInput, WebFetchOutput, SsrfPolicy, MagicFetchResponse.

The package exports map points at ./src/index.ts — consume from TypeScript/Bun with your bundler’s rules for TS source.