web-source-bundler

web-source-bundler fetches a web source, preserves the fetched response, and writes a local Markdown bundle with references and assets.

The tool is inspired by agent web-fetch/web-search pipelines: keep the source provenance and original bytes first, then create a readable Markdown entry for downstream knowledge-base or wiki workflows.

Usage

npx @sanbaiw/web-source-bundler https://example.com ./raw/example
pnpm dlx @sanbaiw/web-source-bundler https://example.com ./raw/example

Options:

web-source-bundler [options] <url> <output-dir>

--no-svg2png   Keep SVG images as-is instead of converting to PNG.
--version      Print the CLI version.

The command writes:

• index.md for the primary source entry.
• assets/ for localized primary-source assets or binary source payloads.
• references/*.md for direct reference pages.
• references/assets/ for binary reference payloads.
• references/references.json for reference provenance.

Cleanup behavior

The bundle aims to be a faithful, readable archive, so conversion applies some zero-config cleanup:

• Defuddle-based content extraction selects the readable page body before Markdown conversion. That keeps article prose, headings, tables, figures, and useful in-article links while cutting unrelated page chrome before direct references are discovered.
• Semantic evidence figures that own a sourced image with non-empty alt text are protected across readable-content extraction. This prevents generic loading-class cleanup from turning claim-bearing <figure><picture><img> structures into empty shells while leaving unrelated non-figure cards eligible for normal chrome removal.
• Semantic interactive figures with multiple role="tabpanel" states are flattened before extraction. Tab controls and hidden nested UI are removed, while every server-provided panel, caption, link, chart label, and inline-SVG text remains in source order for Markdown conversion.
• Markdown assets and navigation use the same localization pipeline as HTML pages. Inline Markdown images are downloaded after discovery (including SVG conversion), while ordinary root-relative Markdown links become absolute browser-navigation URLs.
• Tables with list/<br> cells are flattened to valid single-line GFM rows (cell content joined with <br>) instead of collapsing into an unrenderable multi-line blob.
• Asset hygiene drops tracking pixels, favicons, share badges, and UI icons by content sniffing (a payload that is not really an image is rejected), a byte floor, and a pixel-dimension floor. Genuine figures are kept; assets with unknown dimensions are never dropped on that basis alone. If ordinary Fetch and curl downloads both fail or return a non-image payload for a validated GitHub raw-image URL, an installed git can recover that one public blob by initializing a temporary bare repository and performing an exact shallow, blob-filtered fetch; failures still leave the absolute URL visible for downstream review.
• Readable Markdown shaping keeps one visible top-level source title heading, removes empty-text [](url) links, strips self-referential links, and preserves Markdown output that reads cleanly on its own.
• MDX/JSX wrappers (<CodeGroup>, <Tabs>, <Card>, theme={null}, …) are stripped from docs served as text/markdown, keeping the inner content.
• Reference failures remain visible in references/references.json and an optional failure stub, but the parent keeps its original absolute link until a reference is written successfully. If a localized reference is pruned later, restore or remove the inbound link and remove its page, assets, and manifest entry as one atomic change.

Known limitations

• No JavaScript execution. The fetch returns the pre-render HTML (with a curl fallback). All semantic tab panels already present in that response are preserved, but states injected client-side after load -- e.g. leaderboards rendered from JSON, lazily-loaded citation widgets, or absent carousel panels -- cannot be captured and may appear as a placeholder or only their initial state. Adding a headless browser is intentionally out of scope to keep the tool lightweight.
• GitHub raw recovery is optional and deliberately narrow. It requires an installed git, accepts only validated public GitHub raw URLs with a single-segment branch/tag ref or a full commit SHA, rejects explicit slash-containing ref forms, symlinks, and Git LFS pointer payloads, and never substitutes for other failed asset hosts. Git subprocesses use isolated configuration, share one bounded recovery deadline, and are attempted at most twice per page.

Development

This repo uses Bun for dependency management, building, and tests.

bun install
bun run check

The published package is @sanbaiw/web-source-bundler and exposes one CLI bin: web-source-bundler.