pi-web-md

Secure, keyless web search and Markdown-first page fetching for the Pi coding agent.

It gives Pi three tools:

| Tool | Purpose | | --- | --- | | web_search | Search the public web with keyless providers | | web_fetch | Fetch one public URL as clean Markdown | | batch_web_fetch | Fetch up to ten URLs concurrently |

Install

pi install npm:pi-web-md

Also valid:

pi install git:github.com/dodo-reach/pi-web-md
pi install https://github.com/dodo-reach/pi-web-md
pi install /path/to/pi-web-md
pi -e git:github.com/dodo-reach/pi-web-md

After install, restart Pi or run /reload.

Usage

Ask Pi to use the tools, or call them directly:

web_search({ query: "Node.js fetch documentation", max_results: 5 })
web_fetch({ url: "https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API" })
web_fetch({ url: "https://example.com/docs", selector: "main", focus: "installation" })
batch_web_fetch({ requests: [{ url: "https://example.com" }, { url: "https://example.org" }] })

web_fetch requests native Markdown first. When a server returns HTML, it uses Readability and Turndown with GFM support. Relative links become absolute by default. Large output is saved to a temporary Markdown file instead of filling the model context.

Search uses keyless Exa with DuckDuckGo and Bing fallbacks. No account or API key is required. Keyless providers can apply rate limits or request bot verification, so search availability is not guaranteed.

Fetch options

• selector: extract all matching HTML elements before conversion
• focus: return the most relevant Markdown section or text window
• abs_links: make relative links and images absolute; default true
• output_mode: auto, inline, or file
• timeout_ms: total timeout, including DNS

Batch requests accept the same URL, selector, focus, link, output, and timeout options. The batch tool preserves request order and limits concurrency.

Security and limits

• Allows only public HTTP and HTTPS URLs
• Blocks loopback, private, link-local, metadata, reserved, mapped IPv4, and NAT64 private targets
• Revalidates every redirect and DNS lookup
• Rejects embedded URL credentials
• Limits redirects to five
• Limits each response to 10 MB while streaming
• Rejects PDFs, images, and binary responses
• Applies one total request timeout

This package performs static HTTP fetching. It does not execute JavaScript, open authenticated pages, bypass paywalls, or solve anti-bot challenges.

Independent benchmarks

Reproduction scripts and full results are in benchmarks/README.md.

Highlights from public human-annotated datasets:

• WCXB: 0 extraction errors across 2,008 pages
• WCXB rendered-text F1: 0.7426 development, 0.7951 test
• WebMainBench 545 no-LLM overall: 0.4156
• Mean extraction time: 20–31 ms/page on the recorded Apple M4 test system

The benchmark results are local reproducible runs, not official leaderboard placements. Dataset commits, normalization, hardware, and limitations are reported with the results.

Package layout

pi-web-md/
├── extensions/
│   └── web-md.ts
├── src/
├── test/
├── benchmarks/
├── package.json
├── README.md
└── LICENSE

Development

npm install
npm test
npm run typecheck
npm run pack:check

Requirements

• Node.js 20 or later
• Pi coding agent with extension support
• Public network access for live search and fetch

Credits

Adapted from @mrclrchtr/supi-web (MIT). Search and safe-fetch behavior also use ideas from @bytetrue/pi-web-search (MIT). See THIRD_PARTY_NOTICES.md for license notices.

License

MIT