@nbafrank/pi-webfetch

Keyless web search and fetch tools for pi (an @earendil-works coding agent).

Defaults need no API key, no login, no Ollama:

• Search → DuckDuckGo (HTML parser, with a JSON fallback).
• Fetch → Jina Reader (r.jina.ai), with a raw GET + HTML→text fallback.

Higher-quality providers (SearXNG, Tavily, Brave) are used automatically when their config is present. Everything is implemented on Node built-ins only (fetch, AbortSignal, URL, URLSearchParams) so the core runs under node --experimental-strip-types and is fully unit-testable with zero deps.

Architecture

src/core.ts        dependency-free engine  (search / fetch / providers / parsing)
src/extension.ts   pi layer               (registers web_search + web_fetch tools)
scripts/smoke.ts   offline + live smoke test
tsconfig.json      strict typecheck of core + smoke (no external deps)
tsconfig.extension.json   typecheck the extension against pi's real types

core.ts has no pi/typebox imports, which is what lets it be tested and reused outside pi. extension.ts is the only place that imports @earendil-works/pi-coding-agent and typebox.

Providers

Search (best-then-fallback, in this order)

| provider | when enabled | key required | |-------------------|-------------------------------------|--------------| | searxng | SEARCH_SEARXNG_URL set | no* | | tavily | TAVILY_API_KEY set | yes | | brave | BRAVE_API_KEY set | yes | | duckduckgo-html | always (default) | no | | duckduckgo-json | always (fallback when HTML fails) | no |

* SearXNG format=json may need your instance to allow JSON / a shared secret.

Fetch

| provider | when used | key required | |----------|----------------------|--------------| | jina | default | no | | direct | fallback when Jina unavailable or fails | no |

Environment variables

| var | purpose | default | |-----|---------|---------| | SEARCH_SEARXNG_URL | SearXNG base URL | — (disabled) | | SEARCH_SEARXNG_SECRET | SearXNG anonymous-search secret | — | | TAVILY_API_KEY | Tavily key | — (disabled) | | BRAVE_API_KEY | Brave Search key (X-Subscription-Token) | — (disabled) | | SEARCH_PROVIDER | force a search provider (duckduckgo\|searxng\|tavily\|brave) | auto | | FETCH_PROVIDER | force a fetch provider (jina\|direct) | auto | | SEARCH_TIMEOUT_MS | per-request timeout (ms) | 15000 | | FETCH_TIMEOUT_MS | per-request timeout (ms) | 30000 | | MAX_FETCH_CHARS | max chars returned by fetch | 20000 |

Retry & backoff

httpGet retries transient failures with capped exponential backoff:

• network errors and HTTP 429/5xx → retry, delay retryBaseMs * 2**attempt (default base 400ms, default retries=3 → 3 attempts total).
• honors the Retry-After response header, capped at 8 s.
• abortable: a cancelled AbortSignal stops mid-backoff and surfaces an abort error.
• hard errors (404, 422, auth failures) and JSON-parse failures do not retry — they fail fast so the orchestrator falls through to the next provider.

Core API

import { search, fetch, type SearchItem, type FetchResult } from "./core.ts";

const results: SearchItem[] = await search({
   query: "node fetch api",
   maxResults: 5,             // default 5
   provider: "searxng",       // optional: "duckduckgo" | "searxng" | "tavily" | "brave"
   signal,                    // optional AbortSignal
   onLog: (stage, ok) => {},  // optional progress: "search:brave" | "fetch:jina" ...
});
// SearchItem = { title: string; url: string; content: string }

const page: FetchResult = await fetch({
   url: "https://example.com",
   provider: "jina",          // optional: "jina" | "direct"
   signal,
});
// FetchResult = { title: string; content: string; links: string[] }

Search results are de-duplicated by URL (deduped by origin+path, self-hosts like duckduckgo.com filtered out) and snippets are bounded per result to avoid cross-block contamination.

pi extension (tools)

src/extension.ts registers two model-facing tools that wrap the core:

• web_search — { query, max?, provider? } → numbered results (title, url, bounded snippet).
• web_fetch — { url, provider? } → page title + readable text + linked pages.

Both pass through the agent's abort signal, cap their context size, and return graceful "failed" text on error instead of throwing.

Install

From npm (published package):

# user-global    (all pi sessions)
pi install npm:@nbafrank/pi-webfetch

# or project-local    (this repo only) → writes to .pi/settings.json
pi install -l npm:@nbafrank/pi-webfetch

# try it for the current run only, without installing
pi -e npm:@nbafrank/pi-webfetch

pi installs the package (and runs its npm install for dependencies), then auto-discovers its pi.extensions on the next session. Manage it with pi list, pi update npm:@nbafrank/pi-webfetch, or pi remove npm:@nbafrank/pi-webfetch.

Or, develop locally without publishing — drop / symlink src/extension.ts into a pi extensions directory for auto-discovery + hot-reload (pi supports both .ts and .js):

# user-global  (all pi sessions)
ln -s "$(pwd)/src/extension.ts" ~/.pi/agent/extensions/webfetch.ts

# or project-local  (this repo only)
ln -s "$(pwd)/src/extension.ts" .pi/extensions/webfetch.ts

That's it for running it: pi's extension loader (jiti) compiles the .ts on the fly and resolves @earendil-works/* + typebox from the running pi install automatically, so you do not need npm install or the link:pi symlinks to use the tools — the link step below is only for a local type-check.

Build, typecheck, test

Requires Node ≥ 22.6 (--experimental-strip-types).

npm install                         # devDeps: typescript, @types/node

npm run typecheck                   # strict TS check of core + smoke (no external deps)
npm run typecheck:extension         # type-checks the extension vs pi's REAL types
npm run smoke                       # offline assertions + live keyless DDG search + fetch

typecheck:extension needs the @earendil-works/* + typebox types. They are not on npm (they ship inside pi-coding-agent), so a pre-hook (pretypecheck:extension -> scripts/link-pi.mjs, also npm run link:pi) links them from your global pi install. npm install prunes these extraneous symlinks, so the hook rebuilds them automatically before the check.

npm i -g @earendil-works/pi-coding-agent    # if pi isn't installed globally yet
npm run link:pi                              # (re)link types from the global install