This is fastagent's fork of pi-web-access, published as @fastagent-sh/pi-web-access. It differs from upstream in one fix: each extension instance keeps its own session's results, background fetches and active flag, so a host that serves several conversations from one process (one instance per session) does not clear one conversation's results when another starts or ends (upstream PR #521). A version is the upstream version it is built from; a fix of ours before the next upstream release takes the next patch number. The demo video and banner are left out of the package (6.4 MB). Once upstream ships the fix, fastagent goes back to pi-web-access.

To release: bump version on the fastagent branch, then publish a GitHub Release tagged fastagent-v<version> from that commit. .github/workflows/publish-fastagent.yml tests it and publishes with Trusted Publishing after the npm environment approves.

Pi Web Access

Web search, content extraction, and video understanding for Pi agent. OpenAI/Codex search, zero-config Exa search, Brave, Parallel, TinyFish, Search1API, Searchinfinity, Querit, Tavily, Firecrawl, Jina, SERPdive, Kagi, Bocha, Ollama, AnySearch, XCrawl, Valyu, xAI/Grok, Mistral, Bright Data SERP, SerpBase, SerpApi, Serper, explicit-only Serply, self-hosted SearXNG, degoog metasearch, self-hosted Crawl4AI extraction, keyless DuckDuckGo, keyless Keenable, optional browser-cookie Gemini Web, Kimi Code Plan search, or bring your own API keys.

https://github.com/user-attachments/assets/cac6a17a-1eeb-4dde-9818-cdf85d8ea98f

Why Pi Web Access

Zero Config — Works out of the box with Exa MCP (no API key needed). If you're signed into Pi with a Codex subscription, OpenAI web search can reuse that auth. An active Kimi Code Plan signed in through /login kimi-coding enables explicit Kimi search without a separate Open Platform key. Add API keys or endpoints for OpenAI, Brave, Parallel, TinyFish, Search1API, Searchinfinity, Querit, Tavily, Firecrawl, Jina, SERPdive, Kagi, Bocha, Ollama, Valyu, SerpBase, SerpApi, Serper, explicit-only Serply, Exa, Perplexity, Gemini API, or Mistral for more control; configure a self-hosted SearXNG endpoint or a degoog metasearch instance (public https://degoog.org by default) for private search; or opt into browser-cookie access for Gemini Web.

Video Understanding — Point it at a YouTube video or local screen recording and ask questions about what's on screen. Full transcripts, visual descriptions, and frame extraction at exact timestamps.

Smart Fallbacks — Every capability has a fallback chain. Search tries configured SearXNG first for local/private search. When the active Pi model uses a ChatGPT subscription (openai-codex, or openai with Sign in with ChatGPT), it then tries subscription-backed OpenAI search. Otherwise it tries Exa before OpenAI, then Brave, Parallel, TinyFish, Search1API, Searchinfinity, Querit, Tavily, Firecrawl, Jina, SERPdive, Kagi, Bocha, Ollama, Perplexity, Gemini API, and Gemini Web when browser cookies are enabled. YouTube tries Gemini Web when enabled, then API, then Perplexity. Blocked pages try configured self-hosted Firecrawl or Crawl4AI first. Third-party hosted page fetchers require explicit fetchRouting.allowRemoteHostedProviders opt-in for remote HTTP(S) targets.

GitHub Cloning — GitHub URLs are cloned locally instead of scraped. The agent gets real file contents and a local path to explore, not rendered HTML.

Other Agents — The search and fetch tools also run as a local MCP server for Claude Code, Codex, Cursor, and other MCP clients. See Use from other agents (MCP).

Install

pi install npm:pi-web-access

Works immediately with no API keys — Exa MCP provides zero-config search. If Pi has Codex auth from /login, OpenAI search can also work without a separate key. For more providers or direct API access, add keys to ~/.pi/agent/web-search.json:

{
  "openaiApiKey": "sk-...",
  "braveApiKey": "BSA_...",
  "exaApiKey": "exa-...",
  "tinyfishApiKey": "sk-tinyfish-...",
  "search1apiApiKey": "...",
  "searchinfinityApiKey": "...",
  "queritApiKey": "...",
  "jinaApiKey": "jina_...",
  "bochaApiKey": "sk-...",
  "perplexityApiKey": "pplx-...",
  "geminiApiKey": "AIza...",
  "mistralApiKey": "..."
}

In auto mode (default), web_search tries a configured SearXNG endpoint first for local/private search. When the active Pi model uses a ChatGPT subscription (openai-codex, or openai with Sign in with ChatGPT), it then tries subscription-backed OpenAI search. Otherwise it tries Exa (direct API if keyed, MCP if not) before OpenAI, then Brave, Parallel, TinyFish, Search1API, Searchinfinity, Querit, Tavily, Firecrawl, Jina, SERPdive, Perplexity, Gemini API, and Gemini Web when browser-cookie access is enabled. Exa handles search; curator summary drafts are generated separately by the configured Pi summary model, defaulting to Claude Haiku, Codex Luna, Codex Terra, Gemini 3.6 Flash, GPT-5 mini, then DeepSeek V4 Flash when available. Slow summary drafts fall back to a deterministic result summary after a bounded deadline.

For a third-party Responses-compatible gateway, set openaiResponsesUrl to its full Responses endpoint, or opt into reusing the selected Pi provider's base URL as described below. Without either opt-in, Pi credentials with a custom baseUrl are refused before sending a request, and availability checks mark OpenAI unavailable without blocking other providers. The auth-resolved base URL takes precedence over the model's; gateway Responses/web_search support is not inferred. An explicit endpoint overrides this guard, including an explicit official endpoint. Existing Codex subscription endpoint selection is preserved when provider URL reuse is not active.

Reuse the selected Pi provider URL

Set openaiUseProviderBaseUrl: true to derive the search endpoint from the Pi provider whose credentials were selected. This avoids maintaining the same gateway address in both models.json and web-search.json. The default is false.

{
  "openaiSearchProviders": ["my-openai-provider"],
  "openaiUseProviderBaseUrl": true
}

If openaiResponsesUrl is supplied, this switch is ignored and the existing explicit-endpoint/auth rules apply. Remove that field to use the provider address. Otherwise the switch must be a boolean. The selected provider's auth-resolved baseUrl takes precedence over its model's baseUrl; credentials and URL are resolved together for each search. Candidates with missing or invalid resolved base URLs are skipped in configured order, keeping each candidate's credentials bound to its own URL. Reload Pi after changing its provider configuration.

Only absolute HTTP(S) URLs are accepted. Preserve the origin, port, prefix, and query parameters, and complete the Responses path as follows:

| Provider base URL | Derived Responses endpoint | | --- | --- | | https://gateway.example.com | https://gateway.example.com/v1/responses | | https://gateway.example.com/v1 | https://gateway.example.com/v1/responses | | https://gateway.example.com/team/v1/ | https://gateway.example.com/team/v1/responses | | https://gateway.example.com/v1/responses | https://gateway.example.com/v1/responses |

Codex auth with the official https://chatgpt.com/backend-api base uses /backend-api/codex/responses. When reusing a custom provider URL with Codex credentials, retain its destination and required account headers rather than redirecting to the official host. If openaiUseAlphaSearch is also true, replace the derived /responses suffix with /alpha/search.

If no candidate has usable Pi credentials and a valid provider base URL, this mode fails closed and marks OpenAI unavailable; it does not fall back to the official endpoint or a standalone API key. API-key-only configurations can use openaiResponsesUrl instead. This switch applies to independent OpenAI provider selection; searchRouting.useCurrentModel retains its existing official-endpoint selection and eligibility rules.

Optional OpenAI standalone search

Set openaiUseAlphaSearch: true to use the independent Codex alpha/search protocol instead of Responses-hosted web_search. The default is false; omitting the flag keeps existing Responses behavior. Non-boolean values are rejected.

{
  "openaiUseAlphaSearch": true,
  "openaiResponsesUrl": "https://gateway.example.com/v1/responses",
  "openaiSearchProviders": ["my-openai-provider"]
}

The selected Responses endpoint must end in /responses (an optional trailing slash is accepted). Its path suffix becomes /alpha/search, preserving the host, port, prefix, and query parameters. For example, /v1/responses becomes /v1/alpha/search. By default, Codex subscription auth selects https://chatgpt.com/backend-api/codex/alpha/search; with provider URL reuse enabled, it follows the resolved provider endpoint instead. Existing credential selection, search-model overrides, custom-base-URL safeguards when URL reuse is disabled, and official current-model eligibility rules still apply. This flag does not make an arbitrary gateway support standalone search.

Standalone requests use id, model, and commands.search_query, not a Responses tool call. Plaintext output and structured text_result sources are returned without another model-generated summary. Encrypted-only responses are rejected. numResults defaults to 5 and caps the deduplicated source list at up to 20; recency maps to 1/7/30/365 days, and positive domain filters map to domains. Excluded domains (-example.com) are explicitly unsupported in this mode rather than silently ignored.

Missing endpoint responses (HTTP 404/405/501) and excluded-domain requests follow the existing unsupported fallback policy. Other HTTP errors, cancellation, proxy transport, and the 60-second request deadline retain their existing behavior. There is no automatic retry through Responses. To permit another provider, configure searchRouting.fallbackOn accordingly; explicit provider: "openai" remains strict. Reload Pi after changing configuration.

To route automatic searches through the active Pi model, configure an ordered route without a top-level provider:

{
  "searchRouting": {
    "providers": ["openai", "tavily"],
    "useCurrentModel": true,
    "fallbackOn": ["unsupported", "transient", "quota", "network", "invalid-response"]
  }
}

With useCurrentModel: true, the automatic openai step uses Hosted web_search (or standalone search when openaiUseAlphaSearch is enabled) when the active model is a GPT model backed by an official OpenAI Responses endpoint: openai/openai-responses on HTTPS api.openai.com, or openai-codex/openai-codex-responses on the official ChatGPT Codex endpoint. Third-party gateways, Azure, and other models continue to the next route entry. A tool-level provider or top-level provider remains an explicit override; provider: "openai" keeps the existing independent OpenAI/Codex search-model behavior.

For sandboxed or corporate networks where only the proxy can resolve hostnames, set ssrf.trustEnvProxy to true to skip local DNS preflight for proxied hostnames:

{
  "ssrf": {
    "trustEnvProxy": true
  }
}

This is an opt-in DNS-preflight adjustment, not proxy transport configuration. It applies to requests sent through the proxy set in web-search.json (HTTP(S), socks5h, or socks4a, including a per-call proxy with the same value). When no proxy is configured, it applies to hostnames covered by HTTP_PROXY, HTTPS_PROXY, or ALL_PROXY, for sandboxes that route outbound traffic through those variables themselves; the extension does not route requests through environment proxies. A per-call proxy that differs from the configured one still gets local DNS validation. NO_PROXY hosts still undergo DNS validation, and localhost or literal private IP targets remain blocked.

Optional dependencies for video frame extraction:

brew install ffmpeg   # frame extraction, video thumbnails, local video duration
brew install yt-dlp   # YouTube stream URLs for frame extraction

Without these, video content analysis (transcripts, visual descriptions via Gemini) still works. The binaries are only needed for extracting individual frames as images.

System dependency for curator browser launch on Linux:

| Package | Purpose | Install | |---------|---------|---------| | xdg-utils | Opens the curator UI in your default browser | Debian/Ubuntu: sudo apt install -y xdg-utils · Fedora/RHEL: sudo dnf install -y xdg-utils · Arch: sudo pacman -S xdg-utils |

Without xdg-utils, the curator URL is printed to the tool output so you can copy it into a browser.

Requires Pi v0.37.3+.

Quick Start

// Search the web
web_search({ query: "TypeScript best practices 2025" })

// Fetch a page
fetch_content({ url: "https://docs.example.com/guide" })

// Clone a GitHub repo
fetch_content({ url: "https://github.com/owner/repo" })

// Understand a YouTube video
fetch_content({ url: "https://youtube.com/watch?v=abc", prompt: "What libraries are shown?" })

// Analyze a screen recording
fetch_content({ url: "/path/to/recording.mp4", prompt: "What error appears on screen?" })

Tools

web_search

Search the web via OpenAI, Brave, Parallel, TinyFish, Search1API, Searchinfinity, Querit, Tavily, Firecrawl, Jina, SERPdive, Kagi, Bocha, Ollama, AnySearch, XCrawl, Valyu, xAI, Mistral, Bright Data SERP, SerpBase, SerpApi, Serper, Serply, You.com, keyless Keenable, self-hosted SearXNG, explicit-only degoog, keyless DuckDuckGo, Exa, Perplexity AI, Gemini, or Kimi. The default none workflow makes no summary-generation model call and opens no curator/browser: it returns bounded raw results, identifies the provider used for each query, and stores the full results for retrieval by responseId. The selected search provider may itself be model-backed.

web_search({ query: "rust async programming" })
web_search({ queries: ["query 1", "query 2"] })
web_search({ query: "latest news", numResults: 10, recencyFilter: "week" })
web_search({ query: "...", domainFilter: ["github.com"] })
web_search({ query: "...", provider: "openai" })
web_search({ query: "...", provider: "kimi" })
web_search({ query: "...", provider: "mistral" })
web_search({ query: "...", provider: "all" })
web_search({ query: "...", includeContent: true })
web_search({ queries: ["query 1", "query 2"], workflow: "none" })
web_search({ queries: ["query 1", "query 2"], workflow: "summary-review" })
web_search({ queries: ["query 1", "query 2"], workflow: "auto-summary" })

| Parameter | Description | | ----------- | ------------- | | query / queries | Single query or batch of queries | | numResults | Results per query (default: 5, max: 20) | | recencyFilter | day, week, month, or year | | domainFilter | Limit to domains (prefix with - to exclude) | | category | Exa only: restrict results to a category such as news or research paper; other providers ignore it. Without an API key, if Exa's filtered search is unavailable, the category is added to the query instead | | provider | Configured provider when omitted or set to auto; all searches every eligible provider except Parallel MCP, degoog, DuckDuckGo, Kimi, AnySearch, XCrawl, Valyu, xAI, Mistral, Bright Data, SerpBase, SerpApi, Serper, Serply, You.com, Baizhi, Z.ai, and Keenable simultaneously; otherwise openai, brave, parallel, parallel-mcp, tinyfish, search1api, searchinfinity, querit, tavily, you, firecrawl, jina, serpdive, kagi, bocha, ollama, anysearch, xcrawl, valyu, xai, mistral, brightdata, serpbase, serpapi, serper, serply, baizhi, zai, keenable, searxng, degoog, duckduckgo, exa, perplexity, gemini, or kimi (auto-selects when no provider or routing is configured; Parallel MCP, degoog, DuckDuckGo, Kimi, AnySearch, XCrawl, Valyu, xAI, Mistral, Bright Data, SerpBase, SerpApi, Serper, Serply, You.com, Baizhi, Z.ai, and Keenable are explicit-only) | | includeContent | Fetch full page content from sources in background | | workflow | none (skip curator; fresh-install default), summary-review (open curator and auto-generate a summary draft), or auto-summary (generate a summary without opening the curator) |

Batch searches run up to three queries concurrently. Provider routing and fallback within each query remain sequential. Brave requests share an in-process queue and honor the rate-limit buckets returned in X-RateLimit-*; exhausted short windows delay the next request with a small safety margin, while long quota resets fail fast and keep the advertised cooldown active for later queued calls. HTTP 429 responses are retried once after the advertised Retry-After or rate-limit reset. Brave's 30-second search deadline covers queueing, cooldown waits, and both request attempts. The queue coordinates one Pi process; separate Pi processes still rely on Brave's 429 response.

fetch_content

Fetch URL(s) as readable markdown, exact textual HTTP bodies, direct images, or page-grounded answers. Automatically detects and handles GitHub repos, GitHub PRs and issues, YouTube videos, PDFs, local video files, images, and regular web pages.

fetch_content({ url: "https://example.com/article" })
fetch_content({ urls: ["url1", "url2", "url3"] })
fetch_content({ url: "https://github.com/owner/repo" })
fetch_content({ url: "https://github.com/owner/repo/pull/123#discussion_r456" })
fetch_content({ url: "https://youtube.com/watch?v=abc", prompt: "What libraries are shown?" })
fetch_content({ url: "/path/to/recording.mp4", prompt: "What error appears on screen?" })
fetch_content({ url: "https://youtube.com/watch?v=abc", timestamp: "23:41-25:00", frames: 4 })
fetch_content({ url: "https://example.com/api", mode: "raw" })
fetch_content({ url: "https://example.com/guide", mode: "answer", prompt: "What are the installation steps?" })
fetch_content({ url: "https://example.com/account", auth: "work", mode: "raw" })
fetch_content({ url: "https://example.com/diagram.png" })

| Parameter | Description | | ----------- | ------------- | | url / urls | Single URL/path or multiple URLs | | prompt | Question for video analysis, or the page-local question required by mode: "answer" | | mode | readable (default), raw for exact textual HTTP bodies, or answer for a grounded answer from fetched content | | answerModel | Optional provider/model-id override for answer mode; defaults to the configured fetch.answerProvider + fetch.answerModel pair, or the current enabled Pi model when no pair is configured | | timestamp | Extract frame(s) — single ("23:41"), range ("23:41-25:00"), or seconds ("85") | | frames | Number of frames to extract (max 12) | | forceClone | Clone GitHub repos that exceed the 350MB size threshold |

For a standing answer-mode model, set both fetch.answerProvider and fetch.answerModel in web-search.json; a per-call answerModel takes precedence. Configured answer defaults are opt-in and can send fetched page text to a different provider/model, which may change privacy and cost behavior.

Thanks to @linuxtextadventurer for PR #328.

get_search_content

Retrieve stored content from previous searches or fetches. Search provider answers and every result remain available in full and can be paged with offset and limit or searched with findText. Fetched URL content is stored in full in a private web-search-cache directory under the Pi config directory, not in the session JSONL. This includes fetch_content answer mode, which stores the original page content. The cache has a one-hour lifetime and fixed limits of 128 entries and 128 MiB; when either limit is reached, the oldest entries are removed first. Concurrent Pi processes share this cache and its limits. To give a process its own cache without moving web-search.json, set PI_WEB_ACCESS_CACHE_ROOT to a directory; the cache then lives in web-search-cache inside it, and a session resumed without the same value cannot read content cached there. On macOS and Linux the cache directory and files are kept at permissions 0700 and 0600, respectively. Use findText to locate bounded matching passages without paging through a large page, or use offset and limit to retrieve slices intentionally.

get_search_content({ responseId: "abc123", urlIndex: 0 })
get_search_content({ responseId: "abc123", url: "https://...", offset: 30000 })
get_search_content({ responseId: "abc123", query: "original query" })
get_search_content({ responseId: "abc123", queryIndex: 0, offset: 30000 })
get_search_content({ responseId: "abc123", urlIndex: 0, findText: "installation" })
get_search_content({ responseId: "abc123", urlIndex: 0, findText: ["timeout", "retry"], findMode: "fuzzy" })

findMode supports exact, case-insensitive (default), and fuzzy. Finder output is capped at 20,000 characters with match counts and nearby context. Fitting responses retain their existing format and document order. Overflow responses identify queries as Q1, Q2, and so on, list each full query once, and reserve a representative excerpt for each query whose discovered match span fits the output budget. Queries with oversized spans are listed as having no representative excerpt. findText cannot be combined with offset or limit. The default limit and maximum permitted limit use maxInlineContentChars; search-page continuation guidance shares that overall output budget and may reduce the returned stored-content characters.

source_check

Gather evidence for a claim and return a machine-readable artifact with exact passage citations for manual semantic review. Search results are deduplicated and capped at 20 sources; fetchContent fetches at most 5 pages, while stored and retrieved content remains subject to the configured maxInlineContentChars offset/limit bounds.

source_check({ claim: "The API supports streaming responses" })
source_check({
  claim: "The API supports streaming responses",
  queries: ["API streaming responses documentation", "API streaming limitations"],
  fetchContent: true,
  domainFilter: ["docs.example.com", "-old.example.com"]
})

The artifact preserves the supported, contradicted, unclear, or missing-evidence claim status schema, source quality hints, SHA-256 content hashes, and passage IDs with exact source offsets. It does not infer semantic support or contradiction automatically: retrieved passages produce unclear for manual review, while no passages produce missing-evidence. Search and fetch errors remain in the artifact instead of being silently discarded. Artifacts are stored with the session and retrieved through get_search_content using the returned responseId; paged artifact responses are JSON slices, so request the next offset when needed.

In codemode scripts

Scripts run by Pi's codemode tool (Pi 1.0 or later) get data instead of the text the model sees. await tools.web_search(...) resolves to { responseId, fetchId, queries }, where each query has its answer, error, provider, and results (title, url, snippet). await tools.fetch_content(...) resolves to { responseId, urls }, where each URL has its full content, not limited by maxInlineContentChars, and its error. Failed searches and fetches still resolve to this data, with the reason in error; invalid arguments reject. A script's web_search uses workflow: "none" unless it passes a workflow, so a configured curator does not open, and with includeContent it waits for the pages, readable through fetchId, instead of fetching them in the background. A script that passes workflow: "summary-review" gets the curator and the background fetch, as the model does.

Use from other agents (MCP)

Other agents, such as Claude Code, Codex, Cursor, or Executor, can use web_search, fetch_content, get_search_content, and source_check through a local stdio MCP server. It needs Node.js 22.19 or later and no Pi install:

npx -y pi-web-access

Add it to an mcpServers config (Claude Code, Cursor, and similar clients; Codex takes the same command, args, and env in its config.toml):

{
  "mcpServers": {
    "pi-web-access": {
      "command": "npx",
      "args": ["-y", "pi-web-access"],
      "env": { "BRAVE_API_KEY": "BSA_..." }
    }
  }
}

• The server reads the same web-search.json and provider environment variables as the extension. It lists only the tools enabled there, under their default names; toolNames renames do not apply.
• Pi-only features are not available: the curator and summaries (a configured workflow is ignored), Kimi search, OpenAI search through ChatGPT sign-in or the current Pi model, fetch_content answer mode and video prompts or frames, and direct image fetches. Explicit requests for them return a tool error rather than a fallback, and automatic provider selection only uses providers that work outside Pi.
• includeContent waits for the page fetch before web_search returns.
• The server keeps the 50 most recent results in memory for get_search_content; they are gone when it exits. Restart the server to pick up config changes.
• Install with npm's default settings or --legacy-peer-deps. --omit=peer leaves out zod, which the MCP SDK needs.

Capabilities

GitHub repos

GitHub URLs are cloned locally instead of scraped. The agent gets real file contents and a local path to explore with read and bash. Root URLs return the repo tree + README, /tree/ paths return directory listings, /blob/ paths return file contents.

Repos over 350MB get a lightweight API-based view instead of a full clone (override with forceClone: true). Commit SHA URLs are handled via the API. Clones are cached for the session and wiped on session change. Private repos require the gh CLI. Set githubClone.enabled to false to skip this GitHub-specific clone/API handling; fetch_content remains available, so the URL can continue through the normal HTTP extraction path.

Pull request and issue URLs are rendered as one priority-ordered markdown document instead of scraped HTML. PR views include status, body, checks when gh supports them, review verdicts, linked references, files, commits, conversation comments, review thread comments, truncation markers, and escalation commands. Issue views include state, metadata, body, linked closing PRs when available, and comments. Comment anchors such as #issuecomment-... and #discussion_r... are forced inline. The full rendered document is stored for get_search_content offsets and findText.

fetch_content uses gh pr view or gh issue view first with prompts disabled. It retries with a smaller field set when an older gh does not know a requested field. Public unauthenticated REST is used as a bounded fallback under the same fetchContent.domainPolicy and SSRF rules; REST cannot include checks. Set githubPrIssue.enabled to false to skip PR/issue specialization and keep normal HTTP extraction.

YouTube videos

YouTube URLs are processed via Gemini for full video understanding — visual descriptions, transcripts with timestamps, and chapter markers. Pass a prompt to ask specific questions about the video. Results include the video thumbnail so the agent gets visual context alongside the transcript.

Fallback: Gemini Web when browser cookies are enabled → Gemini API → Perplexity (text summary only). Handles all URL formats: /watch?v=, youtu.be/, /shorts/, /live/, /embed/, /v/.

Local video files

Pass a file path (/, ./, ../, or file:// prefix) to analyze video content via Gemini. Supports MP4, MOV, WebM, AVI, and other common formats up to 50MB for Gemini analysis. Pass a prompt to ask about specific content. If ffmpeg is installed, a thumbnail frame is included alongside the analysis. Timestamp/frame extraction uses ffmpeg directly and can still operate on larger local files.

Fallback: Gemini API (Files API upload) → Gemini Web when browser cookies are enabled.

Video frame extraction

Use timestamp and/or frames on any YouTube URL or local video file to extract visual frames as images.

fetch_content({ url: "...", timestamp: "23:41" })                       // single frame
fetch_content({ url: "...", timestamp: "23:41-25:00" })                 // range, 6 frames
fetch_content({ url: "...", timestamp: "23:41-25:00", frames: 3 })      // range, custom count
fetch_content({ url: "...", timestamp: "23:41", frames: 5 })            // 5 frames at 5s intervals
fetch_content({ url: "...", frames: 6 })                                // sample whole video

Requires ffmpeg (and yt-dlp for YouTube). Timestamps accept H:MM:SS, MM:SS, or bare seconds.

PDFs

With mode: "answer", the answer model receives the extracted PDF Markdown rather than the saved-file notice. The Markdown file is still saved, and the original extracted content remains available through get_search_content; ordinary readable-mode PDF fetches continue to return the file path.

PDF URLs are converted to Markdown and saved under the temporary pi-web-pdf directory by default so the agent can read specific sections without loading the full document into context. Three engines are available, selected with pdf.provider ("auto" is the default):

| Provider | Engine | Trade-offs | | --- | --- | --- | | datalab | Datalab hosted conversion (Marker) | Deterministic layout-aware output — tables, multi-column reading order, headings, math; accurate mode handles scanned pages; may return a parse_quality_score; requires a Datalab key, billed per page with a free monthly credit | | gemini | Gemini API (vision LLM) | Best on scanned/complex pages; LLM transcription can occasionally drift or truncate; requires a Gemini key | | unpdf | Local pdf.js text extraction | Free, offline, no key; flattened text only — no layout, no tables, no OCR |

auto order: Datalab (when a key is configured) → Gemini (when a key is configured) → local unpdf. Datalab runs first for layout-aware conversion. If its request fails — including after free-tier credit is exhausted — the chain continues to Gemini, then unpdf, automatically. Setting pdf.provider to gemini, datalab, or unpdf pins that engine and skips the other remote tiers (an explicit engine still falls back to unpdf when it errors, except for credential/config errors and caller cancellation). No Datalab key means the datalab tier is simply skipped — behavior is unchanged for existing users.

Why Datalab. The hosted converter uses a dedicated extraction engine (Marker) intended to retain document structure such as tables, multi-column reading order, headings, links, and math, where local unpdf extraction only yields flattened text. It is deterministic rather than LLM-based. Completed responses may include a parse_quality_score (0–5) for optional quality gating. Pricing is per processed page: fast / balanced $4 / 1,000 pages; accurate $10 / 1,000 pages. The free tier gives a $10 monthly credit (personal email; $20 with a work email) at 25 requests/minute — roughly 2,500 pages/month free in fast mode or 1,000 in accurate mode. Processing defaults to the US region. EU data residency uses 1.25× usage; opt in with DATALAB_PROCESSING_LOCATION=eu.

Configure Datalab via the web-search config:

{
  "datalabApiKey": "$DATALAB_API_KEY",
  "pdf": {
    "maxSizeMB": 20,
    "maxPages": 100,
    "provider": "auto",      // "auto" | "gemini" | "datalab" | "unpdf"
    "datalabMode": "balanced", // "fast" | "balanced" | "accurate"
    "datalabTimeoutMs": 120000
  }
}

Env vars: DATALAB_API_KEY (or datalabApiKey in config), DATALAB_PROCESSING_LOCATION (us default; eu enables EU data residency at 1.25× usage), DATALAB_MODE (fast / balanced / accurate), and DATALAB_API_BASE (custom gateway). pdf.datalabMode overrides DATALAB_MODE. The default datalabTimeoutMs is 120s and is capped at 300s.

Privacy note: like the Gemini tier, the PDF bytes are sent to the Datalab cloud for conversion. Files are uploaded to the selected region's storage and deleted best-effort after conversion.

Blocked pages

Raw and direct-image HTTP requests use the same SSRF validation, hostname domain policy, redirect checks, timeout, and 5MB streamed response bound as normal extraction. Raw mode returns textual bodies even for non-2xx responses and exposes the HTTP status in tool details; it does not run readability or hosted extraction fallbacks.

fetch_content can opt into local browser-cookie auth with auth: "profile", or auth: true when exactly one authFetch profile exists. Configure profiles in ~/.pi/agent/web-search.json, for example { "authFetch": { "social": ["x.com", "instagram.com"], "work": { "hosts": ["docs.company.com"], "chromeProfile": "Profile 2", "cache": "off" } } }. Auth fetch uses only the local direct HTTP path, requires HTTPS, allows only configured hosts and their subdomains, refuses cross-origin redirects, and never sends cookies or authenticated content to hosted extraction providers. Browser cookie extraction remains opt-in through allowBrowserCookies: true or PI_ALLOW_BROWSER_COOKIES=1.

Proxy (proxy)

web_search, source_check, and fetch_content all accept an optional proxy string (e.g. "http://mcr:4444"). When provided, every outbound HTTP(S) request is routed through curl instead of Node's built-in fetch — this works around Node fetch ignoring HTTP(S)_PROXY env vars and undici ProxyAgent failing the TLS handshake against several common HTTP proxies (ERR_SSL_WRONG_VERSION_NUMBER).

An empty string ("") forces a direct connection even when a config-level proxy is set. Omitting the parameter falls back to the global proxy in ~/.pi/agent/web-search.json.

The config-level proxy applies only to requests made by this extension's tools. Other traffic in the host process — such as the coding agent's own model API calls — is never routed through it, so a configured proxy that is temporarily down cannot break unrelated requests.

// ~/.pi/agent/web-search.json — global proxy for all tools
{
  "proxy": "http://mcr:4444"
}

Localhost, 127.0.0.1, [::1], and any host matching the NO_PROXY environment variable are never proxied.

When Readability fails or returns only a cookie notice, the extension can retry configured Firecrawl extraction, configured Crawl4AI extraction, Jina Reader (handles JS rendering server-side, no API key needed), TinyFish, Search1API, Querit, Kagi Extract, Ollama Web Fetch, Parallel, Bright Data Web Unlocker, Gemini URL Context API, and Gemini Web extraction when browser cookies are enabled. Configure fetchRouting.providers to change the order or set of fetch_content providers. Supported values are http, firecrawl, crawl4ai, jina, tinyfish, search1api, querit, kagi, ollama, parallel, parallel-mcp, brightdata, and gemini; when absent, the default order is unchanged. parallel-mcp is not in the default fetch order and must be listed explicitly. For remote HTTP(S) targets, third-party hosted providers are disabled unless fetchRouting.allowRemoteHostedProviders is true, because hosted services perform their own fetch and can see a different redirect chain than the local safety gate. Firecrawl and Crawl4AI stay available as configured self-hosted extraction services. Firecrawl requests are cache-only by default and require an explicit fresh-scrape opt-in before the Firecrawl server can fetch target URLs. Bright Data Web Unlocker runs last of the remote scraping providers, ahead of only the Gemini fallbacks, because it is billed per request against a paid account; it is skipped unless both a key and an unblocker zone are configured. It applies no minimum-length check, so any non-empty body it returns — including a short consent or paywall stub — is the final answer for that URL and the Gemini fallbacks are not tried. Handles SPAs, JS-heavy pages, and anti-bot protections transparently. Also parses Next.js RSC flight data when present. HTML extraction also surfaces registered discovery relations (service-desc, service-doc, service-meta, api-catalog, describedby) from the HTTP Link header and matching link/a[rel] markup. Readable or rendered content remains primary; on an empty shell, the normal extraction fallbacks run before declared links are returned on their own.

How It Works

web_search(query)
  → configured searchRouting (optionally active-model-aware OpenAI Hosted Search) → automatic provider chain

fetch_content(url)
  → Video file?  Gemini API (Files API) → Gemini Web (if browser cookies enabled)
  → GitHub URL?  Clone repo, return file contents + local path
  → YouTube URL? Gemini Web (if browser cookies enabled) → Gemini API → Perplexity
  → HTTP fetch (asks for markdown first; raw mode does not) → PDF? Datalab → Gemini API → local text extraction, save to temp pi-web-pdf
               → HTML? Readability (+ declared Link/rel discovery) → RSC parser → Firecrawl → Crawl4AI (each if configured) → third-party hosted fallbacks only when fetchRouting.allowRemoteHostedProviders is enabled
               → Text/JSON/Markdown? Return directly

Commands

/websearch

Open the search curator directly. Runs searches and lets you review, add, select results, and approve a summary before it is sent back to the agent — no LLM round-trip needed.

/websearch                                               # empty page, type your own searches
/websearch react hooks, next.js caching                  # pre-fill with comma-separated queries

Results get injected into the conversation when you approve the summary or click "Send selected results without summary". On timeout, the curator auto-submits and falls back to a deterministic summary if no approved draft is present.

In summary review, Approve + auto-summary remaining searches for this prompt approves the current draft and makes later searches that inherit summary-review use auto-summary until the run settles. Explicit workflows still win; the choice stays in memory and does not affect open curator windows.

/curator

Toggle or configure the curator workflow at runtime.

/curator                    # toggle on/off
/curator on                 # enable curator (summary-review)
/curator off                # disable curator (raw results only)
/curator summary-review     # explicit workflow

Persists to ~/.pi/agent/web-search.json and takes effect on the next web_search call. When disabled, web_search returns raw results without opening the curator window.

/search

Browse stored search results interactively. Lists all results from the current session with their response IDs for easy retrieval.

/google-account

Show the active Google account currently authenticated for Gemini Web. If cookie extraction fails, it reports sanitized attempted browser/profile entries and whether the failure was missing required cookies, password-store access, decryption, SQLite, or profile lookup.

Activity Monitor

Toggle with Ctrl+Shift+W to see live request/response activity:

─── Web Search Activity ────────────────────────────────────
  API  "typescript best practices"     200    2.1s ✓
  GET  docs.example.com/article        200    0.8s ✓
  GET  blog.example.com/post           404    0.3s ✗
────────────────────────────────────────────────────────────

Configuration

Config defaults to ~/.pi/agent/web-search.json when neither PI_CODING_AGENT_DIR nor XDG_CONFIG_HOME is set. PI_CODING_AGENT_DIR takes precedence when set; with XDG_CONFIG_HOME, an existing XDG_CONFIG_HOME/pi/web-search.json is preferred, an existing legacy ~/.pi/web-search.json remains usable for compatibility, and the XDG path is used as the new-config target when neither file exists. When neither environment variable is set, an existing ~/.pi/agent/web-search.json is preferred, an existing legacy ~/.pi/web-search.json remains usable for compatibility, and the agent directory is used as the new-config target when neither file exists. Every field is optional.

{
  "openaiApiKey": "sk-...",
  "openaiResponsesUrl": "https://gateway.example.com/v1/responses",
  "braveApiKey": "BSA_...",
  "braveBaseUrl": "https://gateway.example.com/brave/res/v1",
  "exaApiKey": "exa-...",
  "exaBaseUrl": "https://gateway.example.com/exa",
  "parallelApiKey": "...",
  "tinyfishApiKey": "sk-tinyfish-...",
  "search1apiApiKey": "...",
  "tavilyApiKey": "tvly-...",
  "tavilyBaseUrl": "https://gateway.example.com/tavily",
  "youApiKey": "your-ydc-api-key",
  "jinaApiKey": "$JINA_API_KEY",
  "serpdiveApiKey": "sd_live_...",
  "serpdiveModel": "krill",
  "kagiApiKey": "$KAGI_API_KEY",
  "ollamaApiKey": "$OLLAMA_API_KEY",
  "valyuApiKey": "$VALYU_API_KEY",
  "keenableApiKey": "$KEENABLE_API_KEY",
  "serpbaseApiKey": "$SERPBASE_API_KEY",
  "serpapiApiKey": "$SERPAPI_KEY",
  "serperApiKey": "$SERPER_API_KEY",
  "serplyApiKey": "$SERPLY_API_KEY",
  "baizhiApiKey": "$BAIZHI_API_KEY",
  "zaiApiKey": "$ZAI_API_KEY",
  "brightdataApiKey": "$BRIGHTDATA_API_KEY",
  "brightdataSerpZone": "pi_serp",
  "xaiApiKey": "xai-...",
  "xaiSearchTools": ["web_search"],
  "mistralApiKey": "$MISTRAL_API_KEY",
  "mistralSearchModel": "mistral-small-latest",
  "mistralSearchTool": "web_search",
  "searxngBaseUrl": "https://search.example.com",
  "searxngHeaders": {
    "CF-Access-Client-Id": "xxxxxxxx.access",
    "CF-Access-Client-Secret": "xxxxxxxx"
  },
  "degoogBaseUrl": "https://degoog.example.com",
  "degoogApiKey": "$DEGOOG_API_KEY",
  "degoogEngines": ["degoog-org-official-extensions-brave-engine"],
  "firecrawlBaseUrl": "https://crawl.example.com",
  "firecrawlApiKey": "fc-...",
  "firecrawlApiVersion": "v2",
  "firecrawlFreshScrape": false,
  "firecrawlPdfMaxPages": 50,
  "crawl4aiBaseUrl": "https://crawl4ai.example.com",
  "crawl4aiApiToken": "$CRAWL4AI_API_TOKEN",
  "brightdataUnlockerZone": "pi_unlocker",
  "perplexityApiKey": "pplx-...",
  "geminiApiKey": "AIza...",
  "geminiBaseUrl": "https://my-gateway.example.com/gemini",
  "cloudflareApiKey": "...",
  "geminiAuth": "adc",
  "geminiProject": "my-gcp-project",
  "geminiLocation": "us-central1",
  "provider": "openai",
  "searchRouting": {
    "providers": ["openai", "brave", "exa"],
    "useCurrentModel": false,
    "fallbackOn": ["transient", "quota", "network", "invalid-response"]
  },
  "fetchRouting": {
    "providers": ["http", "firecrawl", "crawl4ai", "jina", "tinyfish", "search1api", "querit", "kagi", "ollama", "parallel", "brightdata", "gemini"],
    "allowRemoteHostedProviders": false
  },
  "fetch": {
    "timeout": 30,
    "defaultMode": "readable",
    "allowedModes": ["readable", "raw", "answer"],
    "answerProvider": "openai",
    "answerModel": "gpt-5.6"
  },
  "webSearch": {
    "enabled": true,
    "allowedProviders": ["openai", "brave", "exa"]
  },
  "tools": {
    "webSearch": { "enabled": true },
    "sourceCheck": { "enabled": true },
    "fetchContent": { "enabled": true },
    "getSearchContent": { "enabled": true }
  },
  "toolActivation": "auto",
  "commands": {
    "websearch": { "enabled": true },
    "curator": { "enabled": true },
    "search": { "enabled": true },
    "google-account": { "enabled": true }
  },
  "image": {
    "enabled": true
  },
  "browserCookies": {
    "browser": "helium",
    "profile": "Profile 2"
  },
  "allowBrowserCookies": false,
  "searchModel": "gemini-3.6-flash",
  "summaryModel": "anthropic/claude-haiku-4-5",
  "summaryGenerationDeadlineMs": 30000,
  "summaryInstructions": "- Preserve concrete figures verbatim: prices, limits, versions, dates.\n- Name the product version each claim applies to.",
  "maxInlineContentChars": 30000,
  "workflow": "summary-review",
  "curatorTimeoutSeconds": 20,
  "curatorRemote": {
    "host": "my-box.tailnet.ts.net",
    "bind": "100.101.102.103"
  },
  "autoOpenBrowser": true,
  "githubClone": {
    "enabled": true,
    "maxRepoSizeMB": 350,
    "cloneTimeoutSeconds": 30,
    "clonePath": "/tmp/pi-github-repos"
  },
  "githubPrIssue": {
    "enabled": true
  },
  "youtube": {
    "enabled": true,
    "preferredModel": "gemini-3.6-flash"
  },
  "video": {
    "enabled": true,
    "preferredModel": "gemini-3.6-flash",
    "maxSizeMB": 50
  },
  "pdf": {
    "enabled": true,
    "maxSizeMB": 20,
    "provider": "auto"
  },
  "fetchContent": {
    "domainPolicy": {
      "allow": ["example.com"],
      "deny": ["blocked.example.com"]
    }
  },
  "shortcuts": {
    "curate": "ctrl+shift+s",
    "activity": "ctrl+shift+w"
  },
  "ssrf": {
    "allowRanges": ["198.18.0.0/15"],
    "trustEnvProxy": false
  }
}

maxInlineContentChars accepts integers from 1,000 through 200,000. Missing, invalid, or smaller values use the 30,000-character default so truncation and retrieval guidance always fit.

webSearch.allowedProviders is an optional, non-empty search-provider allowlist with no duplicate entries. When omitted, every current search provider remains permitted. When present, explicit scalar and array requests for providers outside the list fail before any provider request, while auto, all, configured routing, source_check, and Curator use only listed providers. The allowlist does not make explicit-only or paid providers eligible for automatic fallback or all; they must still be named explicitly or included in searchRouting.providers. A configured provider/searchProvider or searchRouting.providers entry outside the allowlist is rejected as invalid configuration. This setting affects search only: it does not restrict fetchRouting, fetch answer models, or summary models.

summaryModel accepts an optional thinking-level suffix, such as anthropic/claude-haiku-4-5:low. Supported suffixes are off, minimal, low, medium, high, xhigh, and max.

summaryInstructions appends custom text to the Requirements section of the summary prompt used by the curator UI and auto-summary mode (e.g. "- Preserve concrete figures verbatim: prices, limits, versions, dates.\n- Name the product version each claim applies to."). The text is passed verbatim to the summary model, so use one - bullet per requirement. It is additive: the built-in guardrails (no invented sources, explicit weak/conflicting-evidence notes, the Sources section) always stay in place, and curator regeneration feedback is still appended afterwards. Missing, non-string, or blank values leave the default prompt unchanged.

All provider API-key fields (openaiApiKey, braveApiKey, parallelApiKey, tinyfishApiKey, search1apiApiKey, searchinfinityApiKey, queritApiKey, tavilyApiKey, youApiKey, jinaApiKey, serpdiveApiKey, kagiApiKey, bochaApiKey, ollamaApiKey, valyuApiKey, serpbaseApiKey, serpapiApiKey, serperApiKey, serplyApiKey, baizhiApiKey, zaiApiKey, keenableApiKey, degoogApiKey, anysearchApiKey, xcrawlApiKey, xaiApiKey, mistralApiKey, brightdataApiKey, firecrawlApiKey, crawl4aiApiToken, exaApiKey, perplexityApiKey, geminiApiKey, datalabApiKey, and cloudflareApiKey) accept explicit credential sources. Use $NAME or ${NAME} to read one named environment variable, or prefix a trusted local shell command with ! to resolve one value at provider request time. Escape $$ as a literal leading $ and $! as a literal leading !:

{
  "openaiApiKey": "!/absolute/path/to/secret-manager read openai",
  "braveApiKey": "${SCOPED_BRAVE_API_KEY}",
  "exaApiKey": "$$literal-key",
  "geminiApiKey": "$!literal-command"
}

This syntax applies to provider credentials only; other configuration fields are not interpolated. firecrawlApiKey, crawl4aiApiToken, kagiApiKey, ollamaApiKey, valyuApiKey, serpbaseApiKey, serpapiApiKey, serperApiKey, serplyApiKey, baizhiApiKey, zaiApiKey, keenableApiKey, degoogApiKey, mistralApiKey, and brightdataApiKey use the same credential-source rules, while braveBaseUrl, exaBaseUrl, tavilyBaseUrl, firecrawlBaseUrl, firecrawlApiVersion, firecrawlFreshScrape, firecrawlPdfMaxPages, crawl4aiBaseUrl, degoogBaseUrl, degoogEngines, brightdataSerpZone, brightdataUnlockerZone, and zaiEndpoint are literal config values.

A command source is not run while the extension loads or registers tools. Each selected provider request runs it again with a five-second timeout, a 16 KiB output limit, a minimized environment, and a one-line non-empty stdout requirement. Command text and stderr are omitted from errors. These commands are trusted local configuration, not a same-user process isolation boundary; use absolute executable paths and protect the config file. OP_SESSION_* and OP_SERVICE_ACCOUNT_TOKEN, when present in Pi's environment, are forwarded to trusted resolver commands so 1Password CLI sessions and service accounts can be reused without storing their credentials in config. For example, "braveApiKey": "!/absolute/path/to/op read 'op://Automation/Brave/credential'" resolves that item after you replace the executable placeholder with your trusted installation's absolute path, but the op:// argument is not an authorization boundary: every configured resolver command that receives the token can exercise all vault and item permissions granted to its service account. Prefer a narrowly scoped service account. An explicit source overrides legacy provider environment variables and fails that provider locally rather than falling back with a stale credential. Direct Google Gemini API requests send the resolved key only in the x-goog-api-key header, never in the URL.

mistralSearchModel defaults to mistral-small-latest and must be a non-empty string. mistralSearchTool defaults to web_search and accepts only web_search or the opt-in web_search_premium tool.

Set braveBaseUrl, exaBaseUrl, or tavilyBaseUrl to route those providers through a compatible HTTPS API gateway. BRAVE_BASE_URL, EXA_BASE_URL, and TAVILY_BASE_URL are the environment-variable equivalents and take precedence over config. Brave appends /web/search; Exa and Tavily append /search. Defaults remain the providers' official API roots. exaBaseUrl applies only to keyed direct API calls; zero-config Exa MCP search continues to use Exa's hosted MCP endpoint. Invalid overrides fail before a request is sent instead of falling back to an official endpoint, and credential headers are removed if a gateway redirects to another origin.

authFetch configures named local browser-cookie auth profiles for explicit fetch_content calls. A profile can be a host array ("work": ["docs.company.com"]) or an object with hosts, optional chromeProfile, redirects: "same-origin", and cache: "session" | "off".

browserCookies selects the Chromium browser preset and profile used for Gemini Web cookies, for example { "browserCookies": { "browser": "helium", "profile": "Profile 1" } }. When browser is set, cookie discovery checks only that browser, which avoids unrelated password-store prompts. Supported preset names are helium, chrome, brave, arc, chromium, and edge, subject to platform availability. Omit browser to keep automatic browser discovery. profile must be a profile directory name. The old top-level chromeProfile field is rejected; move it to browserCookies.profile. Arbitrary profile paths and profilePath are intentionally not supported.

fetchContent.domainPolicy is an optional hostname allow/deny policy for fetch_content target URLs. It is off when omitted. Each bare hostname matches itself and its subdomains; deny wins when a hostname matches both lists. The policy is checked before HTTP(S) target handling and before each redirect followed by this extension's own fetch path. Local file paths and non-HTTP sources are not subject to this policy. It is an additional restriction: the existing SSRF guard still blocks private and internal destinations. Remote extraction services can still perform their own DNS, redirects, and egress after this extension preflights the submitted target URL, so third-party hosted HTTP(S) fallbacks stay disabled unless fetchRouting.allowRemoteHostedProviders is enabled for separately isolated provider deployments.

fetch.timeout is an optional positive finite number of seconds for direct HTTP fetches and the Jina Reader fallback. When omitted, both use a 30-second budget. Fractional values are supported and rounded up to at least 1 millisecond; values that cannot be converted to a finite safe integer delay from 1 through Node's 2,147,483,647 ms timer maximum are rejected. An invalid declared value fails closed with an error naming web-search.json. An internal/per-call timeoutMs override takes precedence over this setting. Other remote extraction fallbacks keep their own documented budgets.

fetch.defaultMode sets the mode used when fetch_content omits mode, and fetch.allowedModes controls which modes the tool exposes and accepts. Valid modes are readable, raw, and answer. By default, the default mode is readable and all three modes are allowed. The default must be included in the non-empty, duplicate-free allowed list. An explicitly requested disabled mode fails before fetching or invoking a model and is never replaced with another mode. Raw mode remains direct HTTP-only and never runs readability, specialized source handling, or hosted extraction fallbacks.

fetch.answerProvider and fetch.answerModel are an optional pair that selects the model used by fetch_content answer mode when no per-call answerModel is supplied. Both values must be non-empty strings and must identify an enabled text-capable model available in Pi's model registry; invalid or partial configuration fails closed. This is opt-in: answering with the configured provider/model can send fetched page text outside the current session and may incur that provider's costs. A per-call answerModel override is resolved first and remains usable even when these configured defaults are malformed.

Set searxngBaseUrl or SEARXNG_BASE_URL to use a self-hosted SearXNG JSON API. A configured endpoint is preferred first in auto mode for local/private search. Its base URL and redirects remain subject to the SSRF guard; add only the narrowest self-hosted range to ssrf.allowRanges when it resolves to a private or synthetic range. Optional searxngHeaders merges extra HTTP headers into each SearXNG request (string values only; invalid header names are ignored), which is useful for reverse-proxy or Zero Trust auth such as Cloudflare Access service tokens (CF-Access-Client-Id / CF-Access-Client-Secret). Configured headers override the default Accept: application/json when the same name is supplied. Thanks to Marcos A. Núñez (@marnunez) for PR #107 and Avinash Kanaujiya (@avinashkanaujiya) for issue #105.

degoog. Set degoogBaseUrl or DEGOOG_URL to use a self-hosted degoog metasearch instance, or omit both to use the public https://degoog.org instance. degoog is keyless and explicit-only: select it with provider: "degoog" or place it in searchRouting; it is never chosen by auto and never participates in provider: "all", so queries do not leave for a third-party metasearch instance unless you ask. When the instance protects its search routes, set degoogApiKey or DEGOOG_API_KEY and the key is sent as Authorization: Bearer <key>; the API key and any degoogHeaders are stripped from cross-origin redirects. degoogEngines restricts the request to specific engine ids from the instance's engine list (an array of strings), which is sent as the engines field on POST /api/search; omitting it uses the instance defaults. recencyFilter maps to degoog's time parameter, and domain filters become site:/-site: query clauses plus a local result filter; multiple allowed domains are grouped as (site:a OR site:b) so the query terms apply to every branch. Only absolute HTTP(S) result URLs are returned as sources, and duplicate URLs are collapsed. A 200 response without a results array is reported as an invalid response, so a configured route can continue when fallbackOn includes "invalid-response", while a genuine empty results array remains a successful empty search. The configured base URL and redirects remain subject to the SSRF guard; add only the narrowest self-hosted range to ssrf.allowRanges when it resolves to a private or synthetic range. Optional degoogHeaders merges extra HTTP headers (string values only; invalid header names are ignored) for reverse-proxy or Zero Trust auth. Values from degoogApiKey and the configured degoogHeaders are redacted from reported upstream error text, and a request that exceeds the 30-second deadline fails as a timeout instead of being recorded as a completed request.

DuckDuckGo. DuckDuckGo HTML search is keyless and explicit-only. Select it with provider: "duckduckgo" or place it in searchRouting; it is never chosen by auto and never participates in provider: "all". Domain filters are enforced locally after DuckDuckGo redirect URLs are decoded. recencyFilter is not guaranteed because the HTML endpoint has no documented stable time parameter. A 200 page with no parseable results is reported as an invalid response, so routing can continue when fallbackOn includes "invalid-response".

Set firecrawlBaseUrl or FIRECRAWL_BASE_URL to use Firecrawl for web_search and as an extraction fallback for fetch_content. Search calls /v2/search by default, requests sources: ["web"], maps numResults to limit, maps recencyFilter to Firecrawl's tbs, and maps domain filters to includeDomains or excludeDomains when possible. With includeContent: true, Firecrawl search adds Markdown scrape options and returns successful result Markdown as inline content. Fetch extraction calls /v2/scrape by default. Set firecrawlApiVersion or FIRECRAWL_API_VERSION to v1 for older self-hosted images. Firecrawl bills PDFs per parsed page; set firecrawlPdfMaxPages to a positive integer to have Firecrawl parse at most that many pages of each PDF in fetch extraction and includeContent search. It is a per-document cap, not a credit budget: several PDFs or repeated requests still add up. Unset keeps the current behavior, and API v1 does not support the cap, so setting it with v1 is a config error. When Firecrawl reports fewer parsed pages than the PDF has, the content ends with a truncation notice naming both page counts. Firecrawl page scraping is cache-only by default (lockdown: true), so the Firecrawl server does not make fresh outbound target requests unless you explicitly set firecrawlFreshScrape: true or FIRECRAWL_FRESH_SCRAPE=1. Enable fresh scraping only for a Firecrawl deployment whose own egress, redirects, DNS rebinding behavior, and internal-network access are isolated or allowlisted; this extension can preflight submitted fetch URLs but cannot control network requests made by the Firecrawl server. A configured Firecrawl API base URL may use localhost, 127.0.0.0/8, or ::1 without adding those loopback ranges to global ssrf.allowRanges; that exception is only for Firecrawl API calls, not submitted fetch/search target URLs. The configured Firecrawl API base URL and redirects are otherwise still validated by the same SSRF guard as other remote requests, and Firecrawl credentials are stripped from cross-origin API redirects.

Crawl4AI. Set crawl4aiBaseUrl or CRAWL4AI_BASE_URL to use a self-hosted Crawl4AI server as a fetch_content extraction fallback. It runs right after Firecrawl and before the hosted providers, and it never participates in web_search because Crawl4AI has no search API. Extraction calls POST /md with the fit markdown filter and returns the response's top-level markdown; the first # heading becomes the title. Crawl4AI 0.9 and later require a bearer token by default, so set crawl4aiApiToken (credential-source syntax applies) or CRAWL4AI_API_TOKEN. The submitted target URL is validated by the local SSRF guard before any request, but the Crawl4AI server then fetches that target from its own network, so leave the base URL unset for URLs that must not be disclosed to that server. A configured Crawl4AI base URL may use localhost, 127.0.0.0/8, or ::1 without adding those ranges to global ssrf.allowRanges; a base URL on another private or synthetic range needs a narrow ssrf.allowRanges entry. The base URL and its redirects are otherwise validated like other remote requests, and the bearer token is stripped from cross-origin API redirects.

Bright Data. Set brightdataApiKey or BRIGHTDATA_API_KEY to use Bright Data-backed features. The SERP search provider also requires brightdataSerpZone or BRIGHTDATA_SERP_ZONE, and the Web Unlocker extraction fallback also requires brightdataUnlockerZone or BRIGHTDATA_UNLOCKER_ZONE. These zone settings are separate and are never substituted for each other: SERP requires a Bright Data zone of type serp, while Web Unlocker requires a zone of type unblocker. Leaving either zone unset keeps that product unavailable, so enabling one Bright Data feature does not opt into the other.

Bright Data search is explicit-only. Select it with provider: "brightdata" or place it in searchRouting; it is never chosen by auto and never participates in provider: "all". The SERP path maps domain filters to Google site: clauses and recency filters to Google tbs parameters, validates the returned SERP envelope, and surfaces provider errors instead of converting them to empty results.

Bright Data Web Unlocker is a paid fetch_content fallback after Parallel and before Gemini. It validates the target URL with the local SSRF guard before resolving credentials or sending any request to Bright Data, validates redirects from the Bright Data API endpoint, and strips authorization across cross-origin API redirects. As with any remote extraction service, Bright Data fetches the submitted target from its own infrastructure; keep brightdataUnlockerZone unset for URLs that must not be disclosed to a third party. Successful Unlocker responses are returned as Markdown, including short pages or consent stubs, because the request has already been billed and discarding the body would hide what Bright Data saw.

Kagi. Set kagiApiKey or KAGI_API_KEY to use Kagi Search API as a normal configured search provider. It maps numResults to Kagi's limit parameter and returns Kagi result snippets; when Kagi includes extracted Markdown in the search response, includeContent exposes it as inline content. Kagi Extract is also available as a fetch_content fallback after Querit and before Ollama/Parallel. The target URL is validated locally before the API request and Kagi authorization is stripped across cross-origin API redirects.

Ollama. Set ollamaApiKey or OLLAMA_API_KEY to use Ollama Cloud Web Search without running a local daemon. The same account key used for Ollama Cloud inference authenticates POST https://ollama.com/api/web_search, with numResults capped to Ollama's documented max of 10. Ollama Web Fetch is available as a fetch_content fallback after Kagi and before Parallel, with local target validation before the Cloud request.

SerpBase. Set serpbaseApiKey or SERPBASE_API_KEY and select provider: "serpbase" to query SerpBase's Google Search Results API. SerpBase is explicit-only: it is never chosen by auto and never participates in provider: "all", because each request can consume paid Google SERP credits. Domain filters are sent as Google site: clauses and reapplied locally; recency maps to Google's tbs time filter.

SerpApi. Set serpapiApiKey or SERPAPI_KEY and select provider: "serpapi" to query SerpApi's Google Search API. SerpApi is explicit-only: it is never chosen by auto or provider: "all", but it can be configured as the named provider or added to searchRouting. Domain filters are sent as Google site: clauses and reapplied locally; recency maps to Google's tbs time filter. Each request consumes SerpAp