pi-search-hub

Unified web search + content extraction extension for pi with 12 backend providers (all working). One web_search tool, one web_read tool, auto-fallback, RRF-ranked combine mode, and credential resolution via env/shell/literal.

Installation

pi install npm:pi-search-hub

Note for DuckDuckGo backend: Requires the ddgs Python package. Install with:

• Linux/macOS: pip3 install ddgs
• Windows: pip install ddgs

Usage

Web Search

After installing, just ask naturally:

Search for recent AI agent frameworks.

What's the latest news on Llama 4?

Or use the tools directly — the agent picks the best configured backend automatically:

• web_search — search the web with auto-fallback or parallel combine mode
• web_read — fetch any URL as clean markdown

Combine Mode

Set combine=true to query ALL enabled backends in parallel with Reciprocal Rank Fusion (RRF) ranking:

Search for "Rust vs Go performance benchmarks" with combine=true to get results from all backends

Combine mode benefits:

• Broader coverage across multiple search indexes
• Results ranked by RRF — position-based scoring across all backends
• Each result shows which backend found it
• URL deduplication with content-aware merge (prefers richest result)
• Useful for comprehensive research or when you want diverse sources

Tradeoff: Uses more API quota per query (all backends are called), but you get more comprehensive results.

Read Web Pages

Fetch any URL as clean markdown — great for extracting article content, docs, or reference pages. Note: web_read uses Jina Reader to fetch and convert URLs to markdown.

Read https://docs.example.com/api-reference

The web_read tool supports:

• objective — CSS selector to target specific content (e.g. "div.article-body")
• keywords — relevant terms to highlight on long pages
• mode — rush for speed (return innerText) or smart (markdown extraction)
• fresh — bypass cache when freshness matters

Supported Backends

| # | Backend | Free Tier | API Key? | How to get key | | --- | --------------------- | ----------------------------- | :------: | ----------------------------------------------------------------- | | 1 | DuckDuckGo | Unlimited (rate-limited) | No | pip install ddgs (Linux/macOS: pip3) | | 2 | Jina AI | Search: key req. web_read: free (no key) | Yes | jina.ai | | 3 | Marginalia Search | Unlimited (rate-limited) | No† | marginalia.nu | | 4 | Tavily | 1,000 calls/month | Yes | tavily.com | | 5 | Serper (Google) | 2,500 free queries (one-time) | Yes | serper.dev | | 6 | Brave | 2,000 queries/month | Yes | brave.com/search/api | | 7 | Firecrawl | 500 free credits | Yes | firecrawl.dev | | 8 | Exa | 1,000 free queries/month | Yes | exa.ai | | 9 | LangSearch | Genuinely free, no CC | Yes | langsearch.com | | 10 | WebSearchAPI.ai | 2,000 free credits | Yes | websearchapi.ai | | 11 | Perplexity Sonar | Paid (usage-based) | Yes | perplexity.ai | | 12 | SearXNG | Self-hosted, unlimited | No | docs.searxng.org |

† Marginalia Search uses public as a shared API key — no registration required, but subject to a shared rate limit.

Jina AI: Search (s.jina.ai) requires a free API key from jina.ai. Content extraction via web_read uses Jina Reader (r.jina.ai) which is free and needs no API key.

Perplexity Sonar supports multiple model variants. Set model in your Perplexity backend config to choose: sonar (default, fast), sonar-pro (higher quality), sonar-deep-research (multi-step reasoning), or sonar-reasoning (DeepSeek R1-based).

SearXNG is a self-hosted metasearch engine. Run your own instance (or use a public one), no API key required. Configure the instance URL in .pi/search.json.

Firecrawl uses api.firecrawl.dev/v2/search with a data.web[] response shape. The v1 endpoint is deprecated.

Exa (March 2026) includes content for the first 10 results per request at no extra cost. Content extraction is enabled by default.

Configuration

Configure backends globally (all projects) or per-project:

Global: ~/.pi/agent/extensions/search.json Project: .pi/search.json (project takes precedence)

{
  "defaultBackend": "auto",
  "backends": {
    "duckduckgo": { "enabled": true },
    "jina": { "enabled": true, "apiKey": "JINA_API_KEY" },
    "marginalia": { "enabled": true },
    "serper": { "enabled": true, "apiKey": "SERPER_API_KEY" },
    "tavily": { "enabled": true, "apiKey": "TAVILY_API_KEY" },
    "brave": { "enabled": true, "apiKey": "BRAVE_API_KEY" },
    "exa": { "enabled": true, "apiKey": "EXA_API_KEY" },
    "firecrawl": { "enabled": true, "apiKey": "FIRECRAWL_API_KEY" },
    "langsearch": { "enabled": true, "apiKey": "LANGSEARCH_API_KEY" },
    "websearchapi": { "enabled": true, "apiKey": "WEBSEARCHAPI_API_KEY" },
    "perplexity": {
      "enabled": true,
      "apiKey": "PERPLEXITY_API_KEY",
      "model": "sonar"
    },
    "searxng": { "enabled": true, "instanceUrl": "http://localhost:8888" }
  }
}

Credential Resolution

The apiKey field supports four formats (following pi-web-providers convention):

| apiKey value | Resolved from | Example | | ------------------------- | --------------------------------------- | ---------------------------------- | | "SERPER_API_KEY" | process.env.SERPER_API_KEY | ALL_CAPS → env var | | "!pass show api/serper" | stdout of shell command (cached) | ! prefix → exec | | "sk-abc123..." | Used as-is | Literal key (backwards compatible) | | (unset) | SEARCH_<BACKEND>_API_KEY env fallback | Auto-enables backend |

Env var references: Any ALL_CAPS string is treated as an environment variable name (not a literal). If the referenced env var is unset, a warning is printed (your literal key is not silently discarded).

Shell commands: Commands prefixed with ! are executed via execSync with a 5s timeout. Results are cached and invalidated when config is reloaded (editing the config file clears the cache).

Convenience env vars: Backends are auto-enabled when these env vars are set (even with no config entry):

export SEARCH_SERPER_API_KEY="sk-..."
export SEARCH_TAVILY_API_KEY="sk-..."
export SEARCH_EXA_API_KEY="sk-..."
# ...

{
  "backends": {
    "serper": { "enabled": true, "apiKey": "SERPER_API_KEY" }
  }
}

To rotate a shell-command key: Update the secret in your password manager, then trigger a config reload (edit the config file, or wait 10s for automatic refresh).

Or use the interactive setup:

/search-setup

Commands

| Command | Description | | ---------------- | ----------------------------------------------------------------- | | /search-setup | Interactive prompt to configure API keys and instance URLs | | /search-status | Show which backends are active, which have keys, and their status |

Tip: After running /search-setup or editing your config, run /reload to activate changes without restarting pi.

How auto mode works

Fallback Mode (default, combine=false)

1. Tries each enabled backend in order from your config
2. If a backend fails (rate limit, auth error, etc.), moves to the next one
3. Jina AI search requires a free API key from jina.ai (get one at jina.ai/reader). DuckDuckGo requires no API key. Both serve as safety nets
4. Returns results from the first backend that succeeds
5. If all backends fail, reports the collected errors

Combine Mode (combine=true)

1. Queries ALL enabled backends in parallel
2. Each backend receives numResults / numBackends as a target
3. Results are merged using Reciprocal Rank Fusion (RRF) — position-based scoring that works across incompatible ranking systems
4. Each result shows its source backend (e.g., *Source: Tavily*)
5. URL dedup prefers the result with the richest content (content > snippet)
6. Backend statistics are displayed (which succeeded, result counts, errors)

RRF Scoring

RRF assigns each result a score of Σ(1 / (60 + rank_i)) across all backends that returned it. Results are ranked by score, then by number of backends that found them. This means a result ranked #1 by one backend and #5 by another beats a result ranked #4 by two backends.

Security

• API keys are stored in local config files only (~/.pi/agent/extensions/search.json or .pi/search.json), never sent to any third party besides the chosen backend
• Env vars and shell commands are supported for credential resolution — the config file is trusted (you own it), but never commit plain API keys to version control
• DuckDuckGo queries use spawned Python subprocess (abortable via signal)
• All HTTP backends have a 30-second timeout; shell commands for credentials have a 5-second timeout
• Error messages are sanitized — API response bodies are truncated and key-like patterns are redacted
• The .pi/ directory is in .gitignore — never commit API keys to version control

Testing

# Run unit tests for backend parsers
npx vitest run backends/parsers.test.ts

# Quick test Jina AI (with your free API key)
curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://s.jina.ai/?q=test&format=json" | jq .

# Quick test via curl with your configured key
curl -X POST "https://api.exa.ai/search" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $KEY" \
  -d '{"query": "test", "numResults": 3, "contents": {"text": true}}'

# Quick test Perplexity Sonar (use "sonar-pro" or "sonar-deep-research" for model)
curl -X POST "https://api.perplexity.ai/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{"model": "sonar", "messages": [{"role": "user", "content": "test"}], "search_context_size": "low"}'

# Quick test Firecrawl (v2 endpoint — code still uses v1)
curl -X POST "https://api.firecrawl.dev/v2/search" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{"query": "test", "limit": 3}'

# Quick test SearXNG (replace URL with your instance)
curl "http://localhost:8888/search?q=test&format=json&count=3"

Adding a new backend

Backends are registered via the BACKEND_DEFS registry in extensions/search-hub.ts. Define a search function and add one entry to the registry:

const BACKEND_DEFS: Record<string, BackendRunner> = {
  // ... existing entries
  mybackend: {
    needsKey: true,
    needsKeyFromConfig: false,
    needsInstanceUrl: false,
    label: "My Backend",
    setupLabel: "My Backend (free tier description)",
    search: async (query, numResults, { key, signal }) => {
      const result = await searchMyBackend(query, numResults, key!, signal);
      return { results: result.results };
    },
  },
};

The registry handles dispatching, key resolution, formatting labels, and setup menu — no other edits needed.

License

MIT