pi-web-search-and-fetch 🌐⚡

Empower your pi coding agent with real-time web intelligence: instant neural search, clean Markdown page extraction, and autonomous deep research.

pi-web-search-and-fetch transforms your pi coding agent into an autonomous web researcher. Say goodbye to outdated training cutoffs and missing documentation — equip your LLM with enterprise-grade web search, high-fidelity webpage retrieval, and multi-query deep synthesis. Built with a modular multi-provider engine, an interactive /ws terminal dashboard, and seamless zero-conflict integration with pi-requesty-provider.

⚡ Superpowers & Features

• 🔍 Real-Time Neural Web Search (web_search) — Search the live web with semantic and category filtering (news, github, research paper, pdf, company, tweet, financial report) to break past knowledge cutoffs.
• 📄 High-Fidelity Web Fetch (web_fetch) — Instantly retrieve live documentation, articles, and repositories, converting raw HTML into clean, token-optimized Markdown.
• 🧠 Autonomous Deep Research (web_deep_search) — Multi-angle iterative exploration that executes parallel queries and synthesizes multi-source findings for complex inquiries.
• 🎛️ Interactive TUI Control Hub (/ws) — Full-featured terminal dashboard to toggle tools on the fly, switch provider backends, and configure API keys with smooth keyboard controls.
• 🚀 Zero-Config Instant Start — Works immediately out of the box with Exa's public MCP endpoint (no API key required) or plug in your own EXA_API_KEY for unlimited bandwidth and deep search.
• 🔌 Decoupled Multi-Provider Architecture — Independent capability interfaces (SearchProvider, FetchProvider, DeepSearchProvider). Mix and match providers freely or implement custom engines with minimal boilerplate.
• 🤝 Smart pi-requesty-provider Synergy — Automatically coordinates with pi-requesty-provider to detect native server-side search models, preventing duplicate searches while keeping markdown extraction active.
• 🔒 Secure Credential Management — Stores API keys safely in ~/.pi/agent/auth.json with strict 0o600 file permissions.

🚀 Quick Installation

From npm (Recommended)

pi install npm:pi-web-search-and-fetch

From GitHub

pi install git:github.com/xinaps-dev/pi-web-search-and-fetch

Local Development

# Clone and build
git clone https://github.com/xinaps-dev/pi-web-search-and-fetch.git
cd pi-web-search-and-fetch
pnpm install

# Run locally in pi
pi -e ./

🎮 Getting Started & The /ws Control Hub

Managing your web tools is effortless with the unified /ws command.

Interactive Hub Dashboard

Run /ws inside pi to open the interactive TUI Control Panel:

┌───────────────────────────────────────────────────────────┐
│  🌐 Web Search and Fetch - Control Panel                             │
├───────────────────────────────────────────────────────────┤
│  [✓] Search (web_search)          : ON  (Provider: exa)   │
│  [✓] Fetch (web_fetch)           : ON  (Provider: exa)   │
│  [ ] Deep Search (deep)          : OFF (Provider: exa)   │
│                                                           │
│  Actions:                                                 │
│  > Assign Providers (3-tool wizard)                       │
│  > Configure Active Provider (Exa API Key / Mode)         │
│  > View Detailed Status                                   │
│  > Exit                                                   │
└───────────────────────────────────────────────────────────┘

• Navigate: Use ↑ / ↓ arrow keys to browse options.
• Toggle / Select: Press Space or Enter to toggle tools or trigger configuration wizards.
• Close: Press Esc or select Exit.

🛠️ Standardized LLM Tools

pi-web-search-and-fetch registers 3 powerful tools with the Pi agent harness:

1. web_search (Enabled by default)

Performs real-time web searches and returns structured results with titles, URLs, snippets, publication dates, and citations.

| Parameter | Type | Required | Description | | :--- | :--- | :--- | :--- | | query | string | Yes | Search keywords or natural language question | | numResults | number | No | Number of results to return (default: 8) | | category | string | No | Content category (company, research paper, news, github, pdf, tweet, financial report) |

2. web_fetch (Enabled by default)

Fetches full web page content from a known URL and converts it into clean, LLM-ready Markdown.

| Parameter | Type | Required | Description | | :--- | :--- | :--- | :--- | | url | string | Yes | Full HTTP/HTTPS URL to retrieve | | maxCharacters | number | No | Maximum character limit for extracted text (default: 15,000) |

3. web_deep_search (Optional)

Agentic multi-query web search for complex questions requiring parallel queries and comprehensive multi-source synthesis.

| Parameter | Type | Required | Description | | :--- | :--- | :--- | :--- | | query | string | Yes | Primary research question | | numResults | number | No | Results per query (default: 10) | | category | string | No | Content filter category | | additionalQueries| string[] | No | Supplementary parallel sub-queries |

⚙️ Configuration & Key Storage

Extension Configuration: ~/.pi/agent/pi-web-search-and-fetch.json

Persistent settings are stored cleanly in your agent directory:

{
  "search": {
    "enabled": true,
    "provider": "exa"
  },
  "fetch": {
    "enabled": true,
    "provider": "exa"
  },
  "deepSearch": {
    "enabled": false,
    "provider": "exa"
  },
  "providers": {
    "exa": {
      "useApiKey": true
    }
  }
}

Authentication & API Keys: ~/.pi/agent/auth.json

API keys are read and stored using Pi's standard authentication store (~/.pi/agent/auth.json) with strict 0o600 file permissions:

{
  "exa": {
    "type": "api_key",
    "key": "your-exa-api-key-here"
  }
}

Key Resolution Hierarchy for Exa:

1. Public MCP Tier: If useApiKey: false, uses Exa's public endpoint (free, no key required).
2. Authenticated Tier: If useApiKey: true:
   • Checks ~/.pi/agent/auth.json (exa.key).
   • Falls back to EXA_API_KEY environment variable.

🧩 Provider Architecture

pi-web-search-and-fetch is designed with a fully decoupled capability model:

┌────────────────────────────────────────────────────────┐
│                   ProviderRegistry                     │
├────────────────────────────────────────────────────────┤
│  SearchProvider      │  FetchProvider   │ DeepSearch   │
│  - exa               │  - exa           │ - exa        │
│  - (custom search)   │  - (custom fetch)│ - (custom)   │
└────────────────────────────────────────────────────────┘

Implementing a Custom Provider

Easily create and register custom providers conforming to ProviderModule:

import type {
  ProviderModule,
  SearchProvider,
  FetchProvider,
  DeepSearchProvider,
  SearchResponse,
  FetchResponse,
} from "pi-web-search-and-fetch";

export const myCustomProvider: ProviderModule = {
  id: "custom",
  name: "My Custom Provider",
  description: "Custom web search and scraper",
  capabilities: ["search", "fetch"],

  searchProvider: {
    id: "custom",
    name: "My Custom Provider",
    description: "Search via Custom API",
    supportsApiKey: true,
    requiresApiKey: true,
    async search(query, options, signal): Promise<SearchResponse> {
      // Implement search logic...
      return {
        query,
        provider: "custom",
        results: [
          { title: "Example", url: "https://example.com", snippet: "..." }
        ],
      };
    },
  },

  fetchProvider: {
    id: "custom",
    name: "My Custom Provider",
    description: "Fetch via Custom Scraper",
    supportsApiKey: false,
    requiresApiKey: false,
    async fetch(url, options, signal): Promise<FetchResponse> {
      // Implement fetch logic...
      return {
        url,
        provider: "custom",
        content: "# Example Page Content\n\n...",
      };
    },
  },
};

🤝 pi-requesty-provider Smart Synergy

When used alongside pi-requesty-provider:

1. pi-web-search-and-fetch inspects ~/.pi/agent/pi-requesty.json and evaluates the active session model on session_start and model_select.
2. If Requesty has native search enabled (nativeSearch: true) in pi-requesty-provider and the active model supports server-side search grounding, pi-web-search-and-fetch automatically suppresses web_search to prevent duplicate web queries and token waste.
3. web_fetch remains fully active, allowing your agent to extract and inspect complete web pages on demand.

🧪 Development & Testing

# Run all unit and integration tests
pnpm test

# Run TypeScript typecheck
pnpm typecheck

📄 License

This project is licensed under the MIT License © xinaps.