pi-web-tools

Web search, content extraction, and GitHub repo cloning for the Pi coding agent.

A lightweight extension providing three tools:

• web_search — Search the web via Exa with snippet extraction
• fetch_content — Fetch any URL and extract clean markdown (HTML via Readability, Jina Reader fallback, GitHub via clone)
• get_search_content — Retrieve stored results from previous searches/fetches

Install

pi install npm:pi-web-tools

Or install from git:

pi install github:coctostan/pi-web-tools

Setup

Exa API Key (required for web_search)

Get a key at exa.ai and set it via environment variable:

export EXA_API_KEY="your-key-here"

Or add it to the config file ~/.pi/web-tools.json:

{
  "exaApiKey": "your-key-here"
}

The environment variable takes precedence over the config file.

GitHub CLI (recommended for fetch_content)

For GitHub repo cloning, install the GitHub CLI:

# Debian/Ubuntu
sudo apt install gh

# Or via conda, brew, etc.
gh auth login

Without gh, the extension falls back to git clone (works for public repos).

Configuration

Config file: ~/.pi/web-tools.json (auto-reloaded every 30 seconds)

{
  "exaApiKey": "your-exa-key",
  "github": {
    "maxRepoSizeMB": 350,
    "cloneTimeoutSeconds": 30,
    "clonePath": "/tmp/pi-github-repos"
  }
}

| Option | Default | Description | |--------|---------|-------------| | exaApiKey | null | Exa API key (env EXA_API_KEY overrides) | | github.maxRepoSizeMB | 350 | Skip cloning repos larger than this | | github.cloneTimeoutSeconds | 30 | Abort clone after this many seconds | | github.clonePath | /tmp/pi-github-repos | Where to store cloned repos |

Tools

web_search

Search the web using Exa. Returns results with snippets and source URLs.

| Parameter | Type | Description | |-----------|------|-------------| | query | string | Single search query | | queries | string[] | Multiple queries (batch) | | numResults | number | Results per query (default: 5, max: 20) |

Example:

Search for "TypeScript 5.8 new features"

fetch_content

Fetch URL(s) and extract readable content as markdown.

| Parameter | Type | Description | |-----------|------|-------------| | url | string | Single URL to fetch | | urls | string[] | Multiple URLs (parallel, max 3 concurrent) | | forceClone | boolean | Force cloning large GitHub repos |

Content extraction pipeline:

1. GitHub URLs → Clone repo (shallow, depth 1), generate tree + README
2. HTML pages → Readability extraction → Markdown conversion
3. Readability fails → Jina Reader fallback (r.jina.ai)
4. Non-HTML → Return raw text

Content over 30,000 characters is truncated with a pointer to get_search_content.

get_search_content

Retrieve full content from a previous web_search or fetch_content result.

| Parameter | Type | Description | |-----------|------|-------------| | responseId | string | ID from a previous tool result | | query | string | Filter by query text | | queryIndex | number | Filter by query index | | url | string | Filter by URL | | urlIndex | number | Filter by URL index |

How GitHub Cloning Works

When fetch_content receives a GitHub URL:

1. Parse — Extracts owner, repo, ref, path, type (root/blob/tree)
2. Size check — Queries repo size via gh api. Skips if over threshold (default 350MB)
3. Clone — Shallow clone (--depth 1) to temp directory, cached for the session
4. Generate — Based on URL type:
   • Root: Full directory tree + README content
   • Tree: Directory listing for the specified path
   • Blob: File content (with binary detection and 100K truncation)

Non-code GitHub URLs (issues, PRs, discussions, etc.) are fetched as normal web pages.

Architecture

index.ts          — Extension entry point, 3 tools, session management
├── config.ts     — Config with 30s TTL cache, env var overrides
├── storage.ts    — LRU storage (max 50 entries, session restore)
├── exa-search.ts — Exa API client
├── extract.ts    — Readability + Jina Reader content extraction
└── github-extract.ts — GitHub URL parsing, clone, tree/content generation

Development

# Install dependencies
npm install

# Run tests
npx vitest run

# Run tests in watch mode
npx vitest

# Load in pi for testing
pi -e ./index.ts

License

MIT