pi-web-toolkit

100% open-source. No required API keys or paid services.

Web research toolkit for pi agents. Search via SearXNG, fetch pages with scrapling, browse interactively via agent-browser, and batch-read sources in parallel. All backends run locally or are self-hosted, with built-in truncation safety and LLM-optimized prompt guidelines.

Features

| Tool | Backend | Purpose | Current Limit | |------|---------|---------|---------------| | web_search | SearXNG | Discover scored, ranked results from multiple engines | 20 results (max 60, auto-pages up to 3 pages) | | web_fetch | scrapling | Fetch a single page as clean markdown | — | | web_batch_fetch | scrapling | Fetch 1–15 pages in parallel for research synthesis (2–5 recommended) | 3 concurrent (max 5) | | web_browse | agent-browser | Interact with a page (click, scroll, fill) then extract content | 25 actions |

Tools Preview

A quick look at how pi renders toolkit calls while an agent searches, fetches, batches, and browses the web.

Install with Pi Agent

Copy and send the prompt below to Pi. It will install this package and its external dependencies for you.

Install pi-web-toolkit and its external dependencies. Complete and verify every
step yourself; do not rely on web browsing or external documentation. Inspect
the machine first and reuse working installations. Ask before using sudo,
changing shell profiles, overwriting configuration, or modifying existing
services or containers.

1. Ensure Node.js 22+, npm, Docker, OpenSSL, curl, uv, and Pi are installed, and
   that Docker is running. Install only missing or incompatible prerequisites.
2. Configure SearXNG:
   - Test SEARXNG_URL when set, then http://localhost:8080.
   - Verify /search?q=test&format=json returns JSON with a results array.
   - If neither endpoint works, first ensure no existing container or config
     would be overwritten, then create a local-only instance by running:

mkdir -p "$HOME/.config/searxng"
cat > "$HOME/.config/searxng/settings.yml" <<'YAML'
use_default_settings: true

search:
  formats:
    - html
    - json
YAML

docker run -d \
  --name searxng \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -e FORCE_OWNERSHIP=false \
  -e SEARXNG_SECRET="$(openssl rand -hex 32)" \
  -v "$HOME/.config/searxng/settings.yml:/etc/searxng/settings.yml:ro" \
  docker.io/searxng/searxng:latest

   - Verify the selected endpoint by running:

SEARXNG_ENDPOINT="${SEARXNG_URL:-http://localhost:8080}"
curl -fsS --get "${SEARXNG_ENDPOINT%/}/search" \
  --data-urlencode "q=test" \
  --data "format=json" |
  grep -q '"results"' && echo "SearXNG JSON API ready"

   - Pi uses http://localhost:8080 by default. Set SEARXNG_URL before starting
     Pi only when using another endpoint.
3. Install and verify Scrapling:
   uv tool install "scrapling[all]"
   scrapling install
   scrapling --help
4. Install and verify agent-browser:
   npm install -g agent-browser
   agent-browser install
   agent-browser doctor
   On Linux, use agent-browser install --with-deps if required.
5. After all dependencies pass verification, install the package:
   pi install npm:pi-web-toolkit

Report what was installed or reused, all verification results, the SearXNG
endpoint Pi will use, and whether Pi must be restarted. Do not report success
until every check passes.

Quick Start

1. Install external dependencies

The commands below assume a POSIX shell with Docker, OpenSSL, curl, uv, and Node.js 22+ with npm.

# SearXNG (for search; local-only instance with the required JSON API)
mkdir -p "$HOME/.config/searxng"
cat > "$HOME/.config/searxng/settings.yml" <<'YAML'
use_default_settings: true

search:
  formats:
    - html
    - json
YAML

docker run -d \
  --name searxng \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -e FORCE_OWNERSHIP=false \
  -e SEARXNG_SECRET="$(openssl rand -hex 32)" \
  -v "$HOME/.config/searxng/settings.yml:/etc/searxng/settings.yml:ro" \
  docker.io/searxng/searxng:latest
export SEARXNG_URL="http://127.0.0.1:8080"

# scrapling (for fetch & batch fetch)
uv tool install "scrapling[all]"
scrapling install

# agent-browser (for browse)
npm i -g agent-browser && agent-browser install
# On Linux hosts missing browser system libraries: agent-browser install --with-deps

Verify dependencies:

# SearXNG
curl -fsS --get "$SEARXNG_URL/search" \
  --data-urlencode "q=searxng" \
  --data "format=json" |
  grep -q '"results"' && echo "SearXNG JSON API ready"

# scrapling
scrapling --help

# agent-browser
agent-browser doctor

2. Install the extension

From npm

pi install npm:pi-web-toolkit

From GitHub

pi install git:github.com/Wade11s/pi-web-toolkit

Configuration

web_search reads its SearXNG endpoint from an environment variable. Set it before starting pi; no build step is required.

| Variable | Default | Used By | Description | |----------|---------|---------|-------------| | SEARXNG_URL | http://localhost:8080 | web_search | Your SearXNG instance endpoint |

Set before starting pi:

export SEARXNG_URL="https://searxng.example.com"

Project Structure

pi-web-toolkit/
├── extensions/
│   ├── index.ts              # Unified entry point — registers all 4 tools
│   ├── utils/
│   │   ├── cli-runner.ts     # Unified CLI process spawning with timeout/AbortSignal
│   │   ├── content-preview.ts # Intelligent content extraction from scraped pages
│   │   ├── output-sink.ts    # Truncation + temp-file fallback
│   │   ├── render-helpers.ts # URL abbreviations, text normalization, error formatting for TUI
│   │   ├── scrapling.ts      # Reusable scrapling CLI wrapper (shared by fetch + batch)
│   │   ├── tool-factory.ts   # Common tool registration patterns
│   │   └── agent-browser.ts  # agent-browser CLI wrapper (shared by web_browse)
│   ├── web_search.ts         # SearXNG search tool
│   ├── web_fetch.ts          # Single-page scrapling fetcher
│   ├── web_batch_fetch.ts    # Parallel scrapling fetcher
│   └── web_browse.ts         # Interactive browser automation (agent-browser)
├── test/
│   ├── agent-browser/        # agent-browser output parser regression tests
│   ├── content-preview/      # Content preview fixtures, baselines & snapshots
│   └── README.md             # Test suite structure and conventions
├── docs/
│   ├── tools.md              # Full parameter specs
│   ├── guide.md              # Decision tree & tool comparison
│   └── agents/               # Issue tracker, triage and domain guidance
├── AGENTS.md
├── CONTEXT.md
├── CHANGELOG.md
├── package.json
├── README.md
├── tsconfig.json
└── LICENSE

Design principles:

• Unified registration — index.ts is the single source of truth for what pi loads.
• Shared utilities — utils/ modules encapsulate CLI spawning, content extraction, output truncation, TUI formatting, and common registration patterns; tool files import only from utils/, never from each other.
• Per-tool isolation — each tool owns its own schema, execute logic, and TUI renderer; no cross-imports except via utils/.
• Runtime config — environment variables are read at execute time, not build time.

Reference

• Tool Reference — Full parameter specs and usage examples for each tool.
• Usage Guide — Decision tree and tool comparison.
• Changelog — Release history and migration notes.

Contributing

# Local development
pi install ./

# Type-check (no build step; pi loads TypeScript directly)
npm run typecheck

# Run tests
npm run test

# Verify external CLI dependencies
scrapling --help
agent-browser doctor

Pull requests welcome. Please keep changes scoped to a single tool or concern and follow Conventional Commits.

License

MIT