pi-spider-tools

Native Pi coding agent tools that expose Spider Cloud scraping, crawling, search, screenshot, and browser-automation capabilities.

It ships two tool sets, registered from separate entry files:

• Core tier (src/spider.ts) — stateless tools that call the Spider Cloud REST API directly.
• Browser tier (src/browser.ts) — stateful remote-browser automation backed by the spider-browser package (WebSocket to Spider's pre-warmed browser fleet).

Core tools

| Tool | Endpoint | Description | | --- | --- | --- | | spider_scrape | POST /scrape | Scrape a single URL; faster/cheaper than crawling. | | spider_crawl | POST /crawl | Crawl a site following links up to a limit/depth. | | spider_search | POST /search | Web search, optionally fetching each result's content. | | spider_links | POST /links | Collect links without returning page content. | | spider_screenshot | POST /screenshot | Capture a page screenshot (base64 PNG in the payload). | | spider_unblocker | POST /unblocker | Fetch anti-bot-protected pages via the stealth unblocker. | | spider_transform | POST /transform | Convert raw HTML to markdown/text (no web request). | | spider_get_credits | GET /data/credits | Check remaining credit balance (free). |

Browser tools

Stateful: spider_browser_open returns a session_id that the other tools take, and spider_browser_close releases it. Open sessions bill until closed — they auto-close after 5 minutes of inactivity and on pi shutdown/reload, and at most 5 may be open at once.

| Tool | Description | | --- | --- | | spider_browser_open | Open a remote browser session; returns a session_id. | | spider_browser_navigate | Navigate the session to a URL and wait for load. | | spider_browser_click | Click an element by CSS selector. | | spider_browser_fill | Fill a form field by CSS selector. | | spider_browser_screenshot | Capture a screenshot (returned as an inline image). | | spider_browser_content | Get the page HTML or visible text. | | spider_browser_evaluate | Execute JavaScript in the page and return the result. | | spider_browser_wait_for | Wait for a selector, navigation, or network idle. | | spider_browser_close | Close the session and stop billing. |

Result display

Core tools cap processing to the first 10 results (the model is told when more were available, e.g. processed first 10 of 31 results) and render a compact summary in the TUI instead of dumping raw JSON — a one-line N results row that expands to a title — url list. The full (capped) JSON is still sent to the model.

Configuration

The extension reads SPIDER_API_KEY from the environment (pi does not load .env itself). Get a key at https://spider.cloud/api-keys. Optional overrides: SPIDER_API_URL (core REST base) and SPIDER_BROWSER_URL (browser fleet WebSocket).

Primary: pass the key inline

Prefix the command with the key — the simplest way to run a one-off (works whether the extension is installed or loaded with -e):

SPIDER_API_KEY=sk-... pi

Optional: auto-load with direnv

To avoid typing the key each time, keep it in a gitignored .env at the workspace root and let direnv load it whenever you enter the directory:

brew install direnv                          # once
echo 'eval "$(direnv hook zsh)"' >> ~/.zshrc # once (then open a new terminal)

echo 'SPIDER_API_KEY=sk-...' > .env          # gitignored
printf 'dotenv\n' > .envrc                    # loads .env
direnv allow

Pi started from this directory then sees the key automatically. (Or, always on: echo 'export SPIDER_API_KEY="sk-..."' >> ~/.zshrc.)

Install

# install from npm (registers both tiers from package.json)
pi install npm:@spider-cloud/pi-spider-tools
pi list

# or try it for a single session without installing
pi -e npm:@spider-cloud/pi-spider-tools

From a local clone (development)

# install by path (both tiers, from package.json)
pi install ./extensions/pi-spider-tools

# or load a single tier ad-hoc; /reload picks up edits
pi -e ./src/spider.ts          # core tools only
pi -e ./src/browser.ts         # browser tools only
pi -e ./src/spider.ts -e ./src/browser.ts   # both

Commands

/spider           open the menu
/spider status    show API key status and which core tools are enabled
/spider tools     interactively toggle which core Spider tools are active
/spider enable     enable all core Spider tools
/spider disable    disable all core Spider tools
/spider help      show help

/spider-browser status   show API key status and open browser sessions
/spider-browser close    close all open browser sessions
/spider-browser help     show browser help

Core tool selection is persisted to pi-spider-tools-settings.json in your Pi agent directory ($PI_CODING_AGENT_DIR, defaulting to ~/.pi/agent).

Development

npm install
npm run typecheck   # tsc --noEmit
npm run check       # biome + typecheck

See parameter details for each endpoint at https://spider.cloud/docs/api.

License

MIT