pi-crawl4ai

A production-ready pi package that integrates Crawl4AI — an open-source LLM-friendly web crawler — as a custom tool and agent prompt.

Install

pi install npm:pi-crawl4ai

Or from git:

pi install git:github.com/romek-rozen/pi-crawl4ai

What's included

• extensions/crawl4ai/ — the full extension source (tool, commands, renderers)
• agents/ — specialized agent definitions for use with subagent/run
• prompts/crawler.md — a dedicated Crawler prompt template
• MIT license

Quick start

1. Install Crawl4AI (inside pi):

   /crawl4ai-install

   This creates a Python venv and installs crawl4ai locally.

2. Verify:

   /crawl4ai-status
   /crawl4ai-test

3. Set up agents (optional, for subagent workflows):

   /crawl4ai-setup-agents

4. Crawl:

   Crawl https://example.com and give me the markdown.

Usage

Direct tool usage

The crawl4ai tool is available to the LLM in any pi session. Just ask:

Crawl https://example.com and give me the markdown.
Crawl https://docs.example.com deeply using BFS, max 10 pages.
Extract all product names and prices from https://shop.example.com as JSON.

Prompt templates

Three prompt templates are available:

/crawl4ai https://example.com                       # general — scrape, crawl, or extract
/crawl4ai-scrape https://example.com                 # single page → clean markdown
/crawl4ai-extract https://example.com product prices # single page → structured JSON

Subagent workflows

After running /crawl4ai-setup-agents, three agents become available for the subagent tool:

Single agent:

Use crawl4ai-scrape to get the content of https://example.com
Use crawl4ai-crawl to explore https://docs.example.com with max 10 pages
Use crawl4ai-extract to get all product prices from https://shop.example.com

Parallel execution:

Run 3 crawl4ai-scrape agents in parallel: one for https://example.com, one for https://docs.example.com, one for https://blog.example.com

Chained workflow:

Use a chain: first have crawl4ai-scrape get the page at https://shop.example.com, then have crawl4ai-extract pull structured product data from {previous}

Tool parameters

| Parameter | Description | |-----------|-------------| | url (required) | Target URL | | output_format | markdown (default), markdown-fit, md, md-fit, json, all | | deep_crawl | bfs, dfs, or best-first | | max_pages | Limit for deep crawl | | question | Natural-language question about the page | | json_extract | LLM extraction prompt (requires LLM provider configured in crawl4ai) | | schema_path | JSON schema file for structured extraction (requires extraction_config) | | extraction_config | Extraction strategy config file (YAML/JSON) (required with schema_path) | | browser_config / crawler_config | Key=value pairs | | bypass_cache | Force fresh crawl | | output_file | Save directly to a custom path | | timeout | Seconds, default 60 |

Commands

| Command | Purpose | |---------|---------| | /crawl4ai-install | Create local venv and install crawl4ai | | /crawl4ai-test | Run a smoke test crawl on example.com | | /crawl4ai-status | Show resolved binary path and version | | /crawl4ai-clear-cache | Remove local .crawl4ai/cache and .crawl4ai/robots | | /crawl4ai-setup-agents | Symlink agents to ~/.pi/agent/agents/ for use with subagent/run |

Agents

Three specialized agents are included for use with the subagent tool:

| Agent | Purpose | Model | |-------|---------|-------| | crawl4ai-scrape | Scrape a single page, extract clean markdown | Sonnet | | crawl4ai-crawl | Crawl multiple linked pages (BFS/DFS/best-first) | Sonnet | | crawl4ai-extract | Extract structured data as JSON (LLM or CSS/XPath) | Sonnet |

To set up agents, run inside pi:

/crawl4ai-setup-agents

This symlinks the agent definitions to ~/.pi/agent/agents/. After setup, you can use them with the subagent tool:

Use crawl4ai-scrape to get the content of https://example.com
Use crawl4ai-crawl to explore https://docs.example.com with max 10 pages
Use crawl4ai-extract to get all product prices from https://shop.example.com

JSON extraction requirements

output_format=json requires an extraction strategy:

• LLM extraction: pass json_extract (e.g. "Extract all product prices"). Requires a configured LLM provider in Crawl4AI (run crwl once interactively or set up ~/.crawl4ai/global.yml).
• CSS/XPath extraction: pass schema_path + extraction_config. Example extraction config YAML:

  type: json-css
  params:
    verbose: true

⚠️ json output with deep_crawl is not supported by Crawl4AI. Use markdown for deep crawls.

Architecture

| File | Responsibility | |------|----------------| | index.ts | Entry point — wires everything into ExtensionAPI | | types.ts | Typebox schemas + TypeScript interfaces | | tool.ts | Tool definition, validation, and execution logic | | args.ts | Maps friendly params to CLI flags | | resolve.ts | Binary discovery, env vars, output path helpers | | runner.ts | Spawns crwl with timeout, abort, and streaming | | render.ts | Custom TUI rendering | | commands.ts | Slash commands (/crawl4ai-*) | | format.ts | Legacy truncation helper |

Development

The extension runs inside pi’s extension loader. After modifying source, reload pi and use /crawl4ai-status or /crawl4ai-test to verify.

Troubleshooting

• "Binary not found" → run /crawl4ai-install or set CRAWL4AI_VENV=/path/to/venv
• "No default LLM provider configured" → configure a provider in Crawl4AI before using json_extract
• "the JSON object must be str, bytes or bytearray, not NoneType" → usually missing extraction_config when using schema_path, or the page has no matching content
• Stale cache → run /crawl4ai-clear-cache
• Timeouts → increase timeout param for slow sites

Links

• npm: https://www.npmjs.com/package/pi-crawl4ai
• GitHub: https://github.com/romek-rozen/pi-crawl4ai

License

MIT — see LICENSE