henyo-pi-web

Web search and content extraction tools for Pi.

Henyo means "genius" in Filipino — because Pi is sharp, and so are you.

Install

Ask your Pi agent:

Install the henyo-pi-web npm package

Or run directly:

pi install npm:henyo-pi-web

pi install automatically resolves npm dependencies (defuddle, jsdom) and registers the six web tools plus the web_tools loader — the tools are lazy and stay inactive until activated via web_tools or /web-tools.

Tools

search_ddg

General web search via DuckDuckGo. Use for news, articles, broad topics, general queries, and as a fallback when no specialized tool fits — keep queries focused and short. Don't use for specific programming errors (→ search_stackoverflow).

Parameters:

• query (string) — Search query (any topic or phrase)
• max (integer, default 10, 1–50) — Max results to return
• noCache (boolean, default false) — Skip cache

search_wikipedia

Encyclopedia knowledge via Wikipedia. Use for definitions, concepts, history — query with short topic names (e.g. "React (software)", "Kubernetes"), not full questions. Don't use for code errors (→ search_stackoverflow).

OpenSearch is prefix/title match — short topic names (e.g. "React (software)") work best; natural-language questions may return 0 results.

Parameters:

• query (string) — Short topic name (e.g. "React", "Kubernetes")
• max (integer, default 10, 1–50) — Max results to return
• noCache (boolean, default false) — Skip cache

search_stackoverflow

Programming Q&A via Stack Overflow. Use for error messages, code patterns, debugging, syntax, API usage — include the full error message and code pattern. Don't use for package lookups (→ search_npm).

Parameters:

• query (string) — Error message or code pattern
• max (integer, default 10, 1–50) — Max results to return
• noCache (boolean, default false) — Skip cache

search_npm

JavaScript package registry search. Use for package names, JS library functionality, dependency lookups — query short and specific (e.g. "react", "state machine"). Don't use for non-JS packages (pip, crates) (→ search_ddg).

Parameters:

• query (string) — Package name or functionality description
• max (integer, default 10, 1–50) — Max results to return
• noCache (boolean, default false) — Skip cache

search_github

GitHub repository and issue search. The query passes raw to the GitHub Search API, so GitHub search qualifiers work — e.g. repo:owner/name, is:issue. Unqualified free-text matches all of GitHub and is noisy; for a targeted in-repo issue hunt, use repo:owner/name <keywords> or fall back to search_ddg with site:github.com/owner/repo <keywords> (e.g. site:github.com/earendil-works/pi session web). Don't use for package docs (→ search_npm).

Parameters:

• query (string) — Repository/library name, issue keywords, or full GitHub search query (qualifiers like repo:owner/name, is:issue pass through)
• max (integer, default 10, 1–50) — Max results to return
• noCache (boolean, default false) — Skip cache

henyo_fetch

Extract clean readable content from any URL. Uses Defuddle-first extraction with Jina quality-check fallback. Handles Cloudflare protection, SPAs, GitHub raw files, JSON, plain text, and binary content detection (PDF, images, archives). Includes SSRF protection. Cached 1 hour.

Parameters:

• url (string) — URL to fetch
• timeout (integer, default 15000) — Timeout in ms (1000–60000)
• noCache (boolean, default false) — Skip cache
• headers (object, optional) — Custom HTTP headers, e.g. { "Authorization": "Bearer token" } (a custom User-Agent — or any header — is honored and overrides the built-in defaults)

Features:

• Content-type aware: handles HTML, JSON, plain text, and binary content
• On 401/403/503 (e.g. Cloudflare blocks), henyo_fetch automatically retries via the Wayback Machine; the result is tagged source: 'wayback' with the snapshot date and original URL. Disable with waybackEnabled: false.
• Smart truncation with configurable heading/content thresholds
• Oversized content returns metadata only (URL, title, source, cache path)
• Politeness delay between requests (configurable min/max)
• Retry with exponential backoff
• cached flag on cached results
• Error categories in details (errorCategory: ssrf, invalid-url, timeout, not-found, forbidden, bad-request, server-error, network, unknown)

TUI Features:

• Source badges — color-coded [defuddle], [jina], [github], [wayback], etc.
• Size labels — human-readable sizes (12.3 KB, 1.45 MB)
• Status indicators — [cached], [truncated], [oversized] badges
• Error cards — categorized errors with actionable messages
• Oversized content card — structured metadata with guidance (reduce threshold, check cache, fresh fetch)
• Collapsible content — press expand key to view full content, collapse to return to header
• TUI settle delay — execute waits 100ms before fetching so the TUI can render the tool's pending state

Activation

The six web tools are lazy: registered at load but inactive until activated for the session. The model activates them via the always-active web_tools tool (purely additive — once enabled, all six stay active); users can force activation with /web-tools. deep_research joins the same lifecycle: it is registered when the henyo-web.skills.deep-research gate is enabled (default) and is hidden/activated together with the six tools. The web_tools result now includes the full web-tools guide (YAML frontmatter stripped, so the query-shaping rules that prevent zero-result failures lead) instead of a condensed routing reference — rendered as a collapsed one-line web tools guide loaded (X KB) header in the TUI — so the model gets failure-prevention guidance in-context immediately after activation. When the henyo-web.skills.web-tools gate is off, the result is the activation line only. Activation is per-session: a new session starts with the web tools hidden again.

For routing, query shaping, and zero-results recovery, load the web-tools skill (see Bundled Skills).

Structure

henyo-pi-web/
├── package.json          # Extension manifest with pi entry point
├── index.ts              # Extension entry point (web tools, lazy web_tools loader + web-tools guide injection, deep_research tool, /web-tools, skills)
├── skills/
│   ├── deep-research/    # Multi-step autonomous research workflow with henyo-pi-web tools
│   │   └── references/   # Reference docs (evidence collection, source credibility, report templates)
│   └── web-tools/        # Routing + query shaping + zero-results protocol skill
├── shared/               # Shared utilities between tools
│   ├── web-tools.ts      # Lazy activation core (WEB_TOOL_NAMES, hide/activate with optional names list)
│   ├── config.ts         # Settings loading, merging & validation
│   ├── cache.ts          # File-based cache for fetch results
│   ├── format.ts         # Result formatting helpers
│   ├── ip-network.ts     # IP/CIDR network matching (SSRF checks)
│   ├── rate-limit.ts     # Per-domain rate limiting store
│   ├── skills.ts         # Skill registration gate
│   ├── user-agents.ts    # Rotating user-agent pool + delay helpers
│   ├── search/           # Search tool implementation (context.ts, execute.ts, queue.ts, trace.ts)
│   │   └── providers/    # Per-engine providers: base.ts, index.ts, duckduckgo, github, npm, stackoverflow, wikipedia
│   └── fetch/            # henyo_fetch pipeline (pipeline, security, extract, html-extraction, content-handlers, detection, github, pdf-extract, retry, truncate, ui, index)
├── tests/                # Unit tests
├── vitest.config.ts      # Vitest test runner config
├── tsconfig.json         # Strict typecheck config (`tsc --noEmit`, part of `pnpm lint`)
├── eslint.config.mjs     # ESLint flat config (mirrors henyo-pi-core)
├── .prettierrc.json      # Prettier rules (identical to henyo-pi-core)
├── .prettierignore       # Prettier ignore list
└── README.md

Testing, coverage & linting

• pnpm test runs the full suite (35 test files) with no coverage collection. pnpm coverage runs the same suite under v8 coverage. vitest.config.ts defines two projects: unit (all tests except the two jiti-based ones) and smoke (index-load.test.ts + fetch-trace.test.ts), which load index.ts through jiti — a second module loader whose transformed copies of shared/*.ts previously collided with vitest's native scripts during the v8 merge (same file URL, different function ranges → spurious uncovered lines; ~84.5% reported vs ~98.2% true). Splitting them into their own project keeps the merge clean and removes the former CLI --exclude flags from the coverage script.
• Linting & formatting mirror the henyo-pi-core tooling stack:
  • pnpm lint runs the strict typecheck (tsc --noEmit against tsconfig.json) plus ESLint (flat config: recommended rules + Prettier integration, no-console and unused-vars at warn level). pnpm lint:fix auto-fixes fixable ESLint findings.
  • Lint is gated at zero warnings (eslint --max-warnings 0) and enforced by CI on every push/PR.
  • pnpm format checks Prettier formatting on shared/**/*.ts, tests/**/*.ts, and index.ts; pnpm format:fix applies it. .prettierrc.json is identical to core's.
• vitest.config.ts sets coverage thresholds at observed-final-minus-0.5 (lines 99.3 / statements 99.32 / branches 97.63 / functions 99.5), so a regression of ≥ 0.5 points fails the coverage run. The residual branch gaps below the thresholds are unreachable defensive code or v8→istanbul mapping artifacts, dispositioned in the coverage plan.
• Unreachable defensive code was deleted rather than papered over with tests: the security.ts IP null-paths, html-extraction.ts trailing fallback, trace.ts over-bound unlink check, and cache.ts eviction guard wrapper in commit 4fb774b (eviction at maxFiles=0 no longer hangs), plus the duckduckgo.ts !html guard in 4dd1205.
• Current coverage (pnpm coverage, shared/**/*.ts): 100% statements, 97.97% branches, 100% functions, 100% lines (1214/1214 statements, 823/840 branches, 143/143 functions, 1101/1101 lines).

Bundled Skills

Both skills are excluded from the model's system prompt by default (disable-model-invocation: true frontmatter), so they cost nothing until needed. Load them explicitly via /skill:deep-research or /skill:web-tools.

/skill:deep-research

A structured methodology for conducting deep, multi-step research — designed to work alongside henyo-pi-web's search and fetch tools. Guides the agent through planning, iterative retrieval, cross-source validation, and synthesis into a structured report with full citations. Use for complex research questions, competitive analysis, literature reviews, or any task requiring thorough investigation beyond a single search. /skill:deep-research and the deep_research tool both return the same SKILL.md (single source of truth) — the tool is the model's self-serve path; /skill: keeps working for users.

Workflow: Plan → Retrieve → Cross-Validate → Synthesize → Report

/skill:web-tools

Routing and usage guide for the six web research tools: the tool routing matrix, query-shaping rules with good/bad examples, the zero-results protocol (rephrase shorter → next tool → report; cooldown and provider-error handling), fallback chains, and runtime behavior (30-min search cache, 1-hour fetch cache, noCache, oversized-fetch envelope). Load before a multi-query research pass or when a web search returns zero or weak results. The web_tools tool's activation auto-injects this guide (frontmatter-stripped, collapsed in the TUI), so the model already has it in-context after activation; henyo-web.skills.web-tools: false yields the activation line only (no guide text). /skill:web-tools still works for users who want to load it explicitly.

Configuration

Optional settings go in ~/.pi/agent/settings.json (shared with the rest of pi's settings). Invalid values are logged to stderr ([henyo-pi-web] ...) and fall back to defaults rather than crashing; a corrupt settings file logs a warning and loads all defaults.

{
  "henyo-web": {
    "fetch": {
      "jinaEnabled": true,
      "waybackEnabled": true,
      "min-delay": 1000,
      "max-delay": 3000,
      "cache-max-files": 100,
      "heading-threshold": 40000,
      "content-threshold": 32000,
      "jina-timeout": 30000,
      "max-response-size": 10485760
    },
    "search": {
      "trace": true,
      "providers": { "stackoverflow": { "api-key": "SO-KEY" } }
    },
    "skills": {
      "deep-research": true,
      "web-tools": true
    }
  }
}

henyo-web.fetch config options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | jinaEnabled | boolean | true | Enable Jina Reader fallback | | waybackEnabled | boolean | true | Auto-fallback to Wayback Machine on 401/403/503 blocks (result tagged source: 'wayback') | | min-delay / max-delay | number | 1000 / 3000 | Politeness delay range (ms) | | cache-max-files | number | 100 | Max cached files per directory | | heading-threshold | number | 40000 | Heading size for smart truncation | | content-threshold | number | 32000 | Max content size; oversize returns metadata only | | jina-timeout | number | 30000 | Jina fallback timeout (ms) | | max-response-size | number | 10485760 | Max response body size (bytes) |

henyo-web.search config options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | trace | boolean | string[] | none | Trace logging for search and fetch: true traces everything, a list of names (duckduckgo, github, stackoverflow, npm, wikipedia, search_ddg, …, henyo-fetch) traces only those. Logs to /tmp/henyo-trace.log with rotation (10 MB, 3 backups). | | providers.<name>.api-key | string | none | Per-provider API key — currently stackoverflow only. An SO StackExchange key raises the SO API quota from the shared anonymous limit to the per-user quota. |

henyo-web.skills config options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | <name> | boolean | true | Registration gate per bundled skill (deep-research, web-tools). false excludes the skill entirely — not in the prompt and not available as /skill:. The deep-research gate also excludes the deep_research tool (not registered, not part of the web_tools//web-tools activation lifecycle). |

Per-provider blocks live under providers, keyed by provider name (stackoverflow, duckduckgo, wikipedia, npm, github) — provider names are reserved.

Tool contract: each search tool returns only its own provider's results. A provider failure surfaces as Provider error (…) or Search cooling down …, never as another provider's results; a genuine no-matches query returns 0 results. Rate-limit cooldowns (built-in per-provider defaults) are enforced and reported, not swallowed.

Trace logging

When henyo-web.search.trace is enabled, every outcome of the whole search and fetch process is appended to /tmp/henyo-trace.log — so "what happened to my search/fetch?" is answerable from one log:

• Search, tool layer (search_ddg, search_wikipedia, search_stackoverflow, search_npm, search_github): cache hits, cooldown blocks, provider errors, aborts, no-results, successes.
• Search, provider layer (duckduckgo, github, stackoverflow, npm, wikipedia): successes and failures — including the rate-limit events that set a cooldown (e.g. error="http-429", error="captcha", error="so-api-rate-limited"), logged at the moment the cooldown is written.
• Fetch (henyo-fetch): successes, cache hits, oversized/size-exceeded, and errors with the errorCategory (ssrf, network, timeout, …).

Example lines:

[2026-08-28T10:15:03.221Z] search_ddg query="react state management" duration=1204ms results=0 status="cooling-down" error="duckduckgo"
[2026-08-28T10:15:03.180Z] duckduckgo query="react state management" duration=1198ms results=0 status="error" error="http-429"
[2026-08-28T10:16:11.902Z] henyo-fetch query="https://example.com/docs" duration=2310ms results=48211 status="ok"

Line format: [timestamp] <provider|tool> query="…" duration=<ms> results=<n> status="…" [error="…"] [instance="…"]. For henyo-fetch, results= is the content size in bytes, not a result count.

status values: ok, cache-hit, cooling-down, no-results, oversized, size-exceeded, aborted, error. error= carries the rate-limit event tag on cooldown-setting events (http-429, http-403, captcha, so-api-rate-limited, so-api-http-<status>, scraper-unavailable), the fetch errorCategory on fetch failures, the blocking provider key on cooldown blocks, and provider error messages otherwise (e.g. No endpoint succeeded).

Requirements

• Node.js (ESM modules)
• Internet access

License

MIT