@sisu-ai/tool-web-fetch

v9.0.0

Published

4 days ago

Fetch web pages and return structured text, HTML, or JSON for LLM consumption.

0High
0Medium
0Low

lejahmie

sisu ai ai-agent agentic tool web-fetch

@sisu-ai/tool-web-fetch

Fetch web pages and return structured text, HTML, or JSON for LLM consumption.

Install

npm i @sisu-ai/tool-web-fetch

Environment / Flags

WEB_FETCH_USER_AGENT or HTTP_USER_AGENT (flag: --web-fetch-user-agent)
WEB_FETCH_MAX_BYTES (flag: --web-fetch-max-bytes) — default 500kB
WEB_FETCH_RESPECT_ROBOTS (flag: --web-fetch-respect-robots) — 1/true (default) to honor robots.txt; set 0/false to disable

Tool

Name: webFetch
Args: { url: string; format?: 'text'|'html'|'json'; maxBytes?: number }
Returns: { url, finalUrl?, status, contentType?, title?, text?, html?, json? }

Behavior

Respects robots.txt by default for the provided User-Agent.
Follows redirects and reads up to maxBytes to avoid huge pages.
If format: 'text' (default) and page is HTML, strips tags (removes script/style) and decodes basic entities; includes title.
If format: 'html', returns raw HTML and title.
If server returns JSON or format: 'json', parses into json.
Non-OK responses return status and a short text body snippet for debugging.

Notes

This is a minimal fetcher to empower summarization / extraction workflows. For deeper crawling, add queueing, URL normalization, and robots.txt handling in upstream middleware.

Community & Support

Discover what you can do through examples or documentation. Check it out at https://github.com/finger-gun/sisu. Example projects live under examples/ in the repo.