SiftWeb

Zero-key web search and safe content extraction for Node.js and Bun. SiftWeb searches the public web and converts HTML pages, plain text, Markdown, JSON, and PDFs into clean Markdown — with no account, no API key, and no browser. Optional adapters expose the same functions as ready-to-use OpenAI, Gemini, and Anthropic tool definitions.

import { fetchContent, search } from "siftweb";

const results = await search("Node.js release notes", { maxResults: 3 });
const page = await fetchContent(results[0].url);

console.log(page.title);
console.log(page.content); // clean Markdown, ads/scripts/nav stripped

Contents

• Why SiftWeb
• Requirements
• Install
• Quick start
• Search
• Fetch content
• Types
• AI tool adapters
• Example: research a topic
• Errors and availability
• Security
• Troubleshooting
• Development
• License

Why SiftWeb

• 🔑 Zero-key search — DuckDuckGo and AnySearch out of the box; no signup, no billing
• 🧹 Clean extraction — Mozilla Readability + Turndown reduce HTML pages to readable Markdown
• 📄 Multi-format — HTML, plain text, Markdown, JSON, and PDF all normalize to the same PageContent shape
• 🛡️ SSRF-safe by default — private IPs, localhost, link-local, and redirect targets are blocked unless you opt out
• ⚡ Bounded and cancellable — timeouts, response-size limits, redirect limits, and AbortSignal support throughout
• 🔌 Bring your own LLM — optional OpenAI / Gemini / Anthropic tool adapters, each a separate entry point that doesn't bloat the core bundle
• 📦 Dual build — ESM and CommonJS with TypeScript declarations for both

Requirements

| | | | --------------------------------- | ----------------------------------------------------------------------------------- | | Runtime | Node.js 18.17+ or a current version of Bun | | Network | Outbound internet access for search and fetching | | Self-hosted search (optional) | A SearXNG instance, if you want a provider you control |

Install

npm install siftweb
yarn add siftweb
pnpm add siftweb
bun add siftweb

Quick start

import { fetchContent, search } from "siftweb";

const results = await search("Node.js release notes", { maxResults: 3 });
console.log(results);
// [{ title, url, snippet }, ...]

if (results[0]) {
  const page = await fetchContent(results[0].url, { maxLength: 20_000 });
  console.log(page.title);
  console.log(page.content);
}

No LLM or AI SDK is involved in this example — search and fetchContent are plain async functions.

Search

search(query, options?)

Returns an array of SearchResult. With the default provider: "auto", SiftWeb tries a configured SearXNG instance first, then falls back to DuckDuckGo and AnySearch until one returns results.

const results = await search("TypeScript NodeNext documentation", {
  maxResults: 5,
  domainFilter: ["typescriptlang.org", "-devblogs.microsoft.com"],
  provider: "auto",
  signal: abortController.signal,
});

domainFilter entries are allowed domains; prefix with - to exclude a domain. Subdomains match their parent domain (docs.python.org matches a filter of python.org). Filtering is applied consistently across every provider.

Use searchWithMetadata when you also need to know which provider answered:

import { searchWithMetadata } from "siftweb";

const { provider, results } = await searchWithMetadata("Bun documentation");

SearchOptions

| Option | Type | Default | Description | | -------------- | ---------------------------------------------------- | -------- | ----------------------------------------------- | | maxResults | number | 5 | Clamped between 1 and 20. | | domainFilter | string[] | — | Allow-list / deny-list (- prefix) of domains. | | provider | "auto" \| "duckduckgo" \| "anysearch" \| "searxng" | "auto" | Explicit providers do not fall back on failure. | | searxngUrl | string | — | Overrides SEARXNG_URL for a single call. | | signal | AbortSignal | — | Cancels the in-flight request. |

Self-hosted SearXNG

Set the SEARXNG_URL environment variable, or pass searxngUrl per call:

const results = await search("privacy tools", {
  provider: "searxng",
  searxngUrl: "https://search.example.org",
});

When SEARXNG_URL is set and provider is left as "auto", SearXNG is tried first, ahead of DuckDuckGo and AnySearch.

Provider-specific functions

searchDuckDuckGo, searchAnySearch, and searchSearXNG are exported directly for callers that want to pin a single provider without going through auto fallback. Each has the same signature as search.

Fetch content

fetchContent(url, options?)

Fetches one public URL and returns normalized PageContent.

const page = await fetchContent("https://example.com/guide.pdf", {
  timeoutMs: 15_000,
  maxLength: 50_000,
  maxResponseBytes: 10 * 1024 * 1024,
  headers: { "Accept-Language": "en" },
  signal: abortController.signal,
});

The returned url is the final URL after redirects. HTML is reduced to its main article with Mozilla Readability where possible, falling back to the full document body. Plain text and Markdown responses are returned as-is (truncated to maxLength), JSON is wrapped in a fenced code block, and PDFs are converted to text.

FetchOptions

| Option | Type | Default | Description | | ------------------ | ------------------------ | -------- | --------------------------------------------------------- | | timeoutMs | number | 30000 | Aborts the request after this many milliseconds. | | maxLength | number | 100000 | Max characters of extracted content before truncation. | | maxResponseBytes | number | 20 MiB | Rejects responses larger than this, by header or by read. | | allowPrivateIps | boolean | false | Disables SSRF protection — see Security. | | headers | Record<string, string> | — | Extra request headers. | | signal | AbortSignal | — | Cancels the in-flight request. |

fetchMultiple(urls, options?)

Fetches several URLs with a bounded worker pool, preserving input order. Per-URL failures are returned inline instead of rejecting the whole batch.

import { fetchMultiple } from "siftweb";

const pages = await fetchMultiple(["https://nodejs.org", "https://bun.sh", "https://deno.com"], {
  concurrency: 3,
  maxLength: 20_000,
});

for (const item of pages) {
  if (item.result) console.log(item.url, item.result.title);
  else console.error(item.url, item.error);
}

concurrency defaults to 3 and is capped at 10. All FetchOptions apply to every URL in the batch.

Types

interface SearchResult {
  title: string;
  url: string;
  snippet: string;
}

interface SearchResponse {
  query: string;
  provider: string;
  results: SearchResult[];
}

interface PageContent {
  title: string;
  url: string;
  content: string;
  byline?: string | null;
  siteName?: string | null;
  excerpt?: string | null;
  length?: number;
}

type FetchResult =
  | { url: string; result: PageContent; error?: never }
  | { url: string; result?: never; error: string };

validateFetchUrl(url, options?), extractHtmlToMarkdown(html, url, maxLength?), and extractPdfToMarkdown(buffer, url, maxLength?) are also exported for callers that want to reuse SiftWeb's SSRF checks or extraction logic on content they already have in hand.

AI tool adapters

Adapters are separate entry points, so they don't increase the size of the core siftweb import. Each contains only tool definitions and an execution helper — no LLM SDK is installed as a dependency.

import { executeOpenAITool, openAITools } from "siftweb/openai";
import { executeGeminiTool, geminiFunctionDeclarations } from "siftweb/gemini";
import { anthropicTools, executeAnthropicTool } from "siftweb/anthropic";

Every adapter exposes two tools: web_search and fetch_content. Pass the exported definitions to the corresponding SDK's tool list, then route tool calls to the matching execution helper.

import OpenAI from "openai";
import { executeOpenAITool, openAITools } from "siftweb/openai";

const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "What's new in Node.js 22?" }],
  tools: openAITools,
});

const call = response.choices[0].message.tool_calls?.[0];
if (call) {
  const result = await executeOpenAITool(call.function.name, call.function.arguments);
  console.log(result); // JSON string ready for a tool message
}

import { GoogleGenerativeAI } from "@google/generative-ai";
import { executeGeminiTool, geminiFunctionDeclarations } from "siftweb/gemini";

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
const model = genAI.getGenerativeModel({
  model: "gemini-1.5-pro",
  tools: [{ functionDeclarations: geminiFunctionDeclarations }],
});

const call = /* functionCall from a model response */;
const result = await executeGeminiTool(call.name, call.args);

import Anthropic from "@anthropic-ai/sdk";
import { anthropicTools, executeAnthropicTool } from "siftweb/anthropic";

const anthropic = new Anthropic();
const message = await anthropic.messages.create({
  model: "claude-3-5-sonnet-latest",
  max_tokens: 1024,
  tools: anthropicTools,
  messages: [{ role: "user", content: "Search for the latest TypeScript release." }],
});

const toolUse = message.content.find((block) => block.type === "tool_use");
if (toolUse) {
  const result = await executeAnthropicTool(toolUse.name, toolUse.input);
}

Example: research a topic

The repository includes a no-LLM demo that searches the web for a topic, then fetches and extracts the top results concurrently, using only search and fetchMultiple:

npm run example:research -- "your topic here"

See examples/research-topic.mjs.

Errors and availability

Search without an API key depends on public provider endpoints. Providers may rate-limit requests, present a challenge page, change their response format, or be temporarily unavailable. auto mode gives you fallback across providers, but it is not a replacement for a search service with a contractual SLA.

When every provider fails, search and searchWithMetadata throw a single AggregateError whose .errors array contains one entry per provider that was tried:

try {
  await search("query");
} catch (error) {
  if (error instanceof AggregateError) {
    for (const providerError of error.errors) console.error(providerError);
  }
}

An explicit provider (anything other than "auto") does not fall back — a failure from that provider is thrown directly.

Security

fetchContent and fetchMultiple are SSRF-safe by default:

• Only http: and https: URLs are allowed
• URLs with embedded credentials (https://user:pass@host/) are rejected
• Requests to localhost, loopback, link-local, private, and other non-public IP ranges are blocked — both for the literal hostname and for every address it resolves to
• Redirects are re-validated at each hop (capped at 5), and sensitive headers (Authorization, Cookie, Proxy-Authorization) are dropped when a redirect crosses origins
• Responses are capped at 20 MiB by default, checked against both the declared Content-Length and the actual bytes read

Pass allowPrivateIps: true to disable the private-network checks for trusted, internal use cases only. Never enable it for a URL supplied by an untrusted user.

SiftWeb performs plain HTTP fetches and does not execute page JavaScript. Sites that require a browser, a login, a CAPTCHA, or an anti-bot challenge may not be extractable.

Troubleshooting

• AggregateError: All configured web-search providers failed... — every provider returned an error or zero results. Inspect error.errors for the per-provider reason; public endpoints occasionally rate-limit or change their response shape. Configure SEARXNG_URL for a provider you control.
• SSRF blocked: ... — the target host resolves to a private, loopback, or link-local address. This is intentional; set allowPrivateIps: true only for trusted, internal URLs.
• Response body is too large / ...exceeded the N-byte limit — raise maxResponseBytes in FetchOptions for large pages or PDFs.
• Content looks stripped or off-topic — some sites need JavaScript to render their main content; Readability then falls back to the raw document body. Verify the page renders meaningful HTML from a plain HTTP GET.
• SearXNG URL is not configured — set the SEARXNG_URL environment variable or pass searxngUrl explicitly when using provider: "searxng".

Development

npm install
npm run check      # typecheck + lint + format:check + test

| Script | Purpose | | --------------------------------- | ----------------------------------------------------------------------------------- | | npm run build | Bundle ESM + CJS + type declarations with tsup. | | npm run dev | Rebuild on change. | | npm run typecheck | tsc --noEmit. | | npm run lint / lint:fix | ESLint over the whole project. | | npm run format / format:check | Prettier write / check. | | npm test | Builds, then runs deterministic unit and integration tests (no live network). | | npm run test:e2e | Also exercises live DuckDuckGo search and https://example.com/ (SIFTWEB_E2E=1). | | npm run check | Everything CI should run: typecheck, lint, format check, and tests. |

License

MIT