searchcast

Keyless search engines over HTTP with a real browser's fingerprint, driven by recipes, with a real browser as the fallback.

Getting keyless web search results today usually means running SearXNG. What SearXNG really contributes is small: engine adapters that turn a results page into title/url/snippet, an HTTP client that looks like a real browser at the TLS and HTTP/2 level (curl_cffi over libcurl-impersonate), and per-engine handling of challenges and failures. searchcast is that, as a Node library plus a small CLI:

• Engines are described by recipes. A declarative recipe is one JSON file that searchcast runs over plain HTTP and the browser runner (@searchcast/browser) runs in a real browser. A code recipe is a JS module for sites that need challenge handling or a non-HTML API.
• All engine traffic goes through libcurl-impersonate, pinned to one explicit Chrome version, with the exact header set that Chrome sends for each kind of request, so the TLS side and the header side describe the same browser (ADR 0001).
• Engines are tried as an ordered engine chain, first answer wins, with a real browser (the browser runner) as the fallback when HTTP is blocked.
• searchcast is not an anonymity tool, and it is built so one can use it safely: the caller injects the proxy, the state store and the recipe set; searchcast makes no network call the caller did not cause and writes nothing to disk on its own (ADR 0002).

Status: in development (0.x; the API may still change between minor versions). Available: the engine chain with its state store, the transport, the declarative recipe runner, code recipes and browser engines, with searchcast query for recipe development, searchcast install-libcurl to install the native library, searchcast doctor to check it, searchcast install-recipes to install a checksum-pinned set of recipes (or one from IPFS, verified against its CID), and searchcast serve to serve recipes from a real browser as a JSON API (with @searchcast/browser installed).

searchcast was called serpcast until 0.2.0, and the name searchcast 0.1.x was the browser runner: see Upgrading from serpcast and Upgrading from searchcast 0.1.x. Every rename is recorded in ADR 0005.

Packages

This repository is a pnpm workspace; every package is released from it.

| package | what it is | license | | ------- | ---------- | ------- | | searchcast | The library (transport, recipe runners, engine chain) and the searchcast CLI. This README. | AGPL-3.0-only | | @searchcast/browser | The browser runner: runs a recipe in a real Chromium or Chrome. An optional peer dependency of searchcast, needed only for library-mode browser engines, searchcast serve and searchcast browser-query. Formerly the searchcast package 0.1.x. | AGPL-3.0-only | | @searchcast/recipe | The recipe schema, its TypeScript types and its validator, zero dependencies. Shared by searchcast and @searchcast/browser so one recipe file describes a site for both; MIT so projects under any license can share the format (ADR 0003). Formerly serpcast-recipe. | MIT | | @searchcast/libcurl-<os>-<arch> | The pinned libcurl-impersonate for one platform (linux-x64, linux-arm64, darwin-x64, darwin-arm64, win32-x64), built in the release workflow from the pinned upstream archive after verifying its sha256. Optional dependencies of searchcast, so npm install searchcast brings the library on those platforms with no separate step and no download at run time; see Installing libcurl-impersonate. | the library's components (MIT, curl, Apache-2.0, BSD-3-Clause, Zlib; on Linux also LGPL-3.0-or-later OR GPL-2.0-or-later, Unicode-DFS-2016), see each package's LICENSE |

Upgrading from serpcast

searchcast 0.2.0 is serpcast 0.6.0 renamed; serpcast is deprecated and gets no further features. Install searchcast instead:

npm uninstall serpcast && npm install searchcast

• The bin is searchcast, with every serpcast command, the same options, output and exit codes (searchcast query, install-libcurl, install-recipes, recipes list, doctor); messages start with searchcast:.

• The API is renamed: createSearchcast, Searchcast, SearchcastOptions, SearchcastError, SearchcastErrorKind, and the embedder entry is searchcast/install. For one release (0.2.x) the old names createSerpcast, Serpcast, SerpcastOptions, SerpcastError and SerpcastErrorKind are still exported, as deprecated aliases of the same values and types (instanceof SerpcastError still works). SearchcastError's name is SearchcastError.

• The environment variable is SEARCHCAST_LIBCURL_PATH. SERPCAST_LIBCURL_PATH is still read for one release, after the new one; searchcast doctor says when it is the one used.

• The data directory is $XDG_DATA_HOME/searchcast (default ~/.local/share/searchcast), where install-libcurl and install-recipes now write. For one release, the library and recipe sets are still read from serpcast's $XDG_DATA_HOME/serpcast when the new directory lacks them (a recipe set in the new directory wins over one of the same name in the old). Nothing moves your files: searchcast doctor lists what it reads from the old directory and prints the command that moves it, for example:

  mv ~/.local/share/serpcast ~/.local/share/searchcast

  (when the new directory already exists, doctor prints one mv per item instead). Embedders can show the same notice with oldDataDirHits().

• Stored sessions are kept: the state-store key serpcast/sessions keeps its name.

• Recipes need no change; the schema package is @searchcast/recipe (formerly serpcast-recipe, same format, same API).

The old names, the old variable and the old directory are no longer read from the next minor after 0.2.x.

Upgrading from searchcast 0.1.x

searchcast 0.1.x was the browser runner. It is now the library @searchcast/browser, which exports what searchcast 0.1.2 exported (Searchcast, SearchcastError, findChrome, ...): replace searchcast with @searchcast/browser in your imports (npm install @searchcast/browser). Its Searchcast and SearchcastError are not this package's (this package's SearchcastError has a kind, the browser runner's a code).

The command line stays on the searchcast bin: searchcast serve, with the same flags, defaults, messages and exit codes (systemd socket activation, --idle-exit, --xvfb, Unix sockets), delegating to @searchcast/browser. After upgrading, install @searchcast/browser next to searchcast, and existing systemd units and SearXNG settings work unchanged:

npm install -g searchcast @searchcast/browser

Without it, searchcast serve exits with code 1 and one line saying to install it. The browser one-shot query that 0.1.x called searchcast query is now searchcast browser-query (searchcast query is the HTTP query), and the SearXNG engine stays at integrations/searxng/searchcast.py in the searchcast package.

Transport (libcurl-impersonate)

Every engine request goes through the transport: a small direct koffi binding to libcurl-impersonate (not impers, see work/notes/findings/impers-fingerprint-vs-curl-cffi.md), pinned to chrome146 with the library's default headers off. Each request carries exactly the header table Chrome 146 on Linux sends for its request kind (document, same-origin-navigation, fetch, script), with cookies from the session placed where Chrome puts them (and, on a fetch, the author headers a caller adds, placed the same way).

import {createTransport} from 'searchcast';

const transport = createTransport({proxy: 'socks5h://127.0.0.1:9050'});
const session = transport.session(); // cookies; session.cookies() is plain JSON
const page = await session.request('https://example.com/', {kind: 'document'});
const api = await session.request('https://example.com/api?q=x', {kind: 'fetch', referer: page.url});
session.documentCookies.set(page.url, 'token=abc; Path=/'); // as the page's script would (document.cookie)
session.close(); // closes the session's connections (cookies are kept)

createTransport(options) takes:

| option | default | meaning | | --------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | libcurlPath | SEARCHCAST_LIBCURL_PATH, LIBCURL_PATH, the data directory (and the old names), the platform package, see below | The libcurl-impersonate shared library (see Finding the library, below). | | proxy | none (direct) | The proxy for all traffic (http://, socks5://, socks5h://, see Proxy and DNS, below). | | strict | true | Refuse to send unless libcurl-impersonate accepts the pinned target (see Strict mode, below). A security guarantee: false sends with a non-browser fingerprint. | | timeoutMs | 15000 (15 s) | Per-request time limit. | | caPath | the library's default | A PEM CA bundle to verify servers against. | | maxBodyBytes | 16 MiB | Largest response body, before and after decoding; larger is a transport error. | | reuseConnections | true | Keep a session's connections open between its requests, as Chrome does (see Connections, below). false: one connection per request, closed after it (serpcast 0.1's behaviour), so a session's requests are not linked at the connection level (its cookies still link them). | | idlePollMs | 5 | How long an in-flight request waits between two looks at sockets that had nothing to do, in ms. Lower costs more CPU per waiting request, higher adds up to that much latency to each network event. | | maxRequestBodyBytes | 1 MiB (MAX_REQUEST_BODY_BYTES) | Largest POST body; larger is a recipe error and nothing is sent. | | preflightCache | true | Remember an allowed CORS preflight for its access-control-max-age, as Chrome does. false: preflight every such request. | | maxPreflightAgeS | 7200 (Chromium's cap) | The longest a preflight is remembered, in seconds. |

Every numeric option must be a positive finite number (a whole number for times in ms passed to libcurl, byte sizes and counts); anything else, or a boolean option that is not a boolean, throws a RangeError naming the option when the transport (or createSearchcast) is created. The defaults are the behaviour of earlier versions, so nothing changes unless you set an option.

• Header tables and sec-fetch-site. document is a typed-URL navigation (sec-fetch-site: none) and same-origin-navigation a link click on the same origin. For fetch and script, sec-fetch-site is derived from the request URL relative to the referer (the page it comes from), as Chrome does: same scheme, host and port is same-origin; same scheme and registrable domain (any port) is same-site (https://cdn.example.com from https://www.example.com); anything else, http to https included, is cross-site. A request that is not same-origin sends only the page's origin as referer (Chrome's default referrer policy), a fetch adds origin, and a cross-site one adds sec-fetch-storage-access: active, as measured in work/notes/findings/sec-fetch-site-by-initiator.md. The registrable domain comes from a small built-in rule, not the public suffix list: the last two labels, or three under a two-letter TLD with a common second level (example.co.uk, example.com.au). It does not know private suffixes such as github.io, so pass fetchSite: 'cross-site' (or 'same-site', 'same-origin') when you know better: session.request(url, {kind: 'script', referer, fetchSite: 'cross-site'}).

• POST (fetch only). session.request(url, {kind: 'fetch', method: 'POST', referer, body, contentType}) sends a page's fetch() POST with Chrome's exact headers (content-length first, content-type after sec-ch-ua, origin always, same-origin included), as measured in work/notes/findings/post-requests.md. body is a string (UTF-8) or a Uint8Array, at most maxRequestBodyBytes (default MAX_REQUEST_BODY_BYTES, 1 MiB); contentType defaults as fetch() does (text/plain;charset=UTF-8 for a string, none for bytes). When Chrome would send a CORS preflight first (another origin, same-site included, with a content-type that is not CORS-safelisted, such as application/json), the session sends it too: an OPTIONS without cookies, on a connection of its own (as Chrome keeps credential-less requests apart), remembered for its access-control-max-age (default 5 s, at most maxPreflightAgeS; preflightCache: false remembers none). If the preflight does not allow the POST, the POST is not sent: a 202/403/429 answer is blocked, 404/410 recipe, another non-2xx transport, and CORS headers that do not allow it (access-control-allow-origin not the page's origin, no access-control-allow-credentials: true, content-type not allowed) are recipe. Navigation (form) POSTs are not offered.

• Author headers (fetch only). session.request(url, {kind: 'fetch', referer, headers: {'API-Key': key}}) (GET or POST) adds headers as a page's script adds them to its fetch(): only the measured name sets, placed as Chrome places them and preflighted when Chrome would preflight them. See Headers a code recipe adds for the rules.

• Connections. A session keeps its connections open between its requests and reuses them, as Chrome does: one HTTP/2 connection per origin (concurrent requests to an origin wait for it rather than opening a second one), through the same proxy tunnel when there is a proxy. Connections are never shared between sessions: sessions (two engines, two callers) must stay unlinkable, and a shared connection would link them at the TLS and IP layer. session.close() closes them (at once when idle, else when the requests in flight settle); a later request opens a new one. An idle session keeps nothing running, so it never keeps the process alive. With reuseConnections: false, each request (a preflight included) opens its own connection and closes it when it settles.

• Proxy and DNS. The proxy URL (http://, socks5://, socks5h://) is passed to libcurl as given, and its scheme decides where DNS is resolved: socks5h:// resolves host names at the proxy, socks5:// resolves them locally, on this host. Callers that want no local DNS must pass socks5h://. With no proxy, the connection is direct: libcurl's proxy environment variables (http_proxy, HTTPS_PROXY, ALL_PROXY, NO_PROXY) are ignored, so the caller's option is the only egress policy.

• Finding the library. In order: the libcurlPath option, SEARCHCAST_LIBCURL_PATH, SERPCAST_LIBCURL_PATH (serpcast's name, read for one release), LIBCURL_PATH, then libcurl-impersonate.so (.dylib, .dll) in searchcast's data directory ($XDG_DATA_HOME/searchcast/, default ~/.local/share/searchcast/), where searchcast install-libcurl puts it, then in serpcast's old data directory ($XDG_DATA_HOME/serpcast/, read for one release; see Upgrading from serpcast), then in the platform package @searchcast/libcurl-<platform> npm installed with searchcast (resolved as Node resolves searchcast's own dependencies; skipped when it is not installed or the platform has none). Nothing else is searched, and the library is never downloaded as a side effect: only searchcast install-libcurl, typed by you, downloads it. The pinned release and its checksums are LIBCURL_IMPERSONATE (libcurl-impersonate 2.1.1). The library is loaded once per process, so its path is process-global: a second instance asking for a different path fails with an impersonation error.

• Strict mode (default on): the first request (or transport.check(), which makes no network call) verifies the loaded library exports curl_easy_impersonate and accepts chrome146; otherwise it fails with an impersonation error saying how to fix it, before any network call. strict: false lets a plain libcurl send requests (with a non-browser TLS fingerprint).

• Errors. Every failure is a SearchcastError with a kind: network failures are transport, the time limit (timeoutMs, default 15 s) is timeout, and a missing or wrong library is impersonation. Aborting with the signal rejects with the signal's reason. The transport follows no redirects and does not interpret status codes; that is the caller's job.

• Platforms. On Linux (and FreeBSD) the library is loaded with RTLD_DEEPBIND, so it uses its own nghttp2 and the HTTP/2 HEADERS frame carries Chrome's PRIORITY flag (asserted in the tests). macOS and Windows have no RTLD_DEEPBIND: TLS impersonation works there, but HTTP/2 fingerprint parity with Chrome is not claimed (unmeasured). Response bodies are decoded with Node's zlib, including zstd (Node 22.15 or later).

Engine chain (createSearchcast)

The library's main entry. A search tries an ordered list of engines and stops at the first answer (results, or an empty match), so a query costs as few requests as possible: engines gate on request volume per exit IP, and querying every engine per search (SearXNG's fan-out) spends that budget. There is no merging or ranking across engines.

import {createSearchcast, SearchcastError} from 'searchcast';
import {loadRecipeFile} from '@searchcast/recipe/node';

const searchcast = createSearchcast({proxy: 'socks5h://127.0.0.1:9050'});
const engines = [loadRecipeFile('./first.json'), loadRecipeFile('./second.json')];
try {
	const {results, engine, failures} = await searchcast.search('some query', {engines, maxResults: 10});
	// results: [{title, url, snippet?, ...}], engine: the name of the recipe that answered,
	// failures: [{engine, error}] for each engine tried before it
} catch (error) {
	if (error instanceof SearchcastError && error.kind === 'exhausted') console.log(error.failures);
	else throw error;
}
await searchcast.clearSessions(); // or clearSessions('first'), by engine name
await searchcast.close(); // closes engine connections, stops a library-mode browser if one was started

createSearchcast(options) takes the transport options (libcurlPath, proxy, strict, timeoutMs, caPath, maxBodyBytes, reuseConnections, idlePollMs, maxRequestBodyBytes, preflightCache, maxPreflightAgeS; checked even when a transport is injected, though they then do not apply) plus the options below, checked the same way (a RangeError at creation):

| option | default | meaning | | --------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------- | | store | in memory | The state store for sessions and cooldowns (below). | | cooldownMs | 5 minutes | How long an engine that answered blocked is skipped (DEFAULT_COOLDOWN_MS). 0: no cooldown. | | sessionIdleMs | 10 minutes | An engine's session is dropped after this long without a search using it (DEFAULT_SESSION_IDLE_MS). | | now | Date.now | The clock for cooldowns and sessions (and the default store). | | transport | created from the transport options | A transport to use instead (tests, or sharing one between instances). | | searchcast | none | How library-mode browser engines start the browser runner. | | decoyGuard | none | The engines (by name) whose answers are checked for decoys, e.g. ['engine-a'], on top of those whose recipe declares decoyProne: true. Or {include?: string[], exclude?: string[]}: include is the array form, exclude names engines never checked, even when their recipe declares decoyProne (exclude wins). | | decoyRule | {top: 5, maxRelevant: 1, prefix: 5} (DEFAULT_DECOY_RULE) | The decoy rule's thresholds, each a positive whole number; set any of them. The defaults are the measured values: changing them is at your own risk. | | keepSessions | true | Keep each HTTP engine's transport session (its open connections) in memory between searches (see Sessions, below). false: every search opens new connections and closes them after; cookies and state still go through the store. | | maxRedirects | 20 (Chrome's limit) | How many redirects a declarative recipe follows; one more is a transport error. 0 follows none. |

search(query, {engines, maxResults?, signal?}): an engine is a declarative recipe, a code recipe or a browser engine, identified by its name. maxResults cuts the answer (each recipe's limit still applies). How each outcome is handled:

• Results or an empty match: returned as {results, engine, failures}. Later engines are not called.
• blocked, recipe, timeout, transport: recorded in failures and the next engine is tried. blocked also starts the engine's cooldown: until it ends, the engine is skipped and listed in failures with a blocked error saying it is cooling down (so an all-skipped chain still says why).
• decoy (only for a guarded engine: named in decoyGuard, or its recipe declares decoyProne: true): its answer was a page unrelated to the query. Recorded in failures and the next engine is tried, with no cooldown (see Decoy guard).
• Every engine failed: the search throws a SearchcastError of kind exhausted, whose failures lists every {engine, error} in order. It never answers [] for that. An empty engines list is exhausted too.
• impersonation is not an engine failure: it means every HTTP engine would search with the wrong fingerprint, so the search is aborted at once with that error (no later engine is tried, including a browser engine). Check error.kind === 'impersonation'.
• Aborting signal rejects with the signal's reason; it is not an engine failure.

Decoy guard

Some engines, one above all, answer some queries with a well-formed page of results unrelated to the query: a decoy (dictionary entries for "why" in answer to a question about Git, a gaming guide for a Debian kernel query). Nothing in the page says so, so without a guard it is returned as the answer. Measured in September 2026, through searchcast and two plain Node clients side by side, the most decoy-prone engine decoyed about half of realistic queries (natural-language questions, 4 to 6 term technical queries) from a residential IP and most of them over Tor. It decoyed the SAME queries for every client, and over Tor a given query decoyed on every repeat; an immediate retry rescued none of 16 decoys. A decoy is a property of the engine, the query and the moment, not of the client, so no transport fixes it.

decoyGuard: ['engine-a'] checks the answers of the engines it names (any kind: declarative, code or browser; off by default). An engine whose recipe declares decoyProne: true is checked without being named: the recipe's author knows whether the site serves decoys, and engine names are chosen by the caller, so naming them is fragile. A declarative recipe declares it in its JSON, a code recipe on its default export, and a library-mode browser engine inherits it from its recipe (an endpoint-mode browser engine has no recipe object here, so name it in decoyGuard). An engine is guarded when either holds, unless it is named in decoyGuard: {exclude: [...]}, which switches the guard off for it (exclude wins over both; passing the engine as {...recipe, decoyProne: false} also works). A decoy answer becomes a decoy failure and the chain moves on to the next engine. It starts no cooldown: decoys are per query, and cooling the engine would drop its good answers to other queries. The rule, isDecoy(query, results), is exported so code recipes and callers can apply it themselves:

• The query's terms are its distinct words of 3 or more characters, minus a fixed list of function words and generic modifiers (why, does, best, using, ...).
• A result is relevant when its title, snippet and URL together carry at least two distinct terms (or the only one), a term matching any word with the same first 5 characters ("replication" matches "replicate").
• A page is a decoy when at most one of its top 5 results is relevant. A query with fewer than 2 terms, or a page with fewer than 3 results, is never judged. The whole answer is judged, before the maxResults cut.

The rule is a port of a guard measured on a live deployment: it flagged every decoy seen and none of about 55 genuine pages. Those measured values (the top 5 results, at most 1 relevant, 5-character prefixes) are the defaults of decoyRule, and isDecoy(query, results, rule?) takes the same override. Other values were not measured: change them at your own risk. The stopword list, the 3-character term minimum and the never-judged floor (fewer than 2 terms or 3 results) are not options.

State store

Cooldowns and sessions live in a state store the caller injects, because where state lives and how it is partitioned (per identity, per process, on disk or not) is a privacy decision (ADR 0002). The default, createMemoryStore(), keeps them in a Map in this instance and writes nothing to disk; searchcast ships no file store. A store is a small async key/value interface with per-key expiry, holding plain JSON:

import type {JsonValue, StateStore} from 'searchcast';

interface StateStore {
	get(key: string): Promise<JsonValue | undefined>; // undefined when absent or expired
	set(key: string, value: JsonValue, options?: {ttlMs?: number}): Promise<void>;
	delete(key: string): Promise<void>;
}

searchcast namespaces its keys per engine name: engine/<name>/session (the engine's cookies and last use) and engine/<name>/cooldown (when it ends), with <name> URL-encoded, plus serpcast/sessions, the list of engines with a session (so clearSessions() finds them in any store). Every value carries the time searchcast relies on and is checked with searchcast's clock, and every set passes a ttlMs so the store can drop it; a store that expires late is still correct. Give each identity its own store (or key prefix) to keep their sessions apart.

Sessions. Each engine gets a transport session whose cookies (and, for a code recipe, its ctx.session state) are loaded from the store before the engine runs and saved after, whatever the outcome (a challenge page may set the cookie that lets the next attempt through). A session unused for sessionIdleMs is dropped, and clearSessions(engine?) drops one engine's session or all of them. Two concurrent searches on the same engine each save their own cookies; the last save wins. The instance also keeps each engine's transport session between searches, so the next search reuses its open connections (never another engine's); they are closed when the engine's session is dropped (idle for sessionIdleMs, clearSessions(), or close()), or when the store holds other cookies for it (another instance saved them). An engine's connections never keep the process alive. With keepSessions: false the instance keeps none: each search makes a new transport session from the store's cookies and closes it when the engine is done.

Declarative recipes over HTTP

runDeclarativeRecipe(recipe, query, {session, signal?, maxRedirects?}) runs one declarative recipe (the same JSON file the browser runner runs in a real browser) over the transport, with the browser runner's semantics except that no script from the page runs:

import {createTransport, runDeclarativeRecipe} from 'searchcast';
import {loadRecipeFile} from '@searchcast/recipe/node';

const transport = createTransport({proxy: 'socks5h://127.0.0.1:9050'});
const {recipe, results} = await runDeclarativeRecipe(
	loadRecipeFile('./web.json'),
	'some query',
	{session: transport.session()},
);
// results: [{title, url, snippet?, ...extra fields}]

It requests navigate.url ({query} replaced by the URL-encoded query) as a typed-URL document navigation, follows redirects itself (at most maxRedirects, default 20, each hop through the session so cookies apply), then decides on the final response, in this order:

1. HTTP 202, 403 or 429, a blockedUrl pattern matching the final URL, or a blocked selector matching the page: a blocked error.
2. Any other non-2xx status: 404 and 410 are recipe errors (the URL template is wrong), everything else is a transport error. The status is in the message.
3. The ready selector matches: the results. Each results.item is read with its fields (visible text by default, href/src resolved to absolute URLs against <base href> or the final URL, like the DOM properties), items missing title or url are skipped, and the list is cut to limit (default 10). If no item has both, it is a recipe error.
4. The empty selector matches: []. This is the only way to get an empty list.
5. Nothing matched: a recipe error (the page does not match the recipe).

A recipe may declare "decoyProne": true when its site sometimes answers with results unrelated to the query; the engine chain then checks its answers (the runner itself ignores the field).

ready is checked before empty, as the browser runner does. Each result is {title, url, snippet?} with every other field passed through as a string; snippet is the first present of the content, snippet and description fields. A recipe that needs a browser (form) is rejected with a recipe error, before any request, telling you to run it through a browser engine. The whole call, redirects included, is bounded by the recipe's timeoutMs (default 15 s, then a timeout error), and aborting signal rejects with its reason.

One deliberate difference from the browser runner. The browser runner keeps polling a live page until its deadline, so a page on which nothing matches, or on which ready matches but no item has both a title and a url, ends in a timeout there. searchcast answers recipe at once in both cases, because a static HTML response will not change.

Other differences come from having no browser: no JavaScript runs, so a site that renders its results with script needs a code recipe or a browser engine; visible text approximates innerText without layout (text in script, style, template, noscript and hidden elements is dropped, block elements separate words, but CSS that hides an element is not seen); and the page is decoded with the content-type charset (UTF-8 by default), not a <meta charset>.

HTML is parsed with htmlparser2 and queried with css-select (cheerio's core, without cheerio): about 2.4 MB installed for the whole dependency closure, about 320 KB of it JavaScript, and it supports the selectors recipes use (combinators, attribute operators, :not, :is, :has). An invalid selector is a recipe error.

Code recipes

A code recipe is a JS module for a site that needs challenge handling or a non-HTML API. A code recipe is code with full Node access: loading one runs it, and nothing stops it from importing fs or opening its own sockets. searchcast hands it only the context below, but which modules to load is your trust decision (ADR 0002). Only load recipes you trust. Recipes live anywhere on disk (private ones never need to be in this repo) and are loaded only from the path you give; searchcast never looks for them.

// ./my-api.mjs: an example for a keyless JSON API. The endpoint is a placeholder:
// point it at an API whose terms allow automated access.
export default {
	name: 'my-api',
	timeoutMs: 10_000, // the whole search; optional, default 15 s
	// decoyProne: true, // optional: the chain checks answers for decoys (see Decoy guard)
	async search(query, ctx) {
		const data = await ctx.http.json(`https://api.example.com/search?q=${encodeURIComponent(query)}`, {kind: 'document'});
		if (data.captcha) ctx.blocked('the API asks for a captcha');
		if (!Array.isArray(data.items)) ctx.recipeError('no "items" in the response');
		return data.items.slice(0, ctx.maxResults).map((item) => ({title: item.title, url: item.link, snippet: item.summary}));
	},
};

import {createSearchcast, loadCodeRecipe} from 'searchcast';
import {loadRecipeFile} from '@searchcast/recipe/node';

const searchcast = createSearchcast({proxy: 'socks5h://127.0.0.1:9050'});
const myApi = await loadCodeRecipe('./my-api.mjs'); // imports (runs) the module
const {results} = await searchcast.search('some query', {engines: [myApi, loadRecipeFile('./fallback.json')]});

loadCodeRecipe(path) imports the ESM module at path (relative to the working directory) and returns its default export, which must be {name, search(query, ctx), timeoutMs?, decoyProne?} (decoyProne a boolean, see Decoy guard); a module that cannot be imported or has another shape is a recipe error. A code recipe goes in the engine chain like a declarative one (same failures, cooldowns and sessions), and runCodeRecipe(recipe, query, {session, state?, signal?, maxResults?}) runs one outside a chain (session needs request, and documentCookies for ctx.cookies). search returns (or resolves to) the results; ctx is:

| member | what it is | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | http.get(url, {kind, referer?, fetchSite?, headers?, timeoutMs?}) | A GET through the engine's transport session: the caller's proxy, the pinned Chrome fingerprint and the header table of kind (document, same-origin-navigation, fetch, script; all but document need a referer; for fetch and script, sec-fetch-site is derived from url and referer unless fetchSite overrides it, see Transport), with the engine's cookies sent and kept. Resolves to the raw response (status, headers, body, text()), whatever its status; redirects are not followed. A fetch may carry headers, see Headers a code recipe adds. | | http.text(url, options) | The body as text. Statuses map as for declarative recipes: 202, 403, 429 are blocked; 404, 410 are recipe; any other non-2xx (3xx included) is transport. | | http.json(url, options) | The body parsed as JSON, statuses as for text. A body that is not JSON (often a challenge page) is a recipe error. | | http.post(url, {kind: 'fetch', referer, body?, contentType?, fetchSite?, headers?, timeoutMs?}) | A page's fetch() POST through the engine's transport session (see POST under Transport): body a string or a Uint8Array (at most 1 MiB), contentType defaulting as fetch() does, headers as below; the CORS preflight Chrome would send first is sent too. kind must be fetch. Resolves to the raw response, whatever its status. | | http.postJson(url, value, {kind: 'fetch', referer, contentType?, fetchSite?, headers?, timeoutMs?}) | POSTs JSON.stringify(value) as application/json (unless contentType says otherwise) and parses the answer as json does, with the same status mapping. A value JSON cannot represent is a recipe error. | | cookies.set(url, cookie), cookies.get(url), cookies.delete(url, name) | The engine's transport-session cookies as a page's script sees them (document.cookie), for a cookie a site sets from JavaScript (a challenge token) rather than with Set-Cookie. set applies cookie (name=value; Path=/; Secure; Max-Age=60; Domain=...) as document.cookie = cookie would on the page at url: the Set-Cookie rules of the store (host-only unless a valid Domain is given, Path defaulting to the URL's directory, Secure only from https, __Secure-/__Host- prefixes), HttpOnly ignored, and never replacing an HttpOnly cookie. It returns false when the cookie is rejected (a browser drops it silently), else true. The cookie is then sent, where Chrome puts it, by every http request it matches, and kept and dropped with the session like any cookie. get(url) is what document.cookie reads there (name=value; ... of the non-HttpOnly cookies sent to url, '' when none); delete(url, name) removes the cookies named name that get(url) shows. Names and values are kept byte for byte (# included). A url that is not http(s) is a recipe error. | | session.get(key), session.set(key, value), session.delete(key) | The engine's own JSON state (a token, a challenge answer), kept in the state store with its cookies: saved after every run whatever the outcome, dropped after sessionIdleMs unused or by clearSessions(). Values are copied; a value that is not plain JSON is a recipe error. | | signal | Aborts on the caller's abort or the recipe's timeoutMs. Every http request already carries it. | | maxResults | The caller's maxResults, when given (the answer is cut to it anyway). | | blocked(message) | Throws a blocked error: the engine's cooldown starts. | | recipeError(message) | Throws a recipe error (the site no longer fits the recipe). |

The results are validated: an array of {title, url, snippet?, ...} with a non-empty title and url and every field a string (fields set to undefined are dropped). Anything else is a recipe error, and so is any throw that is not a SearchcastError (with the original as cause). [] is a valid answer: the module says the site has no results, and the chain stops there, so throw recipeError when the response is not one you understand. The whole search is bounded by timeoutMs (a timeout error); aborting the caller's signal rejects with its reason.

Headers a code recipe adds

A page's script can add its own headers to a fetch() (fetch(url, {headers: {'API-Key': key}})), so a code recipe can too, and no more than that: {kind: 'fetch', referer, headers} on http.get, text, json, post and postJson (and on the transport's session.request). The header table stays Chrome's; declarative recipes have no headers (they describe pages, not scripts). The rules, as measured with Chromium in work/notes/findings/fetch-author-headers.md:

• Allowed: only the name sets whose place in Chrome's header order was measured: {api-key} and {authorization}, one header per request (AUTHOR_HEADER_SETS). Chrome orders a request's author headers by a hash over the whole set of names (with user-agent and the client hints), so each other set, two headers together included, would need its own measurement; they are refused with a recipe error that names the supported sets. Names are case-insensitive (API-Key is sent as api-key), and two spellings of one name are combined as a, b, as Chrome does.
• Placement: right after sec-ch-ua-platform, before user-agent, lowercased, for GET and POST, same-origin, same-site and cross-site alike.
• Preflight: a fetch with an author header to another origin (same-site included) is preflighted, GET as well as POST, as Chrome does: the OPTIONS asks with access-control-request-method (GET or POST) and access-control-request-headers, the non-safelisted header names lowercased, sorted and comma-joined without spaces (api-key,content-type for a JSON POST, api-key alone for a form POST). If the answer's access-control-allow-headers does not list every one of them (compared case-insensitively; * does not count, the request being credentialed), the request is not sent and it is a recipe error naming the missing header. An allowed preflight is remembered per page origin and URL with the header names its answer allowed: a later request whose names are all covered is not preflighted again while it lasts; another one is, and its answer replaces the entry. A same-origin request is never preflighted.
• Refused (a recipe error naming the header, before anything is sent): headers on a document, same-origin-navigation or script request; the Fetch standard's forbidden request headers (cookie, referer, origin, host, content-length, accept-encoding, connection, access-control-request-*, x-http-method-override and the rest), every sec-* and proxy-* name; the names the header table sends itself (user-agent, accept, accept-language, priority, upgrade-insecure-requests; content-type is a POST's contentType); a name that is not an HTTP token; and a value that is not a non-empty printable ASCII string after trimming leading and trailing whitespace as fetch() does (so no CR or LF inside a value).

searchcast ships no code recipe for a real site as part of the package: write your own, for engines whose terms allow automated access. The repo has two examples to start from. To install a set of recipes someone publishes, see Installing recipes.

Example: Marginalia Search

examples/recipes/marginalia.mjs is a code recipe for Marginalia Search, an independent web search engine that offers an API meant for programs. It is an example, not an engine bundled with searchcast: it lives in the repo, outside the published packages, so get it from the repo (a clone, or the file itself).

The terms, from Marginalia's API page (read 2026-09-29 and 2026-09-30): the key public is for experimentation and its rate limit is shared by everyone who uses it (the API answers HTTP 503 when it is hit); for regular use, ask Marginalia for a free personal (non-commercial) key, or buy one on that page. Results are provided under CC-BY-NC-SA 4.0, so what you do with them has to respect that licence.

import {createSearchcast, loadCodeRecipe} from 'searchcast';

const searchcast = createSearchcast();
// Reads MARGINALIA_API_KEY at each search: with it, the current API (key in a header);
// without it, the old API with the shared `public` key.
const marginalia = await loadCodeRecipe('./examples/recipes/marginalia.mjs');
const {results} = await searchcast.search('linear b', {engines: [marginalia], maxResults: 10});

Or copy the file next to your private recipes and load it from there. When MARGINALIA_API_KEY is set, it calls the current API (https://api2.marginalia-search.com/search?query=<query>&count=<n>) with the key in an API-Key header, sent as a page's script would send it (a fetch with an author header, from a page on the API's own origin, so no CORS preflight is needed). Without a key, it calls the URL-keyed API with the shared public key (https://api.marginalia.nu/public/search/<query>?count=<n>), which Marginalia lists as deprecated but working as long as the project does. It maps each result's title, url and description (as snippet), sends maxResults as count (clamped to 1..100), treats HTTP 503 and 429 (the rate limit) as blocked so the engine cools down, and a response without a results array as a recipe error. A personal key travels only in the header, so it never appears in the URLs that error messages name.

Example: Mwmbl

examples/recipes/mwmbl.mjs is a code recipe for Mwmbl, a non-profit, open-source (AGPL-3.0) independent search engine whose search API is public and needs no key. It is an example, not an engine bundled with searchcast: like the Marginalia example, it lives in the repo, outside the published packages.

The limits and terms (read 2026-09-30): without a key, the API allows 1,000 requests a month at 1 request per second, counted per IP address. That quota is shared by everyone who reaches the API from the same address, so behind a Tor exit or a VPN it may already be spent by others. Over the quota the API answers HTTP 429. An optional personal key from Mwmbl raises the quota; set it in MWMBL_API_KEY. Mwmbl's terms forbid scraping and excessive automated queries, and its data is under CC BY-NC-SA 4.0, so what you do with the results has to respect that licence. The recipe does not pace its requests: keep your own search rate within those limits.

import {createSearchcast, loadCodeRecipe} from 'searchcast';

const searchcast = createSearchcast();
// Reads MWMBL_API_KEY at each search; without it, uses the anonymous tier.
const mwmbl = await loadCodeRecipe('./examples/recipes/mwmbl.mjs');
const {results} = await searchcast.search('linear b', {engines: [mwmbl], maxResults: 10});

It calls https://api.mwmbl.org/api/v2/search/?q=<query>, adding &api_key=<key> only when MWMBL_API_KEY is set (Mwmbl takes the key as a query parameter, not a header). It maps each result's title, url and content (as snippet), cuts the list to maxResults (the API takes no count), treats HTTP 429 (the quota) and 503 as blocked so the engine cools down, and a response without a results array as a recipe error. The key is part of the URL, so it appears in the URL that error messages name.

Browser engines

A browser engine runs a recipe in a real browser through the browser runner, @searchcast/browser, so it has a real browser's fingerprint and runs the page's JavaScript. It is the heaviest engine, so it usually goes last in the chain, after the HTTP engines: it answers when they are blocked. It joins the chain like any engine (same failures and cooldowns); it has no searchcast session, since the browser keeps its own cookies in its profile. There are two modes.

Library mode: searchcast runs the browser runner in-process. @searchcast/browser (formerly published as searchcast 0.1.x) is an optional peer dependency, imported only when a library-mode engine first runs (nothing else in searchcast imports it), so users of HTTP engines never install it or Chromium. Without it, the engine fails with a transport error saying npm install @searchcast/browser.

import {createSearchcast} from 'searchcast';
import {loadRecipeFile} from '@searchcast/recipe/node';

const searchcast = createSearchcast({
	proxy: 'socks5h://127.0.0.1:9050', // the browser uses it too
	searchcast: {xvfb: '/usr/bin/Xvfb'}, // or {headless: true}
});
const web = loadRecipeFile('./web.json');
const {results, engine} = await searchcast.search('some query', {
	engines: [web, {name: 'web-browser', searchcast: {recipe: web}}],
});
await searchcast.close(); // stops the browser

The browser is started lazily, on the first search that reaches a library-mode engine, and one browser serves every library-mode engine of the instance; close() stops it. The searchcast option (names follow the searchcast serve flags):

| option | default | meaning | | ------------ | --------------------------------------------------------- | -------------------------------------------------------------------------------- | | chrome | the browser runner's search ($SEARCHCAST_CHROME, then PATH) | The Chromium or Chrome executable. | | xvfb | none | Run the browser headful on a private Xvfb display started from this executable. | | headless | false | Run headless (easier for sites to tell apart from a person). | | profile | a temporary directory | The browser profile directory (cookies and history accumulate there). | | concurrency| the browser runner's (2 tabs) | Maximum simultaneous tabs. | | chromeArgs | none | Extra Chromium arguments. |

• Proxy. searchcast's proxy is the browser's proxy. Chromium does not accept socks5h:// and resolves host names at any SOCKS5 proxy, so socks5h:// is passed as socks5://, which keeps DNS at the proxy (checked with Chromium 151: the host name reaches the proxy, and socks5h:// is not accepted at all). Note that a plain socks5:// also resolves at the proxy in the browser, unlike the HTTP transport, where it resolves locally. With no proxy, the browser connects directly.
• Profile. The browser runner needs a profile directory. Without profile, searchcast creates a private temporary one (searchcast-profile-* in the OS temp directory, mode 0700) when the browser first starts, and deletes it on close() and, synchronously, when the process exits. This is the one disk write searchcast makes itself (ADR 0002). A process killed by a signal it does not handle runs no exit handler, so call close() on shutdown (for example on SIGTERM).
• The search is bounded by the recipe's own timeoutMs inside the browser runner; aborting signal rejects at once with its reason (the tab finishes in the background).

Endpoint mode: searchcast calls a running searchcast serve over HTTP or its Unix socket (GET /search?recipe=<recipe>&q=<query>):

const engines = [
	web,
	{name: 'web-browser', searchcast: {endpoint: '/run/searchcast/searchcast.sock', recipe: 'web'}},
	// or {endpoint: 'http://127.0.0.1:8931'}
];

endpoint is an http:// (or https://) URL or an absolute Unix socket path; recipe is the recipe's name on that server (default: the engine's name); timeoutMs bounds the whole request (default 30 s, twice a recipe's default, to leave room for a cold browser start); maxBodyBytes is the largest answer accepted (default 16 MiB; larger is a transport error). Both are per engine, and a value that is not a positive whole number makes the engine fail with a recipe error (a misconfigured engine). In endpoint mode searchcast cannot control the browser's egress: the searchcast serve service uses its own --proxy, and searchcast's proxy does not apply to it (nor to the request to the endpoint, which goes straight to it). The caller must configure the service's egress to match.

Errors. searchcast serve answers both blocked and recipe with HTTP 502, so searchcast maps the answer's error field, not the status (in library mode, the thrown error's code):

| browser runner error | error kind | | ---------------------------------------------------------- | ------------- | | blocked, recipe, timeout | the same | | input, unknown-recipe (the engine is misconfigured) | recipe | | browser, internal, method, not-found, anything else, a body that is not JSON | transport |

A well-formed answer's results are normalized as for declarative recipes (snippet from content, snippet or description; other string fields pass through); an answer without a results array, or with a result lacking title or url, is a transport error. An endpoint that cannot be reached is transport, one that does not answer in time is timeout.

Installing libcurl-impersonate

searchcast needs the libcurl-impersonate shared library and never fetches it on its own (ADR 0002). On a supported platform, installing searchcast is enough:

The platform package (default). npm install searchcast (or pnpm, yarn) also installs @searchcast/libcurl-<platform> for your platform (Linux x64 and arm64 with glibc, macOS x64 and arm64, Windows x64): an optional dependency that holds only the pinned library, libcurl-impersonate 2.1.1, and its license notices, with no install script. searchcast finds it with no setting, after every other source (see Finding the library under Transport), and strict mode checks it at load like any other. Each searchcast release depends on the exact platform package version it was released with. The packages are built only in this repo's release workflow, from the archives pinned in searchcast's source (LIBCURL_IMPERSONATE), each verified against its sha256 before it is unpacked, and published with npm provenance; nothing is downloaded when you install or run searchcast. Run searchcast doctor to check: its from: line names the package and its version.

Where there is no platform package (another platform, musl, FreeBSD) or npm skipped it (--omit=optional, --no-optional, an install copied from another platform), two other ways:

searchcast install-libcurl downloads the pinned release, libcurl-impersonate 2.1.1 (lexiforest/curl-impersonate), for this platform, verifies its sha256 against the checksum pinned in searchcast's source (LIBCURL_IMPERSONATE), and installs the library as libcurl-impersonate.so (.dylib, .dll) in the data directory, where searchcast finds it with no further configuration:

searchcast install-libcurl [--proxy socks5h://127.0.0.1:9050] [--force]

• Pinned platforms: Linux x64 and arm64 (glibc), macOS x64 and arm64, Windows x64, the same as the platform packages. Anywhere else (musl, FreeBSD), use your own build (below).
• It prints what it downloads from, the verified checksum and where it put the library on stderr, and the installed path alone on stdout.
• --proxy sends the download through your egress (http://, socks5://, socks5h://, as for the transport; socks5h:// resolves host names at the proxy). Without it the connection is direct; proxy environment variables are ignored.
• A checksum mismatch, a failed download or an archive without the library aborts with exit 1 and installs nothing (the data directory is not even created). The library is written to a temporary file and renamed into place, so an interrupted install never leaves a partial library.
• If a library is already installed: an identical one is left alone (exit 0), a different one is kept and the command fails, unless --force replaces it.

Your own build (Nix, a distro package, the one your SearXNG uses): point searchcast at it with SEARCHCAST_LIBCURL_PATH=/path/to/libcurl-impersonate.so (or --libcurl, or the libcurlPath option); nothing is downloaded. For example with Nix (nixpkgs' curl-impersonate is 2.1.1 at the time of writing): SEARCHCAST_LIBCURL_PATH=$(nix build --no-link --print-out-paths nixpkgs#curl-impersonate.out)/lib/libcurl-impersonate.so. It must be libcurl-impersonate 2.1.1 or later (it has to accept the chrome146 target); strict mode refuses anything else.

searchcast doctor reports which library searchcast would load, which setting named it, its version, and whether impersonation is active (exit 0) or not, and why (exit 1). When the library comes from SERPCAST_LIBCURL_PATH or serpcast's old data directory, it says so and names the new one to use; when anything (the library, a recipe set) is read from the old data directory, it lists it on an old data dir: line and prints the exact command that moves it on a move with: line. It moves nothing itself, and a fallback does not make the report unhealthy. It makes no network request unless you add --remote, which requests the fingerprint echo service https://tls.browserleaks.com/json once through the transport (and --proxy, if given) and prints the JA3, JA3N, JA4 and HTTP/2 (Akamai) values it saw. --remote is skipped when impersonation is not active.

$ searchcast doctor --remote
library:       /home/me/.local/share/searchcast/libcurl-impersonate.so
from:          the data directory (searchcast install-libcurl)
version:       libcurl/8.21.0-IMPERSONATE BoringSSL zlib/1.3.1 brotli/1.2.0 zstd/1.5.7 ...
pinned:        libcurl-impersonate 2.1.1
impersonation: active (chrome146)
echo:          https://tls.browserleaks.com/json
ja3:           3b488a06a6c1b27f8195b594c8f1e98a
ja3n:          8e19337e7524d2573be54efb2b0784c9
ja4:           t13d1516h2_8daaf6152771_d8a2da3f94cd
http2:         1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p
http2 hash:    52d84b11737d980aef856699f885ca86

Raw JA3 changes on every connection by design (Chrome permutes its TLS extensions); JA3N, JA4 and the HTTP/2 string are the stable values to compare.

Installing recipes (searchcast install-recipes)

A recipe repository can publish its recipes as one release archive, and searchcast install-recipes installs the whole set in one explicit command. Like install-libcurl, it runs only when you type it (ADR 0002): searchcast never looks for recipes, updates them or loads them on its own.

searchcast install-recipes <url|path> --sha256 <hex> [--name <set>] [--dir <path>] [--proxy <url>] [--force]
searchcast install-recipes ipfs://<cid>[/<path>] [--sha256 <hex>] [--ipfs-gateway <url>]... [--name <set>] [--dir <path>] [--proxy <url>] [--force]
searchcast recipes list [--dir <path>]

--sha256 is required, for a URL and a local file alike (an ipfs:// source is pinned by its CID instead, see Installing from IPFS). Code recipes are code with full Node access (see Code recipes), so the pinned checksum is your trust decision: it says "exactly these bytes, which I have reviewed or whose publisher I trust". Get it from the publisher through a channel you trust, or compute it yourself (sha256sum set.tar.gz) after reviewing the archive. There is no way to install without one