@danypops/web-spider-daemon

Supervised Bun daemon for @danypops/web-spider: an authenticated loopback service for fetch/crawl/search/cache operations. Follows the same architecture as @danypops/papyrus and @danypops/jittor — a single supervised process owns SQLite; every other consumer (Pi extension, CLI) talks to it through an authenticated typed client.

Architecture

web_fetch tool / CLI
      ↓
WebSpiderClient → authenticated loopback daemon (127.0.0.1, bearer token)
      ↓
operation registry (service.ts)
      ↓
ports (PageStore, ...) → SQLite adapters (WAL)

The daemon binds 127.0.0.1 on an OS-assigned port — never a fixed or externally reachable port — and writes its handle only after a successful bind. Every HTTP request (including /health) requires Authorization: Bearer <token>.

Bun-independent consumers can share the authenticated discovery implementation through precompiled Facade exports; neither subpath imports SQLite, Playwright, or daemon process composition:

import { connectWebSpiderClient } from "@danypops/web-spider-daemon/client";
import { resolveWebSpiderPaths } from "@danypops/web-spider-daemon/state";

Both exports are tested under native Node ESM and Jiti with tryNative enabled and disabled.

Storage and service

$XDG_DATA_HOME/web-spider/web-spider.db      # SQLite WAL, daemon-owned
$XDG_STATE_HOME/web-spider/auth-token        # 0600, 64 hex chars
$XDG_RUNTIME_DIR/web-spider/daemon.json      # 0600, { host, port, pid }
$XDG_CONFIG_HOME/systemd/user/armada-web-spider.service # Armada-owned native descriptor

Install

bun install
bun src/cli.ts service install   # registers with Armada, reconciles, and starts the native service
bun src/cli.ts service status
bun src/cli.ts service restart
bun src/cli.ts service stop

Armada owns armada-web-spider.service. On systemd it runs bun <cli.ts> serve with restart-on-failure, NoNewPrivileges=true, PrivateTmp=true, and network-online ordering. These are required portable runtime capabilities: Armada fails explicitly rather than silently dropping them on an unsupported native manager.

Run without installing a service (foreground, for development):

bun src/cli.ts serve

Upgrade

Reinstalling the service (service install) updates Armada desired state against the currently installed bun/cli.ts paths, reconciles the native descriptor, and restarts the daemon; the SQLite database, auth token, local provider keys, and cached pages are untouched by an upgrade. A newer package version that bumps the internal SQLite schema version applies its migration automatically on the next serve startup — schema migrations here are forward-only and versioned via PRAGMA user_version, matching Papyrus/Jittor.

Uninstall

bun src/cli.ts service stop
armada remove web-spider --json

Removing the Armada registration does not delete the SQLite database, auth token, or daemon handle files ($XDG_DATA_HOME/web-spider, $XDG_STATE_HOME/web-spider, $XDG_RUNTIME_DIR/web-spider) — delete those directories directly if a full reset is wanted.

Operations

The current operation registry (see src/service.ts):

| Operation | Description | |---|---| | cache.list | Paginated listing of cached pages (bounded: limit ≤ 100), filterable by grep (substring), domain (exact), tag (auto-extracted), category (curated, see below), and fetchedAfter/fetchedBefore/publishedAfter/publishedBefore time ranges; sortable by fetchedAt/publishedAt/url/domain | | cache.search | BM25F search across cached pages (full chunk text, not a truncated snippet) | | search | Live web search via Brave/Tavily/Exa/Serper/SerpApi/You.com, provider fallback chain, numResults/timeRange/topic/searchEngine/siteFilter/wantFullContent -- the latter two are declarative ("restrict to this domain" / "give me full page content"), routed to whichever configured provider can satisfy them rather than naming one | | search.usage | Per-call usage/cost data each engine itself reported (credits for Tavily, costUsd for Exa, rateLimitHeaders for Brave when present) -- append-only, bounded to the most recent 10,000 rows, filterable by engine. Never a running account balance: no provider's search API exposes one, only what one call cost. | | fetch | Single-page fetch — markdown/lean/links/highlights/tree/source formats, rootSelector/excludeSelectors/tokenBudget, bounded PDF pdfPageStart/pdfPageEnd with an automatic OCR fallback for empty/garbled pages (pdf.ocrPages/pdf.qualityScore), enhanced (Playwright). Robots-blocked pages return { blocked: true, reason: "robots.txt" } instead of throwing. | | quotes | Standalone resource-finder: given a query and an explicit urls list (typically a prior search call's results, ≤ 50), fetches each one and returns ranked, verbatim BM25F quotes per url as "resource cards" ({ url, title?, author?, publishedAt?, quotes } or { url, error } for a url that failed to fetch) -- never an LLM-digested answer. maxQuotesPerUrl (default 3, ≤ 20) bounds each url's own share so one page can't starve the others; maxQuotesTotal (default 15, ≤ 100) bounds the combined count. Each quote carries a citationUrl -- a standards-based URL Text Fragment deep link a real browser navigates to and highlights, distinct from the internal chunkId. Independent of crawl's frontier/depth machinery -- it never discovers new URLs, only extracts from the ones it's given. | | crawl | Depth-bounded BFS crawl — depth (≤ 5), maxPages (≤ 200), sameDomain, discoverOnly, crawlUrls (≤ 50), maxTotalChars, deadlineMs (≤ 300000), same formats as fetch plus a crawl summary. Best-first orders content-likely pages, classifies each page's pageType/contentOk, and reports why the crawl stopped via nextAction. Bounds are enforced server-side regardless of what a caller requests. | | papyrus.ingest | Explicit opt-in: turns already-cached pages (kind: "pages", by URL) or a caller-supplied search-result set (kind: "search") into Papyrus doc artifacts (subtype: "web" / "web-search-result"), optionally linked to an existing artifact via relatesTo. Bounded to 20 items per call. Ingested Docs are immutable service output — never updated in place; re-ingesting the same URL creates a new Doc. Reaches Papyrus only through its own authenticated client, never its SQLite file directly. | | session.create | Launches a named, isolated Playwright browser process with one explicit context shared by that session's tabs — tmux-style session semantics. Named sessions never share a browser/context, while tabs in one session share cookies, cache, and origin storage. Bounded to 5 concurrent sessions; rejects past the ceiling rather than queuing. Defaults to Playwright's bundled Chromium in headless mode. headed: true opens a visible window for human CAPTCHA/login/consent takeover while keeping the same persistent session for later agent automation; forceChromeChannel: true uses the full installed Chrome. | | session.list | Lists active sessions with their snapshotVersion and activity timestamps. | | session.close | Finalizes a named session context-before-browser and returns a structured per-stage report (ok/error/timeout). Success is destination-idempotent; a failure retains the runtime so the same close can be retried instead of losing cleanup ownership. Each stage is bounded to 10 seconds by default. | | session.act | Dispatches one navigate/click/eval/screenshot action against a session's persistent page. Requires the caller's snapshotVersion to match the session's current one — fails closed (HTTP 409) if the page has navigated or changed since the caller last observed it, rather than silently acting on stale state. Committed top-level browser navigation advances the active page's snapshotVersion, including human link/form navigation, reload, and history changes; DOM-only and subframe changes do not. Every call — successful, rejected, or failed — is recorded in an append-only, content-free audit journal (SQLite session_audit_log, bounded to the most recent 10,000 rows): action, outcome, and a redacted target (a sanitized URL for navigate, the selector for click, and a fixed "<script>"/"<screenshot>" placeholder for eval/screenshot — script source and image bytes are never written to the journal, only returned to the caller). | | category.assign | Curated, agent/user-assignable relevance category for a cached page (distinct from domain/tags — a judgment about what a page is for, not mechanical metadata). Creates the category on first use; assigning twice is a no-op. A page can belong to any number of categories at once. | | category.remove | Removes a category from a page. Idempotent — no error if already absent. | | category.rename | Renames a category everywhere it's used in one step (categories have a real id, not free text per page). Renaming into an already-existing name merges the two rather than erroring. | | category.list | Lists every known category with its page count. |

Provider API keys (BRAVE_SEARCH_API_KEY, TAVILY_API_KEY, EXA_API_KEY, SERPER_API_KEY, SERPAPI_API_KEY, YOU_API_KEY) can be read from the daemon's environment for foreground/development launches, but service credentials should use the local key store or Enigma. They are never passed through an operation input or projected into Armada. Every configured keyed provider is round-robined as an equal-tier peer (spreading query volume so no single provider's quota gets hammered first), each with its own cooldown after a rate-limit-shaped failure. With no key, or after configured providers return no results, search uses Firecrawl's official bounded keyless endpoint; its per-IP rate/credit limits remain actionable errors, and an empty/blocked keyless attempt never masks an earlier keyed-provider failure. Throttling (500ms per-domain minimum) and robots.txt checking use daemon-process-wide singletons, replacing the pi-extension's previous per-session instances.

service install deliberately ignores provider API keys and ENIGMA_CLIENT_TOKEN: secrets never enter Armada manifests or generated native descriptors. Configure provider keys with web-spider search-key set <engine>, or use Enigma's shared token file and the persistent opt-in below.

Optional: a local, per-engine key file instead of an env var

web-spider search-key set <engine> stores an API key in its own small file under this daemon's state directory (search-keys/<engine>.json, 0600, plaintext), independent of the process environment entirely:

web-spider search-key set tavily       # hidden prompt, or WEB_SPIDER_SEARCH_KEY_VALUE=... for scripts
web-spider search-key list             # engine names only, never the keys themselves
web-spider search-key remove tavily

This exists because a systemd --user service's env is not actually scoped to what it needs — it inherits the whole desktop session's environment, secrets included. A locally stored key sidesteps that: it lives in a file only this daemon's own state directory holds, and it overrides a same-named env var rather than being overridden by it. Takes effect on the daemon's next restart (resolveSearchEnv() resolves once at startup, not per search call).

Optional: BYOK key stacking (several keys per provider)

search-key set replaces the whole stored list with exactly one key. search-key add stacks an additional key alongside whatever is already stored for that engine instead:

web-spider search-key set tavily     # first key
web-spider search-key add tavily     # a second key, stacked (not a replacement)
web-spider search-key list           # still just engine names, never key values or counts
web-spider search-key test tavily    # live-tests every stored key, reporting valid/rate-limited/invalid by position

With more than one key stored for a provider, that provider's engine is wrapped in a RotatingKeySearchEngine (@danypops/web-spider): a 429 rotates to the next key for the same provider within the same call (60s cooldown for the rate-limited key before it's tried again); a 401/403 marks that key invalid for 300s. Falling back to a different provider only happens once every key for this one is exhausted -- rotation state is in-memory only and resets on the daemon's next restart, same as the env/Enigma tiers' own lifecycle. search-key test makes one real, minimal call per stored key through the daemon (network egress is daemon-owned) and never returns or logs a raw key -- only its position and classified status.

Optional: credentials via Enigma, instead of a static key per provider

Enigma involvement is opt-in. Run web-spider enigma enable to persist the non-secret choice in $XDG_STATE_HOME/web-spider/enigma.json; web-spider enigma disable and web-spider enigma status manage it. Without the opt-in, the daemon never probes for Enigma merely because one is reachable. The legacy WEB_SPIDER_USE_ENIGMA flag remains available for foreground launches and explicit overrides.

With the opt-in enabled, the daemon asks Enigma at startup which provider backends it's registered for (enigma client add) and fills in each one's declared env var from the vault, ahead of whatever the daemon's own environment already has. Nothing is hardcoded — Enigma is the source of truth for both which backends this daemon has and which env var each one maps to (set once, at enigma login apikey --env-var ... time).

Registering the client itself is Enigma's own administrative step (enigma client add web-spider --backends ..., printing a token once) and may need to run under Enigma's own service account rather than yours, depending on how Enigma is deployed — see Enigma's own docs for that part. Once Enigma's shared token file is available to the daemon's account, enable the integration and restart:

web-spider enigma enable
web-spider service restart

enigma login apikey --name Brave --env-var BRAVE_SEARCH_API_KEY
enigma client add web-spider --backends brave,tavily,exa
# -> stores the client registration in Enigma; the daemon uses Enigma's shared token file

Without ENIGMA_CLIENT_TOKEN, Enigma's shared admin-token file is deliberately unreadable outside its own service account — the daemon falls straight through to its own environment's static keys, unchanged from before Enigma existed.

The full ladder

Each engine resolves its key in this order, strongest wins:

1. Enigma, if the persistent opt-in is enabled (or WEB_SPIDER_USE_ENIGMA=1) and a credential is registered
2. This daemon's own local key file (search-key set)
3. The raw process environment (BRAVE_SEARCH_API_KEY, TAVILY_API_KEY, ...)

Each rung is independently optional; using only the env var works identically to before either of the other two existed.

tree.query/tree.path as standalone operations (today folded into fetch(format: "tree")), robots.status, throttle.status, searchEnrich composition, and papyrus.ingest land in follow-up work.

CLI

Every registered operation has a CLI route using the authenticated client only — the CLI never opens SQLite directly. Human-readable output by default; --json prints the exact operation result for scripting.

web-spider fetch <url> [--format markdown|lean|links|highlights|tree|source] [--depth N] [--max-pages N]
                        [--no-same-domain] [--root-selector CSS] [--exclude-selectors CSS,CSS]
                        [--token-budget N] [--pdf-page-start N] [--pdf-page-end N] [--enhanced]
                        [--timeout-ms N] [--query TEXT] [--path DOTPATH] [--top-n N]
                        [--discover-only] [--crawl-urls URL,URL,...] [--max-total-chars N] [--deadline-ms N] [--json]
web-spider search <query> [--num-results N] [--time-range day|week|month|year] [--topic news|general]
                        [--engine brave|tavily|exa|serper|serpapi|you] [--site-filter DOMAIN] [--json]
web-spider usage [--engine NAME] [--limit N] [--json]
web-spider cache list [--grep TEXT] [--domain TEXT] [--tag TEXT] [--fetched-after MS] [--fetched-before MS]
                        [--published-after ISO] [--published-before ISO]
                        [--sort-by fetchedAt|publishedAt|url|domain] [--sort-order asc|desc] [--offset N] [--limit N] [--json]
web-spider cache search <query> [--limit N] [--json]
web-spider category assign <url> <category> [--json]
web-spider category remove <url> <category> [--json]
web-spider category rename <category> <newName> [--json]
web-spider category list [--json]
web-spider papyrus ingest <url...> [--relates-to ARTIFACT_ID] [--json]
web-spider session create <name> [--headed] [--force-chrome-channel] [--json]
web-spider session list [--json]
web-spider session close <name> [--json]
web-spider session act <name> --action navigate --snapshot-version N --url URL [--timeout-ms N] [--json]
web-spider session act <name> --action click --snapshot-version N --selector CSS [--timeout-ms N] [--json]
web-spider session act <name> --action eval --snapshot-version N [--script-file PATH] [--json]
web-spider session act <name> --action screenshot --snapshot-version N [--json]
web-spider daemon diagnose [--history-limit N] [--json]

session act --action eval never accepts the script as a plain flag value (shell history and ps would leak it) — it reads the script from --script-file PATH, or from stdin if that's omitted:

echo "document.title" | web-spider session act agent1 --action eval --snapshot-version 0
web-spider session act agent1 --action eval --snapshot-version 0 --script-file ./check.js

Every session act call needs the session's current snapshotVersion (from session create's response or the previous act call's response) — an out-of-date version is rejected rather than silently acting against a page that may have already navigated elsewhere.

For a human-in-the-loop CAPTCHA/login handoff, create the session with --headed --force-chrome-channel, navigate to the blocked page, let the user complete the step in the visible Chrome window, then continue session act calls against the same session. If the challenge was first discovered in an already-headless session, close and recreate that session headed; Playwright cannot reveal an existing headless process in place.

papyrus ingest requires each URL to already be cached (web-spider fetch <url> first) and requires a running, authenticated Papyrus daemon — it fails closed with Papyrus's own actionable "daemon is not running" message when Papyrus isn't installed or started. It is never automatic: nothing is pushed to Papyrus except in direct response to this explicit call.

fetch and crawl share one command: --depth N (N > 0) or --crawl-urls URL,URL,... (a selective, no-re-discovery crawl of exactly those URLs, even without --depth) routes to the crawl operation, matching the web_fetch tool's own single-entry-point shape. Bounds (depth ≤ 5, maxPages ≤ 200, crawlUrls ≤ 50 entries, deadlineMs ≤ 300000ms, etc.) are enforced by the daemon regardless of what the CLI requests.

daemon diagnose reports this daemon's own current instance identity (instanceId/pid/startedAt/provenance) plus its recent restart history (started/already_running/stopped/crashed events, each carrying an instance id, pid, timestamp, and shutdown reason where applicable) -- bounded to the 50 most recent events, surviving a restart (backed by @danypops/vehicle-server's shared daemon-lifecycle log, a persistent file under this daemon's own state directory). Use it to tell whether the daemon is flapping without reading its state files directly: --history-limit N bounds how much history comes back, defaulting to everything retained.

UI-audit toolkit (internal library, not yet a daemon operation)

src/layout-check.ts measures real rendered geometry (getBoundingClientRect() + getComputedStyle() padding, via a session's page) for a set of CSS selectors and asserts a given layout property is consistent across all of them within a pixel tolerance — reporting the actual disagreeing values, not just pass/fail. Built to catch exactly the kind of bug that motivated this whole toolkit: agent-deck's message bubbles and tool-call card silently drifting to different padding.

src/contrast-check.ts measures each selector's real rendered foreground color and its effective background — resolved by walking up the ancestor chain and alpha-compositing every layer, so a background: transparent element correctly inherits whatever's actually painted behind it — and computes the WCAG 2.1 contrast ratio. Thresholds match doc design-tokens-red-hat-informed-not-red-hat-branded-rm7c: 4.5:1 for text under 18pt/24px, 3:1 for large text (18pt+, or 14pt+ bold) and informative icons/graphics. Catches the bug that motivated the whole toolkit: near-invisible dark-red text on a near-black background in agent-deck's Observability tab.

Neither is yet exposed as a session.*-style operation/CLI command — that lands in the follow-up task that wires both checkers into an actual regression gate run against agent-deck.

Health and readiness

Both endpoints require the bearer token:

curl -H "Authorization: Bearer $(cat $XDG_STATE_HOME/web-spider/auth-token)" \
  http://127.0.0.1:$(cat $XDG_RUNTIME_DIR/web-spider/daemon.json | jq .port)/health

• GET /health — { ok: true, version, schema: { current, required } }
• GET /ready — { ready: true } once the HTTP server has bound; used by packed/other tooling to detect the daemon before calling operations
• GET /api/v1/ops — lists registered operation names
• POST /api/v1/ops — { op, input } → { result }, bounded to SERVICE_MAX_BODY_BYTES (1 MiB) per request

Development

bun test           # 37+ tests across state/db/service/adapter/cli/daemon layers
bun x tsc --noEmit