@bpinternal/site-scout

Website URL discovery + LLM page selection — scouts a site for the pages worth reading. Given a website, it finds candidate URLs, scores them into a prioritized tree, then makes one injected-LLM call to pick the pages best suited to seed a support bot's knowledge base.

What it does

Two independent units, composed:

discover(website, opts, deps) -> MasterMap   (prioritized tree)
select(map, opts, deps)       -> Selection   (best ≤limit URLs)

findKnowledgeUrls() runs guard → discover → select for the common case.

• Discovery pulls candidate URLs from 5 sources — llms.txt / llms-full.txt, sitemap.xml (incl. children), robots.txt, the host's discoverUrls crawl, and a site: web search — normalizes/dedupes them, and builds a score tree: one node per path segment, with aggregate scores and leaf counts so selection can reason about cardinality without walking every leaf.
• Selection renders the tree as a compact numbered list and makes one LLM call (extract) that returns the picked page numbers (best-first), which map back to URLs. The LLM pick is the decision — no deterministic fallback list.

The injection contract (SiteScoutDeps)

site-scout does no I/O and never imports a runtime itself. Every consumer provides the I/O + LLM as deps:

type SiteScoutDeps = {
  // discovery I/O
  fetchText(url, baseHost, timeoutMs?): Promise<string | null>
  fetchJson(url, baseHost, timeoutMs?): Promise<T | null>
  discoverUrls(args): Promise<{ urls: string[]; stopReason: string }>
  webSearch(args): Promise<{ results: SearchResult[] }>
  // the one LLM touchpoint (zai-shaped extract)
  extract(input, schema, { instructions }): Promise<z.infer<schema>>
  // optional record/replay cache around the LLM call (defaults to passthrough)
  cache?(kind, keyParts, fn): Promise<T>
}

@bpinternal/zui is a peer dependency — consumers bring their own (the schema passed to extract uses it).

Usage

import { findKnowledgeUrls } from '@bpinternal/site-scout'

const result = await findKnowledgeUrls(
  {
    website: 'https://example.com',
    limit: 50,
    topics: ['pricing', 'docs'],
    prompt: 'Prioritize developer docs, API reference, pricing.',
    context: { company: 'Example', website: 'https://example.com', overview: '…' },
  },
  deps // your SiteScoutDeps implementation
)
// result.urls        — prioritized URLs, best first
// result.discovered  — total distinct URLs found before selection
// result.stopReason  — 'ok' | 'unsupported_site' | 'no_sources' | 'time_limit_reached'

discover() and select() are also exported for callers that want to show the tree, let a user refine, then select. fetchText/fetchJson/hostOf/ siteOrigin (an SSRF-guarded fetcher + URL helpers) are re-exported so a consumer can reuse them when wiring its own deps.

Record / replay fixtures

The tests are offline and deterministic:

• Network — discovery I/O is wrapped by cachedDeps(group, realDeps), which records/replays each call against src/__fixtures__/<group>.jsonl (one file per site). In replay mode a miss throws (a test can't silently hit the network); in production cachedDeps is a passthrough. Mode is env-driven (URL_FIXTURE_RECORD to record; NODE_ENV=test / VITEST / BUN_TEST / URL_FIXTURE_REPLAY to replay).
• LLM — the select call is keyed and replayed from src/__fixtures__/llm-cache.jsonl. The package e2e (index.e2e.test.ts) loads that file and serves the recorded decision, so no live model is called.

Fixtures are recorded via the app harness (viber-compiler-bot), which owns the real runtime/LLM wiring; this package only replays them.