@leettools/web

Standalone web search, scraping, and crawling utilities. This package depends on @leettools/common and does not know about the LeetTools backend, users, organizations, or knowledge bases.

Search, Scraping, and Crawling

@leettools/web/search provides direct web-search providers and normalized results. @leettools/web/web_scraper provides scraper routing, fallback handling, and per-domain file output. @leettools/web/crawler provides a crawler manager with injectable storage, URL scheduling, crawl actions, links, robots.txt checks, rate limits, retries, and crawl statistics.

The default scraper registry includes Python-compatible scraper names for beautiful_soup, beautiful_soup_simple, web_base_loader, newspaper, crawler4ai, pymupdf, arxiv, and firecrawl. Heavy Python-only extractors are mapped to the closest built-in HTTP/HTML or raw-file scraper unless callers register a more specialized implementation.

import { CrawlerManager, JsonFileCrawlerStore } from '@leettools/web/crawler'
import { WebScraper } from '@leettools/web/web_scraper'

const store = await JsonFileCrawlerStore.open('data/crawler.json')
const scraper = new WebScraper({ outputRoot: 'data/websearch' })
const crawler = new CrawlerManager({ store, scraper })
const summary = await crawler.crawl('https://example.com', { maxUrls: 100 })

Crawler Storage

The crawler keeps durable crawl state behind the CrawlerStore interface. A CrawlerManager coordinates scheduling, robots.txt checks, scraping, retry decisions, and link discovery, but the store owns the shared state. In an app, create one store in the app runtime and pass that same instance to the crawler worker, status UI, ingestion pipeline, and any diagnostics that need to inspect or update crawl state.

import { CrawlerManager, JsonFileCrawlerStore } from '@leettools/web/crawler'
import { WebScraper } from '@leettools/web/web_scraper'

export async function createCrawlerRuntime(dataDir: string) {
  const store = await JsonFileCrawlerStore.open(`${dataDir}/crawler-state.json`)
  const scraper = new WebScraper({ outputRoot: `${dataDir}/websearch` })
  const crawler = new CrawlerManager({
    store,
    scraper,
    rateLimitDelayMs: 3000,
  })

  return { store, scraper, crawler }
}

All components should share that returned store instead of creating private stores. For example, the worker can schedule and crawl URLs while an admin surface reads the same state for progress and failures.

const runtime = await createCrawlerRuntime('data')

await runtime.crawler.addSeedUrl('https://example.com/docs', { priority: 100 })
await runtime.crawler.crawlNext({ maxDepth: 3, waitForReadyMs: 1000 })

const stats = await runtime.store.getStats()
const completed = await runtime.crawler.getCompletedUrls(50)
const failed = await runtime.crawler.getFailedUrls(20)

The store contains five record families:

• sites: one record per domain. Stores robots.txt content, when robots.txt was fetched, the configured user agent, per-domain rate limit delay, and the last crawl time used for rate limiting.
• urls: one record per normalized URL. Stores priority, depth, crawl status, attempt count, last error, last crawl time, saved file path, and content length.
• schedules: due queue entries for URLs. Stores the next scheduled time, priority, retry limit, retry delay, and optional scheduling conditions.
• actions: crawl attempt history. Each crawl attempt creates an action and finishes it as completed, failed, or skipped with result metadata.
• links: discovered source-to-target URL edges, forming the crawl graph.

The URL statuses are pending, scheduled, in_progress, completed, failed, and skipped. Action statuses are pending, in_progress, completed, failed, and skipped.

The normal lifecycle is:

1. addSeedUrl() normalizes the URL, creates or reuses the domain site, creates or reuses the url, and optionally writes a schedule.
2. crawlNext() asks the store for ready schedules, skips terminal URLs, waits for per-domain rate limits, claims the next URL by deleting its schedule, and marks the URL in_progress.
3. A crawl action is written as in_progress; robots.txt is fetched and cached on the site when needed.
4. The configured scraper writes page content to disk. On success, the URL is marked completed with filePath, contentLength, and lastCrawledAt.
5. HTML links are normalized, saved as urls and links, and scheduled until maxDepth is reached.
6. Failures update the URL and action with the error. If attempts remain, the manager writes a retry schedule; otherwise the URL stays failed.
7. Other components read progress through getStats(), listUrls(), listActionsByUrlId(), listReadySchedules(), and link-listing methods.

Store Implementations

InMemoryCrawlerStore keeps all state in process memory. It is useful for tests, one-off runs, and callers that only need crawl state for the lifetime of a single process. It also exposes toSnapshot() so state can be copied into another store or inspected during tests.

JsonFileCrawlerStore persists the same snapshot shape to a JSON file and reloads it on startup. It flushes the whole snapshot after every write, so it is simple and transparent for local apps, demos, and small crawlers that need state to survive restarts.

const store = await JsonFileCrawlerStore.open('data/crawler-state.json')

The JSON store is not a multi-process database. It does not take file locks or perform atomic schedule claiming across separate Node processes. For a production app with multiple workers, high write volume, or shared state across service instances, implement CrawlerStore on the app database and make ready-schedule claiming transactional.

The intended app-level ownership pattern is:

• Create one shared store during app startup.
• Inject it into every crawler-related component.
• Let CrawlerManager mutate crawl state through the store.
• Let other components read from the same store for dashboards, ingestion, retry controls, and diagnostics.
• Replace the store implementation, not the crawler workflow, when moving from local JSON storage to DuckDB/Postgres/another app database.