raidr_crawler

Turn a website into hosted MCP servers and agent skills in one command: crawl it for its JavaScript, recover its original sources where source maps allow, read the code for every API endpoint, analyze the evidence, write one MCP manifest and SKILL.md per API host, validate, and publish to raidr_api.

bunx playwright install chromium
echo 'RAIDR_API_KEY=...' > .env    # raidr_api admin key (queue + publishing); Bun loads .env from the working directory
# Every AI step runs through Claude Code: have the `claude` CLI installed and signed in.
bunx @sudobility/raidr_crawler run https://www.example.com
bunx @sudobility/raidr_crawler run --list sites.txt --out ./runs

The crawl queue

The job list lives on raidr_api, so anything can add work and any number of workers can drain it. The crawler always works in two steps:

1. Enqueue: raidr-crawler enqueue <origin…> | --list <file> [--force] [--priority N] [--mode full|api|routes|audit] [--audit] writes jobs. A site crawled before is reported as already-crawled and not queued again unless --force is given; a site already queued is not duplicated (asking for a different mode makes the queued job full). --mode api writes docs, MCP servers and skills only; --mode routes only the site's page routes, which skips all endpoint work and uses far fewer Claude tokens. --headed-chrome crawls in a visible Chrome window instead of headless Chromium, for sites whose bot checks block headless browsers. --audit adds the security audit (see "Security audit") to the crawl; --mode audit runs only the audit, on the worker's saved bundle of the site when it has one, and is queued for a crawled site without --force.
2. Work: raidr-crawler work [--watch] [--max-jobs N] [--worker id] claims one job at a time, runs the whole pipeline for it, and reports the result. It heartbeats every 5 minutes; a job whose worker dies goes back to the queue when its 30-minute lease expires (3 attempts at most). It exits when the queue is empty, or keeps polling with --watch.

run is exactly those two steps. run --no-publish is the exception: a local dry run with no queue and nothing sent, so testing never marks a site crawled.

A finished job records on the server the crawl time (sites.last_crawled_at, the bundle's start time) and the API hosts that got an MCP server and skill (sites.api_hosts; read them with GET /api/v1/sites/:origin/mcps and /skills). rewrite <out-dir> regenerates and republishes every site's manifests and skills from its saved analysis, without crawling, after the writers change.

Published text (tool and manifest descriptions, skills) describes what the API does and never how the tools were made; the structured evidence field in each tool keeps the provenance for review.

Each generated skill gets the user's site token with raidr_cli's raidr token <apiHost> --print, which signs them in in a browser on their own machine and saves the token to ~/.raidr/config.json; copying it from DevTools is kept only as the fallback for a machine with no browser. That command reads the login page, auth style and signed-in-only endpoints from the host's published api.json.

Security audit

Optional, per job (enqueue --audit, or --mode audit for the audit alone; run --no-publish --audit locally). It is passive: it reads only what the crawl recorded and sends nothing to the site.

1. raidr_processor's rules find candidates in the bundle:
   • secrets: credentials in the site's own JavaScript, internal network hosts, public source maps;
   • client code: HTML, eval or redirects fed from the URL or a message, window message handlers without an origin check, credentials in web storage;
   • headers and cookies: missing or weak CSP, HSTS, framing protection, nosniff, version banners, CORS that trusts any or the null origin with credentials, session cookies without HttpOnly/Secure/SameSite, mixed content;
   • API exposure: secret fields in responses, personal data returned without credentials, stack traces and SQL errors.
2. Claude Code reviews them in batches of 40: it drops false positives and sets each kept issue's severity, confidence and text. Rule, category, CWE, OWASP mapping, evidence and fingerprint always come from the rule. Without Claude Code (or when an answer is unusable twice) the rules' own text is kept.
3. The issues go to <out>/<host>/security.json (report.json gets an audit summary), and the worker posts them to raidr_api's POST /crawls with the crawl's time. An audit failure never fails a crawl job; it fails only a --mode audit job.

Secret values are masked before they are written anywhere. Measured on four saved bundles (2026-10-06): the rules took 1–2 s per site and raised 3–12 candidates; Claude Code's review of gemini.com's 6 cost one call and $0.02 and kept 2.

The run pipeline

run does, for each site:

1. Crawl for JavaScript (see "Getting all the JavaScript"), or reuse a bundle with --reuse / --bundle-dir <dir>.
2. Recover sources: the original files the bundle's source maps carry (most sites ship none), read in memory. --no-reconstruct skips it.
3. Read the code for endpoints: fetch, axios, typed clients (openapi-fetch client.GET('/api/x/{id}')), {url, method} request objects, API path literals, API base URLs, GraphQL operations and how the code attaches credentials. Recovered sources are read too.
4. Analyze: observed traffic plus code endpoints, assigned to API hosts (analysis/analysis.json).
5. Write one manifest.json and SKILL.md per API host, and an api.json with the host's endpoint docs (see "API docs"). A rules writer always runs: observed endpoints get verified inputs, code endpoints get their path, query and a free-form body; telemetry, auth handshakes, feature flags and infrastructure hosts are left out; auth comes from traffic, else from the code. Claude Code (claude -p) then decides which tools are worth keeping and names and describes them; it never edits request mappings, and a bad answer falls back to the rules draft. Before that, the code is parsed for each endpoint's call sites and Claude Code writes its request and response schemas (see "API docs"); Claude Code also labels each host. Claude Code is the crawler's only AI: no API key is needed, and without the claude CLI (or with --no-agent) everything comes from code, traffic and rules.
6. Validate and publish to --api-url / RAIDR_API_URL (default https://api.raidr.app) with RAIDR_API_KEY. --no-publish, or no key, validates only.

Output per site, in <out>/<site host>/: bundle/, analysis/, publish/<apiHost>/{manifest.json,SKILL.md,api.json}, security.json (with --audit), and report.json (each step's result and timing). <out>/run-summary.jsonl gets one line per site as it finishes. A failing step is recorded and the rest continue.

| Flag | Meaning | | --- | --- | | --list <file> | one site per line (see "Crawling a list of sites") | | --out <dir> | default ./raidr-runs | | --reuse | reuse <out>/<host>/bundle/*.zip instead of crawling | | --bundle-dir <dir> | reuse the newest raidr-<host>-*.zip from a crawl directory | | --no-reconstruct | skip source-map recovery | | --no-publish | write and validate, send nothing | | --audit | also run the security audit (run --no-publish; for the queue it goes on the job) | | --api-url, --mcp-url | where to publish; MCP base used in the skills (default: the API URL) | | --no-claude | skip Claude Code's manifest review (schemas and labels still use it) | | --no-agent | no AI at all: docs, manifests and labels from code, traffic and rules | | --model <model> | the model Claude Code runs (RAIDR_AGENT_MODEL) | | --agent-max <n> | endpoints per host Claude Code describes up front, most-called first (default 60; endpoints that become tools are always described) | | crawl flags | everything crawl accepts (--max-pages, --headed, --channel chrome, …) |

API docs

Besides the MCP manifest, each API host gets api.json (ApiDoc from raidr_types, schemaVersion: 2), which raidr_app shows as an API playground. It documents every useful endpoint, not just the curated tools, so a host can have docs without an MCP server. Each endpoint has its whole request as one JSON Schema grouped by path, query, headers and body (any depth), and one schema per response status the site's code handles, every property described. Special headers say where their value comes from (a constant, a cookie, storage, an earlier response, or a recipe for computing it).

How: the code is parsed (@babel/parser) to find where each endpoint is called, with the headers, body, status checks, types and helpers around it (src/code/callsites.ts); that evidence, the recorded calls and a traffic-only baseline go to Claude Code (src/pipeline/agent.ts), whose answer is validated and then checked against every recorded call (crossCheck: missing fields added, conflicting types widened). The evidence is saved to analysis/evidence/<apiHost>.json. MCP tools take the same request (src/pipeline/toolSchemas.ts). Each endpoint says whether it needs nothing, a signed-in user's token or an application key; sign-in endpoints are marked role: 'login'.

When the site publishes an API reference (OpenAPI 3 or Swagger 2, JSON or YAML), the crawler finds it: a spec the pages loaded, a spec URL in the code, a docs page (/docs, developers.<site>) whose Swagger UI or Redoc names one, or a well-known path (/openapi.json, /v3/api-docs, …) on the site's API hosts. Endpoints it documents take its names, types, enums, required fields and descriptions, with traffic still deciding what is really sent; on the site's own hosts, documented endpoints the crawl never met are added to api.json (not as MCP tools). Specs found are saved to <site>/docs/specs.json.

Links between endpoints feed the app's flow map. Observed links come from the traffic: a distinctive value (an id, a token) in one response that a later request reuses. Inferred links connect a {x_id} path parameter to the list endpoint for that resource when nothing was observed. Publishing PUTs the doc to /api/v1/apis/:apiHost.

Commands

| Command | What it does | | --- | --- | | run <origin> / run --list <file> | Enqueue, then work: everything below, per site (see above). | | enqueue / work | The two halves of run (see "The crawl queue"). | | rewrite <out-dir> | Regenerate and republish manifests and skills from saved analyses. | | crawl <origin> | Collects the site's JavaScript: loads a few pages (links, router routes, nav clicks), downloads every lazy chunk their code references, and stops after 3 pages with no new JS. Writes raidr-<host>-<date>.zip and reports whether the site exposes its data to the browser (see Rendering). | | crawl --list <file> | The same for every site in a text file, one after another. | | login <origin> | Opens a visible browser for a manual login and saves Playwright storage state for crawl --storage-state. | | analyze <bundle> | Groups XHR/fetch traffic by API host, filters analytics and ads, infers auth evidence, prettifies chunks and indexes call sites into analysis.json. | | publish <dir> | Validates <dir>/<apiHost>/{manifest.json,SKILL.md} and api.json (either set, or both) and PUTs them to raidr_api with RAIDR_API_KEY. --dry-run validates only. | | install / uninstall | Links the bundled raidr-publish skill into ~/.claude/skills, ~/.codex/skills or ~/.agents/skills. |

crawl, analyze and publish are the steps of run, kept for debugging one stage. The raidr-publish skill in skills/publish/SKILL.md now drives run and reviews its output rather than writing manifests by hand.

Crawl flags

| Flag | Default | Meaning | | --- | --- | --- | | --max-pages | 10 | hard cap on pages | | --max-ms | 600000 | hard cap on wall time | | --stall | 3 | consecutive pages with no new first-party JS before stopping | | --no-chunks | | do not download chunks found in code (pages only) | | --max-chunks | 3000 | cap on chunks downloaded per site | | --clicks | 3 | nav clicks per page | | --novelty chunks\|both | chunks | count new endpoint keys as novelty too | | --storage-state <file> | | reuse a saved login | | --login | | run login first, then crawl with that state | | --headed, --channel chrome | | for sites that block headless Chromium | | --no-click | | skip the nav-click pass (use on authenticated crawls where clicks could have side effects) | | --list <file> | | crawl every site in the file instead of one origin (not with --login / --storage-state) |

Getting all the JavaScript without visiting every page

The goal of a crawl is the site's code: source recovery and MCP analysis read the JavaScript. Visiting pages is a slow way to collect it, because a lazy chunk only loads when its route is opened. So after each page the crawler reads every captured script and HTML page for chunk references and downloads them directly, scanning each new chunk in turn until nothing new turns up:

• Vite import("./Page-a1b2.js") and __vite__mapDeps lists;
• the webpack chunk-URL function ("static/chunks/"+…+{id:"hash"}[e]+".js", Next.js, CRA and plain webpack builds);
• Next.js _buildManifest.js and app-router flight data in HTML;
• any other quoted .js path in code or HTML.

Downloads go through the browser context (the site's cookies apply), only to the site's own domain and hosts that already served its scripts, at most --max-chunks per site and three minutes in total. They are recorded as ordinary Script rows, so a bundle looks the same to the next step. runtime/coverage.json has the totals under chunkDiscovery. On real sites this found 3 to 4 times the JavaScript of a 40-page crawl from 5 to 10 pages (suno.com 172 → 611 scripts, tad.ai 228 → 617).

Every wait in a crawl is bounded (page loads, body fetches, probes, source maps), so one unresponsive server cannot stall a --list run.

Crawling a list of sites

# sites.txt: one site per line; blank lines and # comments are ignored
www.ebay.com
https://hn.algolia.com/      # paths are dropped; duplicates are skipped

raidr-crawler crawl --list sites.txt --out ./capture

Every line is checked before the first crawl starts. Sites run one at a time with the same flags, and one failing site does not stop the others. stdout is a JSON array with one summary per site (or { origin, error }), and stderr ends with the sites grouped by rendering verdict. Each summary is also appended to <out>/crawl-summary.jsonl as soon as its site finishes, so an interrupted run keeps its results. The exit code is 1 if any site failed.

Rendering: is there an API to publish?

Many sites (eBay, most shops and news sites) render pages on the server: the browser receives finished HTML, and the real APIs are only called server-side. Crawling them finds pages and scripts but nothing to turn into MCP tools. crawl and analyze classify every bundle:

| Verdict | Meaning | | --- | --- | | client-api | Pages arrive as shells and fetch their data in the browser. Publishable. | | hybrid | Content arrives in the HTML, but many pages also fetch their own data. Only those endpoints are publishable. | | server-rendered | Pages arrive finished. Calls seen in the browser are header widgets, surveys and tracking. Nothing to publish. | | unknown | Blocked (403/429/503), failed or empty pages. Re-crawl, for example with --headed --channel chrome. |

It is judged per page from what crossed the wire: visible text and hydration state in the HTML (__NEXT_DATA__, Nuxt, Marko, …) against JSON data fetched by the page, ignoring analytics, bot-defence and survey hosts and endpoints that fire on every page. The full evidence is rendering in analysis.json (and runtime/coverage.json in the bundle). The raidr-publish skill stops on server-rendered.

Bundle additions

Beyond the standard raidr format, a crawl writes runtime/cookies.json and runtime/storage.json with cookie and web-storage names only, because redaction removes the whole cookie header and the analysis needs to know which cookie a session lives in. runtime/coverage.json holds the crawl report.

Development

bun install
bun run typecheck
bun run test:unit          # pure tests, run in CI
bun run fixtures:build     # builds fixtures/apps/crawl-sample
bun run test:browser       # needs Chromium; crawls the fixture

Related projects

• raidr_processor — bundle format, redaction and analysis this crawler reuses
• raidr_types — McpManifest and the zod schemas publish validates with
• raidr_api — stores manifests, skills and API docs; hosts the MCP endpoint and the API playground proxy
• raidr_cli — reconstructs a web app from the same bundle format