@walsh/research

Walsh-Research/1.2 spec-compliant fetcher. Reference implementation in TypeScript.

Spec: https://wal.sh/research/bots/compliance-spec
Policy: https://wal.sh/bot/

Install

npm install @walsh/research

Node 18+ required (uses globalThis.fetch, AbortSignal.timeout).

Usage

import { createFetcher } from '@walsh/research';

const { fetch, robots, blocklist } = createFetcher();

const result = await fetch('https://example.com/page');

if (!result.allowed) {
  console.log('denied by', result.deniedBy); // 'blocklist' | 'robots' | 'scope'
} else {
  console.log(result.status, result.body.slice(0, 200));
  console.log('throttle wait:', result.throttleWait, 'ms');
  console.log('retries used:', result.retries);
  console.log('elapsed:', result.elapsed, 'ms');
}

Custom config:

const { fetch } = createFetcher({
  minRequestInterval: 2000,  // 2s between requests per host
  maxRetries: 3,
  backoffBase: 500,
  backoffCap: 30000,
});

Pre-request Gate Pipeline

Every URL passes three gates before an HTTP request is made. The first gate to deny stops the request — no network call is issued for denied URLs.

URL → R3 blocklist → R2 robots.txt → R4 rate limit → fetch (R5 backoff)

R3 blocklist: Checks the canonical operator blocklist at https://wal.sh/.well-known/walsh-research/blocklist.json. Refreshed every 6 hours. Subdomain matching applies: a example.com entry blocks sub.example.com. Last-good list retained on fetch failure.

R2 robots.txt: RFC 9309 parser. Fetched per host, cached 24 hours. Named group (Walsh-Research) overrides wildcard * group (R2a). Longest-match path rule wins (R2b). Crawl-delay honored if present (R2d). Stale cache used up to 2× TTL during revalidation. Fetch failure defaults to allow (RFC 9309 §2.3).

R4 rate limit: Minimum 1 second between requests to the same host. If Crawl-delay is set in robots.txt, the larger of the two values is used.

R5 backoff: Exponential backoff with base 1s, cap 60s, max 5 retries. Retry-After header on 429/503 takes precedence over computed delay.

API

createFetcher(config?)

Returns { fetch, robots, blocklist }.

| Config field | Default | Description | |---|---|---| | userAgent | Mozilla/5.0 (compatible; Walsh-Research/1.2; +https://wal.sh/bot/) | Override UA (breaks R1) | | blocklistUrl | https://wal.sh/.well-known/walsh-research/blocklist.json | Alternate blocklist endpoint | | minRequestInterval | 1000 | Min ms between requests per host | | maxRetries | 5 | Max retries on transient failure | | backoffBase | 1000 | Base backoff in ms | | backoffCap | 60000 | Max backoff in ms |

FetchResult

interface FetchResult {
  url: string;
  status: number;            // 0 if denied before fetch
  headers: Record<string, string>;
  body: string;
  contentType: string;
  allowed: boolean;
  deniedBy?: 'blocklist' | 'robots' | 'scope';
  throttleWait: number;      // ms spent in rate-limit wait
  retries: number;           // retry attempts used
  elapsed: number;           // total wall-clock ms
}

RobotsCache

RFC 9309 robots.txt parser with Walsh-Research-specific extensions.

const robots = new RobotsCache(ttlSeconds?);
const result = await robots.check(url);
// result: { allowed: boolean, crawlDelay?: number, source: string }
robots.clear(); // flush cache

• R2a: Named group (Walsh-Research) overrides * group; wildcard rules reset on match.
• R2b: Longest-match path rule wins.
• R2d: Crawl-delay extracted and returned for the caller to apply.
• R12: Stale-while-revalidate up to 2× TTL. Fetch failure after hard cutoff defaults to allow.

BlocklistCache

const bl = new BlocklistCache(url?, refreshSeconds?);
const blocked = await bl.isBlocked(url);

• R3a: Subdomain matching — example.com entry blocks a.b.example.com.
• R3b: Retains last-good list on fetch failure.
• Fails open (empty list) only when both the fetch fails and the hard cutoff (2× TTL) is exceeded with no prior data.

Compliance Coverage

| Requirement | Description | Status | |---|---|---| | R1 | Sends compliant User-Agent with version and policy URL | ✓ | | R2 | RFC 9309 robots.txt — named group, longest-match, crawl-delay, stale-revalidate | ✓ | | R3 | Operator blocklist — subdomain match, retain-on-failure, 6h refresh | ✓ | | R4 | Per-host rate limit, min 1s, honors Crawl-delay | ✓ | | R5 | Exponential backoff, honors Retry-After, cap 60s, max 5 retries | ✓ | | R6 | Scope enforcement — deniedBy: 'scope' path available | ✓ | | R7 | Schema caching (7d TTL constant defined) | partial | | R8 | Redirect following | ✓ | | R9 | llms.txt convention support | planned | | R10 | Schema TTL (7d) | partial | | R11 | ComplianceReport type exported | partial | | R12 | Stale-while-revalidate, 2× hard cutoff | ✓ |

Dependencies

• [email protected] — bot detection and compliance assessment

License

MIT