@jambudipa/spider

A powerful, Effect-based web crawling framework for modern TypeScript applications. Built for type safety, composability, and enterprise-scale crawling operations.

⚠️ Pre-Release API: Spider is currently in pre-release development (v0.x.x). The API may change frequently as we refine the library towards a stable v1.0.0 release. Consider this when using Spider in production environments and expect potential breaking changes in minor version updates.

🏆 Battle-Tested Against Real-World Scenarios

Spider successfully handles ALL 16 https://web-scraping.dev challenge scenarios - the most comprehensive web scraping test suite available:

| ✅ Scenario | Description | Complexity | |-------------|-------------|------------| | Static Paging | Traditional pagination navigation | Basic | | Endless Scroll | Infinite scroll content loading | Dynamic | | Button Loading | Dynamic content via button clicks | Dynamic | | GraphQL Requests | Background API data fetching | Advanced | | Hidden Data | Extracting non-visible content | Intermediate | | Product Markup | Structured data extraction | Intermediate | | Local Storage | Browser storage interaction | Advanced | | Secret API Tokens | Authentication handling | Security | | CSRF Protection | Token-based security bypass | Security | | Cookie Authentication | Session-based access control | Security | | PDF Downloads | Binary file handling | Special | | Cookie Popups | Modal interaction handling | Special | | New Tab Links | Multi-tab navigation | Special | | Block Pages | Anti-bot detection handling | Anti-Block | | Invalid Referer Blocking | Header-based access control | Anti-Block | | Persistent Cookie Blocking | Long-term blocking mechanisms | Anti-Block |

🎯 View Live Test Results | 📊 All Scenario Tests Passing | 🚀 Production Ready

Live Testing: Our CI pipeline runs all 16 web scraping scenarios against real websites daily, ensuring Spider remains robust against changing web technologies.

🔍 Current Status (Updated: May 2026)

• ✅ Core Functionality: All web scraping scenarios working
• ✅ Type Safety: Full TypeScript compilation without errors
• ✅ Build System: Package builds successfully for distribution
• ✅ Test Suite: 523 tests passing (44 test files, one todo test)
• ✅ Code Quality: Strict ESLint and Effect checks pass with zero diagnostics

✨ Key Features

• 🔥 Effect Foundation: Type-safe, functional composition with robust error handling
• ⚡ High Performance: Concurrent crawling with intelligent worker pool management
• 🤖 Robots.txt Compliant: Automatic robots.txt parsing and compliance checking
• 🔄 Resumable Crawls: State persistence and crash recovery capabilities
• 🛡️ Anti-Bot Bypass: Handles complex blocking mechanisms and security measures
• 🌐 Browser Automation: Playwright integration for JavaScript-heavy sites
• 📊 Built-in Monitoring: Comprehensive logging and performance monitoring
• 🎯 TypeScript First: Full type safety with excellent IntelliSense support

🚀 Getting Started

Installation

npm install @jambudipa/spider effect

Your First Crawl

import {
  CrawlResult,
  makeSpiderConfig,
  SpiderConfig,
  SpiderEventSinkNoop,
  SpiderService,
} from '@jambudipa/spider'
import { Effect, Layer, Sink } from 'effect'

const program = Effect.gen(function* () {
  // Create spider instance
  const spider = yield* SpiderService
  
  // Set up result collection
  const collectSink = Sink.forEach<CrawlResult>(result =>
    Effect.sync(() => {
      if (CrawlResult.isOk(result)) {
        console.log(`Found: ${result.pageData.title}`)
      }
    })
  )
  
  // Start crawling
  yield* spider.crawl(['https://example.com'], collectSink)
})

const config = makeSpiderConfig({ maxPages: 10, maxDepth: 2 })

const runnable = program.pipe(
  Effect.provide(
    SpiderService.layer.pipe(
      Layer.provide(
        Layer.mergeAll(
          SpiderConfig.layerWith(config),
          SpiderEventSinkNoop,
        ),
      ),
    ),
  ),
)

Effect.runPromise(runnable)

📚 Documentation

Comprehensive documentation is now available following the Diátaxis framework for better learning and reference:

🎓 New to Spider?

Start with our Tutorial - a hands-on guide that takes you from installation to building advanced scrapers.

📋 Need to solve a specific problem?

Check our How-to Guides for targeted solutions:

• Authentication - Handle logins, sessions, and auth flows
• Data Extraction - Extract structured data from HTML
• Resumable Operations - Build fault-tolerant crawlers

📚 Need technical details?

See our Reference Documentation:

• API Reference - Complete API documentation
• Configuration - All configuration options

🧠 Want to understand the design?

Read our Explanations:

• Architecture - System design and philosophy
• Web Scraping Concepts - Core principles

📖 Browse All Documentation →

🛠️ Quick Configuration

import { makeSpiderConfig } from '@jambudipa/spider'

const config = makeSpiderConfig({
  maxDepth: 3,
  maxPages: 100,
  maxConcurrentWorkers: 5,
  ignoreRobotsTxt: false, // Respect robots.txt
  requestDelayMs: 1000
})

Core Concepts

Spider Configuration

The spider can be configured for different scraping scenarios:

import { makeSpiderConfig } from '@jambudipa/spider';

const config = makeSpiderConfig({
  // Crawl limits
  maxDepth: 5,
  maxPages: 1000,
  ignoreRobotsTxt: false,

  // Rate limiting
  requestDelayMs: 2000,
  maxConcurrentRequests: 3,
  maxRequestsPerSecondPerDomain: 1,

  // Content handling
  followRedirects: true,

  // User agent
  userAgent: 'MyBot/1.0'
});

Middleware System

Add custom processing with middleware:

import { 
  SpiderService, 
  MiddlewareManager,
  LoggingMiddleware,
  RateLimitMiddleware,
  UserAgentMiddleware 
} from '@jambudipa/spider';

const middlewares = new MiddlewareManager()
  .use(new LoggingMiddleware({ level: 'info' }))
  .use(new RateLimitMiddleware({ delay: 1000 }))
  .use(new UserAgentMiddleware({ 
    userAgent: 'MyBot/1.0 (+https://example.com/bot)' 
  }));

// Use with spider configuration
const config = makeSpiderConfig({
  middleware: middlewares
});

Resumable Scraping

Resume interrupted scraping sessions:

import { 
  CrawlResult,
  makeSpiderConfig,
  ResumabilityService,
  SpiderConfig,
  SpiderEventSinkNoop,
  SpiderSchedulerService,
  SpiderService,
  SpiderStateKey,
} from '@jambudipa/spider';
import { Effect, Layer, Sink } from 'effect';

// Enable resumability in config
const config = makeSpiderConfig({ enableResumability: true, maxPages: 50 });

const collectSink = Sink.forEach<CrawlResult>(result =>
  Effect.sync(() => {
    if (CrawlResult.isOk(result)) console.log(`Scraped: ${result.pageData.url}`)
  })
);

// Initial crawl — saves state automatically
const startCrawl = Effect.gen(function* () {
  const spider = yield* SpiderService;
  yield* spider.crawl('https://example.com', collectSink);
}).pipe(
  Effect.provide(
    SpiderService.layer.pipe(
      Layer.provideMerge(
        Layer.mergeAll(
          ResumabilityService.layer,
          SpiderSchedulerService.layer,
          SpiderConfig.layerWith(config),
          SpiderEventSinkNoop,
        ),
      ),
    ),
  ),
);

// Resume a previous session
const resumeCrawl = Effect.gen(function* () {
  const spider = yield* SpiderService;
  const stateKey = new SpiderStateKey({
    id: 'my-crawl-session',
    timestamp: new Date('2024-01-01'),
    name: 'Example Crawl',
  });
  yield* spider.resume(stateKey, collectSink);
}).pipe(
  Effect.provide(
    SpiderService.layer.pipe(
      Layer.provideMerge(
        Layer.mergeAll(
          ResumabilityService.layer,
          SpiderSchedulerService.layer,
          SpiderConfig.layerWith(config),
          SpiderEventSinkNoop,
        ),
      ),
    ),
  ),
);

See src/examples/07-resumability-demo.ts for a complete example with FileStorageBackend.

Link Extraction

Extract and process links from pages:

import { Effect } from 'effect';
import { LinkExtractorService } from '@jambudipa/spider';

const program = Effect.gen(function* () {
  const linkExtractor = yield* LinkExtractorService;
  
  const result = yield* linkExtractor.extractLinks({
    html: '<html>...</html>',
    baseUrl: 'https://example.com',
    filters: {
      allowedDomains: ['example.com', 'sub.example.com'],
      excludePatterns: ['/admin', '/private']
    }
  });
  
  console.log(`Found ${result.links.length} links`);
  return result;
}).pipe(
  Effect.provide(LinkExtractorService.layer)
);

Interaction-Driven Discovery (v0.15+)

A client-rendered application does not put its asset identities in the delivered markup — they are fetched only once a control is driven. A crawler that reads the DOM finds nothing and reports success.

InteractionDiscoveryService is the second extraction source alongside LinkExtractorService: rather than reading what is there, it operates the page and records what results. Give it a Playwright Page and the controls you want driven, and it returns the requests they revealed, each naming the control that revealed it.

import { InteractionDiscoveryService } from '@jambudipa/spider';

const program = Effect.gen(function* () {
  const discovery = yield* InteractionDiscoveryService;

  const result = yield* discovery.sweep(
    page,
    [
      { id: 'tab-docs', label: 'Documents', selector: '#tab-docs' },
      { id: 'gallery-1', label: 'Gallery, first image', selector: '.thumb:first-child' },
    ],
    {
      // Asset-agnostic by design: narrowing to a class of interest is yours.
      include: (r) => r.resourceType === 'xhr' || r.resourceType === 'media',
      settleMs: 1500,
    }
  );

  for (const request of result.requests) {
    // `revealedBy.label` is what turns an opaque GUID into a nameable thing.
    console.log(request.url, request.revealedBy?.label ?? '(in markup)');
  }
}).pipe(Effect.provide(InteractionDiscoveryService.layer));

Three rules it holds to:

• In-place controls only. A control that navigates away loses the page being measured, and the sweep fails with SweepNavigatedAwayError rather than reporting identities that belong to a different document.
• Absent attribution stays absent. A request seen outside any control's window — an <img> or <video src> in the delivered markup — carries no revealedBy, rather than being credited to whichever control ran last. Set baselineMs and navigate with a commit wait state to capture those.
• No vacuous passes. A sweep that reveals nothing is a failure, not an empty success, and the three ways it can be empty are kept apart: NoOracleDeclaredError (nothing was declared), NoControlsDrivenError (nothing could be driven), NothingRevealedError (drove controls, saw nothing).

Robots.txt: availability is not permission (v0.15+)

RobotsService distinguishes what an origin declared from what it failed to tell you:

| /robots.txt response | Meaning | Outcome | |---|---|---| | 2xx | rules published | parsed and obeyed | | 404 / 410 | no rules published | allowed | | 5xx, timeout, connection failure, body > 512 KiB | nothing is known | refused |

Every verdict carries a reason — no-rules-published, allowed-by-rule, disallowed-by-rule, or robots-unavailable — and RobotsBlockedEvent carries it too, so a transport failure is never mistaken for the target disallowing you. Set ignoreRobotsTxt: true for an origin you control that cannot serve /robots.txt reliably.

Rules resolve by longest-match precedence with Allow beating an equal-length Disallow; * and $ wildcards are supported; patterns match path and query. robots.txt is always fetched without credentials, because it is a public declaration and asking as a signed-in user asks a different question.

API Reference

Core Services

• SpiderService: Main spider service for web crawling
• SpiderSchedulerService: Manages crawling queue and prioritisation
• LinkExtractorService: Extracts and filters links from HTML content
• InteractionDiscoveryService: Discovers identities that only exist after an interaction
• ResumabilityService: Handles state persistence and resumption
• ScraperService: Low-level HTTP scraping functionality

Configuration

• SpiderConfig: Main configuration interface
• makeSpiderConfig(): Factory function for creating configurations

Middleware

• MiddlewareManager: Manages middleware chain
• LoggingMiddleware: Logs requests and responses
• RateLimitMiddleware: Implements rate limiting
• UserAgentMiddleware: Sets custom user agents
• StatsMiddleware: Collects scraping statistics

Storage Backends

• FileStorageBackend: File-based state storage
• PostgresStorageBackend: PostgreSQL storage (requires database)
• RedisStorageBackend: Redis storage (requires Redis server)

Logging and Observability

Spider exposes two independent observability surfaces, both overridable by client code.

1. Diagnostic logs (Effect Logger)

All Effect.log* calls inside Spider (logDebug, logInfo, logWarning, logError) flow through the standard Effect Logger system, with structured fields attached via Effect.annotateLogs. Provide a custom Logger.layer:

import { Effect, Layer, Logger, References } from 'effect';

const myLogger = Logger.make(({ logLevel, message, fiber, date }) => {
  const annotations = fiber.getRef(References.CurrentLogAnnotations);
  // Route to pino, datadog, OpenTelemetry, file, etc.
  console.log(JSON.stringify({
    timestamp: date.toISOString(),
    level: logLevel.toUpperCase(),
    fiberId: fiber.id,
    message: Array.isArray(message) ? message.join(' ') : String(message),
    ...annotations,
  }));
});

const loggingLayer = SpiderService.layer.pipe(
  Layer.provide(
    Layer.mergeAll(
      SpiderConfig.layer,
      SpiderEventSinkNoop,
      Logger.layer([myLogger]),
      Layer.succeed(References.MinimumLogLevel, 'Info'),
    ),
  ),
);

program.pipe(Effect.provide(loggingLayer));

2. Domain events (SpiderEventSink)

Typed lifecycle and progress signals — SpiderStart, SpiderComplete, SpiderError, DomainStart, DomainComplete, PageScraped — are emitted to a SpiderEventSink. The default sink (SpiderEventSinkNoop) discards them. Subscribe by providing your own:

import { Effect, Layer } from 'effect';
import { SpiderEventSink } from '@jambudipa/spider';

const AnalyticsSink = Layer.succeed(SpiderEventSink, {
  emit: (event) => Effect.sync(() => analytics.track(event._tag, event)),
});

const analyticsLayer = SpiderService.layer.pipe(
  Layer.provide(Layer.mergeAll(SpiderConfig.layer, AnalyticsSink)),
);

program.pipe(Effect.provide(analyticsLayer));

SpiderEvent is a discriminated union — switch on _tag for exhaustive handling.

See src/examples/10-custom-logging.ts for a complete example.

Configuration Options

Basic

| Option | Type | Default | Description | |--------|------|---------|-------------| | ignoreRobotsTxt | boolean | false | Skip robots.txt checks | | maxDepth | number | — | Maximum BFS depth from start URL | | maxPages | number | — | Hard page cap per domain | | userAgent | string | 'JambudipaSpider/1.0' | Default user agent string | | followRedirects | boolean | true | Follow HTTP redirects | | respectNoFollow | boolean | true | Honour rel="nofollow" | | enableResumability | boolean | false | Enable crawl state persistence | | allowedDomains | string[] | — | Restrict crawling to these domains | | blockedDomains | string[] | — | Never crawl these domains | | allowedProtocols | string[] | ['http:','https:','file:','ftp:'] | Permitted URL schemes | | normalizeUrlsForDeduplication | boolean | true | Normalise URLs before dedup | | customUrlFilters | RegExp[] | — | Patterns to exclude from crawling |

Rate Limiting / Workers

| Option | Type | Default | Description | |--------|------|---------|-------------| | maxConcurrentWorkers | number | 5 | Worker fibers per domain | | concurrency | number \| 'unbounded' \| 'inherit' | 4 | Inter-domain concurrency | | maxConcurrentRequests | number | 10 | Total concurrent requests | | maxRequestsPerSecondPerDomain | number | 2 | Per-domain rate cap | | requestDelayMs | number | 1000 | Base courtesy delay (ms) | | maxRobotsCrawlDelayMs | number | 2000 | Max robots.txt crawl-delay cap (ms) | | resultChannelCapacity | number \| 'unbounded' | 'unbounded' | Buffered CrawlResult slots between workers and the sink. Set a positive integer to backpressure workers when the sink lags and keep heap flat under slow I/O sinks — see docs/how-to/backpressure.md. | | staleWorkerThresholdMs | number | 300_000 | Worker-health staleness threshold (ms). Override for slow adapters. | | staleWorkerCheckIntervalMs | number | 15_000 | How often the monitor scans for stale workers (ms). | | workerHeartbeatMode | 'per-iteration' \| 'per-attempt' | 'per-iteration' | When 'per-attempt', heartbeat refreshes between retry attempts so long retry chains aren't flagged dead. |

URL Filtering

| Option | Type | Default | Description | |--------|------|---------|-------------| | fileExtensionFilters | FileExtensionFilters | all enabled | Toggle filtering by file category | | technicalFilters | TechnicalFilters | all enabled | Toggle scheme/length/malformed checks | | skipFileExtensions | string[] | — | Legacy: explicit extension blocklist (overrides fileExtensionFilters) |

Advanced Config Objects

| Option | Type | Description | |--------|------|-------------| | domainEquivalence | DomainEquivalenceConfig | www. handling, protocol strictness, subdomain matching | | fetchRetry | FetchRetryConfig | Retry policy: maxAttempts, baseBackoffMs, retryOn | | domainRetry | DomainRetryConfig | In-cycle domain-level retry: re-run residual domains after a backoff (v0.14+) | | crossDomainRedirects | CrossDomainRedirectConfig | Follow cross-domain redirects from start URLs | | userAgentStrategy | UserAgentStrategy | static, rotating, or custom user-agent selection | | httpAdapter | HttpAdapter \| HttpAdapterSelector | Pluggable HTTP fetcher; defaults to the built-in undici path (v0.11+) |

Worker health & long fetches (v0.12+)

The spider's worker loop fires a heartbeat per iteration to drive the dead-worker detector. With slow HttpAdapter implementations (TLS-impersonating clients, sidecar APIs) a single task — bounded by fetchRetry.maxAttempts × adapter timeout + backoff — can run for minutes, which exceeded the pre-0.12 60 s threshold and got busy workers flagged dead mid-fetch.

Two changes address this:

1. STALE_WORKER_THRESHOLD_MS default bumped 60 s → 300 s. Matches the worst case of the default fetchRetry policy: maxAttempts (3) × per-attempt timeout (~45 s) + exponential backoff (1 s + 2 s) ≈ 138 s, with ~160 s headroom. Override per-spider via staleWorkerThresholdMs (positive integer, capped at 2_147_483_647 ms).

2. Opt-in workerHeartbeatMode: 'per-attempt' refreshes the heartbeat on each permitted retry through Schedule.tap. Recommended whenever a single attempt can approach staleWorkerThresholdMs / maxAttempts.

Sample config for a very slow adapter with 5 retries (worst case = 5 × 60 s + (1 + 2 + 4 + 8) s = 315 s ≈ 5.25 min, so 600 s gives ~50% headroom):

makeSpiderConfig({
  httpAdapter: gotScrapingAdapter,            // slow per-attempt fetcher (~60 s timeout)
  workerHeartbeatMode: 'per-attempt',          // refresh between retries
  staleWorkerThresholdMs: 600_000,             // 10 min
  fetchRetry: { maxAttempts: 5, baseBackoffMs: 1000, retryOn: ['timeout', 'connection_refused'] },
})

Note: the default undici adapter returns 5xx responses to the spider as successful fetches (no automatic retry on status code); for status-code-driven retries the adapter must classify them as errors. The retry kinds in the snippet above (timeout, connection_refused) are the kinds reliably surfaced by the built-in adapter.

The standalone WorkerHealthMonitor service (@jambudipa.io/WorkerHealthMonitor) shares the new 300 s default but is independent of SpiderConfig. Consumers wanting a different threshold for that service provide a custom layer via the WithThreshold factory:

import { Effect } from 'effect';
import { WorkerHealthMonitor } from '@jambudipa/spider';

// Stricter 60 s detection for a high-throughput producer.
const program = Effect.gen(function* () {
  const monitor = yield* WorkerHealthMonitor;
  // ...
});

Effect.runPromise(
  program.pipe(Effect.provide(WorkerHealthMonitor.WithThreshold(60_000)))
);

Note on log volume. The spider emits a debug-level event: 'worker_heartbeat' log record for every reportWorkerHealth call. With 'per-attempt' mode under heavy concurrency this can add tens to hundreds of records per second. Default-level loggers filter these out; raise your minimumLogLevel to Debug only when actively diagnosing heartbeat behaviour.

In-cycle domain retry (v0.14+)

fetchRetry retries an individual page-fetch within crawlSingle. When the start URL itself exhausts all retries during a transient burst — cold TLS handshakes, DNS pressure, a noisy neighbour spike — the spider fires DomainCompleteEvent { reason: 'all_fetches_failed', pagesAttempted: 1 } and never revisits that domain. Consumers cannot recover within the public API: domain_complete is terminal per cycle.

domainRetry adds an opt-in second loop layer above fetchRetry. When enabled, Spider.crawl() runs the start-URL list as a multi-pass loop: pass 0 is the existing behaviour, residual domains matching the retryOn predicate are collected, the spider sleeps backoffMs, optional passOverrides are applied, and the residual set is re-run. Each pass emits its own DomainCompleteEvent (now carrying an additive cycle field) and a DomainRetryScheduledEvent fires between passes so consumers see the lifecycle.

makeSpiderConfig({
  fetchRetry: {
    maxAttempts: 3,
    baseBackoffMs: 1000,
    retryOn: ['timeout', 'connection_refused', 'http_5xx', 'http_429'],
  },
  domainRetry: {
    enabled: true,
    maxPasses: 2,                  // 1 = pre-feature behaviour; 2 = pass 0 + one retry
    backoffMs: 60_000,             // sleep between passes
    retryOn: {
      reasons: ['all_fetches_failed'],
      maxPagesAttempted: 1,        // only retry domains that died on the start URL
    },
    passOverrides: {               // optional: apply to retry passes only
      fetchRetry: { maxAttempts: 5, baseBackoffMs: 2000, retryOn: ['timeout', 'connection_refused', 'http_5xx', 'http_429'] },
      // concurrency: 'unbounded',
    },
  },
})

Connection-pool and cache continuity. Retry passes share the same undici connection pool (process-global), robots cache, result channel, and event sink because every pass lives inside the same spider.crawl() invocation. Sockets warmed during pass 0 are available to pass 1; there is no fresh spider instance, no fresh DNS lookup, and no fresh robots fetch. The URL deduplicator is per-pass (each pass gets a fresh localDeduplicator) — that is by design, so pass 1 will actually re-attempt the start URL that pass 0 failed.

Predicate match is conjunctive. A domain is retried when event.reason ∈ retryOn.reasons AND event.pagesAttempted ≤ retryOn.maxPagesAttempted. The pagesAttempted bound is the gate that distinguishes a transient start-URL failure (the intended case) from a domain that scraped partially before structural failure (NOT the intended case).

maxPages is per-pass; stopMode is preserved. Each pass is a fresh crawlSingle invocation with its own URL deduplicator, so maxPages is enforced per-pass — a retry pass gets the full budget, not the remainder from pass 0. SpiderCompleteEvent.totalDomains counts unique start URLs (not pass invocations); totalPages sums scraped pages across all passes since each scrape is real work. Fiber interruption during a retry pass unwinds via Effect's standard interruption, with stopMode semantics preserved.

Validation. makeSpiderConfig rejects, at construction: maxPasses < 1, non-integer maxPasses, enabled: true with maxPasses < 2 (silent no-op footgun), negative backoffMs, empty retryOn.reasons, NaN/negative retryOn.maxPagesAttempted (Infinity permitted), and bad passOverrides.fetchRetry (maxAttempts < 1 or negative baseBackoffMs). Default config (enabled: false, maxPasses: 1) preserves pre-feature behaviour.

Observability. Consume both events by switching on event._tag:

const sink = Layer.succeed(SpiderEventSink, {
  emit: (event) =>
    Effect.sync(() => {
      switch (event._tag) {
        case 'DomainComplete':
          if (event.cycle > 0) console.log(`retry succeeded for ${event.domain} (cycle ${event.cycle})`);
          break;
        case 'DomainRetryScheduled':
          console.log(`scheduling retry for ${event.domain} at ${new Date(event.nextPassAt).toISOString()}`);
          break;
      }
    }),
});

Interrupt Mode (v0.10+)

By default, when a stop condition fires (maxPages reached, queue empty) the spider lets every in-flight fetch finish its full retry schedule before exiting. With production configs (maxAttempts: 5, baseBackoffMs: 1000) a stuck URL can tail for several minutes per worker.

stopMode: 'interrupt' changes this: when a stop condition fires, in-flight fetches are cancelled immediately and workers exit within gracePeriodMs (default 5 seconds).

makeSpiderConfig({
  maxPages: 50,
  stopMode: 'interrupt',            // cancel in-flight on stop
  // or tune grace period:
  // stopMode: { kind: 'interrupt', gracePeriodMs: 3000 },
})

External abort handle

To abort a running crawl programmatically, pass a Deferred<void> via crawl() options. Requires stopMode: 'interrupt' in the config.

import { Deferred, Effect, Fiber, Layer } from 'effect';
import { makeSpiderConfig, SpiderConfig, SpiderEventSinkNoop, SpiderService } from '@jambudipa/spider';

const program = Effect.gen(function* () {
  const stopSignal = yield* Deferred.make<void>();
  const spider = yield* SpiderService;

  // Fork so we can resolve the stop signal concurrently
  const crawlFiber = yield* Effect.fork(
    spider.crawl(['https://example.com'], sink, { externalStopSignal: stopSignal })
  );

  // Abort after 30 seconds
  yield* Effect.sleep('30 seconds');
  yield* Deferred.succeed(stopSignal, undefined);
  return yield* Fiber.join(crawlFiber);
}).pipe(
  Effect.provide(
    SpiderService.layer.pipe(
      Layer.provide(
        Layer.mergeAll(
          SpiderConfig.layerWith(makeSpiderConfig({ stopMode: 'interrupt' })),
          SpiderEventSinkNoop,
        ),
      ),
    ),
  ),
);

Interrupt events

Subscribe to new events emitted in interrupt mode via SpiderEventSink:

| Event | When emitted | |-------|-------------| | WorkerInterruptedEvent | Per interrupted worker fiber — fields: workerId, domain, url, reason | | DomainStoppedEvent | Per domain that stopped — fields: domain, reason, gracefulMs, forced | | SpiderStoppedEvent | When external abort fires — fields: reason, totalDomains, totalPages, wallclockMs |

DomainCompleteEvent.reason gains two new values: 'interrupted' (clean exit within grace period) and 'interrupt_grace_exceeded' (grace period expired, domain force-completed).

Pluggable HTTP Adapter (v0.11+)

By default the spider fetches pages through globalThis.fetch (Node's built-in undici). For sites behind anti-bot CDNs that fingerprint the TLS ClientHello (JA3/JA4), undici's deterministic handshake is rejected before the HTTP layer sees a response. The httpAdapter config option lets you swap in a TLS-impersonating fetcher (got-scraping, curl-impersonate sidecar, etc.) without forking the spider.

Provide a single adapter to apply it to every request, or a selector function to dispatch per-request (e.g. route a small set of anti-bot domains to the alternative adapter while the bulk of the crawl stays on undici):

import {
  defaultUndiciAdapter,
  makeSpiderConfig,
  type HttpAdapter,
  type HttpAdapterSelector,
} from '@jambudipa/spider';
import { gotScrapingAdapter } from './got-scraping-adapter.js'; // your impl

const promoted = new Set(['example.com', 'other-cdn.com']);
const httpAdapter: HttpAdapterSelector = (req) =>
  promoted.has(new URL(req.url).hostname.replace(/^www\./, ''))
    ? gotScrapingAdapter
    : defaultUndiciAdapter;

makeSpiderConfig({
  stopMode: { kind: 'interrupt', gracePeriodMs: 5000 },
  httpAdapter,
});

When httpAdapter is undefined, behaviour matches v0.10 exactly (the built-in defaultUndiciAdapter is used).

Adapter contract

An HttpAdapter is an object with a single fetch method returning an Effect:

interface HttpAdapter {
  readonly fetch: (request: HttpAdapterRequest) =>
    Effect.Effect<HttpAdapterResponse, HttpAdapterError>;
}

Key contract points:

• Cancellable. The returned Effect must be cancellable so stopMode: 'interrupt' propagates. Wrap promise-based libraries with Effect.tryPromise so the auto-injected AbortSignal reaches the underlying request.
• Owns its timeout. The adapter is responsible for honouring request.timeoutMs. The spider does not layer an additional Effect.timeout on top.
• Structured error kinds. HttpAdapterError.kind is drawn from the existing PageFetchErrorKind union (timeout | dns | http_4xx | http_429 | http_5xx | connection_refused | other) so the spider's fetchRetry.retryOn keys keep working unchanged.
• All status codes return as success. The built-in defaultUndiciAdapter returns 4xx and 5xx responses as HttpAdapterResponse with the status code intact (matching v0.10 behaviour where these still get parsed into PageData). Custom adapters may opt into failing with kind: 'http_5xx' for retry semantics.

Error Handling

Fetch errors are surfaced as CrawlResultError values inside the sink — they do not fail the Effect channel. Inspect result.error.kind (a PageFetchErrorKind) to branch on the failure type:

import { CrawlResult } from '@jambudipa/spider';
import { Effect, Sink } from 'effect';

const collectSink = Sink.forEach<CrawlResult>(result =>
  Effect.sync(() => {
    if (CrawlResult.isOk(result)) {
      console.log(`OK: ${result.pageData.title}`);
    } else {
      switch (result.error.kind) {
        case 'timeout':
          console.log(`Timed out: ${result.url}`); break;
        case 'http_4xx':
          console.log(`Client error ${result.error.statusCode}: ${result.url}`); break;
        case 'http_5xx':
          console.log(`Server error ${result.error.statusCode}: ${result.url}`); break;
        case 'dns':
          console.log(`DNS failure: ${result.url}`); break;
        case 'http_429':
          console.log(`Rate limited: ${result.url}`); break;
        default:
          console.log(`Error (${result.error.kind}): ${result.error.message}`);
      }
    }
  })
);

const program = Effect.gen(function* () {
  const spider = yield* SpiderService;
  yield* spider.crawl('https://example.com', collectSink);
});

See src/examples/09-error-handling-recovery.ts for a full example.

Advanced Usage

Custom Middleware

Create custom middleware for specific needs:

import { SpiderMiddleware, SpiderRequest, SpiderResponse } from '@jambudipa/spider';
import { Effect } from 'effect';

class CustomAuthMiddleware implements SpiderMiddleware {
  constructor(private apiKey: string) {}
  
  processRequest(request: SpiderRequest): Effect.Effect<SpiderRequest, never> {
    return Effect.succeed({
      ...request,
      headers: {
        ...request.headers,
        'Authorization': `Bearer ${this.apiKey}`
      }
    });
  }
  
  processResponse(response: SpiderResponse): Effect.Effect<SpiderResponse, never> {
    return Effect.succeed(response);
  }
}

// Use in middleware chain
const middlewares = new MiddlewareManager()
  .use(new CustomAuthMiddleware('your-api-key'));

Performance Monitoring

Monitor scraping performance:

import { WorkerHealthMonitor } from '@jambudipa/spider';

const program = Effect.gen(function* () {
  const healthMonitor = yield* WorkerHealthMonitor;
  
  // Start monitoring
  yield* healthMonitor.startMonitoring();
  
  // Your scraping code here...
  
  // Get health metrics
  const metrics = yield* healthMonitor.getMetrics();
  
  console.log('Performance metrics:', {
    requestsPerMinute: metrics.requestsPerMinute,
    averageResponseTime: metrics.averageResponseTime,
    errorRate: metrics.errorRate
  });
});

Development

# Install dependencies
npm install

# Build the package
npm run build

# Run tests (all scenarios)
npm test

# Run tests with coverage
npm run test:coverage

# Type checking (must pass)
npm run typecheck

# Validate CI setup locally
npm run ci:validate

# Code quality
npm run lint        # Fails on every ESLint or Effect diagnostic
npm run format     # Formats code consistently

🛠️ Contributing & Code Quality

Current State: The codebase is fully functional with comprehensive test coverage and clean linting.

• ✅ Functional Changes: All PRs must pass scenario tests
• ✅ Type Safety: TypeScript compilation must succeed
• ✅ Build System: Package must build without errors
• ✅ Code Style: ESLint configured with Effect-idiomatic rules

Code Quality Commands:

# Check for linting issues
npm run lint

# Fix auto-fixable issues
npm run lint:fix

Repository development guidance

Spider is a TypeScript npm package for Effect-based web crawling. It uses npm, Vite, Vitest, ESLint, Prettier, and the TypeScript compiler.

The former root CLAUDE.md referenced .claude/templates/CLAUDE.md. This README now owns the repository guidance, so update this section when that template or the project stack changes.

Use British English spelling in prose. Keep public names clear and tree-shakeable. Keep runtime dependencies small. Define public package entry points in src/index.ts and maintain their TypeScript declarations.

Add unit tests for public APIs. Add integration tests for important workflows. Use descriptive test names in the form should [behaviour] when [condition]. Validate external input, avoid secrets in code or published artifacts, and consider bundle size and memory use in long-running crawls. Maintain at least 80% test coverage when coverage measurement applies.

The root layout uses src/ for package code and tests, docs/ for user documentation, .claude/ for project tooling, and package.json for the npm contract. Build artifacts belong in dist/; do not treat them as source.

Before a release, run the relevant tests, type checks, build, lint, and documentation checks. Use Changesets for version changes and publishing. When the project stack changes, update this section rather than creating a separate root instruction file. Confirm tests, coverage, security review, documentation, TypeScript compilation, and bundle size before publishing.

Keeping the documentation current

This repository documents itself under three rules, and npm run docs:audit proves all three. CI runs the same command, so a change that breaks a rule fails the build.

1. Every declaration carries a JSDoc block — exported, public, private, or internal alike. Say why the thing exists and what a caller must know. Do not restate the signature.
2. Every folder that holds source files carries a README.md describing what the folder is for. One file, not two: every agent and every reader opens the same text.
3. This block stays in the root README.md. The root AGENTS.md and CLAUDE.md point at that file rather than repeating it.

When you change the code:

• Write or update the JSDoc block in the same commit as the declaration it describes. A stale block is worse than no block, because a reader trusts it.
• Update the folder's README.md when you change what the folder is for, add a new entry point, or move code in or out of it.
• Add a README.md when you add a folder.
• Run npm run docs:audit before you commit.

License

MIT License - see LICENSE file for details.

📚 Complete Documentation

All documentation is organized in the /docs directory following the Diátaxis framework:

• 🎓 Tutorial - Learning-oriented lessons for getting started
• 📋 How-to Guides - Problem-solving guides for specific tasks
• 📚 Reference - Technical reference and API documentation
• 🧠 Explanation - Understanding-oriented documentation

📖 Start with the Documentation Index →

Support

• GitHub Issues - Bug reports and feature requests
• Documentation - Comprehensive guides and reference material
• Tutorial - Step-by-step learning guide

Built with ❤️ by JAMBUDIPA