@xemahq/managed-fetch

This package belongs to Layer 1 — a framework-agnostic HTTP SDK with zero runtime dependencies. It is the managed outbound-fetch / polite-crawl client that any ingestion, monitor, or scraper calls instead of bare fetch. It is fully generic: it knows nothing about any Xema domain concept.

What it is

ManagedFetch wraps the platform fetch with the machinery a well-behaved outbound caller needs:

• Per-host rate limiting — a reservation token bucket per hostname (requestsPerSecond + burst), with a global default and per-host overrides.
• Retry with backoff — exponential backoff + full jitter on retryable statuses (429/502/503/504) and network/timeout errors, honoring Retry-After (capped), bounded by maxRetries. No infinite loops.
• Per-(host × fetcher) circuit breaker — opens after N consecutive call failures, stays open for a cooldown, then half-open probes before closing. States are the closed enum CircuitState = CLOSED | OPEN | HALF_OPEN. Fails fast when OPEN. Keyed per egress fetcher (see below), so a blocked proxy never opens the circuit for direct. getCircuitState(host) reports the direct fetcher.
• Conditional requests — pass prior { etag, lastModified } and the client sends If-None-Match / If-Modified-Since; a 304 Not Modified is a first-class success (result.notModified === true) for cheap change detection.
• User-Agent policy — a configurable, honest, identifiable UA (client-wide or per-request). No anti-bot evasion beyond setting the header.
• Per-request timeouts — via AbortController, with a sane default.
• Health probe — probe(url) → { healthy, status?, latencyMs }.

Usage

import { ManagedFetch } from '@xemahq/managed-fetch';

const client = new ManagedFetch({
  userAgent: 'AcmeCrawler/2.0 (+https://acme.example)',
  rateLimit: { requestsPerSecond: 1, burst: 3 },
  perHostRateLimit: { 'api.example.com': { requestsPerSecond: 5, burst: 10 } },
  retry: { maxRetries: 4, baseDelayMs: 250, maxDelayMs: 15_000 },
  circuitBreaker: { failureThreshold: 5, cooldownMs: 30_000 },
});

const res = await client.fetch('https://example.com/feed.json', {
  conditional: { etag: previousEtag },
});

if (res.notModified) {
  // nothing changed — reuse the cached copy
} else {
  const body = await res.json();
  saveEtag(res.etag);
}

Terminal conditions throw typed errors — never a silent degraded result: CircuitOpenError, RateLimitError, RequestTimeoutError, RetriesExhaustedError, EgressExhaustedError (all extend ManagedFetchError, each carrying a closed ManagedFetchErrorCode).

Optional egress — proxy pools & browser fetch

Some sources IP-block a datacenter, or hide behind a fingerprint WAF that only a real browser gets past. ManagedFetch can route each request through an ordered list of egress strategies — direct, a datacenter/residential proxy pool, or a browser service — rotating a proxy's exit IP on a block and failing over to the next strategy, tracking health + a circuit per (host × fetcher).

Optional by design. Omit egress entirely and every request goes direct, exactly as before. Add a proxy for one host by registering a strategy and pointing that host's policy at it — every other host stays direct.

Core stays zero-dependency. The proxy transport (undici) and the browser backend are injected — the core never imports either. A proxy or browser strategy selected without its transport wired fails fast; it never silently falls back to an unproxied request. A batteries-included undici transport ships in the optional @xemahq/managed-fetch/egress-undici subpath (install undici, an optional peer) — or supply your own dispatcherFactory.

import {
  ManagedFetch,
  createEgressResolver,
  ProxyStrategy,
  BrowserStrategy,
  EgressStrategyKind,
  type BrowserFetchDelegate,
} from '@xemahq/managed-fetch';
import { createUndiciDispatcherFactory } from '@xemahq/managed-fetch/egress-undici';

const resolver = createEgressResolver({
  strategies: [
    new ProxyStrategy('dc-pool', 'Datacenter', {
      kind: EgressStrategyKind.DATACENTER_PROXY,
      proxyUrl: 'http://user:[email protected]:8000',
      rotateUrl: 'http://dc.proxy.example/rotate', // optional on-demand IP swap
    }),
    new BrowserStrategy('browser', 'Headless browser'),
  ],
  // Only these hosts leave `direct`; everything else stays direct.
  policy: [
    { host: '.gov.pt', strategyIds: ['dc-pool', 'direct'] }, // dot-suffix match
    { host: 'www.base.gov.pt', strategyIds: ['browser', 'dc-pool'] }, // most-specific wins
  ],
});

// The undici transport: ProxyAgent per pool + an SSRF-guarded Agent for `direct`
// (refuses any target resolving to a private/loopback/metadata address, per hop).
const dispatcherFactory = createUndiciDispatcherFactory({ ssrf: 'public-web' });

// Wraps any /read-url-style browser service (e.g. Playwright/Crawl4AI). Returns
// a standard Response; map YOUR service's envelope to a status/body here.
const browserFetch: BrowserFetchDelegate = {
  async fetch(url, init) {
    const r = await fetch('http://browser-svc.internal:8010/read-url', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ url }),
      signal: init.signal,
    });
    const { content } = (await r.json()) as { content: string };
    return new Response(content, { status: r.ok ? 200 : 502 });
  },
};

const client = new ManagedFetch({
  egress: {
    resolver,
    dispatcherFactory,
    browserFetch,
    maxRotations: 5, // exit-IP rotations before failover — budgeted apart from retries
    detectSoftBlock: true, // read a bounded body to catch a WAF that answers 200
  },
});

const res = await client.fetch('https://www.base.gov.pt/tender/123');
console.table(client.getHealthMatrix()); // which egress works for which source
// on shutdown: await dispatcherFactory.close();

Key rules:

• Rotation ≠ retry. A retry re-sends the same request from the same IP; a rotation replaces the exit IP — the only move that beats an IP-scoped block. They have separate budgets (maxRotations), so tuning one never silently changes the other.
• Ordered failover. A candidate is skipped when its (host × fetcher) circuit is OPEN; a hard block/failure fails over to the next strategy in the host's policy; exhausting all candidates throws EgressExhaustedError.
• Soft-block detection is opt-in. With detectSoftBlock, a 200/403 carrying a bot-challenge body (or your classifyBody signal) is classified BLOCKED; off, classification is status-only (free). looksLikeBlocked and classifyOutcome are exported for reuse.
• SSRF-guarded direct egress. The egress-undici factory's direct Agent validates every connection (initial request + each redirect hop) against a private/loopback/link-local/metadata blocklist — a public-web target that resolves or 30x-redirects to an internal address is refused. assertUrlAllowed (pre-flight) and isBlockedAddress are exported for standalone use.

Invariants

• State is per-instance / in-memory. The token bucket and circuit breaker coordinate a single process only. Across horizontally-scaled pods each pod keeps its own state — the effective global rate is pods × requestsPerSecond and each pod trips its own breaker. This is intentional; a shared cross-pod budget must be enforced at a shared upstream, not here.
• Fail-fast, no silent fallbacks. Every terminal failure throws a typed error. A malformed URL, an unset transport, or an exceeded rate cap all fail immediately.
• Deterministic timing. Backoff, rate-limit waits, and circuit timing run off an injectable clock (now / sleep / random), so behavior is reproducible and testable with no real sleeping. Timers are used only for the network request timeout — never to drive control flow between retries.