@xemahq/managed-fetch

Polite outbound HTTP client with rate limiting, retries and circuit breaking

Overview

ManagedFetch wraps fetch for ingestion, monitoring and scraping callers: per-host rate limiting, retry with exponential backoff and jitter (honoring Retry-After), a per-host circuit breaker, conditional requests (a 304 is a success, result.notModified), a configurable User-Agent, per-request timeouts and a health probe. It has no runtime dependencies. Every terminal failure throws a typed error that extends ManagedFetchError.

When to use it

• Use it instead of bare fetch for any outbound crawl or polling.
• State (token buckets, breakers) is per instance and in memory; each pod keeps its own, so a shared cross-pod budget must be enforced upstream.

Installation

pnpm add @xemahq/managed-fetch

Usage

import { ManagedFetch } from '@xemahq/managed-fetch';

const client = new ManagedFetch({
  userAgent: 'AcmeCrawler/2.0 (+https://acme.example)',
  rateLimit: { requestsPerSecond: 1, burst: 3 },
  retry: { maxRetries: 4, baseDelayMs: 250, maxDelayMs: 15_000 },
  circuitBreaker: { failureThreshold: 5, cooldownMs: 30_000 },
});

const res = await client.fetch('https://example.com/feed.json', {
  conditional: { etag: previousEtag },
});
if (!res.notModified) {
  const body = await res.json();
}

Configuration

new ManagedFetch(config); every field is optional.

| Option | Meaning | | --- | --- | | userAgent | User-Agent sent unless overridden per request. | | rateLimit | Default token bucket: requestsPerSecond, burst, maxQueueMs?. | | perHostRateLimit | Per-hostname overrides of rateLimit. | | retry | maxRetries, baseDelayMs, maxDelayMs, retryableStatuses?, respectRetryAfter?. | | circuitBreaker | failureThreshold, cooldownMs, halfOpenMaxProbes?. | | timeoutMs | Default per-request timeout. | | fetchImpl | Transport override (defaults to global fetch). | | clock | Override now, sleep, random. | | egress | Optional proxy and browser egress (below). |

Errors: CircuitOpenError, RateLimitError, RequestTimeoutError, RetriesExhaustedError, EgressExhaustedError. Other methods: probe(url), getCircuitState(host), getHealthMatrix().

Optional egress

Without egress, every request goes direct. With it, hosts can be routed through an ordered list of strategies (a proxy pool or a browser service) with failover, exit-IP rotation (maxRotations) and opt-in soft-block detection (detectSoftBlock). Build the resolver with createEgressResolver({ strategies, policy }) using ProxyStrategy and BrowserStrategy. A strategy selected without its transport wired fails fast; it never falls back to an unproxied request.

The undici proxy transport is the @xemahq/managed-fetch/egress-undici subpath: createUndiciDispatcherFactory({ ssrf: 'public-web' }) returns a dispatcherFactory for egress, and its direct agent refuses private, loopback and metadata addresses on every hop.

Guarded public-web fetch

createGuardedFetch (from @xemahq/managed-fetch/egress-undici; needs the optional undici peer) fetches a public-web URL and refuses to reach anything internal. It never reads proxy environment variables and never uses or sets undici's global dispatcher; all configuration is passed as typed options.

import {
  createGuardedFetch,
  GuardedFetchError,
  GuardedFetchFailureReason,
} from '@xemahq/managed-fetch/egress-undici';

const guarded = createGuardedFetch();
try {
  const res = await guarded.fetch('https://example.com/feed.json');
} catch (err) {
  if (err instanceof GuardedFetchError) {
    // err.reason: EGRESS_REFUSED | HOST_UNRESOLVABLE | REDIRECT_REFUSED
  }
}
await guarded.close(); // on shutdown

• Pre-flight assertUrlAllowed: http(s) only, no non-public literal IP, no host with any non-public DNS answer (guarded.assertAllowed(url) runs it alone).
• Connect-time makeSsrfLookup on a private Agent: a host that rebinds to a private, loopback, link-local or metadata address after the pre-flight is refused at connect.
• redirect: 'manual' is forced; any 3xx throws REDIRECT_REFUSED.
• resolveHost and connectLookup options replace name resolution for tests. The transport is not injectable: it is always undici with the guarded Agent.

Peer requirements

undici ^7.29.0, optional: only for the egress-undici subpath.

License

Apache-2.0 © Xema — xema.dev