@bonniernews/fetchy

⚠️ Experimental. This package is under active development and not yet battle-tested in production. APIs and behavior may change between releases — pin your version and read the release notes before upgrading.

A small, pluggable fetch lib built on Node's native fetch stack (undici) — with response caching and keepAlive connection agents that are actually used.

It's a drop-in-shaped replacement for exp-fetch (the got-based wrapper): same buildFetch(behavior) factory, same returned methods, same callback/promise styles. The HTTP engine underneath is undici instead of got.

• ES modules, Node 22+ (developed and tested on Node 24 LTS).
• Minimal dependencies: undici, exp-asynccache, lru-cache.
• No global side effects: importing (or using) fetchy never claims undici's process-wide global dispatcher — every request carries its own dispatcher — so built-in fetch() elsewhere in the process keeps running on Node's bundled undici, unaffected by fetchy being present.
• Connection pooling via a configurable undici Agent (works for http and https).

Install

npm install @bonniernews/fetchy

Usage

import buildFetch from "@bonniernews/fetchy";

const { fetch, get, post, del, stats } = buildFetch({ /* behavior */ });

// Promise
const body = await fetch("https://example.com/resource.json");

// Callback
fetch("https://example.com/resource.json", (err, body) => { /* ... */ });

// Verbs
await post("https://example.com/things", { name: "thing" });

The factory returns fetch, get, post, put, patch, head, options, del, and stats(). Each request method accepts either a URL string or an options object { url, headers, timeout }, an optional body, and an optional callback. With no callback a promise is returned.

Connection agents (keepAlive)

Native fetch ignores Node's http.Agent; pooling is done by an undici Dispatcher. This lib builds one keepAlive Agent per factory and uses it on every request, so connections are reused (TLS handshakes included) for both http:// and https://.

import { Agent } from "undici";

// (a) pass tuning options — a default keepAlive Agent is built for you
buildFetch({ agent: { keepAliveTimeout: 10_000, connections: 10 } });

// (b) pass a ready-made undici Dispatcher (used verbatim)
// ⚠️ `rejectUnauthorized: false` disables TLS certificate verification — it
// exposes you to man-in-the-middle attacks. Use it only against a local test
// server, never in production.
buildFetch({ agent: new Agent({ connect: { rejectUnauthorized: false } }) });

// (c) pass nothing — a keepAlive Agent with sane defaults is still created
buildFetch({});

HTTPS/TLS options go under the agent's connect key (e.g. ca, rejectUnauthorized).

For any Agent it builds, the lib creates one shared undici connector so the TLS session cache is shared across connections — TLS sessions resume even on non-pooled (Connection: close) workloads. If you pass a pre-built undici.Agent yourself, build it with a shared buildConnector(...) to get the same behavior.

Behavior options

Same contract as exp-fetch:

freeze (true), deepFreeze (false), clone (true), cache (AsyncCache, 60s; falsy disables), cacheKeyFn, cacheValueFn, maxAgeFn, onNotFound, onError, onSuccess, onRequestInit, requestTimeFn, cacheNotFound (false), logger, getCorrelationId, correlationIdHeader ("correlation-id"), errorOnRemoteError (true), contentType ("json"), agent, followRedirect (true), httpMethod ("GET"), timeout (20000), headers, retry (0), varyHeaders ([]), maxResponseBytes (0), isRedirectAllowed, transport (auto: "undici", or "fetch" when NODE_ENV === "test"), hooks.beforeRequest.

Caching, manual redirect following (up to 10 hops, with per-hop caching and the followRedirect: false { statusCode, headers } return), cache-control max-age parsing, correlation-id / User-Agent / x-exp-fetch-appname headers, and stats() hit ratio all behave as in exp-fetch.

Default headers

The lib may inject these headers when the caller hasn't set them (the exp-fetch set, plus accept-encoding). The list is exported as DEFAULT_HEADERS (also attached to the default export) for tests/proxies that need to strip or assert on lib-added headers:

| header | value | when | |---|---|---| | correlation-id | from getCorrelationId() | when configured; name follows correlationIdHeader | | x-cloud-trace-context | from getTraceContext() | when configured; name follows traceContextHeader | | x-exp-fetch-appname | the app's package.json name | when the app has a name | | user-agent | <app>/<version> | unless caller-set | | accept-encoding | gzip, deflate | unless caller-set |

import buildFetch, { DEFAULT_HEADERS } from "@bonniernews/fetchy";

getTraceContext (new, opt-in) propagates distributed-tracing context the same way getCorrelationId propagates the correlation id: supply a getter (typically AsyncLocalStorage-backed, reading the incoming request's header) and the value rides on every outgoing request — on Cloud Run this is what stitches the request tree together in Cloud Trace. The default header name is Cloud Run's x-cloud-trace-context; override with traceContextHeader (e.g. "traceparent" for W3C trace context):

buildFetch({ getTraceContext: () => requestContext.get("traceContext") });

Security

The cache is a single, process-wide store. Without care, a shared cache leaks data between callers and a redirect can leak credentials, so the defaults here are conservative:

• Cache key varies on credentials (and method). The default cache key folds in the HTTP method (so a del()/post() is never served a cached GET body) and the authorization, cookie, and proxy-authorization headers, so one caller's authenticated response is never served to another caller with different (or no) credentials. Add more headers with varyHeaders: ["accept-language", ...]. A custom cacheKeyFn(url, body, headers, method) replaces this entirely — you own keying then, including credential separation. (Even so, don't put a personalized/private response in a long-lived shared cache unless you've thought about who else holds the key.)
• Credentials are dropped on cross-origin redirects. When a redirect crosses to a different origin, the authorization, cookie, and proxy-authorization headers are stripped before the next hop, so a redirect to another host can't harvest them. Same-origin redirects keep them.
• isRedirectAllowed(toUrl, fromUrl) — return a falsy value to block a hop (rejects with code: "EREDIRECTBLOCKED"). Validating only the initial URL is not enough to stop SSRF: a redirect can point at an internal or cloud-metadata address. Use this to re-check every hop. Tightening further (DNS-pinning, blocking private IP ranges) is the caller's responsibility.

  buildFetch({ isRedirectAllowed: (to) => new URL(to).hostname !== "169.254.169.254" });

• maxResponseBytes (bytes, 0 = unbounded) — cap the response body read into memory so a hostile or compromised upstream can't exhaust it. A Content-Length over the cap is rejected up front; a missing or lying one is caught while streaming (rejects with code: "EBODYTOOLARGE"). The streamed cap counts decompressed bytes, so a compressed bomb can't defeat it. Off by default to preserve drop-in behavior — set it when fetching untrusted hosts.

  buildFetch({ maxResponseBytes: 10 * 1024 * 1024 }); // 10 MiB

Note that URLs are logged (at debug/info) and embedded in error messages, so avoid putting secrets in query strings. Response bodies in error messages are truncated, but are still passed in full to your onError hook.

Resilience (opt-in)

App-layer resilience primitives for protecting backends, modelled on gaps an SRE review found across our sites (no circuit breakers, no outbound concurrency caps, no stale-if-error, no disconnect propagation). All are off by default — when unset, the request path is unchanged. Per-host state lives for the lifetime of one buildFetch() factory (typically one backend).

• staleIfError (seconds, or an AsyncCache): keep the last-good response and serve it when a later request hits a 5xx, 429, timeout, or network error. Other 4xx, 404, and EBODYTOOLARGE are never masked; onError still fires when a stale value masks a failure (so error-rate monitoring keeps seeing the outage); the served value isn't re-cached as fresh. App-layer stale-if-error. A number of seconds keeps an internal in-process store. Pass your own AsyncCache instance instead (the same contract as cache) to own the store: inspect it, purge an entry (del), pre-seed it, share it between factories, or back it with something remote — retention is then governed by the store's own TTL. Entries are keyed by the same cache keys as the main cache (cacheKeyFn), and it must not be the same instance as cache — colliding keys would serve stale values as fresh, so that config is ignored with a warning.

  buildFetch({ staleIfError: 3600 }); // serve last-good for up to 1h on origin failure
  
  // or own the store, e.g. to purge a bad last-good entry on demand:
  import AsyncCache from "exp-asynccache";
  const staleCache = new AsyncCache(buildFetch.initLRUCache({ age: 3600 }));
  // NB: a url-only key drops the default credential-aware keying (for the main
  // cache too) — only do this when requests carry no per-user credentials.
  buildFetch({ staleIfError: staleCache, cacheKeyFn: (url) => url });
  await staleCache.del("https://api.example/thing");

• circuitBreaker (true or { failureThreshold = 5, resetTimeout = 30000, comparator, failureStatusCodes }): per-host breaker. After failureThreshold consecutive 5xx/timeout/network failures it opens and fails fast (Error with code: "ECIRCUITOPEN", or a stale response if staleIfError is set) for resetTimeout ms, then admits one half-open probe. 4xx/404 count as the backend being healthy — except 429, which is neutral: it neither counts as a failure (rate limits are usually per key, not per origin) nor resets the failure counter (a backend flapping 500/429/500 still trips), and a half-open probe answered 429 doesn't close the breaker — it stays half-open, probing one request at a time until a real success closes it. When a backend rate-limits per host, opt into counting throttling as failures with failureStatusCodes: [429]: listed statuses (4xx only — 5xx always counts, 404 keeps its notFound handling) count toward the threshold and route like a 5xx (masked by staleIfError, onError still fires). For services that answer 200 with an error payload, comparator(content, result) lets the resilience layer see through the status code: return truthy for healthy, falsy to count that 2xx as a failure. A flagged 2xx counts toward the breaker, is never cached as fresh (its max-age is ignored), and never becomes the last-good value — and when staleIfError is enabled and a last-good response exists, that is served instead (stale.served), exactly like a 5xx. Without one the flagged body is still delivered, so existing app-level handling of soft errors keeps working. A throwing comparator counts as healthy and rejects that request only.

  buildFetch({ circuitBreaker: { failureThreshold: 5, resetTimeout: 30000 } });
  // backend rate-limits per host: sustained 429s should trip the breaker
  buildFetch({ circuitBreaker: { failureStatusCodes: [ 429 ] } });
  // backend hides errors behind a 200:
  buildFetch({ circuitBreaker: { comparator: (content) => content?.status !== "error" } });

• maxConcurrentPerHost (+ optional maxQueuePerHost): cap in-flight requests per origin to bound the instances × maxSockets connection multiplier. Beyond the cap requests queue; with maxQueuePerHost set, a full queue sheds (Error with code: "ECONCURRENCYLIMIT"). (undici's Agent connections caps connections at the transport level; this adds request-level queueing/shedding.)

  buildFetch({ maxConcurrentPerHost: 50, maxQueuePerHost: 100 });

• Caller signal (per request): pass an AbortSignal on the request options to cancel in-flight work — e.g. wire req.on("close") so a client disconnect releases backend capacity. It's combined with the timeout; a caller abort is not retried and surfaces as the abort, not ESOCKETTIMEDOUT. Concurrent identical requests share one in-flight load: an abort rejects only that caller's promise, and the shared wire request is cancelled when the last interested caller aborts.

  fetcher.get({ url, signal: req.signal });

The default retry calculateDelay is exponential with jitter (capped at 3s) rather than linear, so a fleet retrying a flapping backend doesn't resynchronize into load spikes. A custom calculateDelay still overrides it.

When a retryable response carries a Retry-After header (delta-seconds or an HTTP-date), it overrides the backoff delay. A server asking for a longer wait than retry.maxRetryAfter (ms; defaults to the request timeout) is not retried — the response is surfaced as-is:

buildFetch({ retry: { limit: 2, maxRetryAfter: 5000 } });
// 429 + "retry-after: 2"  -> retried after 2s (backoff ignored)
// 503 + "retry-after: 60" -> not retried, the 503 is surfaced immediately

onMetric — resilience observability

Pass onMetric(event) to observe the resilience machinery (for counters, gauges, alerts). It's called best-effort — a throwing hook is swallowed (logged at debug) and never affects the request. Each event has a type and the fields below:

| type | fields | when | |---|---|---| | circuit.open | host | breaker trips open (threshold hit, or half-open probe failed) | | circuit.half_open | host | breaker admits a probe after resetTimeout | | circuit.close | host | breaker recovers (probe succeeded) | | circuit.rejected | host | a request was short-circuited by an open breaker | | concurrency.queued | host | all slots busy — the request had to wait | | concurrency.shed | host | the per-host queue was full (ECONCURRENCYLIMIT) | | stale.served | cacheKey | a last-good response was served on failure | | retry | url, attempt, reason ("status"/"network"), statusCode?, retryAfterMs? | a retry was scheduled (retryAfterMs when a Retry-After header set the delay) |

buildFetch({
  circuitBreaker: true,
  staleIfError: 3600,
  onMetric: (e) => {
    if (e.type === "circuit.open") metrics.increment("breaker.open", { host: e.host });
    if (e.type === "stale.served") metrics.increment("stale.served");
  },
});

host is the origin (e.g. https://api.example). Success/error/timing have their own hooks (onSuccess/onError/onNotFound, requestTimeFn) — onMetric is specifically the resilience signal.

Callbacks

The lifecycle callbacks from exp-fetch, unchanged. res is the response ({ statusCode, headers }), content the parsed body, cacheKey the computed key.

function onSuccess(url, cacheKey, res, content) {}   // 2xx
function onError(url, cacheKey, res, content) {}      // > 299 (not 404)
function onNotFound(url, cacheKey, res, content) {}   // 404

onError also fires when staleIfError serves a stale value in place of a 5xx or 429. With errorOnRemoteError: false, error responses resolve with null (like the 404 path).

onRequestInit(requestOptions, cacheKey) runs once before the actual request is made (i.e. on a cache miss) — not on cache hits, and not on subsequent redirect hops. requestOptions is a copy of the request options ({ url, method, responseType, followRedirect, headers }) and does not alter the request. Handy for mocking:

import nock from "nock";

function onRequestInit(requestOptions, cacheKey) {
  const { protocol, host, pathname } = new URL(requestOptions.url);
  nock(`${protocol}//${host}`).get(pathname).reply(200, { mock: true });
}

const { fetch } = buildFetch({ onRequestInit });

requestTimeFn(requestOptions, took) is called after each request with the elapsed milliseconds (the default logs it at debug):

function requestTimeFn(requestOptions, took) {
  console.log("REQUEST", requestOptions.method, ":", requestOptions.url, "took", took, "ms");
}

hooks.beforeRequest

The one got hook carried over (it's widely used to sign/decorate outgoing requests). hooks.beforeRequest is an array of functions (a single function is also accepted), each run — sync or async (awaited) — before every wire attempt: every hop (including redirects) and every retry attempt (got semantics — timestamp/nonce signatures stay fresh on retries). Each receives a mutable request-options object { url, method, headers, body }; mutate it in place to change what's sent:

const { get } = buildFetch({
  hooks: {
    beforeRequest: [ (options) => { options.headers.authorization = sign(options); } ],
  },
});

It runs after the cache key is computed, so mutating headers here does not affect caching — use cacheKeyFn/varyHeaders for that. The options object is this lib's own shape (not a full got options object); got's other hooks (afterResponse, beforeRetry, …) are not supported — passing one logs a warning at build time instead of being dropped silently.

Migrating from exp-fetch

@bonniernews/fetchy is an API-compatible superset of exp-fetch: same buildFetch(behavior) factory, same returned methods (fetch/get/post/put/patch/head/options/ del/stats), same initLRUCache export, and every exp-fetch behavior option is still honored — including the onSuccess/onError/onNotFound/onRequestInit/ requestTimeFn callbacks and hooks.beforeRequest (see Callbacks). Most apps change the import and nothing else.

Two changes you must make:

1. Import as ESM. It's an ES-module package (Node 22+):

   - const buildFetch = require("exp-fetch");
   + import buildFetch from "@bonniernews/fetchy";

   A CommonJS app can await import("@bonniernews/fetchy") instead.
2. Bump nock to v14+ if you mock HTTP with it. exp-fetch ran on got/http, which nock@13 intercepts; @bonniernews/fetchy uses undici, which only nock@14+ reaches (via the global fetch the test transport uses). No test code changes beyond the version bump — or inject undici's MockAgent as agent instead. See Testing with nock.

Default-behavior changes to be aware of (all are deliberate security/correctness fixes; the request/response shapes are otherwise unchanged):

| Behavior | exp-fetch | @bonniernews/fetchy | Notes | |---|---|---|---| | Default cache key | URL + body | method (non-GET) + URL + body + authorization/cookie/proxy-authorization | Verbs no longer share a cache entry (a del() after a cached get() reaches the backend), and credentialed requests no longer share one either (fixes cross-user leakage). A custom cacheKeyFn overrides this — see Security. | | errorOnRemoteError: false | resolves undefined | resolves null (like the 404 path) | Lets error responses be negatively cached (undefined can never be served from cache). | | Cross-origin redirect | forwards all headers | strips credential headers | Same-origin redirects unchanged. | | cacheNotFound: true | caches 404s for ~1 ms (isNumber(true) quirk) | caches 404s per the response's cache-control | Pass a number for an explicit TTL. | | cache-control parsing | substring match (max-age=05→non-cacheable) | per-directive (max-age=05→5 s) | Edge cases only. | | Remote-error message | full util.inspect(body) | body truncated | Prefix "<url> yielded <status>" is preserved; full body still reaches onError. |

New, opt-in (off unless configured, so they don't change migration behavior): staleIfError, circuitBreaker, maxConcurrentPerHost/maxQueuePerHost, per-request signal, varyHeaders, maxResponseBytes, isRedirectAllowed, transport, getTraceContext/traceContextHeader — see Security, Resilience (opt-in), and Default headers.

See the engine-level differences below for the rest.

Differences from the got-based exp-fetch

• Engine: requests go through undici.request in production and the global fetch under test (NODE_ENV === "test"), so nock@14+ intercepts them out of the box (see Testing with nock); you can also inject undici's MockAgent as agent. Override with transport: "undici" / "fetch".
• agent: must be an undici Dispatcher or an Agent-options object (all undici Agent options pass through; the http.Agent-era maxSockets maps to connections). A Node http.Agent is ignored. TLS options move to agent.connect.
• Errors: plain Error (with .statusCode), not VError. Timeout errors carry code: "ESOCKETTIMEDOUT" (and the same message), with the underlying abort as .cause.
• Timeout: an object timeout ({ socket, request, send, ... } — got's phase keys are recognized) collapses to a single deadline (the smallest provided value) that covers the whole exchange including the body read. An explicit 0 disables the deadline.
• retry: a minimal policy (limit/retries, methods, statusCodes, calculateDelay, maxRetryAfter) — honours Retry-After like got, but is not got's full retry engine.
• contentType: JSON and text only. XML is not supported (no xml2js).
• hooks: only hooks.beforeRequest is supported (the one exp-fetch code commonly uses); got's other hooks log a build-time warning. Plus the onRequestInit/onSuccess/onError/onNotFound/requestTimeFn callbacks — see Callbacks.
• Redirects: 303 becomes a body-less GET, and 301/302 rewrite POST to GET (got/browser behavior); 307/308 keep method and body. A 3xx without a Location (e.g. 304) is handled as a plain status, not followed.
• Cloning: uses structuredClone; a non-cloneable payload falls back to the original reference instead of throwing. With deepFreeze: true cache hits skip the clone entirely and return the shared deep-frozen instance (it can't be mutated anyway) — zero per-hit copy cost on hot keys.

Testing with nock

Under test (NODE_ENV === "test") requests go through the global fetch, which nock@14+ intercepts — so existing nock-based test suites work with no configuration:

import nock from "nock";
import buildFetch from "@bonniernews/fetchy";

const { get } = buildFetch(); // NODE_ENV=test -> global fetch
nock("http://api.example").get("/thing").reply(200, { ok: true });
await get("http://api.example/thing"); // intercepted by nock

In production the default is undici.request instead — it's ~1.5–2x faster on the keepAlive path (the WHATWG fetch layer adds per-request overhead; see bench/transport-bench.js). nock doesn't intercept undici.request, but that only matters in tests, where the default is already fetch.

The two transports are normalized to behave identically: caching, redirect following, 404/error handling, maxResponseBytes, compression (both send accept-encoding: gzip, deflate and transparently decompress gzip/deflate/br), repeated headers (comma-joined strings, except Set-Cookie which is an array on both), and the resilience primitives. The keepAlive agent drives connection pooling either way (fetch honours the injected dispatcher), and both read redirects manually (on Node, fetch with redirect: "manual" returns a readable redirect, not an opaque one). test/transport-parity.test.js runs the same assertions through both engines to keep it that way.

Forcing a transport

transport overrides the env-driven default:

• "undici" — always use undici.request (e.g. a perf test that runs under NODE_ENV=test but shouldn't switch engines; or if you mock with MockAgent).
• "fetch" — always use global fetch (e.g. nock in a non-test env).
• a function (url, init) => Promise<{ statusCode, headers, body }> — a custom transport, used verbatim.

Test

npm test

Mocha + Chai. Most behavior is exercised against undici's MockAgent (injected as agent — honoured by both transports); a dedicated suite exercises the default fetch transport against nock; timeout and keepAlive connection-reuse are exercised against a real localhost server.