http-fetch-queue

Fetch HTTP resources, checking for changed etags

A queue of HTTP fetches, each of which is retried and reported on, and none of which downloads again what an etag says has not changed. Built on jobs-queue, so fetches run in priority order and the queue can be asked how much work it is holding.

Fetching is the runtime's own fetch, so Node 18 or later is required and there is nothing to install for it. A caller that needs fetching done differently passes in its own, and carries whatever that takes.

Usage

import HttpFetchQueue from "http-fetch-queue";

const fetchQueue = new HttpFetchQueue({
    maxConcurrents: 10, // how many may fetch at once, unlimited if not set
    maxAttempts: 3, // how many times a fetch may be attempted, once if not set
    timeout: 30000, // how long an attempt may take, unlimited if not set
    fetch: myFetch // fetch to use instead of the runtime's own, for every fetch in the queue
});

const response = await fetchQueue.enqueue(url, fetchOptions, contentOptions, jobConfig);

• fetchOptions: passed to fetch, e.g. {headers: {...}}
• contentOptions:
  • etag: the etag last seen for this resource, so that unchanged content is not downloaded again
  • format: json, base64, or text if not given
  • timeout: how long this fetch may take before it is given up on, in ms
  • contentFromResponse: read the response body differently
  • fetch: fetch to use for this one fetch, whatever the queue was given
• jobConfig: per-fetch overrides of the options given to the queue, plus priority (higher fetches first)

Bringing your own fetch

Fetching is the runtime's own unless a fetch option says otherwise, either on the queue or on a single fetch. Anything answering fetch's contract will do.

The reason to want this is usually a server whose certificate the runtime refuses — self-signed, or expired — which the runtime's fetch gives no way to allow for one request. What it takes to allow one is a decision about how much verifying to skip and where, so it is the caller's to make and the caller's dependency to carry, rather than this module's:

import nodeFetch from "node-fetch";
import https from "node:https";

const insecureAgent = new https.Agent({rejectUnauthorized: false});

const fetchQueue = new HttpFetchQueue({
    fetch: (url, options) => nodeFetch(url, {...options, agent: insecureAgent})
});

A certificate that is refused fails the fetch like any other failure to reach the server, carrying the reason as code, e.g. DEPTH_ZERO_SELF_SIGNED_CERT.

What comes back

{
    headers, // the response headers
    etag, // the response etag, or the one that was sent if the content is unchanged
    content // absent where the content has not changed
}

No content means nothing has changed. That is the case whether the server answered 304 Not Modified or answered 200 with the same etag, so a caller with nothing to do in that case need only ask whether content is there.

Statuses

A status the caller did not ask for — anything outside 200-299, other than the 304 that answers an etag — fails the fetch rather than returning as content. The error carries status, statusText, url and a short body excerpt, and reads, for instance:

HTTP 503 Service Unavailable fetching https://example.com/races: upstream is having a moment

Since it is an error, it is subject to maxAttempts like any other failure.

Failures that never reached a status

A fetch that got no answer at all — a name that would not resolve, a refused connection, a dropped socket — rejects with an error that names the URL and the reason, rather than the bare fetch failed the runtime gives, which reads the same for every one of those causes:

Failed to fetch https://example.com/races: getaddrinfo ENOTFOUND example.com

A body that could not be read, including JSON that turned out not to be JSON, is reported the same way and against the same URL:

Failed to read the response from https://example.com/races: Unexpected token '<', ... is not valid JSON

Both carry url, the original error as cause, and, where the network named one, code (ENOTFOUND, ECONNREFUSED, ECONNRESET and so on) for callers that decide what to retry or what to report from it. Like any other failure, they are subject to maxAttempts.

Timeouts

There are two, and they are not the same:

• contentOptions.timeout aborts the request. This is the one that matters: without it, a server that accepts a connection and then says nothing holds the fetch open indefinitely.
• jobConfig.timeout stops waiting on an attempt and fails it, so it is retried if it has attempts left. The request itself carries on unwatched, since a plain promise cannot be cancelled, which is why the first of the two is the one to set.

Events

fetchQueue.on('fetch:attempt', ({attempt, url, fetchOptions, contentOptions, jobConfig}) => {});
fetchQueue.on('fetch:success', ({attempt, url, response, ...}) => {});
fetchQueue.on('fetch:error', ({attempt, url, error, ...}) => {}); // once per failed attempt

The queue's own events — queued, running, requests — come from jobs-queue, as do fetchQueue.queued.size and fetchQueue.running.size, which are worth exposing anywhere the depth of the queue tells you something about how far behind the fetching is.

Stopping

await fetchQueue.destroy();

Cancels every fetch, queued and running. A fetch enqueued afterwards throws.

Tests

npm test