@millcrest/file-fetcher

Authenticated, segmented, resumable file downloads in the browser.

A Download survives a dropped connection, a page reload, and a browser restart. Bytes stream into an origin-private staging file rather than through memory, so a 4 GB download costs no more RAM than a 4 MB one. Framework-agnostic core; React bindings live at @millcrest/file-fetcher/react.

Why not just use a link

A plain <a href download> already streams to disk and resumes — for free. This library exists for the cases it cannot serve: requests needing an Authorization header, progress and pause/resume inside your own UI, and automatic resume you control.

Status

Working: segmented resumable downloads over concurrent range requests in a Worker; the DownloadManager queue with an enforced concurrency budget, pause/resume/cancel, bounded auto-resume, restore-on-startup and retention sweeping; React hooks; Handoff to the user's filesystem.

Not yet built:

• Digest verification at Handoff. sha256 is accepted and stored but nothing checks it — WebCrypto has no streaming digest, so this needs chunked hashing rather than reading a multi-gigabyte file into memory.
• Content-Disposition filenames. The resolver supplies the name; parseContentDisposition is exported for a resolver that wants to derive one itself.
• Cross-tab progress mirroring. A second tab reports "active in another tab", not a live bar.
• Resume across a reload without host involvement. A resolver is a function and cannot be persisted, so restore() rehydrates and sweeps but does not restart; the host re-enqueues the ids it wants, which getDownloads() lists.

How it works

resolver ──▶ { url, size, validator, sha256? }        fresh on every attempt
                     │
              ┌──────▼───────────────────────────┐
              │ Worker (one per Download)        │
              │  N concurrent range requests     │
              │  verify 206 + Content-Range      │
              │  write at absolute offsets ──────┼──▶ OPFS staging file
              │  flush ─▶ persist complete       │     (truncated to full size)
              └──────┬───────────────────────────┘
                     │ progress / record
              main thread ──▶ Save (user gesture) ──▶ user's filesystem

Four decisions do most of the work, each recorded in docs/adr/:

• Staging in OPFS, not a user-picked file. The File System Access API reaches 28.6% of browsers and no mobile at all; OPFS reaches 93.5% and needs no permission prompt. (ADR 0001)
• A URL resolver, never a stored URL. An S3 URL presigned by a Lambda dies with its role session — often within the hour — so a persisted URL fails exactly when resume matters. (ADR 0002)
• Write, flush, then persist. Reversed, a crash leaves a record claiming bytes that never reached disk, and resume produces a corrupt file. In this order a crash costs a re-download. (ADR 0003)
• Verify every range response before writing a byte. A server that ignores Range returns 200 with the whole object; written at a segment's offset that corrupts the download while progress looks healthy. (ADR 0004)

CONTEXT.md is the glossary. docs/server-requirements.md is the contract a server must satisfy, with the S3 and GCS specifics.

Install

npm install @millcrest/file-fetcher

React is an optional peer dependency (>=18), needed only for the hooks at @millcrest/file-fetcher/react. The core entry point has no dependencies at all.

Usage

The resolver is the only thing you must supply. It returns a currently valid URL plus the metadata the client would otherwise have to read from CORS-exposed headers, and it is called fresh on every attempt — which is what makes an expired presigned URL or bearer token a non-event.

import { DownloadManager } from "@millcrest/file-fetcher";

const manager = new DownloadManager({
  segmentSize: 8 * 1024 * 1024,
  segmentConcurrency: 4,
  maxConcurrentDownloads: 3,
});

await manager.restore(); // rehydrate what survived, sweep what expired

manager.enqueue({
  id: "invoice-2026-07", // stable id, not a URL
  resolve: async (id) => {
    const res = await fetch(`/api/files/${id}/download-url`, {
      headers: { Authorization: `Bearer ${await getToken()}` },
    });
    return res.json(); // { url, size, validator, sha256?, filename? }
  },
});

Then, from a click handler: await manager.save("invoice-2026-07").

Note the resolver is called twice per start — once for the quota preflight, once by the engine — plus once per credential refresh, so it should be cheap and free of side effects beyond minting a URL.

Call manager.dispose() on teardown to abort running downloads and release timers.

For a single download with no queue, createDownload(id, resolve) is a thin façade over a one-slot Manager. Construct it once per application, not once per download: each instance owns a Manager, and two Managers in a tab contend for the same Web Locks.

Validator kinds

validator is compared to detect a changed file, and sent as a precondition. Because a validator is not always an ETag, say which kind it is:

| validatorKind | Precondition sent | Use for | | ------------------- | ---------------------------- | ------------------------------------------------------ | | "etag" (default) | If-Match | S3, and any server whose ETag is stable | | "goog-generation" | x-goog-if-generation-match | GCS x-goog-generation values | | "opaque" | none | anything else — change is detected via Content-Range |

Getting this wrong is not subtle: a GCS generation sent as If-Match fails every segment with 412.

React

import { DownloadManager } from "@millcrest/file-fetcher";
import {
  DownloadManagerProvider,
  useDownloads,
  useDownloadProgress,
  useDownloadControls,
} from "@millcrest/file-fetcher/react";

// One per tab. Two Managers would contend for the same Web Locks.
const manager = new DownloadManager();

function App() {
  return (
    <DownloadManagerProvider manager={manager}>
      <List />
    </DownloadManagerProvider>
  );
}

function Row({ id }: { id: string }) {
  const { bytes, total } = useDownloadProgress(id); // re-renders only this bar
  const { pause, save } = useDownloadControls(id);
  return <progress value={bytes} max={total || 1} onClick={pause} />;
}

function List() {
  return useDownloads().map((d) => <Row key={d.id} id={d.id} />); // no progress here
}

The split is deliberate: chunk callbacks fire hundreds of times a second per download, so useDownloads() carries state but not progress. A tick re-renders one bar, not the table.

Concurrency depends on your server's protocol

On HTTP/1.1 browsers allow only 6 connections per origin, so segmentConcurrency × maxConcurrentDownloads above 6 just queues. Our API serves HTTP/2, so the 4 × 3 defaults multiplex fine. S3 serves HTTP/1.1 and GCS serves HTTP/2 (both measured); CloudFront serves HTTP/2 and HTTP/3. See docs/server-requirements.md. Against S3 directly, prefer 3 × 2.

Progress can move backwards

A segment counts only once fully written and flushed; partially-fetched segments are discarded on interruption. A bar at 41% may resume at 38%. Don't assume monotonicity.

Durability has a one-week ceiling

Safari evicts script-writable storage after ~7 days without interaction with the site, so a download parked awaiting Save can vanish regardless of what this library does. If the staging bytes are gone when a download resumes, the library detects it and re-fetches rather than handing you a zero-filled file.

Bundlers

new Worker(new URL("./worker.js", import.meta.url)) works in webpack 5 and native ESM. Vite's dependency optimizer does not rewrite that pattern inside node_modules, so either exclude the package from pre-bundling:

export default defineConfig({ optimizeDeps: { exclude: ["@millcrest/file-fetcher"] } });

or supply the Worker yourself:

import DownloadWorker from "@millcrest/file-fetcher/worker?worker";
new DownloadManager({ runner: new WorkerRunner({ workerFactory: () => new DownloadWorker() }) });

The same escape hatch covers a CSP that forbids the default worker source.

Development

vp install
vp check                      # format, lint, type check
vp test                       # 153 tests: 149 in Node, 4 in a real browser
vp test run --project node    # skip the browser suite
vp pack                       # build the library
vp pack --watch               # rebuild on change

vp pack is the build — not vp build, which is the Vite application build and fails here for want of an index.html.

The browser suite needs a headless Chromium (playwright install chromium) whose system libraries are present; playwright install --with-deps needs root on Linux.

Publishing runs from CI on a pushed v* tag — see RELEASING.md.

Testing approach

Ports and adapters (ADR 0009): grid arithmetic, retry decisions, response verification, and the whole engine and queue run in Node against in-memory fakes — including a crash between flush() and persisting a segment, an evicted staging file, and a credential refresh that returns a different version of the file.

The browser suite covers what only a real browser can answer: sync-access-handle writes landing at the expected offsets, resume across a Worker restart, and Web Lock exclusion. Known gaps: handOff itself, the WorkerRunner message protocol, withDownloadLock, and the React hooks have no tests.

License

MIT — see LICENSE.