chunkwise

Resumable multipart uploads for S3-compatible storage. Parallel parts, resume from what storage actually holds, per-part integrity checks. Zero dependencies, browser and Node, 4.4 KB gzipped.

npm install chunkwise

Why

Browser uploads to S3 keep getting rebuilt because the good options each ask for something:

• Uppy @uppy/aws-s3 is excellent, but you adopt Uppy — core, plugins, its state model, its UI.
• tus-js-client is excellent, but tus is a protocol: you have to run a tus server.
• evaporate did exactly this job. Its last release was 2017.

chunkwise is the small one. It does not own your UI, your state, your server, or your build. You give it four functions that talk to whatever signs your URLs; it moves the bytes and gets out of the way.

What it does

• Parallel parts with concurrency that tunes itself to the connection (or a fixed number, if you prefer).
• Resume from storage, not from memory. Before every round it asks your backend what is actually stored. A part that was in flight when the connection died may or may not have landed — only storage knows.
• Integrity checked. Each part's MD5 is compared against the ETag storage returns. A silently corrupted part is retried, not completed.
• Pause, resume, abort as real operations.
• Re-signs on retry. A presigned URL that expires mid-upload is refreshed rather than retried to death.
• Stateless. It writes nothing anywhere unless you hand it a store. Your app owns persistence, because your app owns the UI that shows it.
• Browser and Node, same API, from a file, a Blob, or your own source.

Should you use it?

| | | | --- | --- | | What it is | A byte mover. You give it a file and four functions that produce URLs; it uploads the parts, survives the network, and tells you when storage holds the file. | | Use it when | Files are large enough that losing progress hurts (roughly 50 MB and up). Uploads go direct to storage, not through your API. You want resume, integrity and parallelism without adopting a framework. You are on S3, R2, MinIO, B2, Wasabi, or anything else that speaks multipart. | | Skip it when | The file is small — one fetch(url, { method: 'PUT', body: file }) is the right amount of code, and you should not add a dependency for it. You want a ready-made dropzone UI with thumbnails and editing (use Uppy). Uploads must pass through your own server rather than going direct (use tus). Your storage is a product API like Google Drive or Dropbox (use their SDK). | | Problems it removes | A 4 GB upload dying at 87% and starting over. A part that silently corrupts and is only discovered after you have paid to transcode it. A presigned URL expiring in the middle of a slow upload. A hardcoded concurrency that is too high for hotel wifi and too low for fibre. Shipping a framework's worth of JavaScript for a headless need. | | What it will not do | Sign anything (no SigV4, no credentials — see below). Render UI. Run a server. Transcode, thumbnail, or validate your files. Retry forever: a non-retryable 4xx fails immediately instead of burning attempts. |

Credentials, presigned URLs and STS

chunkwise never sees a credential. It PUTs bytes at whatever URL partUrls() returns, with whatever headers you attach.

• Presigned URLs from your server — the normal case, and what the example implements. Nothing reaches the browser but a URL with a short expiry.
• Temporary STS credentials in the client — chunkwise does not sign requests, because a SigV4 implementation would cost more than the rest of the library. partUrls() is your code, so sign there with a small signer (aws4fetch) or the AWS SDK and return the signed URL. headers on each part URL is sent verbatim for exactly this.
• Anything else — a proxy, a CDN upload endpoint, a signed-cookie scheme. If it accepts a PUT at a URL, it works.

Because URLs are requested per batch and re-signed on retry, short expiries are safe. Prefer them.

Quickstart

import { createUpload } from 'chunkwise';

const upload = createUpload({
  source: file, // a browser File/Blob, or a path string in Node
  backend: {
    create:    (meta)         => post('/uploads', meta),
    partUrls:  (id, numbers)  => post(`/uploads/${id}/parts`, { numbers }),
    resume:    (id)           => get(`/uploads/${id}/resume`),
    complete:  (id, parts)    => post(`/uploads/${id}/complete`, { parts }),
  },
});

upload.on('progress', (p) => {
  console.log(`${p.percent}% · ${(p.bytesPerSecond / 1e6).toFixed(1)} MB/s · ${p.etaMs}ms left`);
});

const result = await upload.start();

That is the whole library. upload.pause(), upload.resume() and upload.abort() do what they say.

The backend contract

Four async functions. They are yours to implement, which is why chunkwise works against S3, Cloudflare R2, MinIO, Backblaze B2, Wasabi, or anything else that speaks multipart.

interface Backend<T> {
  create(meta: { name; size; type }): Promise<{ uploadId: string; partSize: number }>;
  partUrls(uploadId: string, partNumbers: number[]): Promise<{ partNumber: number; url: string; headers? }[]>;
  resume(uploadId: string): Promise<{ uploaded: { partNumber; size; etag? }[]; missing?: number[]; bytesStored?: number }>;
  complete(uploadId: string, parts: { partNumber; etag }[]): Promise<T>;
  abort?(uploadId: string): Promise<void>;
}

resume() must report what storage holds, not what the client believes it sent. On S3 that is a ListParts call. This is the single most important thing your backend does — everything resilient about chunkwise rests on it.

A complete Express + AWS SDK implementation is in examples/express-s3-backend.js. It is about 60 lines.

Options

| Option | Default | What it does | | --- | --- | --- | | source | — | File, Blob, a Node path, or your own Source | | meta | — | override name/size/type (a Node path has no content type of its own) | | backend | — | the four functions above | | concurrency | 'auto' | parts in flight, or a fixed number | | minConcurrency / maxConcurrency | 1 / 12 | bounds for 'auto' | | verify | true | compare each part's MD5 against the returned ETag | | retries | 5 | attempts per part, including the first | | store | none | opt-in persistence of the in-progress upload id | | uploadId + partSize | — | resume an upload whose id you kept yourself | | signal | — | an AbortSignal | | fetch | global | your own fetch, for auth headers or instrumentation |

Events

open · progress · part · retry · state · done · error

upload.on('open', ({ uploadId, partSize, partCount }) => { /* persist uploadId if you want resume */ });
upload.on('retry', ({ partNumber, attempt, error, retryInMs }) => {});

Resuming

chunkwise is stateless by default, so pick how you want to remember an upload:

You keep the id. Simplest and most explicit.

upload.on('open', ({ uploadId }) => myDatabase.save(uploadId));
// later
createUpload({ source: file, backend, uploadId, partSize }).start();

Or hand it a store.

import { createUpload, localStorageStore } from 'chunkwise';
createUpload({ source: file, backend, store: localStorageStore() });

A store is { get(key), set(key, record), del(key) } — implement it over IndexedDB, a cookie, or your own server. chunkwise never picks one for you.

Integrity

Every part's MD5 is compared against the ETag storage returns. If they differ, the part is retried; a mismatched part never reaches complete().

This needs nothing from your signer, because a single-part ETag is its MD5. When that is not true — SSE-KMS and SSE-C change the ETag — the check is skipped automatically rather than failing. Set verify: false to skip the hashing entirely.

Multiple files

import { createQueue } from 'chunkwise';

const queue = createQueue({ budget: 6, files: 2 }); // 6 parts in flight across 2 files
queue.on('progress', (p) => console.log(`${p.filesDone}/${p.filesTotal} files · ${p.percent}%`));
for (const file of files) queue.add({ source: file, backend });

Size

| Import | gzipped | | --- | --- | | createUpload | 4.4 KB | | everything | 4.7 KB |

No dependencies, sideEffects: false, ESM and CJS, types included.

License

MIT