Tranquilload

Type-safe, runtime-agnostic file upload library built on Effect. One-shot or multipart. Configure-once, resilient by default.

Tranquilload is a TypeScript library for uploading bytes — from a File, a Node Readable, or any ReadableStream<Uint8Array> — to anywhere that accepts chunks. It is built around two ideas:

1. A small, protocol-agnostic core. The core knows nothing about S3, HTTP, or your backend. It orchestrates the lifecycle of a transfer: chunking, concurrency, retries, abort, progress, persistence, completion. You hand it uploadPart and completeUpload as opaque callbacks.
2. Adapters as configuration presets. An adapter (s3MultipartUpload, simpleHttpUpload, fromFile, …) is a small function that returns the callbacks the core expects. They give you a one-liner for the happy path, without hiding the underlying contract.

It runs on Node 22+, modern browsers, Bun, and Deno — anywhere WHATWG Streams and CompressionStream are available.

Why Tranquilload?

Most upload libraries either expose too little (a single upload(file) call you can't extend), or are tightly coupled to a specific backend (S3-only, tus-only). Tranquilload sits in the middle: declarative configuration on the outside, Effect under the hood. You get every resilience feature from the start, but you don't pay an Effect tax to use the library — Promises and ReadableStream are the default surface.

What "configure-once" gives you:

• Retry policies per error type (network errors retry, application errors don't), composable with Effect.Schedule
• Concurrency control via Effect.Semaphore — back-pressure is free, no Promise.race to write
• Abort interop via AbortSignal — handled as a first-class state, not an exception
• Cross-session resumption — pass an uploadId and an optional reconcileCompletedParts callback, and resume after a crash, a refresh, or a network drop
• Adaptive chunk size — measure throughput per part and shrink/grow the next chunk accordingly (networkMultiplier, computeOptimalPartSize)
• Circuit breaker — stop retrying when the network is clearly gone
• Pipeline composition — compress, hash, encrypt, or any custom Transform between source and uploader
• Closed, exhaustive error union — UploadError is a discriminated union of 9 variants; no catch (e: unknown)
• UploadEvent stream — every state change emits a tagged event; subscribe if you care, ignore if you don't

Install

pnpm add @tranquilload/core @tranquilload/adapters effect
# or npm / yarn / bun

Two packages on npm:

• @tranquilload/core — the protocol-agnostic engine. Usable on its own if you write your own callbacks.
• @tranquilload/adapters — ready-made presets (S3, HTTP, File, Node Readable). Optional.

effect is a peer dependency — installed once, shared across both packages.

Dual-format, types-first. Both packages ship an ESM (.mjs) and a CommonJS (.cjs) build side by side, each with its own declarations (.d.mts / .d.cts). import and require therefore both resolve to correct types under node16/nodenext as well as bundler — there is no @types/* companion to install and no moduleResolution tuning to do. Source maps ship too, with the original TypeScript embedded, so a debugger steps into real source rather than bundled output.

Everything is a subpath. There is no root export: you import from @tranquilload/core/multipart, @tranquilload/adapters/s3MultipartUpload, and so on — six entry points per package, listed under Package layout. This is what keeps bundles small; reaching for the multipart engine never drags in the one-shot path or the compression service. Nothing outside fromNodeReadable imports node:*, so the browser build stays clean.

Each of those three properties is locked by CI against the published tarball — dual-format resolution, tree-shakeability, and the absence of stray node: imports are asserted on a real npm install of the packed artifact, not on the source tree.

Both packages are published from CI with build provenance, so every release is cryptographically traceable to the workflow run that built it.

Requires Node 22+. Older runtimes are missing process.getBuiltinModule, used by the build toolchain.

HTTP streaming requirements. simpleHttpUpload streams the request body by default (with duplex: 'half'), which requires HTTP/2 and a runtime that understands the duplex option (Node 22+, modern browsers). For HTTP/1.x endpoints, opt in to bufferMode: true — see Adapters → simpleHttpUpload.

Quick start

One-shot upload (small file, single HTTP request)

import { uploadOnce } from "@tranquilload/core/oneshot";
import { fromFile } from "@tranquilload/adapters/fromFile";
import { simpleHttpUpload } from "@tranquilload/adapters/simpleHttpUpload";

const { stream } = fromFile(file);
const { upload } = simpleHttpUpload({
  url: "https://api.example.com/upload",
  method: "PUT",
  headers: { "Content-Type": file.type },
});

const { result, events } = uploadOnce({ stream, upload });

await result; // UploadCompleted, or throws a typed UploadError

Multipart upload to S3 (resumable, retried, concurrent)

import { uploadMultipart } from "@tranquilload/core/multipart";
import { fromFile } from "@tranquilload/adapters/fromFile";
import { s3MultipartUpload } from "@tranquilload/adapters/s3MultipartUpload";

const { stream, totalBytes } = fromFile(file);

const s3 = s3MultipartUpload({
  bucket: "my-bucket",
  key: `uploads/${file.name}`,
  s3Client, // any client implementing { createMultipartUpload, completeMultipartUpload }
  getPresignedUrl: async (partNumber, uploadId) =>
    fetch(`/api/sign?uploadId=${uploadId}&part=${partNumber}`).then((r) =>
      r.text(),
    ),
});

const { uploadId, result, events, getProgress } = uploadMultipart({
  stream,
  totalBytes,
  maxConcurrency: 4,
  ...s3, // injects chunkSize + initiate + uploadPart + completeUpload
});

// Persist the uploadId early so you can resume after a refresh
const id = await uploadId;
localStorage.setItem("upload:current", id);

// Subscribe to progress events (optional)
for await (const event of events) {
  if (event._tag === "ProgressTick") {
    console.log(`${event.bytesUploaded} bytes uploaded`);
  }
}

await result;

Resuming an upload after a refresh

The lib produces a ResumeState you persist and pass back. The lib re-validates chunkSize, pipelineIdentity, and the content digest before any byte is uploaded — see Concepts → Resume Safety for what each field guards against.

import type { ResumeState } from "@tranquilload/core/multipart"

// First session — fresh init
const { uploadId, resumeState, result } = uploadMultipart({
  stream,
  totalBytes,
  ...s3,
  getContentDigest: () => `${file.name}|${file.size}|${file.lastModified}`,
  pipelineIdentity: "deflate-v1", // if you set a `pipeline`
});

const state = await resumeState;
localStorage.setItem("upload:current", JSON.stringify(state));

await result;
localStorage.removeItem("upload:current");

// Subsequent session — resume
const stored = localStorage.getItem("upload:current");
if (stored) {
  const parsed = JSON.parse(stored) as ResumeState;
  const { result } = uploadMultipart({
    stream,
    totalBytes,
    ...s3,
    getContentDigest: () => `${file.name}|${file.size}|${file.lastModified}`,
    pipelineIdentity: "deflate-v1",
    reconcileCompletedParts: async () => {
      const res = await fetch(`/api/parts?uploadId=${parsed.uploadId}`);
      return res.json(); // [{ partNumber, etag }, ...]
    },
    resumeFrom: parsed,
  });
  await result;
}

Adaptive chunk size based on network throughput

import { networkMultiplier } from "@tranquilload/adapters/networkMultiplier";
import {
  computeOptimalPartSize,
  S3_MIN_PART_SIZE,
} from "@tranquilload/adapters/optimalPartSize";

const multiplier = networkMultiplier();
const basePartSize = computeOptimalPartSize({
  totalBytes,
  targetPartCount: 100,
  minPartSize: S3_MIN_PART_SIZE,
});

// Wire `multiplier.record(bytes, durationMs)` from your PartCompleted events,
// then `Math.round(basePartSize * multiplier.factor())` is your next chunkSize.

Client-side compression in the pipeline

import { uploadMultipart } from "@tranquilload/core/multipart";
import { compress } from "@tranquilload/core/pipeline";

const { result } = uploadMultipart({
  stream,
  totalBytes,
  ...s3,
  pipeline: compress("deflate-raw"), // any algo CompressionStream supports
});

deflate-raw browser support matrix (verified by the nightly PW-Lib suite — 11.7-E2E-003):

| Engine | CompressionStream("deflate-raw") | |---|---| | Chromium | ✅ supported | | Firefox | ✅ supported | | WebKit (current) | ✅ supported |

Older WebKit releases historically lacked deflate-raw. If you must support a browser without it, fall back to compress("gzip") (universally available) — the upload pipeline is otherwise identical.

Concepts

Two cores, two APIs

| Module | When to use it | Returns | | ------------------------------ | ---------------------------------------------------- | ------------------------------------------- | | @tranquilload/core/oneshot | Whole body fits in one request — simple PUT/POST | { result, events } | | @tranquilload/core/multipart | Large file, resumable, parallel parts | { result, events, uploadId, getProgress } |

There is no forced unification between them. Patterns that turned out to be shared (progress events, abort interop, error union) live in shared modules; everything else stays separate.

Adapters = configuration presets

An adapter is a plain function returning the callbacks the core expects:

function s3MultipartUpload(opts): {
  chunkSize: number;
  initiate: () => Promise<{ uploadId: string }>;
  uploadPart: (partNumber: number, chunk: Uint8Array) => Promise<string>;
  completeUpload: (
    uploadId: string,
    parts: ReadonlyArray<CompletedPart>,
  ) => Promise<void>;
};

Spread it into uploadMultipart({ stream, ...adapter }) and you're done. Want to swap S3 for tus, or for a custom backend? Write a 30-line adapter — the core does not change.

Dual-mode callbacks (Promise or Effect, never required)

Every user-provided callback can return a value, a Promise<T>, or an Effect.Effect<T, UploadError>. The library detects via Effect.isEffect and normalizes internally. You can use the entire library without ever importing effect.

If you do want the full Effect surface, every public function exposes an .effect escape hatch with the Layers left open:

import { uploadMultipart } from "@tranquilload/core/multipart"

const stream = uploadMultipart.effect({ ... }) // Stream<UploadEvent, UploadError, LoggerService>

Errors are data

UploadError is a closed, exhaustive discriminated union (9 variants — one per upload phase, plus ResumeMismatchError for resume validation refusals). Use Match.tag or a switch on _tag. ResumeMismatchError uses an internal reason discriminant — dispatch on it with a nested Match.value:

import { Match } from "effect"

result.catch((err: UploadError) =>
  Match.value(err).pipe(
    Match.tag("InitiateUploadError", () => { /* safe to retry from scratch */ }),
    Match.tag("PartUploadError",     (e) => { /* part ${e.partNumber} failed */ }),
    Match.tag("MaxRetriesExceededError", () => { /* give up */ }),
    Match.tag("ReconcileError",      () => { /* parts state unknown */ }),
    Match.tag("CompleteUploadError", () => { /* parts uploaded, retry .complete() or abort */ }),
    Match.tag("PresignedUrlError",   () => { /* could not obtain a signed URL */ }),
    Match.tag("CircuitOpenError",    () => { /* too many failures, pause */ }),
    Match.tag("AbortError",          () => { /* user cancelled */ }),
    Match.tag("ResumeMismatchError", (e) =>
      Match.value(e.reason).pipe(
        Match.when("version_mismatch",   () => { /* upgrade lib or clear state */ }),
        Match.when("chunksize_mismatch", () => { /* chunkSize changed; start over */ }),
        Match.when("pipeline_mismatch",  () => { /* pipeline changed; start over */ }),
        Match.when("content_mismatch",   () => { /* source content differs; start over */ }),
        Match.exhaustive,
      ),
    ),
    Match.exhaustive,
  ),
)

Resume Safety

ResumeState carries five validation fields. Each one is a tripwire for a silent-corruption class:

| Field | Guards against | |---|---| | version: 1 | Schema evolution — a v1 state passed to a future v2 lib fails fast with ResumeMismatchError("version_mismatch") instead of being silently misinterpreted | | chunkSize | Byte misalignment — changing chunkSize between sessions corrupts the completed object | | pipelineIdentity (opt-in) | Pipeline composition drift — resuming with a different compression algorithm produces a Frankenstein object | | contentDigest (opt-in, from getContentDigest) | Content swap — resuming with a different file of the same name+size uploads wrong bytes against the stored uploadId | | contentDigestCaptured | Persistence-layer field drop — if the original session captured a digest but the persisted state lost it, the lib refuses to resume |

Compression non-determinism caveat. Even with identical pipelineIdentity, a non-deterministic pipeline (e.g. gzip with mtime headers, encryption with random salt) produces different bytes per run. Resume against the same uploaded parts only works if your pipeline is byte-deterministic. Verify before relying on this.

Reconciled-part integrity (the stale-reconcile trust boundary)

On resume, reconcileCompletedParts tells the lib which parts already exist so it can skip re-uploading them. The lib trusts that answer — it forwards each reported part's etag straight to completeUpload without re-checking that the part still exists. That trust has an edge: if the storage backend garbage-collects a part between your reconcile probe and the final completeUpload (e.g. an S3 lifecycle rule that expires incomplete-multipart parts), the commit is rejected and the upload fails with CompleteUploadError at the complete phase.

The library does not auto-detect and re-upload the missing part, by design — and it cannot do so honestly. The complete-phase error does not structurally name which part is gone (parsing it would tie the protocol-agnostic core to S3 error strings), and the skipped part's bytes have already been discarded: the source stream is fully drained by the time completeUpload runs, so an in-band re-upload would mean retaining every reconciled part in memory through to the end — defeating the whole point of resuming. So the trust boundary is documented, with two honest, caller-side remedies.

1. Close the window — verify before you skip. reconcileCompletedParts is where parts are classified as "done". Only report parts you have confirmed still exist (the same ListParts/HeadObject probe you already run), so a GC'd part is treated as a normal missing part and re-uploaded instead of trusted:

reconcileCompletedParts: async () => {
  const parts = await listParts(uploadId); // your backend probe
  // Drop anything the backend cannot confirm is still present.
  return parts.filter((p) => p.etag.length > 0 && p.size > 0);
},

2. Recover after the fact — re-probe and re-invoke. A part can still be GC'd in the (usually day-scale) gap between reconcile and complete. If completeUpload rejects, re-probe and re-run the upload with a fresh source stream; the re-probed reconcile no longer lists the missing part, so it is re-uploaded and the upload completes. The library is idempotent across invocations, so the second run re-reads the source, re-uploads only the now-missing parts, and commits:

import { uploadMultipart } from "@tranquilload/core/multipart";
import { CompleteUploadError } from "@tranquilload/core/errors";

async function uploadWithStaleRecovery(
  openStream: () => ReadableStream<Uint8Array>,
): Promise<void> {
  const run = () =>
    uploadMultipart({
      stream: openStream(), // a FRESH stream each attempt
      chunkSize: 8 * 1024 * 1024,
      reconcileCompletedParts: async () => listParts(uploadId), // re-probed each run
      uploadPart,
      completeUpload,
    }).result;

  try {
    await run();
  } catch (err) {
    if (err instanceof CompleteUploadError) {
      // A reconciled part was GC'd before commit — re-probe + re-drive once.
      await run();
    } else {
      throw err;
    }
  }
}

(A fully GC'd upload — NoSuchUpload on the reconcile itself, not a single part — is the separate reinitOnStale case: see the reinitOnStale option, which re-initiates a fresh multipart from part 1.)

Ingest integrity (the no-checksum trust boundary)

The core deliberately does not checksum the bytes your pipeline produces. It trusts your CompressionService and your source stream and uploads whatever bytes come out — a zero-overhead trust boundary. The consequence: a buggy compressor (or any in-pipeline corruption) silently produces a corrupt object.

A digest of the uploaded bytes cannot catch this on its own — it faithfully matches whatever (corrupt) bytes the pipeline emitted. So the library does not ship a built-in "did my compressor misbehave?" check; that would be a false sense of safety. What you can verify, with no extra library API, is the wire (client → storage): every uploadPart(partNumber, chunk) hands you the exact bytes about to be sent, so you can checksum chunk and forward a server-verified header (e.g. S3's trailing x-amz-checksum-sha256). The server then rejects any corruption introduced between your client and storage:

import { uploadMultipart } from "@tranquilload/core/multipart";

const { result } = uploadMultipart({
  stream,
  chunkSize: 8 * 1024 * 1024,
  initiate,
  completeUpload,
  // You receive the exact post-pipeline bytes for each part — checksum them and
  // let the server verify wire integrity via a trailing checksum header.
  uploadPart: async (partNumber, chunk) => {
    const digest = await crypto.subtle.digest("SHA-256", chunk);
    const checksum = btoa(String.fromCharCode(...new Uint8Array(digest)));
    const res = await fetch(presign(partNumber), {
      method: "PUT",
      headers: { "x-amz-checksum-sha256": checksum },
      body: chunk,
    });
    return res.headers.get("etag")!;
  },
});

This guards the wire, not a buggy pipeline: a checksum computed over the post-pipeline bytes (client- or server-side) cannot tell that the compressor mangled its input. If you need to detect that class of bug, compare a digest of the source bytes you fed in against a digest of the decompressed result, out of band — it is not something a streaming uploader can decide for you.

simpleHttpUpload streaming vs buffered

By default, simpleHttpUpload sends the source as a streaming ReadableStream body with duplex: 'half'. This requires:

• An HTTP/2 endpoint (modern browsers reject HTTP/1.x stream uploads).
• A runtime that accepts the duplex flag on fetch (Node 22+, current browsers). The flag is silently ignored on older runtimes — the request shape is then wrong.

For HTTP/1.x targets or older runtimes, opt in to bufferMode: true. The adapter drains the source into a Blob before sending; no duplex flag is required.

Memory caveat. bufferMode: true holds the entire source in memory. Do not enable for files larger than available memory.

Size-bounded auto-buffer (when you know the size). Rather than toggling bufferMode per environment, hand the adapter the source size and a ceiling and let it choose — the HTTP/1.1-safe buffered path for small sources, streaming for large ones:

simpleHttpUpload({
  url,
  contentLength: file.size,     // known source size
  maxAutoBufferBytes: 8_000_000, // buffer up to 8 MB; stream above it
});

The decision is made before the (single-use) stream is consumed:

• contentLength <= maxAutoBufferBytes → buffered PUT/POST (works on HTTP/1.x, every engine, no manual bufferMode).
• contentLength > maxAutoBufferBytes → streamed PUT/POST (duplex: 'half', HTTP/2) — the large source is never held in memory.

This is why the size must be known up front: a ReadableStream can't be measured without consuming it, and once consumed it can be neither re-streamed nor buffered. maxAutoBufferBytes therefore requires contentLength (the factory throws a TypeError otherwise) and will not blindly buffer an unsized source. bufferMode: true still wins if set (explicit mode beats auto). HTTP/2 detection is intentionally not attempted — the Fetch API exposes no negotiated-protocol signal in the browser, so the size threshold is the honest, memory-safe knob.

The bound is only as honest as contentLength. The buffer-vs-stream choice trusts the size you supply. Understating it can still buffer a larger-than-expected source (the uploaded data is always correct — the Blob is built from the real drained bytes — but the memory ceiling is only as accurate as your number). Pass file.size for a File/Blob.

Events are a stream

uploadMultipart returns an events: ReadableStream<UploadEvent>. Subscribe with for await, pipe to a TransformStream, or ignore it entirely — no overhead if unused. Events: UploadInitiated, PartCompleted, ProgressTick, CircuitOpen, UploadCompleted.

Why effect is a peer dependency

You'll notice the install command asks for effect explicitly:

pnpm add @tranquilload/core @tranquilload/adapters effect

effect is declared as a peerDependency in both packages (not a regular dependency). This is intentional, and the reason is specific to how Effect works.

The core constraint: a single shared effect instance. Context.Tag(key)() returns a new class object every time it is evaluated — two copies of effect in node_modules produce two distinct LoggerService classes for the same key. The Effect runtime's context lookup is keyed by tag.key (a string), so Layer.succeed/yield* Tag would still interop across copies for direct service lookup; but every other invariant breaks:

• Class identity (the Tag class itself) diverges, so brand types, instanceof checks against Tag instances, and code that relies on the class as a sentinel all silently misbehave.
• instanceof for Effect's own classes (Cause, Exit, Fiber, etc.) returns false across copies — error matching and Cause inspection from copy A misclassify values produced by copy B.
• Module-level singletons (default Layers, the Fiber registry, the Schedule default scheduler) are duplicated; runtime state set in copy A is invisible from copy B.
• Version skew: two copies can be different effect versions, so an internal type added in 3.20 is missing in 3.19 and runtime calls fail.
• Bundle bloat: effect is hundreds of KB; two copies double the install footprint and download size.

Declaring effect as a peerDependency forces the package manager to resolve a single shared copy. Tranquilload itself locks this contract down in packages/tranquilload-core/src/peer-dep-contract.test.ts (Story 10.8 / F#77).

The same setup is what every Effect-based library does (@effect/platform, @effect/schema, effect-http, …) for the same reason.

Trade-offs:

| What we get | What it costs | |---|---| | Single shared effect instance → class identity and singletons preserved | One extra package name at install time | | User controls the effect version | You need to keep the peer range honest as Effect evolves | | effect is not bundled into our dist → smaller install, no duplicate code if you already use Effect | Tooling warns on incompatible versions (which is the point) |

The peer range is >=3.19.19 — covers minor/patch updates without requiring us to re-publish.

Package layout

@tranquilload/core
├── /oneshot     — uploadOnce  ({ result, events })
├── /multipart   — uploadMultipart  ({ result, events, uploadId, getProgress })
├── /pipeline    — Transform composition (compress, compose)
├── /services    — CompressionService, LoggerService (injectable Effect Layers)
├── /progress    — UploadEvent types
└── /errors      — UploadError union (9 variants)

@tranquilload/adapters
├── /fromFile             — File           → { stream, totalBytes }
├── /fromNodeReadable     — Node Readable  → { stream }
├── /simpleHttpUpload     — One-shot PUT/POST adapter
├── /s3MultipartUpload    — S3 multipart preset (initiate / uploadPart / completeUpload)
├── /optimalPartSize      — Compute chunk size from total + target part count
└── /networkMultiplier    — Throughput-based dynamic chunk sizing

Each entry point is independently importable for tree-shaking. The core never imports from @tranquilload/adapters — the dependency is one-way.

Development

pnpm install
pnpm turbo build       # build core, then adapters
pnpm turbo test        # vitest (with @effect/vitest)
pnpm -r typecheck      # tsc --noEmit across both packages

See CONTRIBUTING.md for the contribution flow and RELEASE_FLOW.md for the Changesets-driven release process.

To go further

The README intentionally stays surface-level. For the design rationale, the architectural constraints, and the rules a contributor (or AI agent) must follow:

• docs/project-context.md — critical implementation rules, technology stack, anti-patterns, and conventions. Read this before touching the code.
• _bmad-output/brainstorming/brainstorming-session-2026-03-08-001.md — the full brainstorming session that shaped the library: First Principles, SCAMPER, and Cross-Pollination over the 45 ideas that defined the v1 scope and the v2 roadmap.

Version history: core changelog · adapters changelog · GitHub releases.

License

MIT © Schrubitteflau