upload-engine

Production-grade browser upload pipeline. Framework-agnostic core + React bindings. Inspired by Aliyun OSS MultipartUpload, Tencent COS, AWS S3 Multipart Upload, ByteDance veImageX.

Why

Uploading large / multi-format / weak-network files in the browser is a solved problem in cloud storage SDKs but is repeatedly reimplemented (badly) inside each frontend app. upload-engine extracts the production patterns into one framework-agnostic core.

Features

• Magic-number validation — reads the first 512 bytes to block extension spoofing (virus.exe → virus.jpg is caught). Covers PDF / OOXML / OLE2 / PNG / JPEG / MP4 / MP3 and more.
• Adaptive chunking — probes RTT (median of 3 HEAD requests) and upstream bandwidth (1KB probe), then picks clamp(bandwidth × 5s, 256KB, 16MB) per chunk. Resizes at runtime when 3 consecutive chunks are slow / fast.
• Streaming Merkle hashing — File.stream() → TransformStream chunks and SHA-256s in one pass. Each leaf is written by index, not completion order, so concurrent out-of-order completion cannot corrupt the root. First chunk uploads the moment its hash is ready (pipeline parallelism, ~3.25× speedup over hash-then-upload).
• Circuit breaker — three-state (CLOSED / OPEN / HALF_OPEN). After 5 consecutive failures, all chunks stop; after a cooldown it allows one probe; on probe failure the cooldown doubles (cap 60s). Avoids the "exponential backoff keeps hammering a 5xx server" failure mode.
• Adaptive concurrency — initial concurrency from navigator.connection.effectiveType (slow-2g: 1, 2g: 1, 3g: 2, 4g: 4, 5g: 6), refined at runtime by EWMA-smoothed latency and success rate. Capped at the browser's per-host connection limit minus one.
• Image compression (WASM-grade, Worker-based) — OffscreenCanvas + createImageBitmap decode, EXIF auto-orientation via imageOrientation: 'from-image', format auto-selection (avif → webp → png/jpeg with runtime canEncode probe, never encodes a transparent image as JPEG), target-size binary search over quality in [0.4, 0.95] (≤6 iterations). Reverse guard: never returns a "compressed" blob larger than the original.
• Service Worker background sync + IndexedDB resume — closing the tab or losing network does not lose the upload. The SW drains the queue when connectivity returns. Falls back to beforeunload warning + IDB persistence + resume-on-next-visit on browsers without Background Sync.
• StorageAdapter — pluggable upload target. Ships with an Aliyun OSS PostObject adapter (signature policy issued by a local Node signer that keeps AK/SK in .env, never shipped to the browser) and a zero-dependency local Mock OSS for end-to-end dev without a cloud account.
• Scenario presets — universal / document / image / audio / video / ai-image, each with the right whitelist, validators, and chunking policy.

Install

npm install upload-engine
# optional React bindings
npm install react react-dom

Usage

Framework-agnostic core

import { createUploader, PRESETS } from 'upload-engine'

const uploader = createUploader()
uploader.on((event) => {
  // 'validate:ok' | 'chunk:complete' | 'chunk:error' | 'merge:ok' | 'circuit:open' | ...
  console.log(event)
})

const file = document.querySelector('input[type=file]').files[0]
await uploader.upload(file, { config: PRESETS.document })

React

import { useUpload, PRESETS } from 'upload-engine/react'

function App() {
  const { files, upload, dropZoneProps } = useUpload(PRESETS.image)
  return (
    <div {...dropZoneProps}>
      {files.map((f) => (
        <div key={f.id}>
          {f.name} — {f.progress}%
        </div>
      ))}
    </div>
  )
}

Aliyun OSS direct upload

import { createUploader, createOSSAdapter, PRESETS } from 'upload-engine'

const uploader = createUploader({
  adapter: createOSSAdapter({
    signUrl: 'http://localhost:5180/sign', // your local Node signer
    bucket: 'my-bucket',
    region: 'oss-cn-hangzhou',
  }),
})

await uploader.upload(file, { config: PRESETS.universal })

Keys live in server/.env of the dev signer. In production switch to STS temporary credentials (same adapter interface).

Pipeline

File.stream
  └─ Layer 0  magic-number check (first 512 B)
  └─ Layer 1  adaptive chunk size (RTT + bandwidth probe)
  └─ Layer 2  streaming Merkle SHA-256 (TransformStream, in-order leaves)
  └─ Layer 3  circuit breaker (CLOSED / OPEN / HALF_OPEN)
  └─ Layer 4  adaptive concurrency (NetworkInfo + EWMA)
  └─ Layer 5  image compression (Worker + OffscreenCanvas, format auto + target-size bisection)
  └─ Layer 6  Service Worker Background Sync + IndexedDB resume
                ▲
        StorageAdapter (Aliyun OSS / local Mock / …)

Bundles

| entry | size | gzip | |--------------|---------|-------| | core | 12.8 KB | 6.7 KB | | react | 36.3 KB | 9.3 KB | | hash worker | ~1 KB | — | | image worker | ~2.7 KB | — |

Core has zero framework dependencies. React is an optional peer dependency.

Browser support

Chrome 90+, Safari 14.1+, Firefox 90+. DecompressionStream('deflate-raw'), File.stream(), OffscreenCanvas, navigator.connection, optional BackgroundSyncManager — every layer degrades gracefully when an API is missing.

License

MIT