streamarchive

Streaming archive extraction for Node and Bun — libarchive compiled to WebAssembly, with constant memory regardless of archive size.

When the input can seek, zip archives are read via their central directory, so listing entries or extracting a single file never touches the rest of the archive.

Install

npm install streamarchive

Node 18+ and Bun 1.2+. Browser support is designed for — I/O is pluggable — but not yet shipped.

The API is unstable until 1.0; minor versions may break it.

Quick start

import { readArchive } from 'streamarchive';

for await (const entry of readArchive('/data/huge.zip')) {
  if (entry.isFile) {
    const content = await entry.readAll();
    console.log(entry.path, content.length);
  }
}

An entry's data must be consumed before the loop advances — see entry data contract.

Listing and random access

import { listArchive, readArchiveEntry } from 'streamarchive';

// Reads the central directory only — milliseconds on a multi-GB zip.
const entries = await listArchive('/data/huge.zip');

// Seeks to one entry; the rest of the archive is never read.
const result = await readArchiveEntry('/data/huge.zip', 'path/inside/file.pdf');

Remote archives

Pass an http(s):// URL and reads are served by range requests. Listing or extracting one entry from a 24MB remote zip transfers ~0.45MB in the test suite — a figure set by libarchive's seek footprint, not by archive size.

import { listArchive, HttpRangeSource, StreamArchiveReader } from 'streamarchive';

const entries = await listArchive('https://bucket.s3.amazonaws.com/big.zip');

// Auth headers and tuning: open the source explicitly.
const src = await HttpRangeSource.open('https://host/big.zip', {
  headers: { Authorization: `Bearer ${token}` },
  fetchChunkSize: 256 * 1024,
});
for await (const entry of new StreamArchiveReader().entries(src)) {
  /* ... */
}

Streams

A ReadableStream, Node Readable, or async iterable of bytes is read in pure streaming mode:

const response = await fetch('https://host/archive.tar.gz');
for await (const entry of readArchive(response.body)) {
  console.log(entry.path);
}

Input sources

Every API resolves its input automatically:

| Input | Source | Seekable | | --------------------------------------------------- | ----------------- | ----------------------- | | file path string | FileSource | yes | | http(s):// URL string | HttpRangeSource | yes, via range requests | | Uint8Array | BufferSource | yes | | ReadableStream / Node Readable / async iterable | StreamSource | no | | your own ArchiveSource | — | if it implements seek |

New environments implement one interface; the core has no knowledge of files, HTTP, or streams:

interface ArchiveSource {
  /** Fill `target`, return bytes written (0 = EOF). */
  read(target: Uint8Array): Promise<number> | number;
  /** Optional. Implementing it enables random access. */
  seek?(offset: number, whence: 0 | 1 | 2): Promise<number> | number;
  close(): Promise<void> | void;
}

Sources without seek run in streaming mode, where entry sizes may be unknown until read. Sources with seek get the central-directory paths: exact metadata, cheap skipping, single-entry random access.

Supported formats

Verified by test/formats.test.ts against this WASM build — create, extract, byte-compare:

| Format | Status | | --------------------------------------- | -------------------------------------------------------- | | zip, including ZIP64 | streaming + random access | | zip, AES-256 encrypted | with password; wrong or missing password fails cleanly | | tar, and gzip / bzip2 / xz / zstd / lz4 | streaming | | 7z | supported | | cpio, iso9660 | supported | | rar | untested, not claimed |

API

| Function | Purpose | | --------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | readArchive(input, opts?) | Async generator of entries (one-shot) | | readArchive$(input, opts?) | RxJS Observable of entries (one-shot) | | listArchive(input, opts?) | Metadata for all entries, no data reads | | readArchiveEntry(input, path, opts?) | One entry's metadata and content | | new StreamArchiveReader(opts?) | Reusable reader: entries, read$, list, readEntry, extractAll, close | | ArchiveSession | Low-level sequential reader: open, nextEntry, readChunk, skipData, close | | initStreamArchive(opts?) | Explicit WASM init | | FileSource, HttpRangeSource, StreamSource, BufferSource | Built-in sources |

Options for all APIs: bufferSize (default 256KB), streaming (force streaming on seekable input), password, signal, onProgress.

Entry data contract

readAll() and data$ are single-shot and must be consumed before the iteration advances, because archives are sequential streams underneath. Reading late throws ArchiveClosedError rather than returning wrong bytes. Unconsumed entries are skipped efficiently, by seeking where the source allows it.

Archive operations are serialized within a process — one suspended operation per WebAssembly module instance. Use worker threads for parallel extraction.

Errors

Typed classes with stable code values: ArchiveOpenError, ArchiveReadError, ArchiveClosedError, WasmLoadError, IOError, AbortError, plus the isStreamArchiveError() and hasErrorCode() guards. Failures raised by your own ArchiveSource are preserved as error.cause.

Bundlers and single-file executables

The WASM module loads from the package directory by default. Where that layout is gone — bun build --compile, pkg — inject the bytes:

import { initStreamArchive } from 'streamarchive';
import wasmPath from 'streamarchive/lib/wasm/streamarchive.wasm' with { type: 'file' };

await initStreamArchive({ wasmBinary: await Bun.file(wasmPath).arrayBuffer() });

initStreamArchive also accepts a precompiled WebAssembly.Module via wasmModule, or a locateFile override.

Benchmark

10.07GB zip (ZIP64, deflate), 1011 files of incompressible data, every extracted byte verified against SHA-256 manifests. Run in containers so the figures are reproducible rather than machine-specific:

podman run --rm --memory=64m --memory-swap=64m -v "$PWD":/work -w /work \
  node:22-alpine node --experimental-strip-types scripts/benchmark.ts --size-gb=10

| Operation | Result | | ------------------------------------- | ------------------------------------- | | Full extraction, 10.07GB / 1011 files | 0 integrity errors, ~210 MB/s | | list() of 1014 entries | 0.16s, reads 65.6MB of 10.07GB (0.6%) | | readEntry() of the last entry | 0.18s, reads 73.6MB of 10.07GB (0.7%) |

Memory is a hard cgroup limit with swap disabled, so completion is enforced by the kernel rather than observed — the run either finishes inside the cap or the OOM killer ends it. Every image below finished with oom_kill 0.

| Image | Engine | Lowest cap that completes | | ----------------------------------------- | ------ | ------------------------- | | node:22-alpine (Node 22) | V8 | 64MB | | ubi9/nodejs-22 (Node 22.23.1, RHEL 9.8) | V8 | 96MB | | oven/bun:1.2 (Bun 1.2.23) | JSC | 128MB |

What differs between them is the runtime's own idle footprint, not the cost of reading the archive. Both engines also grow their heap to fill whatever memory is available, so an uncapped run measures free RAM more than anything else.

Constant memory is the claim actually under test, and it holds on both engines: a 1GB archive and a 10.07GB archive peak within ~1MB of each other on Bun (229.9MB vs 228.9MB uncapped), and within ~3MB on Node. Under a 64MB cap on Node, peak RSS is 60.2MB against a 58.1MB idle baseline — streaming 10.07GB costs about 2MB.

Building from source

The WASM artifacts in lib/wasm/ are committed and ship in the package. Rebuilding them requires Podman; the container build is reproducible and produces byte-identical output.

npm install
npm run build:wasm          # container build, ~10-15 min on first run
npm run build               # TypeScript
npm test                    # requires bsdtar for fixture creation
node scripts/benchmark.ts   # needs ~20GB free disk

License

MIT — see LICENSE.

The published .wasm statically links libarchive, zlib, bzip2, xz, Zstandard, LZ4, and Mbed TLS; the loader is emitted by Emscripten. Their licences are in THIRD-PARTY-LICENSES.md.