@portable/streams
v0.1.0
Published
Collect, combine, and bridge web streams and iterables, in any JavaScript runtime.
Maintainers
Readme
@portable/streams
Collects, combines, and transforms web streams and iterables, and makes your own transforms cancellable. It works without ReadableStream.from and for await over a stream, which some browsers lack. Runs in browsers, Cloudflare Workers, Node 24+, Bun, and Deno.
Replaces get-stream, concat-stream, bl (as a collector), stream-to-array, merge2, merge-stream, through2, and the limit handling of raw-body. @portable/streams/node replaces Readable.toWeb from node:stream.
npm install @portable/streamsimport * as Streams from '@portable/streams';
async function* rows() {
yield 'a';
yield 'b';
}
const upper = Streams.transform(async function* (source: AsyncIterable<string>) {
for await (const chunk of source) {
yield chunk.toUpperCase();
}
});
await Streams.toText(Streams.fromIterable(rows()).pipeThrough(upper)); // 'AB'Collect
Each collector reads a ReadableStream, an async iterable (such as a Node Readable) or an iterable to the end. It releases the stream's lock when it finishes, and cancels the stream if it stops early.
| Function | Returns |
| --- | --- |
| toBytes(source, { maxBytes, signal }) | Uint8Array. String chunks are encoded as UTF-8. |
| toBlob(source, { maxBytes, type, signal }) | Blob with the given type. String chunks are encoded as UTF-8. |
| toText(source, { maxBytes, encoding, signal }) | string. Byte chunks are decoded with TextDecoder (encoding defaults to 'utf-8'), and string chunks are kept. |
| toJson(source, { maxBytes, encoding, signal }) | JSON.parse of the text, typed unknown. |
| toArray(source, { signal }) | The chunks, unchanged. |
toBytes, toBlob, toText, and toJson take Uint8Array and string chunks, and reject with a TypeError for any other chunk. When the source yields more than maxBytes bytes (string chunks count as their UTF-8 length), the collector cancels the source and rejects with MaxBytesError, whose maxBytes property holds the limit. Aborting signal cancels the source and rejects with signal.reason.
const body = await Streams.toJson(request.body!, { maxBytes: 100_000 });Combine
| Function | Does |
| --- | --- |
| concat(sources, { signal }) | One stream with every chunk of the first source, then every chunk of the second, and so on. A source is read only after the one before it ends. |
| merge(sources, { signal }) | One stream that reads every source in the array at once and passes chunks on as they arrive. |
| limitBytes(maxBytes, { signal }) | A TransformStream that passes chunks through and fails with MaxBytesError once more than maxBytes bytes have passed. |
| tee(source, count, { highWaterMark, unit, signal }) | count streams that each receive every chunk of source. |
Sources are streams, async iterables, or iterables. An error in one source errors the output and cancels the others, and cancelling the output cancels every source.
tee splits in the other direction, and differs from ReadableStream.prototype.tee() in three ways:
- It makes any number of branches. The platform's
tee()makes two, so three branches need a chain of two calls. - It reads the source only as fast as the slowest branch. Once a branch holds
highWaterMarkunread chunks,teestops reading the source until that branch reads, so the fastest branch is at most that far ahead. The platform'stee()keeps reading and buffers without limit for a branch that isn't read.highWaterMarkdefaults to 16 chunks. Withunit: 'bytes'it counts bytes, asmaxBytesdoes, and defaults to 65536. - Cancelling a branch removes it, and
teethen reads only as fast as the slowest remaining branch. The source is cancelled when every branch is cancelled, with the reason from the last one. The platform'stee()does the same, with both reasons in an array.
An error in the source errors every branch. Aborting signal errors every branch with signal.reason and cancels the source. Branches receive the same chunk objects, as with the platform's tee() on a stream that isn't a byte stream, so copy a chunk with chunk.slice() before changing it. Because of the limit, if you read one branch to the end before you read the next, that read stalls after highWaterMark chunks. Read the branches at the same time, or pass highWaterMark: Infinity.
const upload = (await request.formData()).get('file') as File;
const [forExif, forStorage, forResize] = Streams.tee(upload.stream(), 3, {
highWaterMark: 1_000_000,
unit: 'bytes',
});
await Promise.all([
Exif.parseStream(forExif), // cancels forExif once it has the metadata
bucket.put(key, forStorage),
bucket.put(`${key}.thumb`, resize(forResize)),
]);Bridge streams and iterables
| Function | Does |
| --- | --- |
| fromIterable(iterable) | A ReadableStream that pulls one value per read. Cancelling it calls the iterator's return(), so a generator's finally blocks run. Uses ReadableStream.from when the runtime has it. |
| toAsyncIterable(stream, { preventCancel }) | An async iterator over a stream. Leaving a for await loop early cancels the stream, or with preventCancel: true only releases the lock. Uses stream.values() when the runtime has it. |
| transform(async function* (source) { ... }, { signal }) | A { readable, writable } pair for pipeThrough, built from an async generator that reads source and yields output. The generator runs only as fast as the output is read. An error thrown in it errors the output and cancels the input. Cancelling the output ends source and runs the generator's finally blocks. |
| map(fn, { signal }), filter(fn, { signal }), tap(fn, { signal }) | TransformStreams that call fn(chunk, index). fn may return a promise, and chunks stay in order. |
| batch(size, { signal }) | A TransformStream that groups chunks into arrays of size, with a shorter last array. |
Aborting signal errors a transform with signal.reason, and pipeThrough then cancels its source. Aborting the signal of concat or merge errors the output with signal.reason and cancels every source. fromIterable and toAsyncIterable take no signal. Cancel the stream that fromIterable returns, or pass signal to the pipeTo that reads it, and leave a toAsyncIterable loop early to cancel its stream.
const lines = Streams.transform(async function* (source: AsyncIterable<string>) {
let rest = '';
for await (const chunk of source) {
const parts = (rest + chunk).split('\n');
rest = parts.pop()!;
yield* parts;
}
if (rest) {
yield rest;
}
});Cancel your own TransformStream
abortable(transformer, { signal }) wraps a Transformer so that aborting signal errors its TransformStream with signal.reason, and pipeThrough then cancels the source. It removes its listener on the signal when the stream ends, is cancelled, or fails because the transformer threw, and returns the transformer unchanged when there is no signal. The stream transforms in lines, base64, csv, and the other Portable packages use it to support their signal option.
class UpperCaseStream extends TransformStream<string, string> {
constructor(options: { signal?: AbortSignal | undefined } = {}) {
super(
Streams.abortable(
{ transform: (chunk, controller) => controller.enqueue(chunk.toUpperCase()) },
options,
),
);
}
}Convert Node streams
@portable/streams/node converts a Node Readable to a ReadableStream, so web stream APIs and the other Portable packages can read the output of a Node library. It imports node:stream, so it runs in Node, Bun, and Deno, and not in browsers or Cloudflare Workers.
import { createReadStream } from 'node:fs';
import * as NodeStreams from '@portable/streams/node';
const text = await Streams.toText(NodeStreams.toWeb(createReadStream('notes.txt')));| Function | Does |
| --- | --- |
| toWeb(readable, { signal }) | A ReadableStream of plain Uint8Arrays from a Readable of bytes. Buffer chunks become Uint8Arrays, and the strings from a Readable with setEncoding are encoded back to the bytes they came from. Any other chunk errors the stream with a TypeError. |
| toWebObjects(readable, { signal }) | A ReadableStream of the values of an object-mode Readable, typed unknown. Buffer chunks become Uint8Arrays. |
Both read from the Readable only when the stream's reader asks for a chunk, so a slow reader pauses the Readable. An error or a destroy() on the Readable errors the stream, and cancelling the stream destroys the Readable with the cancel reason. Aborting signal errors the stream with signal.reason and destroys the Readable with it.
A Buffer chunk becomes a Uint8Array over the same memory, so the conversion copies no bytes. A small Buffer and the Uint8Array made from it share Node's buffer pool with other buffers. Don't transfer the chunk's ArrayBuffer, because that detaches every other buffer in the pool.
For the other conversions, use the ones in node:stream: Readable.fromWeb for a ReadableStream that a Node API needs to read; Writable.toWeb and Writable.fromWeb for writables; and Duplex.toWeb and Duplex.fromWeb for transforms. Keep the work in web streams and convert only where a Node API needs a Node stream, such as a file:
import { createWriteStream } from 'node:fs';
import { Writable } from 'node:stream';
import * as Csv from '@portable/csv';
const response = await fetch('https://example.com/report.csv');
await response.body!
.pipeThrough(new Csv.DecoderStream())
.pipeThrough(Streams.map((row: readonly string[]) => `${row.join('\t')}\n`))
.pipeThrough(new TextEncoderStream())
.pipeThrough(new CompressionStream('gzip'))
.pipeTo(Writable.toWeb(createWriteStream('report.tsv.gz')));Types
| Type | Description |
| --- | --- |
| Streams.Source | What the collectors and combinators read: a ReadableStream, an async iterable, or an iterable. |
| Streams.abortable.Transformer | The Transformer that abortable takes and returns. It adds the optional cancel method, which TypeScript's DOM types don't have yet. |
Why use this over get-stream, concat-stream, and merge2
The libraries above were written for Node streams. This package reads web streams with getReader(), so it works in every runtime listed above, and it has no dependencies outside @portable/*.
toBytescollected 6.8 MB of web stream chunks about 7.6× faster than get-stream, and about 1.4× faster thanResponse.arrayBuffer().toBlobran about 1.7× faster thanResponse.blob(), andtoTextran at the same speed as get-stream.maxBytescancels the source as soon as the limit is crossed. get-stream'smaxBufferkept buffering (get-stream #42), and raw-body's limit becameNaNforArrayBufferchunks, which turned it off.- An error in any source of
concatormergeerrors the output and cancels the other sources. merge-stream and merge2 didn't propagate errors (merge-stream #24, merge2 #7), and concat-stream never called back when the stream errored (concat-stream #50). - Collectors don't need async iteration over
ReadableStream, which get-stream needed and Chrome lacked (get-stream #116). - The collectors reject a number chunk with a
TypeError. In concat-stream, a number chunk exposed uninitializedBuffermemory (concat-stream #57).
Compared with Readable.toWeb
@portable/streams/node has only the conversion that Node doesn't already do well. Node's Readable.fromWeb, Writable.toWeb, Writable.fromWeb, Duplex.toWeb, and Duplex.fromWeb follow backpressure and propagate errors, and in a benchmark on Node 24 they ran faster than our versions of them, so this package doesn't wrap them. On Node 24, NodeStreams.toWeb differs from Readable.toWeb in these ways:
Readable.toWebpasses the strings of aReadablewithsetEncodingthrough as strings, so aReadableStreamthat should hold bytes holds hex or base64 text.NodeStreams.toWebencodes them back to the original bytes.Readable.toWebread 128 KB from aReadablebefore anything read the stream.NodeStreams.toWebreads from theReadableonly when the stream is read, and it read 6.8 MB about 1.7× faster.Readable.toWebtakes nosignal.Readable.toWebreturns aReadableStream<any>.NodeStreams.toWebreturnsReadableStream<Uint8Array<ArrayBuffer>>, andtoWebObjectsreturnsReadableStream<unknown>for an object-modeReadable.
Operations per second from pnpm bench, higher is better, in Node 24.22.0, on Intel(R) Xeon(R) Processor @ 2.10GHz (4 cores). Each benchmark lists only the libraries that can do the same work, fastest first. The relative speed compares each library with @portable/streams, and the bars are scaled to the fastest library.
| Library | Version | Minified and gzipped | Runtime dependencies | Node built-ins | TypeScript types |
| --- | --- | ---: | ---: | --- | --- |
| @portable/streams | – | 1.7 kB | 2 | none | included |
| get-stream | 9.0.1 | 2.1 kB | 2 | Buffer | included |
| concat-stream | 2.0.0 | 15 kB | 7 | buffer, events, util | @types/concat-stream |
| bl | 7.0.12 | 2.0 kB | 0 | buffer, stream | included |
| stream-to-array | 2.3.0 | 0.9 kB | 1 | none | @types/stream-to-array |
| raw-body | 4.0.0 | 4.8 kB | 7 | Buffer, async_hooks | included |
| merge2 | 1.4.1 | 1.2 kB | 0 | stream | @types/merge2 |
| merge-stream | 2.0.0 | 0.8 kB | 0 | stream | @types/merge-stream |
| through2 | 5.0.11 | 25 kB | 10 | buffer, events | included |
Sizes cover the imports in the usage example, or the equivalent imports of each library, bundled for the browser by scripts/bundle-size.ts. Runtime dependencies count every package installed with the published package. A library that uses Node built-ins needs polyfills in a browser and the nodejs_compat flag in Cloudflare Workers.
maxBytesalways counts bytes. get-stream'smaxBuffercounts characters forgetStreamand elements forgetStreamAsArray, andtoArrayhas no limit.- raw-body's
limitaccepts a string such as'1mb'.maxBytestakes a number of bytes, such as1_000_000. - get-stream's
MaxBufferErrorstores the data read so far inbufferedData. A collector that fails has already cancelled its source and dropped what it read, so its errors don't include that data. - Byte collectors accept only
Uint8Array(includingBuffer) andstringchunks. get-stream also acceptedArrayBuffer,DataView, and other typed arrays, and concat-stream accepted numbers. - Arrays and other iterables are valid sources. get-stream rejected them.
- merge2 called with several arguments, as in
merge2(a, b, c), reads the sources one after another, which isconcat.mergereads them at the same time. - merge2's
.add()and the{ end: false }option have no equivalent, so pass every source when callingconcatormerge. mappasses onundefinedlike any other value. through2 dropped it. Usefilterto drop chunks.- A
transformgenerator starts when the output is first read. A through2 transform starts when the first chunk is written.
