genish
v1.0.0
Published
Lean async-iterable utility library: composable operators with concurrent prefetch and ordered output.
Downloads
68
Readme
genish
A lean library of composable operators for async iterables. Build pipelines from small functions, run work concurrently when it pays off, and keep the type story in built-in types only.
Example
import { filter, map, pipe, range, toArray } from 'genish'
const pipeline = pipe(
filter((n: number) => n % 2 === 0),
map(async (n) => fetch(`/api/items/${n}`).then((r) => r.json()), 4),
map((item) => item.name),
)
const names = await toArray(pipeline(range(0, 100)))Operators
Each operator is a curried factory returning a function from a source to an
async generator. The exception is unique, which is itself a direct
source -> generator function.
map(fn, concurrency?) — transform each value; fn may be sync or async.
With concurrency > 1, runs up to that many invocations in parallel and
yields results in source order. Defaults to 1 (sequential).
filter(predicate) — keep values whose predicate returns truthy.
flatMap(fn) — fn returns an iterable, async iterable, or promise of one;
its values are flattened into the stream.
take(n) — yield at most n values, then close.
drop(n) — skip the first n values, then yield the rest.
tap(fn) — invoke fn for each value, pass the value through unchanged.
buffer(n) — eagerly prefetch up to n values from the source so the
consumer always has values ready.
batch(n) — collect values into arrays of size n; the final array may be
shorter when the source does not divide evenly.
uniqueBy(keyFn) — yield the first occurrence of each derived key.
unique — identity-keyed dedup, used directly without parens.
spread() — flatten a source of iterables one level.
Composition
const pipeline = pipe(map(fn1), filter(p), buffer(5))
const out = await toArray(pipeline(source))pipe accepts up to ten operators with full type inference between
stages; past that the result falls back to the loose unknown shape.
A sink (e.g. toArray, reduce(fn, init), drain(fn)) can be passed as
the last argument to terminate the pipeline. The result is then a function
from source to Promise<R> instead of an async generator:
const sum = pipe(
filter((n: number) => n % 2 === 0),
reduce((acc, n) => acc + n, 0),
)
const total = await sum(range(0, 10))Sinks
toArray(source) — drain into an array.
reduce(fn, initial)(source) — fold to a single value.
Sources
range(start, stop, step?) — lazy half-open numeric sequence with optional
step (defaults to 1, supports negative for reverse iteration).
Anything else that satisfies Iterable<T> or AsyncIterable<T> works as a
source: arrays, sets, maps, generators, Node readable streams.
Pitfalls
pipe(map(asyncFn), buffer(n)) is not concurrent. Async generators
serialise their next calls internally, so buffering a sequential map
just queues sequential work. Pass a concurrency argument to map when
the goal is parallel execution.
unique and uniqueBy retain every seen key for the lifetime of the
iteration, so they leak on unbounded sources. Bound the upstream first or
use a different approach.
Pipe's typed overloads stop at ten stages. Pipelines of eleven or more fall back to a loose signature with no inference. If you hit this, split the pipeline into two piped halves.
When a source rejects mid-stream, the pipeline halts and propagates the error. Speculative work that buffer or concurrent map had already started is silently abandoned rather than promoted to unhandled rejections.
Sync variants
A parallel set of operators that work entirely on synchronous Iterable<T>
sources (no Promise, no AsyncIterable) is published under
genish/sync:
import { filter, map, pipe, range, toArray } from 'genish/sync'
const pipeline = pipe(
filter((n: number) => n % 2 === 0),
map((n) => n * 10),
)
const out: number[] = toArray(pipeline(range(0, 10)))Sync operators take Iterable<T> and return Generator<T>; sinks return
the value directly instead of a Promise. Predicates and mapper functions
must be sync. The sync subset includes batch, chunkBy, drain, drop,
dropWhile, filter, find, flatMap, map, max, min, pipe,
range, reduce, spread, take, takeWhile, tap, toArray,
unique, uniqueBy. Operators that are inherently async — buffer,
paginateCursor, paginateOffset, rateLimit, and concurrent variants
of map/drain — are only available from the main genish entry.
