llm-stream-mux

Race, fallback, merge, and tee over any stream — generic over T, zero runtime dependencies, Web Streams throughout.

A standalone TypeScript stream-orchestration layer for LLM pipelines: coordinate multiple AsyncIterable<T> or ReadableStream<T> sources with race, fallback, ensemble/merge, and bounded tee — raw bytes or parsed events, with no baked-in event model.

Orchestrate streams — not another hand-rolled Promise.race on fetch.

Status: 1.1.0 — stable; §9 runtime exports and §6.3 MuxErrorCode frozen as of 1.0.0 under semver. 972 tests via pnpm verify. Spec: docs/proposal.MD · stability: docs/STABILITY.md · security: SECURITY.md

Contents

• Positioning
• Why not hand-roll?
• Edge-case showcase
• Why use this
• Install
• Quickstart
• Architecture
• Strategies at a glance
• Documentation
• Examples
• Non-goals
• Development

Positioning

llm-stream-mux is the N↔1 orchestration layer only: race, fallback, merge, and tee over streams you already opened. You keep your HTTP client, auth, parsing (llm-stream-assemble), security filters, and UI.

| Library | Shape | Responsibility | | ---------------------------------------------------------------------- | ----- | ----------------------------------------------------- | | llm-stream-assemble | 1→1 | Parse one provider stream → typed events (format) | | llm-stream-guard | 1→1 | Redact / tool policy (safety) | | llm-stream-mux | N↔1 | Race, fallback, merge, tee (orchestration) |

No npm coupling between the three — compose in userland (cookbook).

Why not hand-roll?

Multi-stream coordination fails in production for predictable reasons:

• Promise.race on fetch does not cancel losing HTTP bodies or buffer pre-usable frames.
• Native ReadableStream.tee() grows memory without bound when one branch is slow.
• Failover after partial output splices two responses unless you define a commit point.
• Naïve merge loops drop settled reads when using Promise.race on pending iterators.

Concrete contracts and test IDs: docs/edge-cases.md.

Edge-case showcase

Three orchestration footguns mux addresses by design:

• Unbounded tee() → mux tee with block / bounded / drop.
• Junk-first race winner → isUsable + ordered pre-usable buffer flush.
• Merge read loss → one pending read per source; re-arm after consume.

Walkthrough: docs/edge-cases.md.

Why use this

• Zero runtime dependencies.
• Generic over T — Uint8Array byte mode or any parsed event type.
• Four strategies with semantic hooks (isError, isUsable, isFinal, mapEach).
• Observable — onSourceEvent, MuxResult, stable MuxError codes.
• Runtime-agnostic — Node 22+, Bun, Deno, Cloudflare Workers (Web Streams only).

Install

npm install llm-stream-mux
# or: pnpm add llm-stream-mux

Pin: npm install [email protected]. Public API frozen at 1.0.0 (STABILITY).

Requirements: Node.js 22+ · see compatibility matrix.

Quickstart

Byte mode — race two provider bodies

import { race } from "llm-stream-mux";

const winner = race<Uint8Array>([resA.body!, resB.body!], { signal, timeoutMs: 5000 });

for await (const chunk of winner) {
	// first usable stream wins; losers cancelled
}

Event mode — merge tagged outputs

import { merge } from "llm-stream-mux";

for await (const tagged of merge<MyEvent>(
	{ gpt: streamA, claude: streamB },
	{
		isError: (e) => e.type === "error",
		onSourceEvent: (s) => log(s),
	},
)) {
	if (tagged.kind === "value") render(tagged.source, tagged.value);
}

Strategy picker: docs/usage-guides.md · quick-decision diagram.

Architecture

• Diagram index: docs/img/README.md (19 diagrams)
• Frozen API surface: api-frozen-surface.svg
• Public API types (P0): public-api-types.svg
• Edge matrix §H: edge-matrix-h.svg
• Interop paths: interop-matrix.svg
• Ecosystem stack: ecosystem.svg
• Byte vs event placement: byte-event-modes.svg
• Merge Tagged<T> flow: merge-tagged.svg
• Strategy decision tree: quick-decision.svg

Strategies at a glance

| Function | Shape | Use when | | -------------------- | ----- | ----------------------------------------------- | | race(sources) | N→1 | Fastest usable stream wins; cancel losers | | fallback(sources) | N→1 | Primary → backup; lazy thunks for true failover | | merge / ensemble | N→1 | Parallel models; Tagged<T> per source | | tee(source, n) | 1→N | Fan-out with bounded backpressure |

Helpers: collect, toReadable, toAsyncIterable.

Full API: proposal §9.

Documentation

• Proposal & roadmap — normative spec + P0–P10
• API stability (frozen at 1.0.0)
• Security policy
• Release templates
• Usage guides — per-strategy recipes
• Integration cookbook — pair with assemble/guard (docs only)
• Edge-case contracts
• Runtime compatibility
• Comparison matrix
• Testing strategy
• FAQ
• Performance notes

Examples

Runnable samples: examples/node-fetch/ — race.ts, fallback.ts, merge.ts, tee.ts (fake streams; pnpm typecheck:examples after build).

For parsing examples see llm-stream-assemble/examples.

Non-goals

• No HTTP client, auth, or retry of the same request.
• No provider parsing (assemble).
• No content redaction / tool policy (guard).
• No UI, agent loop, or tool execution.

Development

./scripts/setup-githooks.sh   # once per clone — strip AI co-author trailers
pnpm install
pnpm verify

CI runs pnpm verify on Node 22 and 24; smoke-runtimes.yml smoke-tests Bun + Deno.

| Command | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | pnpm verify | portability + deps + lint + typecheck + build + test + smoke + docs + diagrams + format | | pnpm build | tsup → ESM + CJS + declarations in dist/ | | pnpm test | Vitest — LSM-REL-*, LSM-TYP-*, LSM-CORE-*, LSM-TEE-*, LSM-RACE-*, LSM-FB-*, LSM-MERGE-*, LSM-X-*, LSM-EDGE-*, LSM-EDGE-P0-*, LSM-SRC-* | | pnpm verify:portability | forbid Node-only / ReadableStream.from patterns in src/ | | pnpm smoke:package | ESM/CJS import from npm pack tarball | | pnpm smoke:runtimes | Node (+ optional Bun/Deno) tarball smoke — --skip-optional locally, --ci in Actions | | pnpm smoke:consumer | ESM + CJS downstream consumer smoke from tarball | | pnpm smoke:published | post-pack publish simulation (--node22 / --node24 / --all-runtimes) | | pnpm verify:pre1 | maintainer gate: verify + release:prep + smoke:runtimes + smoke:consumer + smoke:published | | pnpm verify:deps | fail if runtime dependencies added | | pnpm diagrams:build | render docs/img/*.mmd → .svg | | pnpm diagrams:check | SVGs present and newer than .mmd | | pnpm typecheck:examples | Typecheck examples/ against built dist/ | | pnpm release:prep | pre-tag checks (version, CHANGELOG, npm pack, examples) |

Author

Ladislav Kostolny — [email protected] · GitHub @01laky

License

MIT — see LICENSE. Copyright (c) 2026 Ladislav Kostolny.