@zakkster/lite-bake-stream

Streaming byte-level JSON to LBK1 binary containers. Zero-GC, tree-shakeable, gigabyte-scale.

Status: v1.7.1. LBK1 format frozen at format_version: 1. Qualified on an 8 GB soak with zero GC and byte-exact preservation across 590 million cells.

The gigabyte-scale JSON front door the ecosystem was missing

You have a gigabyte of JSON and a program that has to read it. JSON.parse cannot get you there: V8 caps a single string near 512 MB, and well under that ceiling the intermediate object graph shreds your heap. This package is the front door -- a chunk-safe UTF-8 JSON SAX scanner that reads bytes off a stream and emits a flat, seekable LBK1 binary container, without ever materializing the object graph.

npm install @zakkster/lite-bake-stream

import { serialize, deserialize } from '@zakkster/lite-bake-stream';

const container = serialize(ndjsonBytes);
const reader = deserialize(container);
reader.get(0, 'id');                     // 1
reader.findShards('id', { min: 500 });   // shard pruning via zone maps

Table of contents

• Why this exists
• What you get
• The core surface
• API reference
• Composability
• Zero-GC design notes
• Design decisions worth knowing
• Testing
• What this is not
• Ecosystem
• License

Why this exists

The enemy is JSON.parse, not any one downstream format. Two hard walls stop it on large input:

• The string ceiling. V8 refuses to hold a single string beyond roughly 512 MB. A gigabyte file cannot even be read into memory as text, let alone parsed.
• Object-graph heap death. Even under the ceiling, JSON.parse allocates a fully-realized object graph -- every object, array, string, and boxed number. For a gigabyte of records that graph is many gigabytes of live heap, and the garbage collector spends the run trying (and failing) to keep up.

lite-bake-stream never builds the graph. It scans bytes as they arrive, packs each record into typed-lane shard buffers (or preserves it as an opaque blob), and writes an LBK1 container whose Reader gives you random access without re-parsing. The input-size ceiling becomes your disk, not your heap.

What you get

Two ingest modes share one top-level API. deserialize() auto-dispatches on the container's flag bit, so a consumer never has to know which mode wrote the bytes.

• Schema mode (default): F64/U32 lane packing, per-shard zone-maps for query pruning, byte-exact preservation for flat records. Random access to millions of rows per second on a zero-GC hot path.
• Preserve mode (serialize(input, { preserve: true })): opaque byte blobs. The module does not crack open records, it moves them intact. Deeply nested API JSON, arrays, mixed types -- any valid JSON round-trips byte-for-byte identical. The reader exposes getBytes(i) (zero-alloc view), getString(i), and getJSON(i).

Compiling gigabytes of API responses with two-to-three levels of nesting? Preserve mode. Flat records that want columnar-style range queries? Schema mode.

The core surface

finalize() returns the whole container as one ArrayBuffer -- peak memory is O(container). For bounded-memory output, writer.finalizeToSink(sink, { layout }) emits to a caller sink and returns { totalRows, shardCount, schema | mode, bytesWritten, layout } (no buffer). A sink is any object with a synchronous write(bytes); layout: 'stream' also needs writeAt(bytes, position) for one header backpatch. The contract is satisfiable by fs.write(fd, buf, 0, len, pos) and the File System Access API's createWritable().write({ type: 'write', position }); the test suite uses an in-memory sink of the same shape. No sink class is exported -- the contract is public, the convenience implementation is internal.

Two ways to drive a stream, with different peaks:

• Two-step, bounded RAM -- call writer.beginStream(sink, { layout: 'stream' }) BEFORE feeding, feed the tokenizer, then writer.finalizeToSink(sink, { layout: 'stream' }). Each shard is written to the sink as it finalizes and its bytes are dropped, so peak memory is 2*targetShardBytes + shardCount*(40 + 16*T) + schemaBlockBytes -- O(targetShardBytes + directory), never O(container). This is the mode the 500 MB gate proves.
• One-shot, buffered -- call writer.finalizeToSink(sink, opts) alone (no beginStream). Correct and simplest, but the shards are buffered first, so peak is containerBytes + sum(shard bytes) -- O(container), the same as finalize().

layout: 'prefix' is the classic layout, byte-identical to finalize(). layout: 'stream' places the schema/directory/zone-map trailer and footer after the payloads with one header backpatch; a default-emitted stream container is a legal v1 container (format_version stays 1) that every shipped reader, checkContainer, and mergeContainers accept. A malformed sink (non-object, missing write, missing writeAt for stream, or an async/thenable return) throws W_BAD_SINK; a sink that throws mid-emission fails the writer closed and rethrows the source error verbatim (a retry then hits W_FINALIZED).

Optional CRC-32C (Castagnoli, table-driven, zero deps). Opt in with { crc: true } on the writer (or serialize writer opts); coverage is [0, footer_off), folded per emitted chunk. Readers expose verifyCrc() -> 'ok' | 'absent' (a mismatch throws R_BAD_CRC) on Reader/PreserveReader (sync) and RangeReader (async). The open option { verifyCrc: true } (also deserialize(bytes, opts)) fails closed on both mismatch (R_BAD_CRC) and absence (R_CRC_ABSENT) -- null is not zero. 0xFFFFFFFF means absent and stays legal. mergeContainers recomputes the CRC iff every input carried one, else emits absent. See decisions/0009-streaming-emission.md.

Future lane kinds and payload modes (an I64 exact-integer lane, a columnar payload mode, a container-level string table) land via the format's forward-compat seams -- min_reader_version on ShardEntry, reserved FieldDescriptor flags, the metadata_off block wrapper -- without a format_version bump.

API reference

Every subpath is a standalone import; the browser reader never pulls the writer. Every subpath also exports a VERSION const.

Root (@zakkster/lite-bake-stream)

• serialize(input, opts?) -> Uint8Array -- ingest bytes into an LBK1 container. opts.preserve selects preserve mode; opts.writer forwards writer options.
• deserialize(bytes, opts?) -> Reader | PreserveReader -- auto-dispatch on the container flag bit. Plus class re-exports and StringTableError.

Schema mode

• Tokenizer (/tokenizer) -- chunk-safe UTF-8 JSON SAX scanner. feed(chunk), end(), sink callbacks fire synchronously.
• Writer (/writer) -- LBK1 shard emitter; plugs into the Tokenizer. F64/U32 lanes, per-shard string tables and zone maps, beginStream/finalizeToSink.
• Reader (/reader) -- container parser, sync, zero-alloc get(row, field) plus zone-map query APIs shardBounds(shardIdx, field) and findShards(field, range).
• RangeReader (/range-reader) -- HTTP Range lazy shard loading; synchronous query pruning after open. Optional { signal } cancels in-flight range I/O (R_ABORTED); once it fires the reader is dead for new I/O.
• MultiReader (/multi-reader) -- logical union over N Readers sharing a schema.
• splitNDJSON, compilePart, compileInParts, mergeContainers (/split) -- worker-agnostic split, compile, and merge.

Preserve mode

• PreserveTokenizer (/preserve-tokenizer) -- NDJSON record-boundary scanner with JSON-aware depth tracking, chunk-safe.
• PreserveWriter (/preserve-writer) -- opaque byte-blob sink, pre-allocated shard buffer, zero-GC record path.
• PreserveReader (/preserve-reader) -- getBytes(i) (zero-alloc view), getString(i), getJSON(i).

Shared

• StringTable (/string-table) -- byte-level UTF-8 interning primitive.
• ingestStream(readableStream, opts?), ingestFile(file, opts?) (/file-ingest) -- browser helpers piping a ReadableStream<Uint8Array> (e.g. File.stream()) through the Tokenizer + Writer, returning a Reader. The per-chunk onProgress callback's state object is reused across calls (mutated in place, zero per-chunk allocation) -- copy it if retained past the callback.

Zero-copy view contract. A byte view returned by getBytes(i) (and any shard raw-access getter) is a plain subarray over the container's ArrayBuffer: it pins that buffer for as long as the view is reachable. Copy the bytes out (getBytes(i).slice()) if you need the container itself to be collectable.

Constants

Lane kinds (SPEC 4.2):

| Kind | Name | Bytes | | ---: | :---- | ----: | | 1 | F64 | 8 | | 2 | F32 | 4 | | 3 | U32 | 4 | | 4 | U8 | 1 |

Ceilings:

| Limit | Value | | :----------------------------------- | -----------: | | String-table blob bytes (u32) | 4294967295 | | String-table entry count | 4294967294 | | u64 offset safe Number() cast | 2^53 - 1 |

Error-code families (thrown with a stable code; the full inventory is pinned by the torture gate):

| Prefix | Class | | :----- | :----------------------------------------------------------------- | | E_ | TokenizerError (e.g. E_NUMBER_OVERFLOW) | | W_ | WriterError (W_MIXED_LANE_TYPES, W_LANE_MISMATCH, W_BAD_SINK, W_FINALIZED) | | R_ | ReaderError / RangeReaderError (R_ROW_OUT_OF_RANGE, R_OFFSET_TOO_LARGE, R_BAD_CRC, R_CRC_ABSENT, R_ABORTED, ...) | | M_ | MultiReaderError (M_ROW_OUT_OF_RANGE) | | S_ | SplitError | | ST_ | StringTableError (ST_BLOB_OVERFLOW) | | P*_ | PreserveTokenizerError / PreserveWriterError / PreserveReaderError |

Composability

The /split subpath turns one logical dataset into worker-parallel or checkpointed parts, and the readers merge them back into a single view:

import { splitNDJSON, compileInParts, mergeContainers } from '@zakkster/lite-bake-stream/split';
import { RangeReader } from '@zakkster/lite-bake-stream/range-reader';

// 1. split NDJSON into byte-aligned parts (record boundaries preserved)
const parts = splitNDJSON(ndjsonBytes, { partBytes: 64 * 1024 * 1024 });

// 2. compile each part (fan out to workers if you like), then merge
const containers = compileInParts(parts);
const merged = mergeContainers(containers);

// 3. open the merged container and prune shards before fetching them
//    (pass { signal } to cancel in-flight range I/O when the consumer detaches)
const reader = await RangeReader.open(adapterOver(merged), { signal: controller.signal });
const shards = reader.findShards('id', { min: 500, max: 999 });
for (const s of shards) reader.get(s.firstRow, 'id');

Zero-GC design notes

The hot path allocates nothing steady-state. Buffers are pre-allocated and reused across the churn:

| Operation | Steady-state allocation | | :-------------------- | :---------------------- | | feed(chunk) | 0 (views into the tokenizer's internal buffer) | | per row | 0 (typed-lane staging is pre-allocated) | | per shard roll | 0 (working buffers reused across rolls) | | Reader.get(row, f) | 0 for F64; U32 string fields resolve through a cached decoder |

Qualified at 8 GB. Overnight soak, M1 MacBook Pro, both release gates armed:

| | | | :-- | --: | | Source | 8.00 GB NDJSON | | Container | 4.89 GB (61.1% of source) | | Rows | 98,367,702 | | Ingest throughput | 110.0 MB/s | | Major GC | 0 | | Minor GC | 0 | | Total heap allocation | 499.2 KB | | Cells verified | 590,206,212 | | Preservation mismatches | 0 |

499 KB of heap allocation to compile 8 GB of JSON -- total, for the whole run, not per shard, not per second. Zero garbage collections of any kind. Every one of 590 million declared cells round-tripped byte-exact.

Bounded streaming, measured. The full-tier gate drives the two-step bounded path, streaming 500 MB in 8 MiB chunks into a counting sink, and asserts peakRss - baselineRss <= 4*targetShardBytes + 64*shardCount + 64 MiB -- measured 32.1 MiB against the 96.0 MiB bound at 41 shards (the buffered prefix path is ~704 MiB at the same 500 MB).

Boot latency: the crossover. A persisted query cache boots one of two ways: decode + JSON.parse the whole cache, or open a preserve container and getJSON(i) lazily per entry. Both start from bytes already in hand. Measured 2026-09-02, node v26.3.1, M1 MacBook Pro, deterministic corpus, interleaved rotating order; the reproduce-within band is 15% over the boot-to-all cells (a second run landed inside it):

| Cache size | Whole-cache JSON.parse (all) | Preserve-bake lazy (first entry) | Preserve-bake (all) | | :-- | --: | --: | --: | | 10 KB | 0.035 ms | 0.001 ms | 0.046 ms | | 100 KB | 0.346 ms | 0.004 ms | 0.459 ms | | 1 MB | 3.96 ms | 0.076 ms | 5.29 ms | | 10 MB | 45.6 ms | 0.60 ms | 54.2 ms | | 20 MB * | 105 ms | 1.65 ms | 109 ms | | 40 MB * | 233 ms | 3.10 ms | 220 ms |

Medians; * = bench-only bracket rows above the charter's 10 MB ladder, added only to locate the boot-to-all crossover. Plain JSON has no lazy path -- the whole cache must parse before the first entry is readable, so its first-entry column would equal its boot-to-all column and is omitted. Two regimes. Boot-to-first-entry: the preserve open is O(1), so lazy per-entry boot beats the whole-cache parse at or below the 10 KB floor and at every size above it (0.001 ms vs 0.035 ms at 10 KB; 0.60 ms vs 45.6 ms at 10 MB). Boot-to-all (full eager hydration): one big JSON.parse wins until the crossover between 20 MB and 40 MB, where per-entry bake overtakes it -- the interval between adjacent measured sizes, never interpolated. Below the crossover, for full-cache hydration, use plain JSON; reach for the bake container when you boot lazily (first-entry is O(1) at any size) or when the cache outgrows the crossover. bench/bench-crossover.js reproduces this table.

Tokenizer throughput is ~55% of JSON.parse (237 MB/s vs 416 MB/s on a 50 MB fixture). That is the trade: you give up some raw speed and you get an input-size ceiling bounded by your disk instead of your heap.

Design decisions worth knowing

Each call is an on-disk record under decisions/:

• 0001 -- reserve string-table index 0 as the empty string.
• 0002 -- depth-0 non-object records are refused, not dropped.
• 0003 -- duplicate schema field names are refused at freeze.
• 0004 -- a container is verified at the door; a lying pointer is corruption.
• 0005 -- absent names are refused, untracked names fall back.
• 0006 -- the UTF-8 door is documented-permissive, not a tokenizer gate.
• 0007 -- the sample window is byte-true, not record-count.
• 0008 -- JSON null is lane-neutral.
• 0009 -- streaming emission, sinks, and optional CRC-32C.
• 0010 -- why a second (preserve) mode exists at all.
• 0011 -- per-shard re-intern on drain and the byte-true sample window.
• 0012 -- the Clinger fast path and the two-tier F64 guarantee.

Testing

The gates run at scales chosen to fit different hardware and time budgets:

| Command | Scale | Use | | :-- | :-- | :-- | | npm test | 561 tests across 38 files | Dev loop, every save | | npm run torture | 44 fast scenarios (~1 MB each) | Before every commit | | npm run torture:full | 47 scenarios incl. the 500 MB soak | Before every publish | | npm run soak | 100 MB with preservation gate | Sanity check | | npm run soak:500 | 500 MB | Pre-publish scale check | | npm run soak:1gb | 1 GB | Real-hardware qualification | | npm run soak:overnight | 8 GB | Overnight caffeinate run |

The test tree includes a 66-fixture RFC 8259 conformance corpus, a property-based robustness fuzz, and two structural guards: an ASCII-law scanner (test/AsciiLaw.test.js) and an API-surface drift guard (test/ApiSurface.test.js), each also wired into the torture tier with an armed control that proves it can fail. Two release gates decide a publish: zero major GC AND every declared cell round-trips exactly. If either fails, no publish.

What this is not

No JSON5, JSONC, comments, or trailing commas. No streaming field updates, no compression, no native (non-JS) producers. LBK1 is this package's own container format; non-JSON producers are welcome as long as they emit a well-formed container.

Ecosystem

The demo/ directory is a single-file HTML + module-JS app: ingest a JSON file, compile it to LBK1, and browse the result, with a zone-maps panel that visualizes the query-pruning fetch-set reduction. Run it with npm run demo.

License

MIT (c) 2026 Zahary Shinikchiev