stream-json

stream-json is a micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual data items (keys, strings, and numbers) can be streamed piece-wise. A SAX-inspired event-based API is included.

Components:

• Parser — streaming JSON parser producing a SAX-like token stream.
  • Optionally packs keys, strings, and numbers (controlled separately).
  • The main module creates a parser decorated with emit().
• Filters edit a token stream:
  • Pick — selects matching subobjects, ignoring the rest.
  • Replace — substitutes matching subobjects with a replacement.
  • Ignore — removes matching subobjects entirely.
  • Filter — filters subobjects while preserving the JSON shape.
• Streamers assemble tokens into JavaScript objects:
  • StreamValues — streams successive JSON values (for JSON Streaming or after pick()).
  • StreamArray — streams elements of a top-level array.
  • StreamObject — streams top-level properties of an object.
• Essentials:
  • Assembler — reconstructs JavaScript objects from tokens (EventEmitter).
  • Disassembler — converts JavaScript objects into a token stream.
  • Stringer — converts a token stream back into JSON text.
  • Emitter — re-emits tokens as named events.
• Utilities:
  • emit() — attaches token events to any stream.
  • withParser() — creates parser + component pipelines.
  • Batch — groups items into arrays.
  • Verifier — validates JSON text, pinpoints errors.
  • FlexAssembler — Assembler with custom containers (Map, Set, etc.) at specific paths.
• JSONL (JSON Lines / NDJSON) — ⚠️ deprecated; use stream-chain's JSONL directly. stream-json's JSONL is now a thin re-export of stream-chain's (which carries the full reviver / errorIndicator API) and is slated for removal in a future major — JSONL yields whole objects per line and belongs in stream-chain, not in this token-oriented library.
  • jsonl/Parser — parses JSONL into {key, value} objects. Faster than parser({jsonStreaming: true}) + streamValues() when items fit in memory.
  • jsonl/Stringer — serializes objects to JSONL text. Faster than disassembler() + stringer().
• JSONC (JSON with Comments):
  • jsonc/Parser — streaming JSONC parser with comment and whitespace tokens, plus optional comma tokens (streamCommas) for faithful round-trip editing.
  • jsonc/Stringer — converts JSONC token streams back to text; with useCommas it reproduces comma placement (incl. trailing commas) exactly.
  • jsonc/Verifier — validates JSONC text, pinpoints errors.

All components are building blocks for custom data processing pipelines. They can be combined with each other and with custom code via stream-chain.

Distributed under the New BSD license.

Introduction

import chain from 'stream-chain';

import {parser} from 'stream-json';
import {pick} from 'stream-json/filters/pick.js';
import {ignore} from 'stream-json/filters/ignore.js';
import {streamValues} from 'stream-json/streamers/stream-values.js';

import fs from 'node:fs';
import zlib from 'node:zlib';

const pipeline = chain([
  fs.createReadStream('sample.json.gz'),
  zlib.createGunzip(),
  parser(),
  pick({filter: 'data'}),
  ignore({filter: /\b_meta\b/i}),
  streamValues(),
  data => {
    const value = data.value;
    // keep data only for the accounting department
    return value && value.department === 'accounting' ? data : null;
  }
]);

let counter = 0;
pipeline.on('data', () => ++counter);
pipeline.on('end', () => console.log(`The accounting department has ${counter} employees.`));

stream-json 3.x is ESM-only and requires Node.js 22+. The default Node-flavored entries (stream-json/...) attach both .asStream (Node Duplex) and .asWebStream (Web Streams pair) on every component, since modern Node and Bun support both stream flavors natively. For browser bundles, import from the stream-json/web/... subpath instead — it pulls no Node-stream code into the dep graph. Advanced consumers can also import from stream-json/core/... to get bare factories with no adapters attached. See Migrating from 2.x to 3.x.

See the full documentation in Wiki.

Companion projects:

• stream-csv-as-json streams huge CSV files in a format compatible with stream-json: rows as arrays of string values. If a header row is used, it can stream rows as objects with named fields.

Installation

npm install --save stream-json
# or: yarn add stream-json

Use

The library is organized as small composable components based on Node.js streams and events. The source code is compact — read it to understand how things work and to build your own components.

Bug reports, simplifications, and new generic components are welcome — open a ticket or pull request.

Release History

• 3.3.0 File I/O components (parseFile, stringerToFile, verifyFile), faithful JSONC comma round-trip (streamCommas / useCommas), JSONL delegated to stream-chain.
• 3.2.0 Improvements in TS typings, faster JSON parser.
• 3.1.0 Web Streams parity sweep.
• 3.0.0 Moved to ESM using stream-chain 4.x. See Migrating from 2.x to 3.x.
• 2.1.0 new: jsonc/Verifier — validates JSONC text with exact error locations. Parser performance improvements (pre-allocated token singletons).
• 2.0.0 major rewrite: functional API based on stream-chain 3.x, bundled TypeScript definitions. New: JSONC parser/stringer, FlexAssembler. See Migrating from 1.x to 2.x.

The full history is in the wiki: Release history.