stream-csv-as-json

stream-csv-as-json is a micro-library of Node.js stream components for creating custom CSV processing pipelines with a minimal memory footprint. It can parse CSV files far exceeding available memory, streaming individual primitives using a SAX-inspired API.

stream-csv-as-json is a companion project for stream-json and stream-chain. It uses the same token protocol ({name, value} tokens) and works seamlessly with stream-json filters, streamers, and general infrastructure. This means you can combine CSV parsing with stream-json utilities like streamValues, Filter, Pick, and Ignore for powerful data processing pipelines.

Components

• parser — streaming CSV parser producing a SAX-like token stream.
  • Optionally packs values into single tokens or streams them piece-wise.
  • The main module provides a convenience factory with event emission.
• asObjects — uses the first row as field names, converts subsequent rows to object tokens.
• stringer — converts a CSV token stream back to CSV text.

All components are building blocks for flexible pipelines. They can be combined with custom functions, stream-chain, and stream-json utilities.

Installation

npm install stream-csv-as-json

Quick start

Examples use ESM (import). CommonJS (require) is also supported — see Modules below.

import fs from 'node:fs';
import zlib from 'node:zlib';
import chain from 'stream-chain';
import {parser} from 'stream-csv-as-json';
import asObjects from 'stream-csv-as-json/as-objects.js';

const pipeline = chain([
  fs.createReadStream('sample.csv.gz'),
  zlib.createGunzip(),
  parser(),
  asObjects(),
  data => {
    if (data.name === 'keyValue' && data.value === 'accounting') return data;
    if (data.name !== 'keyValue') return data;
    return null;
  }
]);

let counter = 0;
pipeline.on('data', data => {
  if (data.name === 'endObject') ++counter;
});
pipeline.on('end', () => console.log(`Found ${counter} matching rows.`));

Using .withParser() for a combined pipeline

import fs from 'node:fs';
import chain from 'stream-chain';
import asObjects from 'stream-csv-as-json/as-objects.js';

const pipeline = chain([fs.createReadStream('data.csv'), asObjects.withParser()]);

pipeline.on('data', token => console.log(token));

Using .asStream() for direct piping

import fs from 'node:fs';
import parser from 'stream-csv-as-json/parser.js';

fs.createReadStream('data.csv')
  .pipe(parser.asStream())
  .on('data', token => console.log(token));

Modules

This package supports both ESM (import) and CommonJS (require).

ESM (recommended):

import {parser} from 'stream-csv-as-json';
import asObjects from 'stream-csv-as-json/as-objects.js';
import stringer from 'stream-csv-as-json/stringer.js';

CommonJS:

const {parser} = require('stream-csv-as-json');
const asObjects = require('stream-csv-as-json/as-objects.js');
const stringer = require('stream-csv-as-json/stringer.js');

See the full documentation in the Wiki.

API at a glance

| Module | Factory | Stream wrapper | | ---------------------------------- | -------------------- | ------------------------------------------- | | stream-csv-as-json | make(options) | Returns a Duplex stream with event emission | | stream-csv-as-json/parser.js | parser(options) | parser.asStream(options) | | stream-csv-as-json/stringer.js | stringer(options) | stringer.asStream(options) | | stream-csv-as-json/as-objects.js | asObjects(options) | asObjects.asStream(options) |

parser options

| Option | Default | Description | | -------------------------------- | ------- | --------------------------------------------------- | | packStrings / packValues | true | Emit stringValue tokens with the complete value | | streamStrings / streamValues | true | Emit startString/stringChunk/endString tokens | | separator | ',' | Field separator character |

stringer options

| Option | Default | Description | | ------------------------------- | ------- | ---------------------------------------------------------- | | useStringValues / useValues | false | Use packed stringValue tokens instead of streamed chunks | | separator | ',' | Field separator character |

asObjects options

| Option | Default | Description | | ------------------------------- | --------- | ----------------------------------------------------- | | packKeys / packValues | true | Emit keyValue tokens | | streamKeys / streamValues | true | Emit startKey/stringChunk/endKey tokens | | useStringValues / useValues | false | Use packed stringValue tokens for header collection | | fieldPrefix | 'field' | Prefix for unnamed/extra fields |

TypeScript

TypeScript declarations (.d.ts) are included and provide full type information for all modules.

License

BSD-3-Clause

Release history

• 2.0.1 Added direct dependency on stream-chain. Documentation updates.
• 2.0.0 Major rewrite: functional API (stream-chain 3.x), source in src/, TypeScript declarations, tape-six tests. See Migration guide.
• 1.0.5 technical release: updated deps.
• 1.0.4 technical release: updated deps.
• 1.0.3 technical release: updated deps.
• 1.0.2 technical release: updated deps, updated license's year.
• 1.0.1 minor readme tweaks, added TypeScript typings and the badge.
• 1.0.0 the first 1.0 release.