streamline.js

A JavaScript package that helps to reads and processes a stream line-by-line in order or in batches in parallel.

Install

$ npm install --save @rpidanny/streamline.js

Usage

Functions

processLine(readStream, handler, concurrency)

| Parameters | Default | Description | | ------------- | ------- | ---------------------------------------------------------------------------------------------- | | readStream | None | Any Node.js ReadStream object | | handler | None | A processing function that is called on every line of the stream with the line as the argument | | concurrency | 1 | The number of concurrent execution of the handler |

processJson(readStream, handler, concurrency)

| Parameters | Default | Description | | ------------- | ------- | -------------------------------------------------------------------------------------------------------------- | | readStream | None | Any Node.js ReadStream object | | handler | None | A processing function that is called on every line of the stream with the parsed JSON object as the argument | | concurrency | 1 | The number of concurrent execution of the handler |

processCsv(readStream, handler, concurrency)

| Parameters | Default | Description | | ------------- | ------- | ---------------------------------------------------------------------------------------------------- | | readStream | None | Any Node.js ReadStream object | | handler | None | A processing function that is called on every line of the stream with the parsed CSV as the argument | | concurrency | 1 | The number of concurrent execution of the handler |

Example

const readStream = fs.createReadStream(`jsonlFile.json`)
await processJson(
  readStream,
  async (item: Record<string, unknown>) => {
    console.log(item)
  },
  2,
)

Classes

Base classes

• Streamline
• StreamlineCsv
• StreamlineJson

These are abstract classes that can be inherited to create complex classes to process read streams.

Forked classes

These classes are derived from Base classes which is suitable for some common use cases.

• SimpleCsvProcessor
• SimpleJsonProcessor
• SimpleCsvProcessor