batch-utils

Take an iterable (such as an array) of values, and batch the values into multiple fixed-length arrays.

Both sync and async iterable generators are available.

What is it for?

Imagine needing to parse a huge CSV file and loading it into a database. The file is too big to fit into memory. You could just read the file line by line, but sending individual INSERTs to a SQL database is slow.

@mangosteen/batch-utils allows you to to batch the entries in preparation before executing a batch SQL insert. This provides significant performance speed-up.

Installation

With npm do:

$ npm install @mangosteen/batch-utils

Basic Usage

import { batchItemsAsync } from '@mangosteen/batch-utils';

(async () => {
    const values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
    const batches = batchItemsAsync(values, 3);

    for await (const batch of batches) {
        console.log(batch);
        // console.log(typeof batch); // Array<string>
    }
})();

This code will output the following arrays:

[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
[10]

Advanced Usage

import fs from 'fs';
import { lineByLine } from '@mangosteen/line-by-line';
import { batchItemsAsync } from '@mangosteen/batch-utils';

(async () => {
    const inputStream = fs.createReadStream('./shakespeare.txt');
    const lines = lineByLine(inputStream);
    const lineBatches = batchItemsAsync(lines, 100);

    for await (const batch of lineBatches) {
        // sql.batchInsert(batch);
        console.log(typeof batch); // Array<string>
    }
})();

API

batchItemsAsync(items, desiredLength)

Generates arrays of length desiredLength containing the values from items. The final array may be shorter than desiredLength if there are not enough values in items.

items: AsyncIterable<T> : Iterable of items to batch up into fixed-size arrays.

desiredLength: number : The desired length of the generated arrays.

forEachBatchAsync(items, desiredLength, onBatch)

Performs the specified action for each array generated from items iterable.

The generated arrays will be of length desiredLength, except for the last array which may be shorter.

items: AsyncIterable<T> : Iterable of items to batch up into fixed-size arrays.

desiredLength: number : The desired length of the generated arrays.

onBatch: (batch: T[], batchIndex: number) => Promise<void> : Action to perform for each array.

batchItemsSync(items, desiredLength)

Generates arrays of length desiredLength containing the values from items.

The final array may be shorter than desiredLength if there are not enough values in items.

items: Iterable<T> : Iterable of items to batch up into fixed-size arrays.

desiredLength: number : The desired length of the generated arrays.

forEachBatchSync(items, desiredLength, onBatch)

Performs the specified action for each array generated from items iterable.

The generated arrays will be of length desiredLength, except for the last array which may be shorter.

items: Iterable<T> : Iterable of items to batch up into fixed-size arrays.

desiredLength: number : The desired length of the generated arrays.

onBatch: (batch: T[], batchIndex: number) => void : Action to perform for each array.