Split JSON

A small Node.js module that splits large JSON files (with an array of objects) into smaller part files.

It does the following:

• Opens a read stream to the large JSON file
• Reads each top-level item from the array, one at a time
• Writes each smaller list of items into a part file with sequencial numbering

For more information, please read the Usage section.

Warning

I'll be implementing changes as quickly as possible, if you have a bug report or feature request, please read the Feedback section.

I'll ensure that patch (0.0.x) updates won't break your code, but major (x.0.0) and minor (0.x.0) ones might.

Always check this README file before upgrading to the latest version.

Installation

To install the package, run this inside your project's folder.

$ npm i @codetailor/split-json

Usage

The module exports a Promise that must be handled on your side.

// Javascript imports
const split = require('split-json');
const path = require('path');

// Typescript imports
import split from 'split-json';
import * as path from 'path';

// Sample data, replace with your own
const inputFilePath = path.join('data', 'input', 'large-json-file.json'); // Path to the large JSON file
const outputFolder = path.join('data', 'output'); // Path to the folder for the part files
const outputPrefix = 'part-'; // Prefix for the part filenames
const maxItemsPerFile = 1; // (optional) Maximum number of items in each part file (default: 10000)
const minPartNumberLength = 3; // (optional) Minimum length of the part file number (ex: 4 -> 0001) (default: 4)

// Then/catch version
split(inputFilePath, outputFolder, outputPrefix, maxItemsPerFile, minPartNumberLength)
  .then(() => {
    // Insert your code here, part files have been created
  })
  .catch(console.error);

// Async/await version
(async () => {
  try {
    await split(inputFilePath, outputFolder, outputPrefix, maxItemsPerFile, minPartNumberLength);

    // Insert your code here, part files have been created
  }

  catch (error) {
    console.error(error);
  }
})();

With the following example input file in data/input/large-json-file.json:

[
  { "id": 1 },
  { "id": 2 },
  { "id": 3 },
  { "id": 4 }
]

The module should generate the following part files in the data/output folder:

part-001.json

[{"id":1}]

part-002.json

[{"id":2}]

part-003.json

[{"id":3}]

part-004.json

[{"id":4}]

Important notes:

• The input file must contain an array of objects in valid JSON format
• Each generated part file will contain an array of maxItemsPerFile items in minified JSON format

Errors

In case of error, the function will reject the promise with the respective error.

Custom error messages generated by the module are:

• Input file not found when the module can't open or find the input JSON file

Feedback

All bug reports and feature requests are welcome, and should be submitted through one of the following channels:

• Create a Github issue here
• Tweet me at @ricardonunesdev

All requests will be created as Github issues, if you don't use that channel.

License

MIT © 2023 Ricardo Nunes