jsonburst

Split large JSON files into smaller ones by size or count.

Takes a JSON array or JSON Lines (.jsonl) file and splits it into multiple output files, each under a configurable size limit, or into a target number of roughly equal chunks. Handles 1 GB+ files via streaming; never loads the whole file into memory.

Features

• 📦 Two split modes - by max size (--size 100MB) or by chunk count (--chunks 10)
• 🌊 Streaming engine - processes multi-GB files without loading them into memory
• 🔀 Dual format support - JSON arrays and JSON Lines (NDJSON)
• 🎯 Format-preserving - JSON array input becomes JSON array output; JSON Lines becomes JSON Lines
• 📏 Human-readable sizes - 500KB, 10MB, 1.5GB, no byte-counting needed
• ⚡ Zero extra dependencies - only Commander, Chalk, and Ora (all trusted, well-maintained)
• 🛡️ Graceful error handling with actionable suggestions

Quick start

# Install globally
npm install -g jsonburst

# Split a large JSON array into 100 MB chunks
jsonburst huge.json -s 100MB

# Output lands in ./jsonburst_output/:
#   huge.001.json
#   huge.002.json
#   huge.003.json
#   ...

Installation

From npm (recommended)

npm install -g jsonburst

From source

git clone https://gitlab.com/frvnkenstein/jsonburst.git
cd jsonburst
npm install
npm run build
npm link          # makes `jsonburst` available globally

Prerequisites

• Node.js ≥ 18

Usage

jsonburst <file> [options]

Arguments:
  <file>                  path to the JSON file to split

Options:
  -s, --size <size>       max size per output file (e.g. 500KB, 10MB, 1GB)
  -c, --chunks <n>        split into N roughly equal output files
  -o, --output <dir>      output directory (default: ./jsonburst_output)
  -n, --name <name>       base name for output files (default: input filename)
  --no-pretty             output compact JSON instead of pretty-printed
  -v, --verbose           enable detailed logging
  -h, --help              display help
  -V, --version           display version

Examples

# Split by size, 100 MB per chunk
jsonburst large.json -s 100MB

# Split into exactly 10 chunks
jsonburst large.json -c 10

# JSON Lines input, custom output dir
jsonburst data.jsonl -s 50MB -o ./chunks

# Compact (single-line) output for JSON arrays
jsonburst huge.json -s 1GB --no-pretty

# Custom base name for output files
jsonburst huge.json -s 500MB -n mydata
# → mydata.001.json, mydata.002.json, ...

# Verbose logging
jsonburst large.json -c 5 --verbose

Output format

JSON array input (default: pretty-printed)

Input: data.json

[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"},{"id":3,"name":"Carol"}]

Output (with --size small enough to split):

data.001.json:

[
  {"id":1,"name":"Alice"},
  {"id":2,"name":"Bob"}
]

data.002.json:

[
  {"id":3,"name":"Carol"}
]

JSON array input (compact mode: --no-pretty)

data.001.json:

[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]

JSON Lines input

Input: data.jsonl

{"id":1,"name":"Alice"}
{"id":2,"name":"Bob"}
{"id":3,"name":"Carol"}

Output: Each chunk is valid JSON Lines with the same format.

How it works

Input File (.json / .jsonl)
        │
        ▼
  Format Detector ─── array or jsonlines?  file size?
        │
        ▼
  Config Resolver ─── --size → maxBytes per chunk
        │              --chunks → compute maxBytes = fileSize / N
        ▼
  Stream Splitter ─── depth-tracking state machine (arrays)
        │              or readline (JSON Lines)
        │              rotates output files at size threshold
        ▼
  Output ─── <name>.001.json, <name>.002.json, ...

1. Detect: Reads the first 4 KB to determine whether the input is a JSON array (starts with [) or JSON Lines (starts with {).
2. Resolve: Computes the max bytes per chunk from the --size or --chunks option. Defaults to 10 MB.
3. Split (arrays): A streaming state machine tracks brace/bracket depth and string state character-by-character. Elements are accumulated from the outermost array and flushed to disk whenever the accumulated byte size exceeds the limit. Handles any nesting depth and escaped strings correctly.
4. Split (JSON Lines): Reads line-by-line via readline, groups lines until the size threshold is met, then flushes.

Memory usage is proportional to one chunk size, not the input file size. A 1 GB file split into 100 MB chunks uses ~100 MB of memory at peak.

--chunks vs --size

The --chunks flag computes maxBytes = ceil(fileSize / N) and then splits exactly like --size. Because element boundaries are always respected, the actual number of output files may differ slightly from the requested N.

Limitations

• JSON arrays only at the top level. The input must be a JSON array or JSON Lines. A single JSON object with top-level keys is treated as JSON Lines (one line, one object).
• Elements cannot be split. If a single array element is larger than the chunk size, it will be placed in its own chunk (which will exceed the limit). A warning is printed when this happens.
• Not a validator. jsonburst assumes the input is valid JSON. Malformed input will produce garbage output or a cryptic error.

Development

git clone https://gitlab.com/frvnkenstein/jsonburst.git
cd jsonburst
npm install

# Run in development mode (no build needed)
npm run dev -- ../large.json -s 100MB --verbose

# Run tests
npm test

# Watch mode
npm run test:watch

# Build
npm run build

License

MIT, see LICENSE.