bote

A fast, modern and low-memory approach to processing a big JSON:

npm install @botejs/core

import { fileURLToPath } from 'node:url';
import { open, fromFile } from '@botejs/core';

// 181 MB GeoJSON:
// { type: "...", features: [{ properties: { STREET: "..." }}] }
const filePath = fileURLToPath(new URL('../citylots.json', import.meta.url));

await using cursor = await open(fromFile(filePath));

const byStreet = await cursor
  .iter('features', {
    select: ['properties', 'STREET'],
  })
  .reduce((tally, street) => {
    if (typeof street === 'string') {
      tally.set(street, (tally.get(street) ?? 0) + 1);
    }
    return tally;
  }, new Map());

console.log([...byStreet].sort((a, b) => b[1] - a[1]).slice(0, 10));

Given a seekable or forward source and a path, it retrieves values out of a JSON, without loading the whole thing in-memory.

Here's a run of snippet above (Apple M1 Pro 2021, default settings, RUNS=100):

| method | mean time (seconds) | mean peak footprint (MB) | | --------------------------- | ------------------- | ------------------------ | | bote: v0.8 | 0.517 ± 0.018 s | 40.3 ± 2.5 | | JSON.parse: v22 | 0.816 ± 0.031 s | 648.9 ± 2.4 | | JSONStream: v1.3 | 4.452 ± 0.052 s | 57.9 ± 3.9 | | @streamparser/json: v0.0.22 | 5.103 ± 0.084 s | 47.9 ± 2.3 | | oboe.js: v2.1 | 8.566 ± 0.295 s | 100.0 ± 4.6 | | stream-json: v3.4.0 | 13.346 ± 0.569 s | 207.6 ± 8.4 |

For comparison notes, go here.

Features

• Modern AsyncIterator API with helpers that emulate the tc39 ones
• Validate with Standard Schema, avoiding those pesky unknowns
• Supports multiple sources of data (e.g. file, network, stream) or write a custom one. (see sources.js for the built-in ones)
• For forward-only sources, there's support for replaying/buffering, allowing navigation to previous values

Supported

• Node.js >= 22.18.0
• ESM-only
• Platforms
  • macOS (Apple Silicon aarch64 and Intel x86_64)
  • Linux x64 (x86_64, glibc)
  • Windows x64 (x86_64, MSVC)
  • More if requested :)

Documentation

Coming soon. Check the ./examples folder for usages. I've also heavily JSDoc'ed the hell out of the API so have fun playing around with it for now.

Status

Pre-1.0. Still in development and APIs may change based on feedback, bugs and holy divinations from the coding gods.

After a lot of chaos, I'm finally-kinda-sorta happy with the public API. Major breaking changes seems to be slowing down so feedback from the community and dogfooding on my end is what's next.

License

MIT.