turbocsv 🚀

An ultra-fast, production-ready CSV processing engine written in Rust and optimized for Node.js and Bun using NAPI-RS.

Designed for high-performance systems and big-data streaming, turbocsv uses multi-threaded Rayon parallelism, quote-aware binary chunk splitting, in-engine filtering, and zero-copy TypedArray output to deliver unmatched CSV processing speeds.

Key Features

• ⚡ Blazing Fast: Built in Rust leveraging SIMD instruction sets, low-level tokenizers, and multi-core data-parallelism via rayon.
• 🧵 Non-Blocking Async: Performs parsing in background OS threads (libuv threadpool), keeping the Node.js / Bun main event loop fully responsive.
• 🧠 Intelligent Schema Inference: Scans initial rows dynamically to infer types (Integer, Float, Boolean, DateTime, String, Null) and parses directly into native Rust primitives.
• 💾 Zero-Copy Columnar Output: Options to output directly in columnar structures: numeric columns map directly to JavaScript TypedArrays (BigInt64Array, Float64Array, Uint8Array), and strings are stored in a contiguous shared buffer with offsets.
• 🧹 In-Engine Filtering & Selection: Provide filters and column selections to drop unmatched rows and skip ignored fields before generating JavaScript objects, saving massive V8 garbage collector (GC) overhead.
• 🌿 Dynamic JSON Flattening Writer: High-speed CSV serializer supporting custom delimiters, quotes, and deep-nested JSON structure flattening using dot-notation.
• 🛡️ Error Handling Strategies: Granular recovery strategies: StopOnError, SkipAndLogMalformedRows (with rollback states), or CoerceInvalidValues.

Installation

npm install turbocsv

Note: Precompiled binaries are automatically loaded based on your platform and CPU architecture.

API Reference

1. Synchronous & Asynchronous Parsing

import { parseCsvSync, parseCsvAsync, JsParserOptions } from 'turbocsv';

// Parse options type definition
interface JsParserOptions {
  delimiter?: string;          // Default: ','
  quote?: string;              // Default: '"'
  escape?: string;             // Escape character
  comment?: string;            // Ignore lines starting with this char
  doubleQuote?: boolean;       // Double quotes as escape (Default: true)
  hasHeaders?: boolean;        // Autodetect header keys (Default: true)
  relaxed?: boolean;           // Relaxed RFC 4180 parsing for malformed rows
  columnSelect?: string[];     // Select subset of columns to parse, ignoring others
  errorStrategy?: 'StopOnError' | 'SkipAndLogMalformedRows' | 'CoerceInvalidValues';
  inferRows?: number;          // Number of rows to read for type inference (Default: 100)
  chunkSize?: number;          // Records per parsing worker thread chunk (Default: 50000)
  format?: 'row' | 'column';   // Output format (Default: 'row')
  filters?: {                  // In-engine row filtering
    column: string;
    op: 'eq' | 'neq' | 'gt' | 'gte' | 'lt' | 'lte' | 'contains' | 'startswith';
    value: string;
  }[];
}

Row-Based Parsing (Array of Objects)

Returns an array of V8 objects. Excellent for standard data consumption.

const csvData = `name,age,active,joined
Alice,30,true,2026-01-15
Bob,25,false,2025-06-20`;

const result = parseCsvSync(csvData, { format: 'row' });
console.log(result.rows);
/*
[
  { name: 'Alice', age: 30, active: true, joined: 1768435200000 },
  { name: 'Bob', age: 25, active: false, joined: 1750377600000 }
]
*/

Columnar-Based Parsing (Maximum Throughput)

Returns TypedArrays for numbers/booleans and a flat buffer for strings. Eliminates object allocation overhead.

const result = parseCsvSync(csvData, { format: 'column' });
console.log(result);
/*
{
  headers: [ 'name', 'age', 'active', 'joined' ],
  types: [ 'String', 'Integer', 'Boolean', 'DateTime' ],
  columns: {
    name: { 
      data: <Buffer 41 6c 69 63 65 42 6f 62>, // AliceBob
      offsets: Int32Array [ 0, 5, 5, 8 ] 
    },
    age: BigInt64Array [ 30n, 25n ],
    active: Uint8Array [ 1, 0 ],
    joined: BigInt64Array [ 1768435200000n, 1750377600000n ]
  },
  nulls: {
    name: Uint8Array [ 0, 0 ],
    age: Uint8Array [ 0, 0 ],
    active: Uint8Array [ 0, 0 ],
    joined: Uint8Array [ 0, 0 ]
  },
  length: 2
}
*/

To decode strings zero-copy from the flat buffer:

const nameCol = result.columns.name;
const firstString = nameCol.data.toString('utf8', nameCol.offsets[0], nameCol.offsets[1]); // "Alice"

Asynchronous Parallel Parsing (libuv threadpool)

const csvBuffer = fs.readFileSync('huge.csv');

parseCsvAsync(csvBuffer, {
  format: 'column',
  chunkSize: 100000, // Process 100k rows in parallel chunks
  filters: [
    { column: 'age', op: 'gte', value: '18' }
  ]
}).then((result) => {
  console.log(`Processed ${result.length} rows asynchronously!`);
});

2. High-Speed Serialization (CSV Writing)

Supports recursive flattening of deep JavaScript nested structures into dot-notation CSV headers.

import { writeCsvString, JsWriterOptions } from 'turbocsv';

const records = [
  { name: "Alice", details: { age: 30, city: "New York" }, tags: ["rust", "wasm"] },
  { name: "Bob", details: { age: 25, city: "San Francisco" }, tags: ["js"] }
];

const options: JsWriterOptions = {
  delimiter: ',',
  doubleQuote: true
};

const csvString = writeCsvString(records, options);
console.log(csvString);
/*
details.age,details.city,name,tags.0,tags.1
30,New York,Alice,rust,wasm
25,San Francisco,Bob,js,
*/

Alternatively, output directly to a Node.js Buffer using writeCsv(records, options) to avoid JS string allocations before writing to file.

Advanced Performance Rationale

1. Rayon Quote-Aware Splitting

Instead of parsing the CSV sequentially, turbocsv uses a zero-allocation single-pass scan of the file to determine record boundary offsets (correctly ignoring newlines inside quotes). It then divides the buffer slices among multiple Rust threads via rayon to parse cells and infer schemas in parallel, avoiding event loop blockage.

2. Eliminating V8 Object Allocation Overhead

Generating millions of JavaScript objects:

{ name: 'Alice', age: 30 }

results in huge garbage collection pauses. By utilizing Columnar Format, the Rust engine writes directly to native memory vectors and wraps them into TypedArrays (e.g. Float64Array, BigInt64Array) and standard Node Buffer slices. This memory is transferred directly to V8 with zero copies, achieving maximum possible memory throughput.

3. In-Engine Filtering

Evaluating filters inside the JavaScript runtime requires parsing the raw strings into V8 structures first. By applying filters inside Rust, turbocsv discards non-matching rows instantly, completely avoiding JS heap allocation for dropped records.

License

MIT