convert-buddy-js

A high-performance, streaming-first parser, converter and transformer for CSV, XML, NDJSON, and JSON.
convert-buddy-js is a TypeScript wrapper around a Rust → WASM core, designed for throughput and low memory overhead on large files, with unified APIs for Node.js and modern browsers.

Table of Contents

• Install
• Quick Start
• API Overview
  • Simple API
  • Streaming API
  • Instance-based API
  • Low-level API
• Stats & Monitoring
• Format Detection & Structure Analysis
• Real-World Examples
• Formats & Configuration
• Transformations & Field Mapping
  • Transform Basics
  • Transform Modes
  • Field Mapping
  • Compute Functions
  • Type Coercion
  • Error Handling
  • Transform Examples
• How it Works
• Benchmarks
• License

Status & Quality

Why Convert Buddy

What it optimizes for

• Performance: Rust/WASM fast-path parsing and conversion
• Streaming-first: convert without loading entire inputs into memory
• Unified multi-format API: one interface for CSV / XML / NDJSON / JSON
• Cross-platform: Node.js and modern browsers

When you might not want it

• Tiny inputs where WASM initialization overhead dominates
• Highly specialized format features
• Environments where WASM is restricted

Install

npm install convert-buddy-js

Quick Start

For small files: Simple one-liner conversion

import { convert, convertToString } from "convert-buddy-js";

// From URL
const json = await convertToString("https://example.com/data.csv", {
  outputFormat: "json",
});

// From File (browser)
const file = fileInput.files![0];
const ndjson = await convertToString(file, { outputFormat: "ndjson" });

// From string data
const csv = "name,age\nAda,36";
const out = await convertToString(csv, { outputFormat: "json" });

// Returns Uint8Array instead of string
const bytes = await convert(file, { outputFormat: "json" });

For streaming & large files: Streaming with automatic backpressure

import { ConvertBuddy } from "convert-buddy-js";

const buddy = new ConvertBuddy();

// Process records as they're converted
const controller = buddy.stream("large-file.csv", {
  outputFormat: "ndjson",
  onRecords: async (controller, records) => {
    // Automatically waits for async operations
    await saveToDatabase(records);
    console.log(`Saved ${records.length} records`);
  },
  onDone: (finalStats) => {
    console.log(`Processed ${finalStats.recordsOut} records in ${finalStats.durationMs}ms`);
  }
});

// Optional: await final stats
const stats = await controller.done;
console.log(`Throughput: ${stats.throughputMbPerSec.toFixed(2)} MB/s`);

Instance-based API (reuse global config)

import { ConvertBuddy } from "convert-buddy-js";

const buddy = new ConvertBuddy({
  debug: true,
  maxMemoryMB: 512,
});

// Simple conversion
const result = await buddy.convert("https://example.com/data.csv", {
  outputFormat: "json",
});

// Or use streaming for large files
const controller = buddy.stream("https://example.com/data.csv", {
  outputFormat: "json",
  onRecords: (controller, records, stats) => {
    console.log(`Progress: ${stats.throughputMbPerSec.toFixed(2)} MB/s`);
  }
});

Platform entrypoints

Browser

import { ConvertBuddy } from "convert-buddy-js/browser";

const buddy = new ConvertBuddy();
const file = document.querySelector<HTMLInputElement>('input[type="file"]')!.files![0];

// Simple conversion
const json = await buddy.convertToString(file, {
  inputFormat: "auto",
  outputFormat: "json",
});

// Or stream large files
const controller = buddy.stream(file, {
  outputFormat: "json",
  onRecords: (ctrl, records) => {
    displayRecords(records);
  }
});

Node.js

import { ConvertBuddy } from "convert-buddy-js/node";

const buddy = new ConvertBuddy();

// Simple conversion
const json = await buddy.convertToString("input.csv", {
  outputFormat: "json",
});

// Or stream large files
const controller = buddy.stream("input.csv", {
  outputFormat: "ndjson",
  onRecords: async (ctrl, records) => {
    await saveToDatabase(records);
  }
});

await controller.done; // Wait for completion

API Overview

Convert Buddy offers multiple layers of control, from one-liners to fully managed streaming.

1. Simple API (for small files)

import { convert, convertToString } from "convert-buddy-js";

const json = await convertToString(input, { outputFormat: "json" });

Supported input types:

• URLs (string)
• Browser File / Blob
• Uint8Array / Node Buffer
• Raw strings
• ReadableStream<Uint8Array>
• Node.js streams
• Node.js file paths

2. Streaming API (for large files & real-time processing)

Automatic backpressure management

import { ConvertBuddy } from "convert-buddy-js";

const buddy = new ConvertBuddy();

const controller = buddy.stream("large-file.csv", {
  outputFormat: "ndjson",
  recordBatchSize: 500, // Records per batch (default: 500)
  
  onRecords: async (controller, records, stats, totalCount) => {
    // Stream automatically waits for async operations
    await processRecords(records);
    console.log(`Processed ${totalCount} records so far`);
  },
  
  onData: (chunk, stats) => {
    // Optional: get raw output chunks
    writeToFile(chunk);
  },
  
  onError: (error) => {
    console.error("Conversion failed:", error);
  },
  
  onDone: (finalStats) => {
    console.log(`Complete! ${finalStats.recordsOut} records in ${finalStats.durationMs}ms`);
  }
});

// Optional: await final stats
const finalStats = await controller.done;
console.log(`Throughput: ${finalStats.throughputMbPerSec.toFixed(2)} MB/s`);

Manual flow control

const controller = buddy.stream(dataSource, {
  outputFormat: "json",
  onRecords: (controller, records) => {
    if (needsToSlowDown()) {
      controller.pause();
      
      // Later...
      setTimeout(() => controller.resume(), 1000);
    }
  }
});

// Cancel processing
controller.cancel("User requested cancellation");

Fire-and-forget processing

// No need to await if you don't need stats
buddy.stream(fileUrl, {
  outputFormat: "ndjson",
  onRecords: (ctrl, records) => {
    displayRecords(records);
  },
  onDone: () => {
    console.log("Processing complete!");
  }
});

// Continues processing in the background

Parser-only mode (no output)

// Set emitOutput to false to parse without generating output data
const controller = buddy.stream(csvFile, {
  emitOutput: false,
  onRecords: async (ctrl, records) => {
    // Just process records, no output chunks generated
    await validateAndStore(records);
  }
});

Standalone stream function

import { stream } from "convert-buddy-js";

// Use without creating a ConvertBuddy instance
const controller = stream(input, {
  inputFormat: "csv",
  outputFormat: "json",
  onRecords: handleRecords
});

3. Instance-based API

import { ConvertBuddy } from "convert-buddy-js";

const buddy = new ConvertBuddy({
  profile: true,
  progressIntervalBytes: 1024 * 1024,
});

const out = await buddy.convert(file, { outputFormat: "ndjson" });
console.log(buddy.lastStats()); // Get conversion stats

4. Low-level API (for advanced use cases)

Manual chunked streaming

import { ConvertBuddy } from "convert-buddy-js";

const converter = await ConvertBuddy.create({
  inputFormat: "xml",
  outputFormat: "ndjson",
  xmlConfig: { recordElement: "row", includeAttributes: true },
});

converter.push(new Uint8Array([/* bytes */]));
converter.push(new Uint8Array([/* bytes */]));

const final = converter.finish();
console.log(converter.stats());

Node.js Transform stream

import { createNodeTransform } from "convert-buddy-js/node";
import { createReadStream, createWriteStream } from "node:fs";

const transform = await createNodeTransform({
  inputFormat: "csv",
  outputFormat: "ndjson",
  csvConfig: { hasHeaders: true },
  profile: true,
});

createReadStream("input.csv")
  .pipe(transform)
  .pipe(createWriteStream("output.ndjson"));

Web Streams

import { ConvertBuddyTransformStream } from "convert-buddy-js";

const transform = new ConvertBuddyTransformStream({
  inputFormat: "csv",
  outputFormat: "ndjson",
});

const response = await fetch("/data.csv");
const output = response.body?.pipeThrough(transform);

Stats & Monitoring

Every conversion provides detailed statistics for monitoring performance and progress.

Stats Object

All streaming operations provide real-time stats:

interface BuddyStats {
  bytesIn: number;           // Total bytes consumed
  bytesOut: number;          // Total bytes generated
  recordsOut: number;        // Total records processed
  batchesOut: number;        // Number of batches emitted
  durationMs: number;        // Elapsed time in milliseconds
  throughputMbPerSec: number; // Processing speed (MB/s)
  startedAt: number;         // Timestamp when started
  endedAt?: number;          // Timestamp when completed
  isPaused: boolean;         // Current pause state
  isDone: boolean;           // Whether processing is complete
  error?: string;            // Error message if failed
}

Accessing Stats

During streaming:

const controller = buddy.stream(input, {
  outputFormat: "json",
  onRecords: (controller, records, stats) => {
    // Stats snapshot at this point in time
    console.log(`Progress: ${stats.recordsOut} records`);
    console.log(`Speed: ${stats.throughputMbPerSec.toFixed(2)} MB/s`);
  }
});

// Get current stats at any time
const currentStats = controller.stats();

// Wait for final stats
const finalStats = await controller.done;
console.log(`Total: ${finalStats.recordsOut} records`);
console.log(`Duration: ${finalStats.durationMs}ms`);
console.log(`Avg speed: ${finalStats.throughputMbPerSec.toFixed(2)} MB/s`);

After simple conversion:

const buddy = new ConvertBuddy({ profile: true });
await buddy.convert(input, { outputFormat: "json" });

const stats = buddy.lastStats();
console.log(`Processed ${stats.recordsOut} records in ${stats.durationMs}ms`);

Stats Immutability

All stats objects returned are frozen (immutable) to prevent accidental modification:

const stats = controller.stats();
stats.recordsOut = 999; // Error: Cannot modify frozen object

Format Detection & Structure Analysis

Convert Buddy can automatically detect input formats and analyze data structure.

Auto-detect Format

import { detectFormat } from "convert-buddy-js";

// From any input source
const format = await detectFormat(fileOrUrlOrStream);
console.log(format); // "csv" | "json" | "ndjson" | "xml" | "unknown"

// With options
const format = await detectFormat(input, {
  maxBytes: 256 * 1024,  // Sample up to 256KB
  preferredFormat: "csv" // Hint if ambiguous
});

Supported inputs:

• Files (browser File / Blob)
• URLs (string)
• Streams (ReadableStream, Node.js streams)
• Buffers (Uint8Array, Buffer)
• Strings

Detect Structure

Analyze the structure of your data:

import { detectStructure } from "convert-buddy-js";

const structure = await detectStructure(input);

console.log(structure);
// {
//   format: "csv",
//   confidence: 0.95,
//   details: {
//     hasHeaders: true,
//     delimiter: ",",
//     quote: '"',
//     estimatedColumns: 5,
//     estimatedRows: 1000,
//     sampleFields: ["name", "email", "age", "city", "country"]
//   }
// }

For CSV:

• Detects delimiter (,, ;, \t, |)
• Identifies quote character
• Determines if headers are present
• Samples field names

For XML:

• Identifies record elements
• Detects namespace usage
• Finds attribute patterns

For JSON/NDJSON:

• Determines if newline-delimited or array
• Analyzes nesting depth
• Samples field names

Use Auto-detection in Conversions

// Input format defaults to "auto"
const json = await buddy.convertToString(unknownFile, {
  outputFormat: "json"
});

// Explicitly use auto-detection
const controller = buddy.stream(input, {
  inputFormat: "auto",  // Detect automatically
  outputFormat: "ndjson",
  onRecords: handleRecords
});

Real-World Examples

Batch Processing with Database Inserts

import { ConvertBuddy } from "convert-buddy-js";

const buddy = new ConvertBuddy();

const controller = buddy.stream("large-export.csv", {
  outputFormat: "ndjson",
  recordBatchSize: 1000, // Process 1000 records at a time
  
  onRecords: async (controller, records, stats) => {
    // Batch insert to database
    await db.users.insertMany(records);
    console.log(`Inserted ${stats.recordsOut} records (${stats.throughputMbPerSec.toFixed(2)} MB/s)`);
  },
  
  onError: (error) => {
    console.error("Import failed:", error);
    // Rollback or cleanup
  },
  
  onDone: (finalStats) => {
    console.log(`Import complete! ${finalStats.recordsOut} total records in ${finalStats.durationMs}ms`);
  }
});

await controller.done; // Wait for completion

Real-time Data Validation

const validRecords = [];
const invalidRecords = [];

const controller = buddy.stream(uploadedFile, {
  outputFormat: "json",
  onRecords: (ctrl, records) => {
    for (const record of records) {
      if (validateRecord(record)) {
        validRecords.push(record);
      } else {
        invalidRecords.push(record);
      }
    }
  },
  onDone: () => {
    console.log(`Valid: ${validRecords.length}, Invalid: ${invalidRecords.length}`);
  }
});

Progress Bar for File Uploads

const controller = buddy.stream(file, {
  outputFormat: "ndjson",
  onRecords: (ctrl, records, stats) => {
    // Update progress bar
    const progressPercent = (stats.bytesIn / file.size) * 100;
    updateProgressBar(progressPercent, stats.throughputMbPerSec);
  }
});

Streaming API Response

// Node.js server endpoint
app.get('/export/users', async (req, res) => {
  const buddy = new ConvertBuddy();
  
  res.setHeader('Content-Type', 'text/csv');
  res.setHeader('Content-Disposition', 'attachment; filename="users.csv"');
  
  const controller = buddy.stream(getUsersStream(), {
    outputFormat: "csv",
    onData: (chunk) => {
      res.write(chunk);
    },
    onDone: () => {
      res.end();
    },
    onError: (error) => {
      res.status(500).send(error.message);
    }
  });
});

Browser File Converter

// Convert and download in browser
import { ConvertBuddy } from "convert-buddy-js/browser";

async function convertAndDownload(file, outputFormat) {
  const buddy = new ConvertBuddy();
  const chunks = [];
  
  const controller = buddy.stream(file, {
    outputFormat,
    onData: (chunk) => {
      chunks.push(chunk);
    },
    onDone: (stats) => {
      // Create blob and download
      const blob = new Blob(chunks, { type: getMimeType(outputFormat) });
      const url = URL.createObjectURL(blob);
      const a = document.createElement('a');
      a.href = url;
      a.download = `converted.${outputFormat}`;
      a.click();
      URL.revokeObjectURL(url);
      
      console.log(`Converted in ${stats.durationMs}ms`);
    }
  });
}

Formats & Configuration

Supported

• csv
• xml
• ndjson
• json
• auto

CSV options

{
  csvConfig: {
    delimiter: ",",
    quote: '"',
    hasHeaders: true,
    trimWhitespace: false,
  }
}

XML options

{
  xmlConfig: {
    recordElement: "row",
    trimText: true,
    includeAttributes: true,
  }
}

Performance options

{
  chunkTargetBytes: 1024 * 1024,
  parallelism: 4,
  profile: true,
  debug: false,
}

Transformations & Field Mapping

Convert Buddy includes a powerful Rust-based transform engine that enables field-level transformations, computed fields, type coercion, and data reshaping during conversion — all executed in WASM for maximum performance with zero JavaScript overhead.

Transform Basics

Apply transforms using the transform option in any conversion:

import { ConvertBuddy } from "convert-buddy-js";

const buddy = new ConvertBuddy();

const result = await buddy.convert(csvData, {
  outputFormat: "json",
  transform: {
    mode: "replace",
    fields: [
      { targetFieldName: "id", required: true },
      { targetFieldName: "full_name", compute: "concat(first, ' ', last)" },
      { targetFieldName: "age", coerce: { type: "i64" }, defaultValue: 0 },
      { targetFieldName: "email", compute: "lower(trim(email))" },
    ],
    onMissingField: "null",
    onCoerceError: "null",
  },
});

Key Features:

• 🚀 WASM performance — all transforms execute in Rust/WASM
• 🔄 Streaming — transforms work on line-by-line records without buffering
• 🧮 30+ compute functions — string manipulation, math, logic, type conversions
• 🎯 Type coercion — convert strings to numbers, booleans, timestamps
• ⚡ Zero memory overhead — no intermediate objects or copies in JS
• 🛡️ Error policies — control behavior for missing fields and coercion failures

Transform Modes

mode: "replace" (default)
Outputs only the fields you explicitly map. Original fields are discarded.

// Input:  { "first": "Alice", "last": "Smith", "age": 30, "city": "NYC" }
// Output: { "full_name": "Alice Smith", "age": 30 }

transform: {
  mode: "replace",
  fields: [
    { targetFieldName: "full_name", compute: "concat(first, ' ', last)" },
    { targetFieldName: "age" },
  ],
}

mode: "augment"
Preserves all original fields and adds/overwrites mapped fields.

// Input:  { "name": "Alice", "age": 30 }
// Output: { "name": "Alice", "age": 30, "name_upper": "ALICE", "is_adult": true }

transform: {
  mode: "augment",
  fields: [
    { targetFieldName: "name_upper", compute: "upper(name)" },
    { targetFieldName: "is_adult", compute: "gte(age, 18)" },
  ],
}

Field Mapping

Each field in the fields array defines a mapping:

type FieldMap = {
  targetFieldName: string;        // Output field name
  originFieldName?: string;       // Input field name (defaults to targetFieldName)
  required?: boolean;             // Fail if missing
  defaultValue?: any;             // Fallback value
  coerce?: Coerce;                // Type conversion
  compute?: string;               // Expression to compute value
};

Basic field pass-through:

{ targetFieldName: "name" }  // Maps input "name" to output "name"

Rename fields:

{ targetFieldName: "fullName", originFieldName: "full_name" }  // Rename full_name → fullName

Required fields:

{ targetFieldName: "id", required: true }  // Fail conversion if "id" is missing

Default values:

{ targetFieldName: "status", defaultValue: "active" }  // Use "active" if field is missing or null

Computed fields:

{ targetFieldName: "display", compute: "concat(first, ' ', last)" }

Type coercion:

{ targetFieldName: "age", coerce: { type: "i64" } }  // Convert string "30" → number 30

Combined:

{
  targetFieldName: "age_years",
  originFieldName: "age",
  coerce: { type: "i64" },
  defaultValue: 0,
  required: false,
}

Compute Functions

Transform expressions support 30+ built-in functions for data manipulation:

String Functions

| Function | Description | Example | |----------|-------------|---------| | concat(...) | Join strings | concat(first, " ", last) → "Alice Smith" | | lower(s) | Lowercase | lower("HELLO") → "hello" | | upper(s) | Uppercase | upper("hello") → "HELLO" | | trim(s) | Remove whitespace | trim(" text ") → "text" | | trim_start(s) | Remove leading whitespace | trim_start(" text") → "text" | | trim_end(s) | Remove trailing whitespace | trim_end("text ") → "text" | | substring(s, start, end) | Extract substring | substring("hello", 0, 3) → "hel" | | replace(s, old, new) | Replace text | replace("foo bar", "foo", "baz") → "baz bar" | | len(s) | String length | len("hello") → 5 | | starts_with(s, prefix) | Check prefix | starts_with("https://...", "https") → true | | ends_with(s, suffix) | Check suffix | ends_with("file.pdf", ".pdf") → true | | contains(s, substr) | Check substring | contains("hello world", "world") → true | | pad_start(s, len, char) | Left-pad | pad_start("42", 5, "0") → "00042" | | pad_end(s, len, char) | Right-pad | pad_end("x", 3, "_") → "x__" | | repeat(s, n) | Repeat string | repeat("ab", 3) → "ababab" | | reverse(s) | Reverse string | reverse("hello") → "olleh" | | split(s, delim) | Split to array | split("a,b,c", ",") → ["a","b","c"] | | join(arr, delim) | Join array | join(["a","b"], ",") → "a,b" |

Math Functions

| Function | Description | Example | |----------|-------------|---------| | +, -, *, / | Arithmetic operators | price * 1.1 | | round(n) | Round to nearest integer | round(3.7) → 4 | | floor(n) | Round down | floor(3.9) → 3 | | ceil(n) | Round up | ceil(3.1) → 4 | | abs(n) | Absolute value | abs(-5) → 5 | | min(...) | Minimum value | min(a, b, c) → smallest | | max(...) | Maximum value | max(a, b, c) → largest |

Logic & Comparison

| Function | Description | Example | |----------|-------------|---------| | if(cond, true_val, false_val) | Conditional | if(age >= 18, "adult", "minor") | | not(bool) | Boolean NOT | not(active) → opposite | | eq(a, b) | Equals | eq(status, "active") → true/false | | ne(a, b) | Not equals | ne(a, b) | | gt(a, b) | Greater than | gt(age, 18) | | gte(a, b) | Greater than or equal | gte(age, 18) | | lt(a, b) | Less than | lt(price, 100) | | lte(a, b) | Less than or equal | lte(score, 50) |

Type Checking

| Function | Description | Example | |----------|-------------|---------| | is_null(val) | Check if null | is_null(optional_field) → true/false | | is_number(val) | Check if number | is_number(val) → true/false | | is_string(val) | Check if string | is_string(val) → true/false | | is_bool(val) | Check if boolean | is_bool(val) → true/false |

Type Conversion

| Function | Description | Example | |----------|-------------|---------| | to_string(val) | Convert to string | to_string(42) → "42" | | parse_int(s) | Parse string to integer | parse_int("123") → 123 | | parse_float(s) | Parse string to float | parse_float("3.14") → 3.14 |

Utility Functions

| Function | Description | Example | |----------|-------------|---------| | coalesce(...) | First non-null value | coalesce(alt1, alt2, "default") | | default(val, fallback) | Fallback for null | default(optional, "N/A") |

Nested expressions:

compute: "upper(trim(concat(first, ' ', last)))"
// Input: { first: "  alice  ", last: "  smith  " }
// Output: "ALICE SMITH"

Complex logic:

compute: "if(gte(age, 18), 'adult', if(gte(age, 13), 'teen', 'child'))"

Type Coercion

Convert field types automatically:

type Coerce =
  | { type: "string" }
  | { type: "i64" }      // 64-bit integer
  | { type: "f64" }      // 64-bit float
  | { type: "bool" }
  | { type: "timestamp_ms"; format?: "iso8601" | "unix_ms" | "unix_s" };

String coercion:

// Number → String: 42 → "42"
// Boolean → String: true → "true"
// Null → String: null → ""
{ targetFieldName: "age_str", originFieldName: "age", coerce: { type: "string" } }

Integer coercion (i64):

// String → Integer: "42" → 42
// Float → Integer: 3.7 → 3 (truncates)
// Boolean → Integer: true → 1, false → 0
{ targetFieldName: "count", coerce: { type: "i64" } }

Float coercion (f64):

// String → Float: "3.14" → 3.14
// Integer → Float: 42 → 42.0
{ targetFieldName: "price", coerce: { type: "f64" } }

Boolean coercion:

// String → Boolean: "true"/"false" → true/false, "1"/"0" → true/false
// Number → Boolean: 0 → false, non-zero → true
{ targetFieldName: "active", coerce: { type: "bool" } }

Timestamp coercion:

// ISO8601 string → Unix timestamp (ms)
{
  targetFieldName: "created_ts",
  originFieldName: "created_at",
  coerce: { type: "timestamp_ms", format: "iso8601" }
}
// "2024-01-15T10:30:00Z" → 1705315800000

// Unix seconds → milliseconds
{
  targetFieldName: "updated_ts",
  coerce: { type: "timestamp_ms", format: "unix_s" }
}
// 1705315800 → 1705315800000

Transform Error Handling

Control behavior when fields are missing or coercion fails:

onMissingField — What to do when a field is missing:

• "error" — Fail the entire conversion
• "null" — Insert null for missing fields
• "drop" — Omit the field from output

transform: {
  fields: [{ targetFieldName: "optional_field" }],
  onMissingField: "null",  // Missing fields become null
}

onMissingRequired — What to do when a required field is missing:

• "error" — Fail the conversion (default)
• "abort" — Stop processing this record

transform: {
  fields: [{ targetFieldName: "id", required: true }],
  onMissingRequired: "error",  // Fail if "id" is missing
}

onCoerceError — What to do when type coercion fails:

• "error" — Fail the entire conversion
• "null" — Set field to null on coercion failure
• "dropRecord" — Silently skip records that fail coercion

transform: {
  fields: [{ targetFieldName: "age", coerce: { type: "i64" } }],
  onCoerceError: "null",  // "invalid" → null instead of throwing
}

Skip invalid records:

// Drop records where age cannot be parsed as integer
transform: {
  fields: [
    { targetFieldName: "name" },
    { targetFieldName: "age", coerce: { type: "i64" } },
  ],
  onCoerceError: "dropRecord",  // Skip bad records silently
}

Transform Examples

Example 1: Clean and normalize user data

const controller = buddy.stream("users.csv", {
  outputFormat: "json",
  transform: {
    mode: "replace",
    fields: [
      // Required ID
      { targetFieldName: "id", required: true },
      
      // Normalize name
      { 
        targetFieldName: "full_name", 
        compute: "trim(concat(first_name, ' ', last_name))" 
      },
      
      // Lowercase email
      { targetFieldName: "email", compute: "lower(trim(email))" },
      
      // Parse age as integer with default
      { 
        targetFieldName: "age", 
        coerce: { type: "i64" }, 
        defaultValue: 0 
      },
      
      // Active status as boolean
      { targetFieldName: "active", coerce: { type: "bool" } },
    ],
    onMissingField: "null",
    onCoerceError: "null",
  },
  onRecords: async (ctrl, records) => {
    await db.users.insertMany(records);
  },
});

Example 2: Calculate derived fields

// Add computed fields to existing data
transform: {
  mode: "augment",  // Keep all original fields
  fields: [
    // Calculate total with tax
    { 
      targetFieldName: "total_with_tax", 
      compute: "round(price * (1 + tax_rate))" 
    },
    
    // Discount percentage
    { 
      targetFieldName: "discount_pct", 
      compute: "round((original_price - price) / original_price * 100)" 
    },
    
    // Is on sale?
    { 
      targetFieldName: "on_sale", 
      compute: "lt(price, original_price)" 
    },
  ],
}

Example 3: Data validation and filtering

// Only keep records with valid email and age >= 18
const validRecords = [];

const controller = buddy.stream(csvData, {
  outputFormat: "ndjson",
  transform: {
    mode: "replace",
    fields: [
      { targetFieldName: "email", required: true },
      { targetFieldName: "age", coerce: { type: "i64" }, required: true },
      { 
        targetFieldName: "is_adult", 
        compute: "gte(age, 18)" 
      },
    ],
    onMissingRequired: "error",
    onCoerceError: "dropRecord",  // Skip records with invalid age
  },
  onRecords: (ctrl, records) => {
    // Filter for adults only
    const adults = records.filter(r => r.is_adult);
    validRecords.push(...adults);
  },
});

Example 4: Format conversion with field mapping

// Convert XML to CSV with field restructuring
const controller = buddy.stream("data.xml", {
  inputFormat: "xml",
  outputFormat: "csv",
  xmlConfig: { recordElement: "user" },
  transform: {
    mode: "replace",
    fields: [
      { targetFieldName: "user_id", originFieldName: "id" },
      { targetFieldName: "name", compute: "concat(firstName, ' ', lastName)" },
      { targetFieldName: "email_domain", compute: "split(email, '@')[1]" },
      { 
        targetFieldName: "created_date", 
        originFieldName: "createdAt",
        coerce: { type: "timestamp_ms", format: "iso8601" }
      },
    ],
  },
  onData: (chunk) => {
    fs.appendFileSync("output.csv", chunk);
  },
});

Example 5: Streaming with real-time transforms

// Process large file with transforms in streaming mode
let processedCount = 0;

const controller = buddy.stream("massive-dataset.csv", {
  outputFormat: "ndjson",
  recordBatchSize: 1000,
  transform: {
    mode: "replace",
    fields: [
      { targetFieldName: "id" },
      { targetFieldName: "name_upper", compute: "upper(name)" },
      { targetFieldName: "price", coerce: { type: "f64" } },
      { 
        targetFieldName: "price_category",
        compute: "if(gt(price, 100), 'premium', if(gt(price, 50), 'mid', 'budget'))"
      },
    ],
    onMissingField: "drop",
  },
  onRecords: async (ctrl, records, stats) => {
    processedCount += records.length;
    console.log(`Transformed ${processedCount} records at ${stats.throughputMbPerSec.toFixed(2)} MB/s`);
    
    // Stream directly to database
    await db.products.insertMany(records);
  },
});

await controller.done;

Example 6: Complex field transformations

transform: {
  mode: "replace",
  fields: [
    // Combine multiple fields with formatting
    {
      targetFieldName: "address",
      compute: "concat(street, ', ', city, ', ', state, ' ', zip)"
    },
    
    // Conditional logic
    {
      targetFieldName: "shipping_fee",
      compute: "if(eq(country, 'US'), 5.00, if(eq(country, 'CA'), 7.50, 12.00))"
    },
    
    // Nested string operations
    {
      targetFieldName: "username",
      compute: "lower(trim(replace(email, '@', '_at_')))"
    },
    
    // Math with multiple fields
    {
      targetFieldName: "bmi",
      compute: "round(weight / ((height / 100) * (height / 100)))"
    },
    
    // String manipulation chain
    {
      targetFieldName: "display_name",
      compute: "concat(upper(substring(first, 0, 1)), '. ', last)"
    },
  ],
}

Performance Notes

• ✅ Zero JS overhead — transforms execute entirely in Rust/WASM
• ✅ Streaming — processes line-by-line, no buffering of entire dataset
• ✅ Type safety — expression parser validates syntax at compile time
• ✅ Memory efficient — uses WASM linear memory, no JS objects created
• ⚡ Throughput — typically adds <10% overhead vs. raw conversion

For extremely complex transformations (e.g., API lookups, database joins), consider using onRecords callback for post-processing in JavaScript.

How it works

• Rust core implements streaming parsers and conversion
• WASM bindings generated via wasm-bindgen
• TypeScript wrapper exposes high-level APIs and stream adapters

What ships in the npm package

• Prebuilt WASM binaries
• Compiled JS / TypeScript output

Rust sources, demos, and benchmarks live in the repository but are not published in the npm package.

Benchmarks (repository)

cd packages/convert-buddy-js
npm run bench
npm run bench:check
npm run bench:competitors

License

MIT