@bernierllc/csv-import-service

Service layer for orchestrating CSV import operations using @bernierllc core packages.

Overview

The CSV Import Service provides a comprehensive solution for importing and processing CSV data with built-in validation, progress tracking, and error handling. It orchestrates multiple @bernierllc core packages to provide a unified CSV import experience.

Features

• Multiple Input Sources: File paths, buffers, strings, and streams
• Flexible Parsing: Configurable delimiters, quotes, headers, and encoding
• Data Validation: Schema-based validation with detailed error reporting
• Progress Tracking: Real-time progress updates with callback support
• Batch Processing: Efficient processing of large datasets
• Error Recovery: Continue processing on errors with configurable limits
• TypeScript Support: Full type safety with Zod schema validation

Installation

npm install @bernierllc/csv-import-service

Quick Start

import { CSVImportService } from '@bernierllc/csv-import-service';

const service = new CSVImportService({
  delimiter: ',',
  headers: true,
  batchSize: 1000
});

// Import from string
const result = await service.import({
  type: 'string',
  data: 'name,email,age\nJohn,[email protected],30'
});

console.log(`Processed ${result.summary.processedRows} rows`);

API Reference

CSVImportService

Constructor

constructor(config?: Partial<CSVImportConfig>)

Creates a new CSV import service instance with optional configuration.

Methods

import(source, validationSchema?, progressCallback?)

Imports CSV data from various sources.

async import(
  source: ImportSource,
  validationSchema?: ValidationSchema,
  progressCallback?: ProgressCallback
): Promise<ImportResult>

Parameters:

• source: Input source (file, buffer, string, or stream)
• validationSchema (optional): Schema for data validation
• progressCallback (optional): Callback for progress updates

Returns: ImportResult with processed data, errors, and metadata

updateConfig(updates)

Updates the service configuration.

updateConfig(updates: Partial<CSVImportConfig>): void

reset()

Resets the service state for a new import operation.

reset(): void

Configuration Options

interface CSVImportConfig {
  // File handling
  encoding: 'utf8' | 'utf16le' | 'latin1' | 'ascii';
  skipEmptyLines: boolean;
  trimWhitespace: boolean;
  
  // Parsing
  delimiter: string;
  quote: string;
  escape: string;
  headers: boolean;
  skipLinesStart: number;
  skipLinesEnd: number;
  
  // Validation
  strictMode: boolean;
  allowExtraFields: boolean;
  allowMissingFields: boolean;
  
  // Processing
  batchSize: number;
  maxRows?: number;
  
  // Error handling
  continueOnError: boolean;
  maxErrors: number;
  
  // Output
  includeLineNumbers: boolean;
  includeRawData: boolean;
}

Import Sources

File Source

{
  type: 'file',
  path: string
}

Buffer Source

{
  type: 'buffer',
  data: Buffer,
  filename?: string
}

String Source

{
  type: 'string',
  data: string,
  filename?: string
}

Stream Source

{
  type: 'stream',
  stream: NodeJS.ReadableStream,
  filename?: string
}

Validation Schema

Define validation rules for your CSV data:

const validationSchema = {
  name: { 
    type: 'string', 
    required: true, 
    minLength: 2 
  },
  email: { 
    type: 'email', 
    required: true 
  },
  age: { 
    type: 'number', 
    required: true, 
    min: 0, 
    max: 150 
  }
};

Examples

Basic Import with Validation

import { CSVImportService } from '@bernierllc/csv-import-service';

const service = new CSVImportService({
  delimiter: ',',
  headers: true,
  continueOnError: true
});

const validationSchema = {
  name: { type: 'string', required: true },
  email: { type: 'email', required: true },
  age: { type: 'number', required: false }
};

const result = await service.import(
  { type: 'file', path: './data.csv' },
  validationSchema
);

if (result.success) {
  console.log(`Successfully processed ${result.summary.processedRows} rows`);
  console.log(`${result.errors.length} validation errors found`);
} else {
  console.log('Import failed:', result.errors);
}

Progress Tracking

const service = new CSVImportService({
  batchSize: 500
});

const progressCallback = (progress) => {
  console.log(`Progress: ${progress.percentage}% (${progress.processed}/${progress.total})`);
  console.log(`Stage: ${progress.stage}`);
};

const result = await service.import(
  { type: 'file', path: './large-data.csv' },
  undefined,
  progressCallback
);

Custom Configuration

const service = new CSVImportService({
  delimiter: '|',
  encoding: 'latin1',
  skipEmptyLines: false,
  batchSize: 2000,
  maxErrors: 50,
  includeRawData: true
});

const result = await service.import({
  type: 'string',
  data: csvContent,
  filename: 'custom.csv'
});

Stream Processing

import { createReadStream } from 'fs';

const stream = createReadStream('./data.csv');
const result = await service.import({
  type: 'stream',
  stream,
  filename: 'data.csv'
});

Error Handling

const service = new CSVImportService({
  continueOnError: true,
  maxErrors: 100
});

const result = await service.import(source, validationSchema);

// Process successful rows
result.data.forEach(row => {
  console.log(`Line ${row.line}:`, row.data);
  if (row.warnings.length > 0) {
    console.log('Warnings:', row.warnings);
  }
});

// Handle failed rows
result.errors.forEach(errorRow => {
  console.log(`Failed line ${errorRow.line}:`, errorRow.errors);
});

Result Structure

The import method returns an ImportResult object:

interface ImportResult {
  success: boolean;
  summary: {
    totalRows: number;
    processedRows: number;
    failedRows: number;
    warningRows: number;
    duration: number;
    startTime: Date;
    endTime: Date;
  };
  data: ProcessedRow[];
  errors: FailedRow[];
  warnings: ImportError[];
  metadata: {
    filename?: string;
    fileSize?: number;
    encoding: string;
    delimiter: string;
    headers?: string[];
    config: CSVImportConfig;
  };
}

Performance

• Batch Processing: Large files are processed in configurable batches
• Memory Efficient: Streaming support for large datasets
• Configurable Limits: Control memory usage with maxRows and batchSize
• Progress Tracking: Monitor performance with real-time progress updates

Dependencies

This package orchestrates the following @bernierllc core packages:

• @bernierllc/csv-parser - CSV parsing and formatting
• @bernierllc/csv-validator - Data validation (in simplified form)
• @bernierllc/file-handler - File operations (referenced but using Node.js fs directly)

Error Types

Common error codes returned by the service:

• REQUIRED_FIELD_MISSING - Required field is empty or missing
• INVALID_EMAIL_FORMAT - Email field doesn't match email pattern
• INVALID_NUMBER_FORMAT - Number field contains non-numeric data
• PROCESSING_ERROR - General processing error
• IMPORT_FAILED - Overall import failure

Best Practices

1. Use validation schemas for data quality assurance
2. Set appropriate batch sizes based on memory constraints
3. Monitor progress for long-running imports
4. Handle errors gracefully with continueOnError
5. Configure appropriate limits to prevent resource exhaustion

See Also

• @bernierllc/csv-parser - CSV parsing functionality
• @bernierllc/csv-validator - Data validation utilities
• @bernierllc/file-handler - File operations

License

Copyright (c) 2025 Bernier LLC. All rights reserved.