@bernierllc/csv-import

Service package that orchestrates CSV core packages to provide higher-level CSV import functionality with schema management, error handling, and import workflows.

Installation

npm install @bernierllc/csv-import

Usage

Basic Import

import { CSVImport, createSchema } from '@bernierllc/csv-import';

// Create CSV import instance
const csvImport = new CSVImport();

// Create schema
const schemaId = await csvImport.createSchema({
  name: 'Users Import',
  description: 'Schema for importing user data',
  fields: [
    {
      name: 'email',
      type: 'email',
      required: true,
      validation: {
        pattern: /^[^\s@]+@[^\s@]+\.[^\s@]+$/
      }
    },
    {
      name: 'firstName',
      type: 'string',
      required: true,
      validation: {
        minLength: 1,
        maxLength: 50
      }
    },
    {
      name: 'lastName',
      type: 'string',
      required: true,
      validation: {
        minLength: 1,
        maxLength: 50
      }
    },
    {
      name: 'age',
      type: 'number',
      required: false
    }
  ]
});

// Create import job
const job = await csvImport.createImport({
  schemaId,
  options: {
    batchSize: 100,
    validateOnly: false,
    autoMap: true,
    skipErrors: false,
    maxErrors: 10
  },
  callbacks: {
    onProgress: (progress) => {
      console.log(`Progress: ${progress.progress}%`);
    },
    onComplete: (result) => {
      console.log(`Import completed: ${result.validRows} valid rows`);
    },
    onError: (error) => {
      console.error('Import error:', error.message);
    }
  }
});

// Start import
const fileData = {
  name: 'users.csv',
  size: 1024,
  type: 'text/csv',
  content: 'email,firstName,lastName,age\[email protected],John,Doe,30',
  buffer: Buffer.from('email,firstName,lastName,age\[email protected],John,Doe,30')
};

const result = await csvImport.startImport(job.id, fileData);

if (result.success) {
  console.log(`Successfully imported ${result.validRows} rows`);
} else {
  console.log(`Import failed with ${result.errors.length} errors`);
  result.errors.forEach(error => {
    console.log(`Row ${error.row}: ${error.message}`);
  });
}

Advanced Usage with Retry Logic

import { CSVImport } from '@bernierllc/csv-import';

const csvImport = new CSVImport();

// Create import with retry configuration
const job = await csvImport.createImport({
  schemaId: 'your-schema-id',
  options: {
    batchSize: 50,
    retryFailed: true,
    retryOptions: {
      maxRetries: 3,
      initialDelayMs: 1000,
      maxDelayMs: 30000,
      backoffFactor: 2
    }
  }
});

const result = await csvImport.startImport(job.id, fileData);

Progress Tracking

// Subscribe to progress updates
const unsubscribe = csvImport.progressTracker.subscribeToProgress(
  job.id,
  (progress) => {
    console.log(`Status: ${progress.status}`);
    console.log(`Progress: ${progress.progress}%`);
    console.log(`Processed: ${progress.currentRow}/${progress.totalRows}`);
    console.log(`Valid: ${progress.validRows}, Invalid: ${progress.invalidRows}`);
    
    if (progress.estimatedTimeRemaining) {
      console.log(`ETA: ${progress.estimatedTimeRemaining}ms`);
    }
  }
);

// Start import
await csvImport.startImport(job.id, fileData);

// Clean up subscription
unsubscribe();

Schema Management

// Create basic schema
const basicSchemaId = await csvImport.schemaManager.createBasicSchema(
  'Simple CSV',
  ['name', 'email', 'phone']
);

// Create email-specific schema
const emailSchemaId = await csvImport.schemaManager.createEmailSchema(
  'Email List Import'
);

// Update schema
await csvImport.updateSchema(schemaId, {
  description: 'Updated description',
  fields: [
    // ... updated fields
  ]
});

// Validate schema
const validation = await csvImport.validateSchema(schema);
if (!validation.isValid) {
  console.log('Schema errors:', validation.errors);
}

// List all schemas
const allSchemas = await csvImport.schemaManager.listSchemas();

Import Management

// Pause import
await csvImport.pauseImport(job.id);

// Resume import
await csvImport.resumeImport(job.id);

// Cancel import
await csvImport.cancelImport(job.id);

// Get import history
const history = await csvImport.getImportHistory({
  status: 'completed',
  limit: 10,
  offset: 0
});

console.log(`Found ${history.length} completed imports`);

Error Handling and Recovery

import { ErrorHandler } from '@bernierllc/csv-import';

const errorHandler = new ErrorHandler();

// Get error suggestions
result.errors.forEach(error => {
  const suggestion = errorHandler.getSuggestion(error);
  if (suggestion) {
    console.log(`Error: ${error.message}`);
    console.log(`Suggestion: ${suggestion}`);
  }
});

// Generate error report
const errorReport = errorHandler.generateErrorReport(result.errors);
console.log('Error Summary:', errorReport.summary);
console.log('Errors by Field:', errorReport.byField);
console.log('Errors by Code:', errorReport.byCode);

// Get recovery options
result.errors.forEach(error => {
  const options = errorHandler.getRecoveryOptions(error);
  console.log(`Recovery options for ${error.code}:`, options);
});

API Reference

CSVImport

Main service class that orchestrates CSV import operations.

Methods

• createImport(config: ImportConfig): Promise<ImportJob> - Create a new import job
• startImport(jobId: string, fileData: FileData): Promise<ImportResult> - Start processing an import
• pauseImport(jobId: string): Promise<void> - Pause an active import
• resumeImport(jobId: string): Promise<void> - Resume a paused import
• cancelImport(jobId: string): Promise<void> - Cancel an import
• createSchema(schema: SchemaData): Promise<SchemaId> - Create a new schema
• updateSchema(schemaId: SchemaId, updates: Partial<CSVSchema>): Promise<void> - Update existing schema
• validateSchema(schema: CSVSchema): Promise<SchemaValidationResult> - Validate schema
• getImportProgress(jobId: string): Promise<ImportProgress> - Get import progress
• getImportHistory(options?: HistoryOptions): Promise<ImportHistory[]> - Get import history

Types

ImportConfig

interface ImportConfig {
  schemaId?: SchemaId;
  options?: ImportOptions;
  callbacks?: ImportCallbacks;
}

ImportOptions

interface ImportOptions {
  batchSize?: number;
  validateOnly?: boolean;
  autoMap?: boolean;
  skipErrors?: boolean;
  maxErrors?: number;
  retryFailed?: boolean;
  retryOptions?: RetryOptions;
}

ImportResult

interface ImportResult {
  jobId: string;
  success: boolean;
  totalRows: number;
  processedRows: number;
  validRows: number;
  invalidRows: number;
  errors: ImportError[];
  warnings: ImportWarning[];
  processingTime: number;
  completedAt: Date;
}

CSVSchema

interface CSVSchema {
  id: SchemaId;
  name: string;
  description?: string;
  fields: SchemaField[];
  validationRules?: ValidationRule[];
  mappingRules?: MappingRule[];
  createdAt: Date;
  updatedAt: Date;
}

Configuration

Import Options

• batchSize (default: 100) - Number of rows to process in each batch
• validateOnly (default: false) - Only validate data without processing
• autoMap (default: true) - Automatically map CSV columns to schema fields
• skipErrors (default: false) - Skip rows with errors and continue processing
• maxErrors (default: undefined) - Maximum number of errors before stopping
• retryFailed (default: false) - Enable retry logic for failed rows

Retry Options

• maxRetries (default: 3) - Maximum number of retry attempts
• initialDelayMs (default: 1000) - Initial delay between retries
• maxDelayMs (default: 30000) - Maximum delay between retries
• backoffFactor (default: 2) - Exponential backoff multiplier

Error Codes

Common Error Codes

• REQUIRED_FIELD - Required field is missing or empty
• INVALID_EMAIL - Invalid email format
• INVALID_NUMBER - Invalid numeric value
• INVALID_DATE - Invalid date format
• VALUE_TOO_LONG - Value exceeds maximum length
• VALUE_TOO_SHORT - Value is below minimum length
• DUPLICATE_VALUE - Duplicate value found
• OUT_OF_RANGE - Value is outside acceptable range
• PARSING_ERROR - Error parsing CSV data
• SYSTEM_ERROR - System or processing error

Dependencies

This package orchestrates the following core packages:

• @bernierllc/csv-parser - CSV parsing functionality
• @bernierllc/csv-validator - Data validation
• @bernierllc/csv-mapper - Column mapping
• @bernierllc/file-handler - File operations
• @bernierllc/retry-policy - Retry logic

Performance

Benchmarks

• Small files (< 1MB) - < 5s processing time
• Large files (> 100MB) - < 300s processing time
• Memory usage - < 50MB for 1GB file
• Optimal batch size - 1000 rows per batch

Optimization Tips

1. Use appropriate batch sizes for your data volume
2. Enable auto-mapping to reduce processing time
3. Set reasonable error limits to avoid processing invalid data
4. Use retry logic only when necessary
5. Subscribe to progress updates for better user experience

See Also

• @bernierllc/csv-parser - Core CSV parsing functionality
• @bernierllc/csv-validator - Data validation utilities
• @bernierllc/csv-mapper - Column mapping utilities
• @bernierllc/file-handler - File handling utilities
• @bernierllc/retry-policy - Retry logic utilities

License

Copyright (c) 2025 Bernier LLC. All rights reserved.