CSV Redis Loader

A powerful Node.js package that allows you to load CSV data into Redis with customizable key-value formation, partitioning, and locking mechanisms.

Installation

npm install csv-redis-loader

Repository

GitHub: https://github.com/punnayansaha07/csv-redis-loader

Features

• 🔄 Load CSV files into Redis with customizable parsing rules
• 🔑 Flexible key-value formation based on user-defined columns
• 🔒 Built-in locking mechanism to prevent concurrent loads
• 📦 Support for data partitioning
• 🚀 High-performance batch processing
• 🔍 Easy data retrieval with flexible querying options

Quick Start

import CSVRedisLoader from 'csv-redis-loader';

// Create a loader instance with default configuration
const loader = new CSVRedisLoader();

// Load a CSV file into Redis
await loader.loadCSV('path/to/your/file.csv');

// Get a value using key parts
const value = await loader.getValue(['ID', 'Name'], 'path/to/your/file.csv');

Configuration Options

The loader can be configured with various options:

const loader = new CSVRedisLoader({
  // Redis connection configuration
  redisConfig: {
    host: 'localhost',
    port: 6379,
    password: 'your-password',
    db: 0
  },
  
  // CSV parsing configuration
  parserConfig: {
    delimiter: ',',
    columns: true,
    skip_empty_lines: true
  },
  
  // Key-value formation configuration
  keyValueConfig: {
    keyColumns: ['ID', 'Name'],     // Columns to use for key formation
    valueColumns: ['Salary'],       // Columns to use for value formation
    keyDelimiter: ':',              // Delimiter for key columns
    valueDelimiter: ',',            // Delimiter for value columns
    partition: 'employees'          // Optional partition name
  },
  
  keyPrefix: 'csv:'                 // Prefix for all Redis keys
});

Usage Examples

1. Basic Usage

import CSVRedisLoader from 'csv-redis-loader';

const loader = new CSVRedisLoader();

// Load CSV file
await loader.loadCSV('employees.csv');

// Get value using string key
const employee = await loader.getValue('1:John', 'employees.csv');

// Get value using array of key parts
const salary = await loader.getValue(['1', 'John'], 'employees.csv');

2. Custom Key-Value Formation

const loader = new CSVRedisLoader({
  keyValueConfig: {
    keyColumns: ['ID', 'Department'],
    valueColumns: ['Salary', 'Email'],
    keyDelimiter: '-',
    partition: 'employees'
  }
});

// Load CSV file
await loader.loadCSV('employees.csv');

// Get specific fields
const employeeData = await loader.getValue(['1', 'Engineering'], 'employees.csv');
// Returns: { Salary: '75000', Email: '[email protected]' }

3. Working with Partitions

const loader = new CSVRedisLoader({
  keyValueConfig: {
    keyColumns: ['ID'],
    valueColumns: ['Name', 'Salary'],
    partition: '2023'
  }
});

// Load multiple files with different partitions
await loader.loadCSV('2023/employees.csv');
await loader.loadCSV('2023/departments.csv');

// Get values from specific partition
const employee = await loader.getValue('1', '2023/employees.csv');

API Reference

getValue Formats

The getValue method supports multiple formats for retrieving data:

1. Using String Key

// Single string key
const value = await loader.getValue('1:John', 'employees.csv');
// Returns: { Salary: '75000', Email: '[email protected]' }

// With custom delimiter
const loader = new CSVRedisLoader({
  keyValueConfig: {
    keyColumns: ['ID', 'Name'],
    keyDelimiter: '-'
  }
});
const value = await loader.getValue('1-John', 'employees.csv');

2. Using Array of Key Parts

// Array of key parts
const value = await loader.getValue(['1', 'John'], 'employees.csv');
// Returns: { Salary: '75000', Email: '[email protected]' }

// With multiple key columns
const loader = new CSVRedisLoader({
  keyValueConfig: {
    keyColumns: ['ID', 'Department', 'Name']
  }
});
const value = await loader.getValue(['1', 'Engineering', 'John'], 'employees.csv');

3. Single Value Column

// Returns just the salary as a string
const loader = new CSVRedisLoader({
  keyValueConfig: {
    keyColumns: ['ID'],
    valueColumns: ['Salary']
  }
});
const salary = await loader.getValue('1', 'employees.csv');
// Returns: "75000"

4. Multiple Value Columns

// Returns an object with specified value columns
const loader = new CSVRedisLoader({
  keyValueConfig: {
    keyColumns: ['ID'],
    valueColumns: ['Name', 'Salary', 'Email']
  }
});
const employee = await loader.getValue('1', 'employees.csv');
// Returns: { Name: 'John', Salary: '75000', Email: '[email protected]' }

5. With Partition

// Using partition in key formation
const loader = new CSVRedisLoader({
  keyValueConfig: {
    keyColumns: ['ID'],
    valueColumns: ['Salary'],
    partition: '2023'
  }
});
const salary = await loader.getValue('1', '2023/employees.csv');
// Returns: "75000"

6. No Value Columns Specified

// Returns all columns as an object
const loader = new CSVRedisLoader({
  keyValueConfig: {
    keyColumns: ['ID']
  }
});
const employee = await loader.getValue('1', 'employees.csv');
// Returns: { ID: '1', Name: 'John', Department: 'Engineering', Email: '[email protected]', Salary: '75000' }

Note: The return type will be:

• string when only one value column is specified
• Record<string, string> when multiple value columns are specified or no value columns are specified
• null when the key doesn't exist

loadCSV(filePath: string): Promise<void>

Loads a CSV file into Redis. Uses a locking mechanism to prevent concurrent loads.

getAllValues(filePath: string): Promise<Record<string, Record<string, string>>>

Retrieves all values for a given file.

clearAll(filePath: string): Promise<void>

Clears all data for a given file from Redis.

isLocked(filePath: string): Promise<boolean>

Checks if a file is currently being loaded.

CSV File Format

The package expects CSV files with headers. Example:

ID,Name,Department,Email,Salary
1,John,Engineering,[email protected],75000
2,Jane,Marketing,[email protected],65000

Error Handling

The package throws errors in the following cases:

• Failed to acquire lock for loading
• Invalid file path
• Redis connection issues
• Invalid key formation

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT