ChunkNorris.js

ChunkNorris.js maps CSV's into a specified structure using an efficient workflow to make uploading CSV's to a server painless and reliable. It was designed to handle very large CSV files with millions of rows quickly and efficiently. The library does not validate the data in files, just that a file structure conforms to a defined schema.

How it works

ChunkNorris.js has two public functions. One to create a dialog UI and another that executes batches of promises concurrently. The library is fully typed and written using Solid.js. By default chunks are a maximum of 500KB, but can be configured as desired.

Installation

NPM

npm i chunk-norris

PNPM

pnpm i chunk-norris

Getting Started

Create a dialog element and mount it to a DOM element.

import { createDialog, type Field } from 'chunk-norris';

// Create the fields schema, which determines how a data file should be mapped.
const fields: Field[] = [{
    label: 'Email',
    key: 'email',
    alternates: [
        'email_address',
        'Email Address'
    ],
    validators: [{
        validate: 'required'
    }]
},{
    label: 'First',
    key: 'first',
    alternates: [
        'first_name',
        'First Name'
    ],
    validators: [{
        validate: 'required_with',
        fields: ['last']
    }]
},{
    label: 'Last',
    key: 'last',
    alternates: [
        'last_name',
        'Last Name'
    ],
    validators: [{
        validate: 'required_with',
        fields: ['first']
    }]
}]

// Create a dialog element that provides the UI to map the file to the schema.
const { mount, toggle } = createDialog({
    fields,
    open: true, // Open the dialog on mount. 
    onComplete(data, meta) {
        // Do something after the dialog completes its process.
        console.log(chunks, meta);
    }
});

// Mount the element to the DOM.
mount(document.getElementById('chunk-norris') as Element);

// Bind a click event to a button that toggles opening and closing the dialog.
document.getElementById('toggle')?.addEventListener('click', () => toggle());

Creating a Task Runner

ChunkNorris.js provides a task runner that executes promises concurrently.

Assume chunks has been set by the onComplete callback and you are ready to send the data to the server. In this example, create a task runner to send the chunks with 5 concurrent requests that retries up to 2 times if the server fails.

import { createRunner } from 'chunk-norris';

// Create a task runner function.
const run = createRunner(
    chunks.map(chunk => () => fetch('https://your-api-endpoint', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify(chunk),
    })
), 5, 2);

// Execute the function.
run({
    onProgress(result, percent) {
        console.log('onProgress', percent)
    },
    onError(error) {
        console.log('onError', error)
    },
    onFinish(success) {
        console.log('onFinish', success)
    }
})