LLM-Data-Tool

A powerful and lightweight TypeScript utility library for comprehensive data manipulation and processing, especially useful for handling data from Large Language Models (LLMs).

Features

• Data Formatting: formatData - Shape raw data according to a defined schema.
• Data Cleaning: cleanData - Remove unwanted fields from your data.
• Deep Cleaning: deepClean - Recursively remove null and undefined values.
• Data Validation: validateData - Validate data against a schema.
• Data Transformation: transformData - Convert data to different formats like JSON or a formatted string.
• Data Mapping: mapData - Remap keys in your data.
• Type Casting: castTypes - Convert data fields to specified types (number, boolean, string).
• Object Flattening: flattenObject - Flatten nested objects into a single level.
• Object Unflattening: unflattenObject - Restore a flattened object to its original nested structure.
• HTML Stripping: stripHtml - Remove HTML tags from a string.
• Markdown Conversion: markdownToHtml - Convert Markdown to HTML.
• Markdown Stripping: stripMarkdown - Remove Markdown formatting.
• Data Aggregation: aggregateData - Perform sum, average, or count operations on an array of objects.
• Data Grouping: groupBy - Group an array of objects by a specific key.
• Data Sorting: sortData - Sort an array of objects by a key in ascending or descending order.

Installation

npm install llm-data-tool

Usage

import LLMDataTool from 'llm-data-tool';

const tool = new LLMDataTool();

// Example: Formatting and Cleaning Data
const rawData = { id: 1, name: '  John Doe  ', email: '[email protected]', extra: 'garbage' };
const schema = { id: 'number', name: 'string', email: 'string' };

const formattedData = tool.formatData(rawData, schema);
// => { id: 1, name: '  John Doe  ', email: '[email protected]' }

const cleanedData = tool.cleanData(formattedData, ['extra']);
// => { id: 1, name: '  John Doe  ', email: '[email protected]' }


// Example: Aggregating Data
const sales = [
  { product: 'A', amount: 100 },
  { product: 'B', amount: 150 },
  { product: 'A', amount: 50 },
];

const totalAmount = tool.aggregateData(sales, 'amount', 'sum');
// => 300

// Example: Grouping Data
const groupedByProduct = tool.groupBy(sales, 'product');
// => {
//      'A': [ { product: 'A', amount: 100 }, { product: 'A', amount: 50 } ],
//      'B': [ { product: 'B', amount: 150 } ]
//    }

API Reference

formatData(rawData: Data, schema: Schema): Data

Formats raw data according to the provided schema, only keeping keys present in the schema.

cleanData(data: Data, rules: string[]): Data

Removes keys from the data object based on the provided rules (an array of keys to remove).

deepClean(data: any): any

Recursively removes all null and undefined properties from an object or elements from an array.

validateData(data: Data, schema: Schema): boolean

Validates if the data conforms to the schema, checking for key existence and basic type matching.

transformData(data: Data, targetFormat: 'json' | 'string'): any

Transforms the data into either a JSON string or a simple key-value string representation.

mapData(data: Data, mapping: { [newKey: string]: string }): Data

Maps data from old keys to new keys.

castTypes(data: Data, schema: { [key: string]: string }): Data

Casts values in the data object to the types defined in the schema (e.g., 'number', 'boolean').

flattenObject(obj: Data): Data

Flattens a nested object into a single-level object with dot-separated keys.

unflattenObject(data: Data): Data

Converts a flattened object back into a nested object.

stripHtml(html: string): string

Removes all HTML tags from a string.

markdownToHtml(markdown: string): string

Converts basic Markdown syntax (headings, bold, italic, lists) to HTML.

stripMarkdown(markdown: string): string

Removes Markdown formatting syntax from a string.

aggregateData(data: Data[], key: string, operation: 'sum' | 'average' | 'count'): number

Performs an aggregation operation on an array of objects.

groupBy(data: Data[], key: string): { [key: string]: Data[] }

Groups an array of objects based on the value of a specified key.

sortData(data: Data[], key: string, order: 'asc' | 'desc' = 'asc'): Data[]

Sorts an array of objects by a given key.

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

License

ISC