LLMTrim

A library for trimming tokens in encoding and decoding in LLM (Large Language Model) applications.
It provides utilities to compress text by removing stopwords, punctuation, spaces, and applying stemming, with the ability to reverse the process as much as possible.

Features

• Token trimming: Remove stopwords, punctuation, and spaces from text.
• Stemming: Supports Porter and Lancaster stemmers via natural.
• Negation preservation: Keeps negation words even when removing stopwords.
• Customizable: Options for language, stemming, and what to remove.
• Decoder: Attempts to reconstruct trimmed text.

Installation

npm install llmtrim

Usage

Encoder

import { Encoder } from 'llmtrim';

const encoder = new Encoder();
const text = "The quick brown fox jumps over the lazy dog.";
const trimmed = encoder.trim(text, {
  removeSpaces: true,
  removeStopwords: true,
  removePunctuation: true,
  stemmer: 'porter'
});
console.log(trimmed); // Output: quickbrownfoxjumpsoverlazydog

Decoder

import { Decoder } from 'llmtrim';

const decoder = new Decoder();
const original = decoder.detrim(trimmed, {
  removeSpaces: true,
  removeStopwords: true,
  removePunctuation: true,
  stemmer: 'porter'
});
console.log(original);
// Output: The quick brown fox jumps over the lazy dog.

Options

| Option | Type | Description | |---------------------|--------------------------------------|---------------------------------------------------------------------------------------------| | stemmer | 'porter' | 'lancaster' | Selects the stemming algorithm to use. | | language | 'english' | Sets the language for stopword removal (currently only English is supported). | | removeSpaces | boolean | If true, removes all spaces from the text. | | removeStopwords | boolean | If true, removes common stopwords (e.g., "the", "and") from the text. | | removePunctuation | boolean | If true, removes punctuation characters from the text. |

Testing

Run the tests using:

npm test

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

License

This project is licensed under the MIT License. See the LICENSE file for details.