prompt-compressor

Rule-based prompt compression — reduce LLM token usage without a model, without an API, without sending data anywhere.

Install

npm install prompt-compressor

Quick start

import { compress } from 'prompt-compressor';

const { output, stats } = compress(systemPrompt, 'low');

console.log(stats);
// { tokensBefore: 1200, tokensAfter: 820, saved: 380, reductionPct: 31 }

Risk levels

Pick a level — every algorithm at or below that risk runs automatically.

| Level | What runs | Typical saving | |---|---|---| | 'none' | Whitespace, unicode normalization, indentation | 5–15% | | 'very-low' | + Filler removal, phrase shortening, JSON/SQL minify | 15–30% | | 'low' | + Hedging removal, code comments, deduplication, stack trace truncation | 25–45% | | 'medium' | + Fuzzy dedup, TF-IDF, TextRank (may drop sentences) | 30–70% |

compress(text, 'none')      // safest
compress(text, 'very-low')
compress(text, 'low')       // recommended default
compress(text, 'medium')    // most aggressive

Custom options

Override specific algorithms or parameters:

const { output, stats } = compress(text, {
  level: 'low',         // base level

  // enable something above your level
  tfidf: true,
  tfidfRatio: 0.7,      // keep 70% of sentences

  // disable something in your level
  codeComments: false,  // keep code comments intact
});

In a MCP server

import { compress } from 'prompt-compressor';

// Before every Bedrock call:
const { output } = compress(systemPrompt, 'low');

const response = await bedrock.invokeModel({
  body: JSON.stringify({ system: output, messages }),
  ...
});

Individual functions

All algorithms are exported individually:

import {
  removeFiller,
  tfidfCompress,
  deduplicateLines,
  fuzzyDeduplicate,
  estimateTokens,
} from 'prompt-compressor';

const cleaned = removeFiller(text);
const compressed = tfidfCompress(cleaned, 0.7);
console.log(estimateTokens(compressed));

Variable protection

Prompt variables are automatically detected and preserved — compression never breaks your templates.

Supported: {{var}}, {var}, ${var}, <tag>, [[var]], %VAR%

// This is safe:
compress("Please {{user_name}}, utilize this in order to process {context}", 'low');
// → "Please {{user_name}}, use this to process {context}"
// Variables untouched, filler compressed.

Result shape

{
  output: string,
  stats: {
    tokensBefore: number,
    tokensAfter: number,
    saved: number,
    reductionPct: number,   // 0–100
  }
}

Algorithms reference

| Algorithm | Risk | What it does | |---|---|---| | compressWhitespace | none | Collapse spaces, tabs, excess blank lines | | normalizeUnicode | none | Smart quotes, em-dash, zero-width chars | | cleanPunctuation | none | !!! → !, ??? → ? | | stripLicenseHeaders | none | Remove copyright blocks from code | | normalizeIndentation | none | 4-space → 2-space in code blocks | | removeFiller | very-low | Remove "please note that", "e.g.", "i.e." clauses | | shortenPhrases | very-low | "in order to" → "to", "utilize" → "use" (50+ rules) | | numberWordsToNumerals | very-low | "twenty" → "20" | | normalizeDates | very-low | "Monday, Dec 25th 2023" → "2023-12-25" | | minifyJSON | very-low | Minify JSON code blocks | | minifySQL | very-low | Remove SQL comments, collapse whitespace | | removeHedging | low | Remove "perhaps", "it seems", "one might say" | | removeMetaCommentary | low | Remove "Here is...", "I hope this helps" | | removeParentheticals | low | Remove "(as mentioned above)", "(see below)" | | stripCodeComments | low | Remove //, /* */, # comments | | stripDocstrings | low | Remove Python/JSDoc documentation blocks | | removeNullJsonFields | low | Strip null/empty fields from JSON | | stripBase64AndHex | low | Replace long hashes with placeholders | | truncateStackTraces | low | Keep 4 frames, replace rest with count | | deduplicateLines | low | Remove duplicate lines | | deduplicateSentences | low | Remove duplicate sentences | | fuzzyDeduplicate | medium | Jaccard-based near-duplicate removal | | ngramDeduplicate | medium | Bigram/trigram near-duplicate removal | | levenshteinDeduplicate | medium | Edit-distance near-duplicate removal | | tfidfCompress | medium | Keep top sentences by TF-IDF score | | textRankCompress | medium | Keep top sentences by PageRank centrality | | extractiveCompress | medium | Keep top sentences by Term Frequency |

License

MIT