smol-logs

A CLI tool to compress verbose text output (logs, test output, build logs) into a human-readable, AI-friendly format. Reduces token usage when working with AI tools by intelligently encoding repetition.

For AI Agents (Claude, etc.)

When running commands that produce verbose output, pipe through smol to reduce tokens:

npm test 2>&1 | smol
npm run build 2>&1 | smol
docker logs container_name 2>&1 | smol

When to use:

• Test runs with many similar log lines
• Build output with repeated warnings/errors
• Any command outputting >100 lines

Reading compressed output:

• ↑ repeated N times = previous line appeared N consecutive times
• ↑ appears N times = previous line appeared N times throughout (non-consecutive)
• "template with {var}" + ↑ pattern matched N times: var = a, b, c = N similar lines differing only in the variable parts

Installation

npm install -g smol-logs

Or run directly with npx:

npx smol-logs

Usage

# Pipe input from stdin
cat test.log | smol

# Read from file
smol test.log

# Write to file
smol test.log -o compressed.txt

# Adjust similarity threshold for pattern detection
smol test.log -t 0.9

# Exclude metadata header
smol test.log --no-meta

How It Works

smol uses three compression strategies:

1. Run-Length Encoding (RLE)

Collapses consecutive identical lines:

Input:

Loading config...
Loading config...
Loading config...
Server started

Output:

Loading config...
  ↑ repeated 3 times
Server started

2. Global Deduplication

Removes non-consecutive duplicate lines:

Input:

  console.log
    Message 1
  console.log
    Message 2
  console.log
    Message 3

Output:

  console.log
  ↑ appears 3 times
    Message 1
    Message 2
    Message 3

3. Pattern Templates

Detects near-duplicate lines and extracts variable parts:

Input:

[INFO] Server ready on port 8080
[INFO] Server ready on port 8081
[INFO] Server ready on port 8082

Output:

"[INFO] Server ready on port {port}"
  ↑ pattern matched 3 times: port = 8080, 8081, 8082

Output Format

=== COMPRESSION: 1247 → 89 lines (93%) ===

Loading config...
  ↑ repeated 3 times

Server started

"[INFO] Server ready on port {port}"
  ↑ pattern matched 5 times: port = 8080, 8081, 8082, 8083, 8084

"[ERROR] Connection failed: {error}"
  ↑ pattern matched 12 times: error = timeout, refused, reset, ...3 more
  ↑ time range: 10:15:06 - 10:17:23

Done

Format Elements

| Element | Syntax | Meaning | |---------|--------|---------| | Repeated line | line + ↑ repeated N times | Line appeared N consecutive times | | Deduplicated | line + ↑ appears N times | Line appeared N times (non-consecutive) | | Pattern | "template" + ↑ pattern matched | Near-duplicate lines with variables | | Unique line | just the line | Single occurrence |

CLI Options

| Option | Description | Default | |--------|-------------|---------| | -o, --output <file> | Output file (stdout if not provided) | - | | -t, --threshold <n> | Similarity threshold for patterns (0-1) | 0.8 | | --no-meta | Exclude compression stats header | false | | -p, --max-patterns <n> | Maximum patterns to extract | 10 | | -v, --verbose | Show compression statistics | false |

Examples

Compress Test Output

npm test 2>&1 | smol

Compress Build Logs

npm run build 2>&1 | smol -o build-summary.txt

Use with AI Tools

# Compress and copy to clipboard (macOS)
cat verbose.log | smol | pbcopy

# Compress and copy to clipboard (Windows)
type verbose.log | smol | clip

API Usage

You can also use the compressor programmatically:

import { compress, compressVerbose } from 'smol-logs';

const input = `
Loading...
Loading...
Server started
`;

// Basic compression
const output = compress(input);
console.log(output);

// With options
const output2 = compress(input, {
  threshold: 0.9,
  includeMeta: false,
});

// With statistics
const { output: result, stats } = compressVerbose(input);
console.log(`Compression: ${stats.totalLines} → ${stats.compressedItems} items`);

Development

# Clone the repository
git clone https://github.com/anthropics/smol-logs
cd smol-logs

# Install dependencies
npm install

# Run tests
npm test

# Build
npm run build

# Run in development
npm run dev sample.log

License

MIT