contextcalc

v1.4.4

Published

4 months ago

A fast CLI tool and library for analyzing codebase structure and token counts for LLM context management

0High
0Medium
0Low

gulivan

cli library api tokens llm context codebase analysis tokenizer

ContextCalc 📏

`tree` like CLI tool and library for token counting.

GitHub License NPM Unpacked Size NPM Version NPM Downloads

Screenshot

https://github.com/user-attachments/assets/3c9556b3-3876-46f7-9a44-707dd8c85000

ContextCalc 📏 - tree like CLI tool but with token counts.

Installation

npm install -g contextcalc
# or
npx contextcalc [path] [options]

Quick Start

# Tree view
npx contextcalc .

# Flat file list sorted by token count  
npx contextcalc . --output flat

# Flat file list sorted by token count (only .txt, .md and etc.)
npx contextcalc . --mode docs

# Tree with visual bars
npx contextcalc . --bars

Input Sources

ContextCalc supports multiple input sources for maximum flexibility:

Directory Analysis (default)

Analyze entire directories with hierarchical structure:

npx contextcalc ./src --depth 2
npx contextcalc . --output flat --min-tokens 1000 # Get the biggest files

Single File Analysis

Analyze individual files directly:

npx contextcalc README.md
npx contextcalc src/cli.ts --metrics tokens,lines,size

Stdin Pipe

Pipe content from other commands or files:

cat README.md | npx contextcalc
echo "Your text here" | npx contextcalc --metrics tokens,lines
git diff | npx contextcalc --metrics tokens --no-colors

Clipboard Content

Analyze content directly from your clipboard:

# Copy some text, then analyze it:
npx contextcalc --from-clipboard
npx contextcalc --from-clipboard --metrics tokens
npx contextcalc --from-clipboard --metrics size --no-colors

Output Formats

Tree View (default)

Shows hierarchical structure with absolute percentages:

npx contextcalc . --depth 2

 ~/git/contextcalc ╱ main  npx -y contextcalc@latest --depth 2 .                                                        ✔ ╱ 06:59:13 
. (23.4k tokens, 87.6KB) (100.0%)
└── src (12.8k tokens, 50.0KB) (54.5%)
    ├── src/formatters (4.0k tokens, 14.9KB) (17.0%)
    ├── src/core (2.9k tokens, 12.4KB) (12.5%)
    ├── src/cli.ts (2.8k tokens, 339 lines, 12.5KB) (12.1%)
    ├── src/utils (2.5k tokens, 8.5KB) (10.8%)
    ├── src/types (484 tokens, 1.8KB) (2.1%)
└── test (4.7k tokens, 17.4KB) (20.0%)
    ├── test/enhancedTreeFormatter.test.ts (2.0k tokens, 240 lines, 7.2KB) (8.7%)
    ├── test/cli.test.ts (1.8k tokens, 229 lines, 7.1KB) (7.8%)
    ├── test/fileDetector.test.ts (371 tokens, 33 lines, 1.5KB) (1.6%)
    ├── test/pathUtils.test.ts (223 tokens, 19 lines, 793B) (1.0%)
    ├── test/hasher.test.ts (218 tokens, 26 lines, 832B) (0.9%)
└── README.md (2.2k tokens, 283 lines, 7.8KB) (9.4%)
└── CHANGELOG.md (1.1k tokens, 62 lines, 2.9KB) (4.8%)
└── CLAUDE.md (747 tokens, 111 lines, 2.5KB) (3.2%)
└── .github (592 tokens, 2.2KB) (2.5%)
    ├── .github/workflows (592 tokens, 2.2KB) (2.5%)
└── package.json (454 tokens, 55 lines, 1.3KB) (1.9%)
└── eslint.config.mjs (281 tokens, 44 lines, 1.1KB) (1.2%)
└── LICENSE (221 tokens, 21 lines, 1.0KB) (0.9%)
└── tsconfig.json (193 tokens, 29 lines, 713B) (0.8%)
└── .releaserc.json (103 tokens, 14 lines, 413B) (0.4%)
└── .claude (46 tokens, 129B) (0.2%)
    ├── .claude/settings.local.json (46 tokens, 10 lines, 129B) (0.2%)
└── index.ts (8 tokens, 1 lines, 30B) (0.0%)

Summary: 23.4k tokens, 31 files, 87.6KB
Completed in 0.21s

Flat View

Lists all files sorted by token count (perfect for finding large files):

npx contextcalc . --output flat --min-tokens 500

src/formatters/enhancedTreeFormatter.ts 1.8k tokens, 226 lines, 6.5KB (13.1%)
src/core/scanner.ts 1.5k tokens, 226 lines, 6.7KB (11.5%)
src/cli.ts 1.2k tokens, 131 lines, 5.3KB (9.3%)
src/utils/fileDetector.ts 922 tokens, 123 lines, 3.0KB (6.7%)

JSON Output

Structured data for scripts and automation:

npx contextcalc . --output json

CSV Output

Export data in comma-separated values format for spreadsheets and data analysis:

npx contextcalc . --output csv

Path,Tokens,Lines,Size(bytes),Type,Percentage
src/cli.ts,2834,339,12504,ts,12.1
src/formatters/enhancedTreeFormatter.ts,1847,226,6498,ts,7.9
src/core/scanner.ts,1544,226,6706,ts,6.6
README.md,2208,283,7806,md,9.4
"Total",23421,1247,87604,"",100.0

AST Output (Code Symbol Extraction)

Extract and display code structure as symbols (functions, classes, interfaces, etc.) using tree-sitter AST parsing:

npx contextcalc . --output ast

src/core/scanner.ts (1750 tokens, 258 lines, 7.6KB)
├─ ← from "node:path" { join, relative } line 1
├─ C DirectoryScanner lines 11-258
│  ├─ v cache line 12
│  ├─ v tokenizer line 13
│  ├─ ƒ constructor(projectPath: string, mode: AnalysisMode, ...): void lines 19-32
│  ├─ ƒ async initialize(useGitignore: boolean, ...): Promise<void> lines 34-45
│  ├─ ƒ async scan(): Promise<ScanResult> lines 47-69
│  └─ ƒ static calculatePercentages(nodes: Node[]): Node[] lines 237-257
└─ c CACHE_VERSION line 7

Found 326 symbols across 16 files

Supported Languages:

| Language | Extensions | Functions | Classes | Interfaces | Types | Enums | Imports/Exports | |----------|-----------|-----------|---------|------------|-------|-------|-----------------| | TypeScript | .ts, .tsx | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | JavaScript | .js, .jsx, .mjs, .cjs | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | | Python | .py, .pyi | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |

Notes:

JavaScript supports all symbol types except interfaces, types, and enums (not part of JS spec)
Python supports functions, classes, and imports; interfaces/types/enums are not native Python constructs
All languages support nested symbols (e.g., class methods, nested functions)

Symbol Icons:

ƒ Function/Method - with parameters, return types, async/generator flags
C Class - with extends, implements, abstract modifiers
I Interface - with extends and members
T Type - type aliases and definitions
E Enum - with member values
v Variable - with type annotations
c Constant - with type annotations
← Import - showing source and imported items
→ Export - showing exported items

Features:

Performance: AST results are cached - subsequent runs are fast
Selective: Only enabled with --output ast flag
Smart: Respects --max-size limit to prevent OOM on large files
Compatible: Works with all existing filters (--mode, --depth, --min-tokens)

Examples:

# Analyze TypeScript/JavaScript project structure
npx contextcalc src --output ast --mode code

# Find all classes and functions (depth 2 to see class members)
npx contextcalc . --output ast --depth 2

# Analyze Python codebase
npx contextcalc . --output ast --mode code

Options Reference

| Option | Description | Default | |--------|-------------|---------| | -o, --output <format> | Output format: tree, flat, json, csv, ast | tree | | --mode <mode> | Files to analyze: all, code, docs | all | | --max-size <size> | Maximum file size to analyze (e.g., 10M, 500k) | 10M | | --sort <by> | Sort by: tokens, size, name | tokens | | --depth <n> | Tree depth levels (0=root only, 1=root+children, etc.) | unlimited | | --min-tokens <n> | Hide files with fewer than n tokens | 0 | | --metrics <list> | Metrics to display: tokens,lines,size,percentage | tokens,lines,size,percentage | | --percentages | Show absolute percentages (enabled by default) | true | | --no-percentages | Disable percentage display | | | --relative-percentages | Show percentages relative to parent directory | false | | --bars | Show visual weight bars | false | | --no-colors | Disable colored output | | | --no-gitignore | Ignore .gitignore files | | | --no-default-ignores | Disable default ignore patterns | | | --from-clipboard | Read content from system clipboard | |

Usage Examples

Basic Usage

# Analyze current directory
npx contextcalc .

# Analyze specific directory
npx contextcalc ./src

# Analyze single file
npx contextcalc README.md

# Get help
npx contextcalc --help

File Filtering

# Only analyze code files (.js, .ts, .py, etc.)
npx contextcalc . --mode code

# Only analyze documentation files (.md, .txt, .rst, etc.)
npx contextcalc . --mode docs

# Limit by file size (skip large files)
npx contextcalc . --max-size 1M

# Hide small files (less than 100 tokens)
npx contextcalc . --min-tokens 100

# Ignore .gitignore patterns
npx contextcalc . --no-gitignore

# Disable all default ignore patterns
npx contextcalc . --no-default-ignores

Output Customization

# Flat list sorted by token count
npx contextcalc . --output flat

# Tree view with visual bars
npx contextcalc . --bars

# Limit tree depth
npx contextcalc . --depth 2

# Sort by file size instead of tokens
npx contextcalc . --sort size

# Sort alphabetically
npx contextcalc . --sort name

# Show only specific metrics
npx contextcalc . --metrics tokens,lines

# Clean output for scripts
npx contextcalc . --no-colors --no-percentages

# Relative percentages (% of parent directory)
npx contextcalc . --relative-percentages

Performance & Analysis

# Find the largest files in your project
npx contextcalc . --output flat --min-tokens 1000

# Export analysis data for further processing
npx contextcalc . --output json > analysis.json
npx contextcalc . --output csv > analysis.csv

# Quick overview of just top-level directories
npx contextcalc . --depth 1

# Focus on heavyweight files only
npx contextcalc . --output flat --min-tokens 2000 --sort tokens

Input Source Comparison

# All these should give the same token count for the same content:

# File analysis
npx contextcalc README.md --metrics tokens

# Stdin pipe
cat README.md | npx contextcalc --metrics tokens

# Copy README.md content to clipboard, then:
npx contextcalc --from-clipboard --metrics tokens

Advanced Workflows

# Git workflow: analyze staged changes
git diff --cached | npx contextcalc --metrics tokens --no-colors

# Find files that exceed token limits for LLM context
npx contextcalc . --output flat --min-tokens 4000

# Analyze documentation structure
npx contextcalc ./docs --mode docs --bars --depth 3

# Compare token usage between branches
git show main:README.md | npx contextcalc --metrics tokens
git show feature-branch:README.md | npx contextcalc --metrics tokens

# Process multiple files through stdin
find . -name "*.md" -exec cat {} \; | npx contextcalc --metrics tokens,lines

# Generate reports for different file types
npx contextcalc . --mode code --output json > code-analysis.json
npx contextcalc . --mode docs --output csv > docs-analysis.csv

# Clean analysis for CI/CD pipelines
npx contextcalc . --output flat --no-colors --min-tokens 500 | head -10

Library Usage

ContextCalc can be imported and used as a library in your Node.js applications for programmatic token counting.

⚠️ Bundling Notice: If you're bundling an application that uses contextcalc, mark tiktoken as external in your bundler configuration to avoid WebAssembly loading issues. See DOCS.md for detailed guidance.

Installation as Library

npm install contextcalc
# or
yarn add contextcalc
# or
bun add contextcalc

📖 Complete API Reference: See DOCS.md for detailed documentation, advanced examples, and integration guides.

Basic API

import { countTokens, countTokensWithOptions } from 'contextcalc';

// Simple token counting
const tokens = countTokens("Hello, world!");
console.log(tokens); // 4

// Works with any data type
const jsonTokens = countTokens({ message: "Hello", data: [1, 2, 3] });
console.log(jsonTokens); // 24

const bufferTokens = countTokens(Buffer.from("Hello"));
console.log(bufferTokens); // 2

// Advanced options
const result = countTokensWithOptions("Line 1\nLine 2", {
  includeLines: true,
  format: 'formatted'
});
console.log(result); // { tokens: 6, lines: 2, formatted: "6" }

Common Use Cases

import { 
  countTokens,
  countTokensBatch,
  countTokensFromFile,
  estimateTokens
} from 'contextcalc';

// Batch processing multiple inputs
const inputs = ["Hello", "World", { key: "value" }];
const results = countTokensBatch(inputs);
console.log(results); // [2, 1, 8]

// File processing
const fileTokens = await countTokensFromFile('./README.md');
console.log(fileTokens); // 1247

// Quick estimation for large datasets
const estimate = estimateTokens("Large text content...");
console.log(estimate); // Fast approximate count

Function Reference

| Function | Purpose | Example | |----------|---------|---------| | countTokens(input) | Basic token counting | countTokens("Hello") → 2 | | countTokensBatch(inputs) | Multiple inputs efficiently | countTokensBatch(["a", "b"]) → [1, 1] | | countTokensFromFile(path) | File processing | await countTokensFromFile("file.txt") | | estimateTokens(input) | Quick estimation | estimateTokens("text") → 1 | | countWithLines(input) | Include line count | countWithLines("Hi\nBye") → {tokens: 3, lines: 2} |

TypeScript Support

import { 
  countTokens,
  type TokenInput,
  type TokenCountResult 
} from 'contextcalc';

const input: TokenInput = "Hello, TypeScript!";
const tokens: number = countTokens(input);

License

MIT