Why tokenlean?

AI coding assistants are powerful, but they have a fundamental constraint: context windows. Every file read, every search result consumes tokens. This matters because:

| Problem | Impact | |-------------|---------------------------------------| | Cost | More tokens = higher API costs | | Quality | Overstuffed context = worse responses | | Speed | Larger contexts = longer processing | | Limits | Hit the ceiling = lost information |

tokenlean provides 39 specialized CLI tools that give you exactly the information needed — no more, no less.

Instead of reading a 500-line file    →  tl-exports (~50 tokens)
Instead of reading all type files     →  tl-types (~100 tokens)
Instead of guessing impact            →  tl-impact (know for sure)

Installation

npm install -g tokenlean

git clone https://github.com/edimuj/tokenlean.git
cd tokenlean
npm link

Requirements

• Node.js >= 18.0.0
• ripgrep (rg) — for search-based tools
• git — for history/blame/diff tools

Quick Start

# Get project overview
tl-structure

# Check file sizes before reading
tl-context src/api/

# Get function signatures without bodies
tl-symbols src/utils.ts

# Understand what a module exports
tl-exports src/lib/

# Check what would break if you change a file
tl-impact src/core/auth.ts

AI Agent Integration

Add tokenlean instructions to your AI tool's config:

| AI Tool | Config File | Command | |----------------|-----------------------------------|------------------------------------------------| | Claude Code | CLAUDE.md | tl-prompt >> CLAUDE.md | | Cursor | .cursorrules | tl-prompt --minimal >> .cursorrules | | GitHub Copilot | .github/copilot-instructions.md | tl-prompt >> .github/copilot-instructions.md | | Windsurf | .windsurfrules | tl-prompt --minimal >> .windsurfrules |

The --minimal flag produces a compact version that uses fewer tokens.

Tools Reference

Before Reading Files

Understand code structure without reading full implementations.

| Tool | Description | Example | |----------------|------------------------------------------|---------------------------| | tl-structure | Project overview with token estimates | tl-structure src/ | | tl-context | Estimate token usage for files/dirs | tl-context src/api/ | | tl-symbols | Function/class signatures without bodies | tl-symbols src/utils.ts | | tl-types | Full TypeScript type definitions | tl-types src/types/ | | tl-exports | Public API surface of a module | tl-exports src/lib/ | | tl-component | React component analyzer (props, hooks) | tl-component Button.tsx | | tl-docs | Extract JSDoc/TSDoc documentation | tl-docs src/utils/ | | tl-entry | Find entry points and main files | tl-entry src/ | | tl-schema | Extract DB schema from ORMs/migrations | tl-schema | | tl-stack | Auto-detect project technology stack | tl-stack |

Before Modifying Files

Understand dependencies and impact before making changes.

| Tool | Description | Example | |-----------------|---------------------------------------------|-------------------------------------| | tl-impact | Blast radius — what depends on this file | tl-impact src/auth.ts | | tl-deps | Show what a file imports (with tree mode) | tl-deps src/api.ts --tree | | tl-related | Find tests, types, and importers | tl-related src/Button.tsx | | tl-flow | Call graph — what calls this, what it calls | tl-flow src/utils.ts | | tl-coverage | Test coverage info for files | tl-coverage src/ | | tl-complexity | Code complexity metrics | tl-complexity src/ --threshold 10 | | tl-style | Detect coding conventions from code | tl-style src/ |

Understanding History

Track changes and authorship efficiently.

| Tool | Description | Example | |----------------|----------------------------------|--------------------------| | tl-diff | Token-efficient git diff summary | tl-diff --staged | | tl-history | Recent commits for a file | tl-history src/api.ts | | tl-blame | Compact per-line authorship | tl-blame src/api.ts | | tl-hotspots | Frequently changed files (churn) | tl-hotspots --days 30 | | tl-pr | Summarize PR/branch for review | tl-pr feature-branch | | tl-changelog | Generate changelog from commits | tl-changelog --from v1 |

Finding Things

Search and discover code patterns.

| Tool | Description | Example | |--------------|------------------------------------|--------------------------| | tl-example | Find diverse usage examples | tl-example useAuth | | tl-search | Run pre-defined search patterns | tl-search hooks | | tl-secrets | Find hardcoded secrets & API keys | tl-secrets --staged | | tl-todo | Find TODOs/FIXMEs in codebase | tl-todo src/ | | tl-env | Find environment variables used | tl-env --required-only | | tl-unused | Find unused exports/files | tl-unused src/ | | tl-api | Extract REST/GraphQL endpoints | tl-api src/routes/ | | tl-routes | Extract routes from web frameworks | tl-routes app/ | | tl-npm | Quick npm package lookup/compare | tl-npm express fastify |

Utilities

| Tool | Description | Example | |-----------------|------------------------------------------|-----------------------------| | tl-cache | Manage ripgrep result cache | tl-cache stats | | tl-config | Show/manage configuration | tl-config --init | | tl-context7 | Look up library docs via Context7 API | tl-context7 react "hooks" | | tl-name | Check name availability (npm/GH/domains) | tl-name myproject -s | | tl-npm | Quick npm package lookup/compare | tl-npm express fastify | | tl-playwright | Headless browser content extraction | tl-playwright example.com | | tl-prompt | Generate AI agent instructions | tl-prompt --minimal | | tl-run | Smart command runner with summaries | tl-run "npm test" |

Common Options

All tools support these flags:

-l N, --max-lines N    Limit output to N lines
-t N, --max-tokens N   Limit output to ~N tokens
-j, --json             Output as JSON (for piping)
-q, --quiet            Minimal output (no headers/stats)
-h, --help             Show help

Configuration

Create .tokenleanrc.json in your project root or ~/.tokenleanrc.json globally:

{
  "output": {
    "maxLines": 100,
    "maxTokens": null
  },
  "skipDirs": [
    "generated",
    "vendor"
  ],
  "skipExtensions": [
    ".gen.ts"
  ],
  "importantDirs": [
    "domain",
    "core"
  ],
  "importantFiles": [
    "ARCHITECTURE.md"
  ],
  "searchPatterns": {
    "hooks": {
      "description": "Find React hooks",
      "pattern": "use[A-Z]\\w+",
      "glob": "**/*.{ts,tsx}"
    }
  }
}

{
  "output": {
    "maxLines": 100,
    "maxTokens": null
  },
  "skipDirs": [
    "generated",
    "vendor"
  ],
  "skipExtensions": [
    ".gen.ts"
  ],
  "importantDirs": [
    "domain",
    "core"
  ],
  "importantFiles": [
    "ARCHITECTURE.md"
  ],
  "searchPatterns": {
    "hooks": {
      "description": "Find React hooks",
      "pattern": "use[A-Z]\\w+",
      "glob": "**/*.{ts,tsx}"
    }
  },
  "hotspots": {
    "days": 90
  },
  "structure": {
    "depth": 3
  },
  "cache": {
    "enabled": true,
    "ttl": 300,
    "maxSize": "100MB",
    "location": null
  }
}

Config values extend built-in defaults (they don't replace them).

Caching

tokenlean caches expensive ripgrep operations with git-based invalidation — automatically invalidates on commits or file changes.

tl-cache stats      # View cache statistics
tl-cache clear      # Clear cache for current project
tl-cache clear-all  # Clear all cached data

Disable with TOKENLEAN_CACHE=0 or in config: {"cache":{"enabled":false}}

Example Workflows

tl-structure                    # Get the lay of the land
tl-entry                        # Find entry points
tl-exports src/lib/             # Understand the public API
tl-docs src/utils/              # Read documentation, not code
tl-types src/types/             # Understand data shapes
tl-schema                       # Understand the database

tl-impact src/core/auth.ts      # What would break?
tl-deps src/core/auth.ts        # What does it depend on?
tl-related src/core/auth.ts     # Find the tests
tl-coverage src/core/auth.ts    # Is it well tested?
tl-complexity src/core/auth.ts  # How complex is it?

tl-component src/Button.tsx     # Props, hooks, dependencies
tl-symbols src/Button.tsx       # Function signatures
tl-history src/Button.tsx       # Recent changes
tl-blame src/Button.tsx         # Who wrote what

tl-complexity src/ --threshold 15  # Complex functions
tl-unused src/                     # Dead code
tl-todo                            # Outstanding TODOs
tl-hotspots                        # Frequently changed (unstable?)

tl-secrets                         # Scan for hardcoded secrets
tl-secrets --staged                # Only check staged files
tl-secrets --min-severity high     # Only high severity issues

tl-pr feature-branch               # Summary of branch changes
tl-pr 123                          # GitHub PR #123 (needs gh CLI)
tl-pr --full                       # Include files, stats, commits

tl-changelog --unreleased          # What's new since last tag
tl-changelog v0.1.0..v0.2.0        # Between versions
tl-changelog --format compact      # Quick summary

tl-name coolproject awesomelib     # Check npm, GitHub, domains
tl-name myapp -s                   # Suggest variations if taken
tl-npm express fastify koa         # Compare framework options

tl-run "npm test"                  # Summarize test results
tl-run "npm run build"             # Extract build errors only
tl-run "eslint src/"               # Summarize lint violations
tl-run "npm test" -j               # Structured JSON output

tl-context7 react "useEffect"      # Look up React docs
tl-context7 nextjs "app router"    # Next.js docs
tl-context7 express -s             # Search for library
tl-npm lodash --deps               # Check package dependencies
tl-npm chalk --versions            # Version history

tl-playwright example.com                 # Extract page text
tl-playwright example.com -s "h1,h2,h3"  # Extract headings only
tl-playwright example.com --screenshot p  # Save screenshot
tl-playwright example.com --eval "title"  # Evaluate JS expression

Design Principles

1. Single purpose — Each tool does one thing well
2. Minimal output — Show only what's needed
3. Token-conscious — Every tool saves context tokens
4. Composable — Tools work together with JSON output for piping
5. Fast — No heavy parsing or external services
6. Universal — Works with JS/TS projects, most tools support Python/Go too

Other tools for Claude Code

| Project | Description | |------------------------------------------------------------------------|--------------------------------------------------------------------------------| | claude-mneme | Persistent memory for Claude Code — remember context across sessions | | claude-simple-status | Minimal statusline showing model, context usage, and quota | | vexscan-claude-code | Security scanner protecting against untrusted plugins, skills, MCPs, and hooks |

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT © Edin Mujkanovic