toks

Count tokens in files, like wc for LLMs.

Installation

Requires Node.js 22+.

npm install -g toks

Usage

# Count tokens in files
toks file1.txt file2.txt

# With line ranges
toks file.txt:1-100      # lines 1-100
toks file.txt:50         # line 50 only
toks file.txt:50-        # line 50 to end
toks file.txt:-50        # first 50 lines

# From stdin
echo "hello world" | toks
cat file.txt | toks

# Change tokenizer encoding
toks --encoding cl100k_base file.txt
toks -e o200k_base file.txt

Output

    1234  file1.txt
    5678  file2.txt
    6912  total

Options

| Flag | Description | |------|-------------| | -e, --encoding <name> | Tokenizer encoding (default: o200k_base) | | -h, --help | Show help | | -v, --version | Show version |

Supported Encodings

• o200k_base — GPT-4o, GPT-5, o1, o3, o4 (default)
• cl100k_base — GPT-4, GPT-3.5-turbo
• p50k_base — Codex models
• r50k_base — GPT-3 (davinci)

Design

See DESIGN.md for implementation details.

License

MIT

Built with Claude Code (Claude Opus 4.5).