Countokens

A command-line tool, similar to cloc (Count Lines of Code), but for counting tokens across files — with OpenAI/tiktoken models, raw encodings, Hugging Face Hub tokenizers, or a local tokenizer.json.

Features

• Recursively scans directories to find files (including extensionless files and dotfiles).
• Counts tokens with auto-detected tokenizers (default: gpt-4o-mini via tiktoken).
• Supports OpenAI model names, tiktoken encodings, Hugging Face org/name repos, and local tokenizer paths.
• Provides both human-readable and JSON output formats.
• Allows custom ignore patterns (globs).
• Ignores node_modules/**, .git/**, and dist/** by default.
• Skips binary or unreadable files (NUL bytes / invalid UTF-8) and reports how many were skipped.
• Supports tree view output (--tree).
• Limits tree depth with -d, --depth <number> option.

Installation

You can install countokens globally using npm:

npm install -g countokens

Alternatively, you can run it directly without installation using npx:

npx countokens [options] [path]

Usage

countokens [options] [path]

Arguments:

• path: The root directory to scan for files. Defaults to the current directory (.).

Options:

• -m, --model <name>: Tokenizer to use (default: "gpt-4o-mini"). Accepts an OpenAI model name, tiktoken encoding, Hugging Face org/name, or a local tokenizer.json path.
• -i, --ignore <globs>: Provide a comma-separated list of glob patterns to ignore. Example: -i "*.log,coverage/**"
• --json: Output the results in JSON format (includes total count and per-file counts). When combined with --tree, JSON wins and a warning is printed.
• --tree: Display the token counts in a file tree structure.
• -d, --depth <number>: When used with --tree, limit how many directory levels to display (default: all levels).
• -V, --version: Display the package version.
• -h, --help: Display help information.

Examples

1. Count tokens in the current directory using the default model:

npx countokens

Output (Example):

Token count (gpt-4o-mini): 19,643 tokens

yarn.lock             12,410
README.md             1,700
test/scan.test.ts     1,018
src/format.ts         768
src/scan.ts           626
src/cli.ts            621
.gitignore            465
package.json          405
src/tree.ts           391
test/tree.test.ts     366
test/options.test.ts  205
test/binary.test.ts   157
src/options.ts        155
tsconfig.json         133
LICENSE               111
src/binary.ts         97
.gitattributes        15

2. Count tokens in a specific directory (./src) using gpt-4:

npx countokens --model gpt-4 ./src

2b. Use a tiktoken encoding name:

npx countokens -m cl100k_base

2c. Use a Hugging Face Hub tokenizer (downloaded once, then cached):

npx countokens -m HuggingFaceTB/SmolLM2-135M

2d. Use a local tokenizer directory or tokenizer.json file:

npx countokens -m ./my-tokenizer
npx countokens -m ./my-tokenizer/tokenizer.json

3. Count tokens, ignoring log files and the build directory:

npx countokens -i "*.log,build/**"

4. Get JSON output:

npx countokens --json

Output (Example):

{
  "total": 19643,
  "files": {
    ".gitattributes": 15,
    ".gitignore": 465,
    "LICENSE": 111,
    "package.json": 405,
    "README.md": 1700,
    "tsconfig.json": 133,
    "yarn.lock": 12410,
    "src/binary.ts": 97,
    "src/cli.ts": 621,
    "src/options.ts": 155,
    "src/scan.ts": 626,
    "src/format.ts": 768,
    "test/options.test.ts": 205,
    "src/tree.ts": 391,
    "test/scan.test.ts": 1018,
    "test/binary.test.ts": 157,
    "test/tree.test.ts": 366
  }
}

5. Display token counts as a file tree:

npx countokens --tree --depth 1

Output (Example):

Token count (gpt-4o-mini): 19,643 tokens

├─ src (2,658)
├─ test (1,746)
├─ yarn.lock (12,410)
├─ README.md (1,700)
├─ .gitignore (465)
├─ package.json (405)
├─ tsconfig.json (133)
├─ LICENSE (111)
└─ .gitattributes (15)

6. Display tree up to a maximum depth of 2:

npx countokens --tree --depth 2

Output (Example):

Token count (gpt-4o-mini): 19,643 tokens

├─ src (2,658)
│  ├─ format.ts (768)
│  ├─ scan.ts (626)
│  ├─ cli.ts (621)
│  ├─ tree.ts (391)
│  ├─ options.ts (155)
│  └─ binary.ts (97)
├─ test (1,746)
│  ├─ scan.test.ts (1,018)
│  ├─ tree.test.ts (366)
│  ├─ options.test.ts (205)
│  └─ binary.test.ts (157)
├─ yarn.lock (12,410)
├─ README.md (1,700)
├─ .gitignore (465)
├─ package.json (405)
├─ tsconfig.json (133)
├─ LICENSE (111)
└─ .gitattributes (15)

7. Display tree for ./src at depth 1 (direct children of that folder):

npx countokens ./src --tree --depth 1

Output (Example):

Token count (gpt-4o-mini): 2,658 tokens

├─ format.ts (768)
├─ scan.ts (626)
├─ cli.ts (621)
├─ tree.ts (391)
├─ options.ts (155)
└─ binary.ts (97)

Ignoring Files

By default, countokens ignores:

• node_modules/**
• .git/**
• dist/**

Add more patterns with -i / --ignore as a comma-separated list of globs.

Tokenization / --model resolution

-m / --model is resolved in this order:

1. Local path — a tokenizer.json file, or a directory containing tokenizer.json (and optionally tokenizer_config.json)
2. Hugging Face repo — org/name (fetched from the Hub, then cached under ~/.cache/countokens/tokenizers/)
3. tiktoken OpenAI model — e.g. gpt-4o-mini, gpt-4 (model list)
4. tiktoken encoding — e.g. cl100k_base, o200k_base

Default is gpt-4o-mini (offline, via tiktoken).

For gated Hugging Face repos, set HF_TOKEN or HUGGING_FACE_HUB_TOKEN.

Note: Proprietary Anthropic/Gemini tokenizers are only supported when a compatible Hugging Face / tokenizer.json is available. This CLI does not call provider token-count APIs.

Development

yarn install
yarn typecheck
yarn test
yarn build

Contributing

Issues and pull requests are welcome! Please refer to the GitHub repository and the issue tracker.

License

WTFPL — do whatever you want. See LICENSE.