Token Tree CLI

token-tree is a command-line tool that recursively scans a directory, counts the number of tokens in each file according to a specified GPT model, and displays the result as a color-coded tree. This is useful for understanding the token cost of a project or a specific directory when working with OpenAI's language models.

Installation

You can install the package globally using npm:

npm install -g token-tree

Usage

The CLI can be run using the token-tree command.

token-tree [model] <directory>

• [model] (optional): The GPT model to use for token counting. Defaults to gpt-4. See supported models below.
• <directory> (optional): The directory to analyze. Defaults to the current directory (.).

Examples

# Analyze the current directory using the default gpt-4 model
token-tree

# Analyze a specific directory
token-tree ./my-project

# Analyze the current directory with a specific model
token-tree gpt-3.5-turbo

# Analyze a specific directory with a specific model
token-tree gpt-3.5-turbo ./my-project

Supported Models

This tool supports a wide range of OpenAI models for accurate token counting. Some of the supported models include:

• gpt-4 (default)
• gpt-4o
• gpt-4-turbo
• gpt-4-32k
• gpt-3.5-turbo
• gpt-3.5-turbo-16k

... and many more. The token counting is powered by the gpt-tokens library.

Example Output

Here is an example of what the output looks like. The directory names and their total token counts are colored blue, while file names and their token counts are colored green.

my-awesome-project/ (750 tokens)
├── src/ (400 tokens)
│   ├── index.ts (150 tokens)
│   └── api/ (250 tokens)
│       └── routes.ts (250 tokens)
├── .gitignore (20 tokens)
├── package.json (200 tokens)
├── README.md (80 tokens)
└── tsconfig.json (50 tokens)

Grand total: 750 tokens (model: gpt-4)

License

MIT