@huen123/llm-token-counter

Count prompt text and simple chat-message tokens for mainstream LLM families without making remote API calls.

@huen123/llm-token-counter is a Node.js-first npm package for quick token budgeting across OpenAI, Anthropic, Gemini, Mistral, Cohere, and Llama model families. Every result includes both the token count and metadata that tells you whether the count is exact or estimated.

Install

npm install @huen123/llm-token-counter

What It Counts

• Plain text with countTokens({ model, input })
• Structured chat messages with countChatTokens({ model, messages })
• Supported model metadata with getModelInfo(model)
• Curated canonical model families with listSupportedModels()

Supported Precision

| Provider family | Plain text | Chat messages | Strategy | | --- | --- | --- | --- | | OpenAI (gpt-*, o*, chatgpt-*) | Exact | Estimated | openai-tiktoken | | Anthropic (claude-*) | Estimated | Estimated | anthropic-tokenizer | | Google (gemini-*) | Estimated | Estimated | gemini-char4 | | Mistral (mistral-*) | Estimated | Estimated | cl100k-heuristic | | Cohere (command-*, aya-*) | Estimated | Estimated | cl100k-heuristic | | Meta (llama-*) | Estimated | Estimated | cl100k-heuristic |

OpenAI plain-text counts use the model tokenizer directly. Chat-message counts are estimated for every provider because message wrappers vary across APIs and versions.

Quick Start

import {
  countChatTokens,
  countTokens,
  getModelInfo,
  listSupportedModels,
} from "@huen123/llm-token-counter";

const promptResult = countTokens({
  model: "gpt-4o",
  input: "hello world!",
});

console.log(promptResult);
// {
//   requestedModel: 'gpt-4o',
//   resolvedModel: 'gpt-4o',
//   provider: 'openai',
//   family: 'gpt-4o',
//   tokenCount: 3,
//   precision: 'exact',
//   strategy: 'openai-tiktoken'
// }

const chatResult = countChatTokens({
  model: "claude-3-5-sonnet-latest",
  messages: [
    { role: "system", content: "Be concise." },
    { role: "user", content: "Summarize this paragraph." },
  ],
});

console.log(chatResult.precision); // "estimated"

console.log(getModelInfo("chatgpt-4o-latest"));
console.log(listSupportedModels());

API

countTokens({ model, input })

Counts tokens for a plain string and returns:

type TokenCountResult = {
  requestedModel: string;
  resolvedModel: string;
  provider: "openai" | "anthropic" | "google" | "mistral" | "cohere" | "meta";
  family: string;
  tokenCount: number;
  precision: "exact" | "estimated";
  strategy:
    | "openai-tiktoken"
    | "anthropic-tokenizer"
    | "gemini-char4"
    | "cl100k-heuristic";
};

countChatTokens({ model, messages })

Accepts only simple string chat messages:

type ChatMessage = {
  role: "system" | "user" | "assistant";
  content: string;
};

Chat counts add a lightweight per-message overhead estimate on top of each message content count.

getModelInfo(model)

Resolves aliases to canonical families and returns:

type ModelInfo = {
  resolvedModel: string;
  provider: "openai" | "anthropic" | "google" | "mistral" | "cohere" | "meta";
  family: string;
  precision: "exact" | "estimated";
  strategy:
    | "openai-tiktoken"
    | "anthropic-tokenizer"
    | "gemini-char4"
    | "cl100k-heuristic";
  aliases: string[];
};

listSupportedModels()

Returns the curated canonical families bundled in the current package release.

Aliases and Unknown Models

• Known aliases are normalized to canonical families before counting.
• Unknown models throw an error with close suggestions instead of silently guessing providers.
• New provider releases usually only need registry updates in a new package version.

Unsupported in v1

• Tool-call accounting
• Multimodal inputs such as images, files, and audio
• Full provider request-body parsing
• Pricing estimates
• Remote model-catalog updates
• Browser runtime support

Development

npm test
npm run build
npm pack

Publish Checklist

npm test
npm run build
npm publish --access public

prepublishOnly already runs the build and test steps before npm publish.