tiktoken-context-window

Zero-dependency context-window token counter for tiktoken-powered AI agents. Computes accurate per-message and per-conversation budgets for OpenAI-style message histories. Ships for both Node.js and Python with identical results.

Why

AI coding agents (Claude Code, Cursor, Codex, Gemini CLI) call the tiktoken encoder to estimate context-window usage. But tiktoken encodes text — it does not count tokens in already-built message histories. Every agent framework ends up re-implementing num_tokens_from_messages() and small errors (off-by-one on role separators, missing the NAME: / ROLE: prefixes tiktoken adds for tool/function messages) silently corrupt budget estimates.

This library wraps a user-supplied tiktoken encoding and applies the exact tiktoken reference rule (3 base + per-message 3 + tokens of role, content, optional name, function-call envelope, plus 1 trailing newline) so you get accurate budgets without re-deriving the schema yourself.

Bring your own tiktoken

This package has zero runtime dependencies. You provide the tiktoken (Node) or tiktoken (Python) encoder; we just apply the rule.

# Node
npm install tiktoken-context-window tiktoken

# Python
pip install tiktoken-context-window tiktoken

Both tiktoken 0.7+ (Python) and 1.0+ (Node) are supported.

Quick start

Node.js (ESM)

import { createContextWindow, tokenBudget, DEFAULT_ENCODING } from 'tiktoken-context-window';
import tiktoken from 'tiktoken';

const encoding = tiktoken.get_encoding('cl100k_base');

const ctx = createContextWindow(encoding, [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'Fix the bug in auth.py' },
  { role: 'assistant', content: 'I found the issue in line 42...' },
  { role: 'tool', name: 'read_file', content: 'def authenticate(...):\n    ...' },
  { role: 'user', content: 'Thanks, that fixed it.' },
]);

ctx.totalTokens;        // e.g. 64
ctx.remainingTokens;    // limit - totalTokens (clamped to 0)
ctx.pctUsed;            // 0.0–1.0 (may exceed 1.0 when over budget)
ctx.estimatedCost();    // '$0.00016' (gpt-4o prompt pricing)
ctx.canAddMessage({ role: 'user', content: 'Hi' });  // true / false

const budget = tokenBudget(encoding, ctx.messages, 'gpt-4o');
// → { used: 64, limit: 128000, pct: 0.0005 }

Python

from tiktoken_context_window import ContextWindow, token_budget, DEFAULT_ENCODING
import tiktoken

encoding = tiktoken.get_encoding("cl100k_base")

ctx = ContextWindow(encoding, [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Fix the bug in auth.py"},
    {"role": "assistant", "content": "I found the issue in line 42..."},
    {"role": "tool", "name": "read_file", "content": "def authenticate(...):\n    ..."},
    {"role": "user", "content": "Thanks, that fixed it."},
])

print(ctx.total_tokens)        # e.g. 64
print(ctx.remaining_tokens)
print(ctx.pct_used)
print(ctx.estimated_cost())    # '$0.00016'
print(ctx.can_add_message({"role": "user", "content": "Hi"}))

budget = token_budget(encoding, ctx.messages, "gpt-4o")
# {'used': 64, 'limit': 128000, 'pct': 0.0005}

CLI audit (Python, zero runtime dependencies)

python3 scripts/count_context.py --model gpt-4o --file session_history.json
# Model: gpt-4o
# Context limit: 128,000 tokens
# Tokens used:  12,847
# Utilisation:  10.0%
# Status:       SAFE

The CLI uses tiktoken if it's installed (accurate) and falls back to a coarse 4-chars-per-token approximation otherwise (emits a warning to stderr).

API

Encoding

DEFAULT_ENCODING = 'cl100k_base'. This matches the spec and the gpt-3.5-turbo / gpt-4-turbo family. For gpt-4o / gpt-4o-mini, current tiktoken releases actually map those models to o200k_base; this library still defaults to cl100k_base for reproducibility. Pass a different encoding object if you need the native per-model encoding:

import tiktoken from 'tiktoken';
const encoding = tiktoken.encoding_for_model('gpt-4o'); // o200k_base
const ctx = createContextWindow(encoding, messages);

Model limits (tokens)

| Model | Limit | Source | |--------------------|-----------|-----------------------------------------------------------| | gpt-4o | 128,000 | https://platform.openai.com/docs/models/gpt-4o | | gpt-4o-mini | 128,000 | https://platform.openai.com/docs/models/gpt-4o-mini | | gpt-4-turbo | 128,000 | https://platform.openai.com/docs/models/gpt-4-turbo | | gpt-3.5-turbo | 16,385 | https://platform.openai.com/docs/models/gpt-3-5-turbo |

Override per call with createContextWindow(encoding, msgs, { limit: N }) or tokenBudget(encoding, msgs, 'gpt-4o', { limit: N }).

Pricing (USD per 1k prompt tokens)

| Model | $/1k | Source | |--------------------|------------|----------------------------------------------| | gpt-4o | $0.0025 | https://openai.com/api/pricing/ | | gpt-4o-mini | $0.00015 | https://openai.com/api/pricing/ | | gpt-4-turbo | $0.01 | https://openai.com/api/pricing/ | | gpt-3.5-turbo | $0.0005 | https://openai.com/api/pricing/ |

Pricing is a snapshot from August 2026; refresh the table in src/index.cjs / tiktoken_context_window/__init__.py if OpenAI changes it.

Operations

• canAddMessage(msg) / can_add_message(msg) — boolean: would adding this message keep us within limit?
• trimToBudget(target) / trim_to_budget(target) — removes oldest user/assistant messages FIFO until pct_used ≤ target. Preserves system (index 0) and never removes tool/function messages. Raises ContextOverflowError if the target cannot be reached because only protected messages remain.
• estimatedCost(model?) — formatted $X.XXXXX string.

Encoding

The token-counting rule is uniform across encodings; the user picks the encoding and we apply the rule:

total = 3                              # base overhead per API call
for each message:
    total += 3                         # tokens_per_message
    total += tokens(encoding, role)
    total += tokens(encoding, content)
    if name and role in {'tool','function'}:
        total += tokens(encoding, name) + 1
    if function_call:
        total += tokens(encoding, function_call.name)
        total += tokens(encoding, ' ROLE:')
        total += 1
    total += 1                         # trailing newline

This mirrors the official tiktoken Python reference for cl100k_base and is verified by a parity test (tests/parity.test.mjs) that runs identical message arrays through both runtimes and asserts zero divergence across 50+ messages × 3 seeds.

CLI output

For machine readability, the CLI prints one fact per line:

Model: <model>
Context limit: <limit> tokens
Tokens used:  <used>
Utilisation:  <pct>%
Status:       <SAFE|WARNING|DANGER|OVERFLOW>

Status thresholds: < 70% SAFE, 70–90% WARNING, 90–100% DANGER, > 100% OVERFLOW. Exit code is 0 only when SAFE; WARNING/DANGER/OVERFLOW all exit 1 so the CLI can be used as a CI gate.

Limitations

• Spec vs current tiktoken: this library defaults to cl100k_base per the v0.1.0 spec; current tiktoken (≥0.7 Python, ≥1.0 Node) maps gpt-4o/gpt-4o-mini to o200k_base. Pass tiktoken.encoding_for_model(...) for native per-model encoding.
• Model limits: the per-model context limits use 128k for the GPT-4o family (OpenAI's docs) rather than the 127216 example value from the pre-release spec. Override via the limit option if you need that exact number for legacy reasons.
• No streaming: this library counts tokens in already-built message histories. It does not stream-decode partial messages.
• No tool-use schema validation: function/tool message shapes are accepted as-is. We count tokens for whatever fields you provide.
• Not affiliated with OpenAI: this is an independent library; the pricing and limits tables are best-effort snapshots of OpenAI's public docs and may be stale.

Tests

# Node
npm install
npm test

# Python
pip install -e ".[dev]"
pytest

Both suites assert ≥100 passing tests across acceptance criteria AC-01 through AC-35 (see tests/COVERAGE.md for traceability).

Project layout

src/index.mjs              ESM facade
src/index.cjs              CommonJS implementation
index.d.ts                 Type declarations
tiktoken_context_window/   Python package
tests/                     Node tests (node:test)
python_tests/              Python tests (pytest)
scripts/count_context.py   Zero-dep CLI audit tool
fixtures/sample.json       Example session history
tests/COVERAGE.md          Acceptance criteria + test mapping

License

MIT — see LICENSE.