token-delta

bundlesize, but for prompts. Diff two versions of a prompt, see the token delta and the dollar-cost delta across the model providers you care about — and fail CI when a prompt quietly balloons.

$ npx token-delta prompt.txt prompt-v2.txt --models gpt-4o,claude-sonnet-4-6,gemini-2.5-pro --calls-per-day 10000

prompt.txt → prompt-v2.txt
────────────────────────────────────────────────
Tokens (openai-o200k):  842 → 1,203  (+361, +42.9%)
Tokens (anthropic):     861 → 1,231  (+370, +43.0%)
Tokens (estimated):     ~830 → ~1,187  (+~357, +43.0%)

Cost per call (input only):
  gpt-4o             $0.002105 → $0.003008  (+$0.000903)
  claude-sonnet-4-6  $0.002583 → $0.003693  (+$0.001110)
  gemini-2.5-pro     ~$0.001038 → ~$0.001484  (+$0.000446)

At 10,000 calls/day this change costs an extra $4.46–11.10/day (≈ $133.80–333.00/month).
────────────────────────────────────────────────
Pricing as of: openai 2026-07-17 · anthropic 2026-07-17 · google 2026-07-17.
~ = estimated via chars/4 heuristic (no public tokenizer for this model), not an exact count.

v0.1: real tokenizers for OpenAI (js-tiktoken, o200k_base) and Anthropic (@anthropic-ai/tokenizer); everything else (e.g. Gemini) is a clearly-marked chars/4 estimate. Note that Anthropic has not published the tokenizer for Claude 3+ models, so Claude counts are close approximations rather than guaranteed-exact.

Install / run

npx token-delta <before> <after>         # no install
# or
npm install -g token-delta
token-delta <before> <after>

Requires Node 18+ (enforced via engines; npx/npm warn on older versions). Pure JS — no native compilation, no WASM loading quirks — so zero-install npx works anywhere.

Usage

Inputs: files, strings, stdin, git

Two inputs are always required — a "before" and an "after". They can come from:

# Two files
token-delta prompt.txt prompt-v2.txt

# Literal strings (must not be mixed with file arguments — that's an error)
token-delta --a "You are a helpful assistant." --b "You are a terse assistant."

# Stdin, explicitly with '-' in either position…
cat prompt-v1.txt | token-delta - prompt-v2.txt     # stdin is "before"
cat prompt-v2.txt | token-delta prompt-v1.txt -     # stdin is "after"

# …or implicitly: one file + piped stdin means stdin is "before", file is "after"
cat prompt-v1.txt | token-delta prompt-v2.txt

# Stdin can only ever fill ONE side — this errors:
cat prompt.txt | token-delta                         # error: two inputs required

# Working tree vs last committed version (inside a git repo)
token-delta prompt.txt --git

Flags

| Flag | Meaning | |---|---| | --models <list> | Comma-separated model names matching pricing-table keys. Default: gpt-4o,claude-sonnet-4-6 (the output says so when defaulted). | | --fail-on-increase <percent> | Exit code 1 if token count grows strictly more than this percent. With multiple models, the largest increase across tokenizers is used (conservative). An empty→non-empty change counts as infinite and always trips the gate. | | --calls-per-day <n> | Adds an extrapolated daily/monthly cost-impact line (and dailyProjection in JSON). | | --json | Machine-readable output (versioned schema below); suppresses the human table. | | --pricing <path> | Merge your own pricing JSON over the built-in tables (partial overrides supported — see Pricing). | | --a <text> / --b <text> | Compare literal strings instead of files. Conflicts with file arguments. | | --git | Compare <before>'s last committed version (git show HEAD:<file>) against the working tree. Takes exactly one file. | | --help / --version | The usual. --help also lists built-in models with per-provider pricing dates. |

CI usage

Exit codes (the contract CI scripts depend on)

| Code | Meaning | |------|---------| | 0 | Success, no threshold exceeded (or no --fail-on-increase given) | | 1 | Token increase exceeded --fail-on-increase threshold | | 2 | Invalid CLI arguments (incl. malformed --pricing files) | | 3 | File not found / unreadable input (incl. non-UTF-8 content) | | 4 | Git error (not a repo, no committed version, git not installed, …) | | 5 | Unsupported model name (not in pricing table and no --pricing override covers it) |

Pre-commit hook

Copy into .git/hooks/pre-commit (and chmod +x it):

#!/bin/sh
# Fail the commit if any staged prompt file grew more than 20% in tokens.
for f in $(git diff --cached --name-only -- 'prompts/*.txt'); do
  [ -f "$f" ] || continue   # skip deletions
  npx token-delta "$f" --git --fail-on-increase 20 || {
    echo "Prompt $f grew >20% in tokens. Re-commit with --no-verify to override."
    exit 1
  }
done

GitHub Actions

No dedicated action yet (planned once there's demand) — the CLI is enough:

name: prompt-budget
on: pull_request

jobs:
  token-delta:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Fail if any changed prompt grew >20% in tokens
        run: |
          git fetch origin ${{ github.base_ref }}
          for f in $(git diff --name-only origin/${{ github.base_ref }}...HEAD -- 'prompts/*.txt'); do
            [ -f "$f" ] || continue
            git show "origin/${{ github.base_ref }}:$f" > /tmp/base.txt 2>/dev/null || continue
            npx token-delta /tmp/base.txt "$f" --fail-on-increase 20 --calls-per-day 10000
          done

Pricing

Built-in prices live in per-provider JSON files at the package root — pricing/openai.json, pricing/anthropic.json, pricing/google.json — merged at load time. Each file carries its own asOf date, since providers change prices on different schedules. Seeded (verified 2026-07-17) with:

gpt-4o, gpt-4o-mini, claude-sonnet-4-6, claude-haiku-4-5, claude-opus-4-8, gemini-2.5-pro, gemini-2.5-flash

Each entry carries the metadata that keeps logic switch-free:

{
  "provider": "openai",
  "asOf": "2026-07-17",
  "models": {
    "gpt-4o": {
      "family": "gpt-4o",
      "tokenizer": "openai-o200k",
      "inputPer1M": 2.50,
      "outputPer1M": 10.00,
      "currency": "USD"
    }
  }
}

The tokenizer field names a tokenizer adapter (openai-o200k, anthropic, or estimated — defaults to estimated when omitted). Adding a model never touches tokenizer code — it's a pricing entry pointing at an adapter.

Overriding prices (--pricing)

Your file is merged over the built-ins, per model, per field — so partial overrides work:

token-delta a.txt b.txt --pricing ./my-prices.json

{
  "asOf": "2026-08-01",
  "models": {
    "gpt-4o": { "inputPer1M": 1.25 },
    "my-custom-model": { "provider": "acme", "inputPer1M": 0.80, "outputPer1M": 3.20 }
  }
}

Here gpt-4o keeps its built-in tokenizer/provider metadata and only the price changes; my-custom-model is added (tokenizer defaults to estimated). A bare {"model": {...}} map without the models wrapper also works.

Only input-token cost is computed — output length is speculative until you run the prompt.

JSON output schema (--json)

The stable contract, versioned via schemaVersion — downstream tooling should branch on it if this ever changes:

{
  "schemaVersion": 1,
  "inputs": {
    "before": { "label": "prompt.txt",    "tokensByModel": { "gpt-4o": 842 } },
    "after":  { "label": "prompt-v2.txt", "tokensByModel": { "gpt-4o": 1203 } }
  },
  "models": [
    {
      "model": "gpt-4o",
      "tokenizer": "openai-o200k",       // adapter name; "estimated" for heuristics
      "isEstimated": false,
      "tokens": {
        "before": 842, "after": 1203, "delta": 361,
        "percent": 42.87                 // null when before==0 and after>0 (infinite)
      },
      "cost": {                          // input-token cost per call, in `currency`
        "before": 0.002105, "after": 0.003008, "delta": 0.000903, "currency": "USD"
      },
      "dailyProjection": {               // null unless --calls-per-day given
        "callsPerDay": 10000, "extraCostPerDay": 9.03
      }
    }
  ],
  "failOnIncrease": {                    // null unless --fail-on-increase given
    "threshold": 20, "exceeded": true
  },
  "generatedAt": "2026-07-17T12:00:00.000Z"
}

Numbers are emitted at full precision — round at the consumer.

Library use

The CLI is a thin wrapper over an exported, layered API (tokenizers know nothing about dollars; pricing knows nothing about tokenization; only the diff layer combines them):

import {
  computeDiff, singlePart, loadPricing,          // orchestration
  countTokens, OpenAITokenizerAdapter,           // tokenizer layer
  costFromTokens,                                // pricing layer
} from "token-delta";

const pricing = loadPricing();
const result = computeDiff(
  singlePart("old", oldPrompt),
  singlePart("new", newPrompt),
  { models: ["gpt-4o"], pricing }
);

// Or use the layers independently:
const tokens = countTokens(oldPrompt, new OpenAITokenizerAdapter());
const cost = costFromTokens(tokens, pricing.models["gpt-4o"]);

Internally the diff engine operates on PromptParts (an ordered list of named text segments per side); the two-file CLI just builds single-part instances. A future --parts system.txt,rag.txt,... mode is additive.

For contributors

• ESM throughout: "type": "module", TypeScript compiled to ESM (module: NodeNext), .js extensions on relative imports. Keep new code ESM.
• Node 18+ minimum (engines field; no runtime check needed — npm/npx warn).
• npm test — vitest suite (unit + CLI end-to-end).
• npm run bench — baseline benchmark against ~70k-token inputs. Current baseline on a dev laptop: ~1.6s cold (lazy tokenizer init dominates), ~0.4s warm, ~1.6s full CLI spawn.

Not in v0.1 (on purpose)

• Models beyond the seed list (add your own via --pricing)
• Output-token prediction beyond the --calls-per-day extrapolation
• Multi-part CLI flags (--parts) — internals support it, surface comes later
• A packaged GitHub Action, web UI, dashboards, telemetry

License

MIT