tokenfmt

Token-aware code formatter for LLM context windows. Strips ~47% of tokens from TS/JS and ~37% from Python, without changing what the code does.

tokenfmt reformats source the way an LLM wants to read it: no comments, tight whitespace, and (optionally) shorter identifiers. It parses your code, transforms the AST, then re-verifies that the output still parses, still type-checks, and is structurally equivalent to the input. If any gate fails, the run aborts. You never get silently broken code back.

src.ts ──► parse ──► strip comments ──► (rename) ──► compact ──► line cap
                                                                     │
              ┌──── parse + AST equivalence + tsc / py_compile ──────┘
              ▼
          out.ts (atomic write)

TL;DR

The Problem

LLM context windows are scarce and metered. Standard formatters (prettier, black) are designed for humans. They preserve comments, indent generously, and never rename. When you paste code into a prompt, you pay token cost for whitespace, explanatory comments, and verbose identifier choices the model doesn't need to do its job.

The Solution

tokenfmt rewrites the same source into a denser, semantically-identical form. The output:

• Parses cleanly (Babel for .ts/.tsx/.js/.jsx, CPython's ast for .py/.pyi)
• Type-checks cleanly (TS) or compiles cleanly (Python)
• Is AST-equivalent to the input (modulo renames, when enabled)
• Tokenizes ~47% smaller for TS/JS, ~37% smaller for Python, measured against prettier / black baselines

Why use tokenfmt?

| Feature | tokenfmt | prettier | black | LLM minifiers | | ------------------------------------ | --------------------- | --------- | --------- | ------------- | | Token reduction (vs human formatter) | +47% TS/JS, +37% Py | 0% | 0% | varies, lossy | | AST-equivalent output | yes | yes | yes | usually no | | Post-format tsc / py_compile | yes | no | no | no | | Three risk tiers (safe → aggressive) | yes | no | no | no | | Multi-language | TS, JS, TSX, JSX, Py | JS/TS | Python | varies | | Atomic file writes (no .tmp orphan)| yes | n/a | n/a | n/a | | Per-content-hash token cache | yes | n/a | n/a | n/a | | stdin streaming | yes (tokenfmt -) | yes | yes | varies |

Quick example

# Format a TS file, balanced level, write to stdout
tokenfmt --level balanced src/UserService.ts

# Atomic write (no .tmp orphan on crash)
tokenfmt -o out.ts src/UserService.ts

# Overwrite the input atomically, but keep JSDoc on exported bindings
tokenfmt --in-place --keep-jsdoc src/UserService.ts

# stdin pipeline
cat src/UserService.ts | tokenfmt -

# Aggressive level: also rewrites exported bindings as `internal as Original` aliases
tokenfmt --level aggressive src/UserService.ts

# Python is auto-detected by extension
tokenfmt src/cli_tool.py

# Token counts (cached by content SHA on disk)
npm run measure -- src/UserService.ts
# → o200k=1040  cl100k=1035  anthropic=1307  bytes=4923

# Validate that a file parses + type-checks without formatting it
npm run validate -- src/UserService.ts
# → OK UserService.ts

Design philosophy

1. AST equivalence is non-negotiable. Every transform runs through parse → mutate → re-parse → equivalence check. If the output isn't structurally equivalent to the input (modulo declared renames), the pipeline throws. There is no "best effort" mode. Broken code never reaches your disk.

2. Risk is opt-in via --level. safe only strips comments and whitespace. balanced adds local-identifier renaming. aggressive also rewrites exports as export { internal as Original }. Each level is a separate, deliberate budget for behavioral risk.

3. The output must still type-check. TS files run through an in-memory tsc program after transformation. Python files run through py_compile plus a token-stream equality check between pre- and post-transform. A formatter that emits files your build can't accept is useless.

4. Content-addressed caching. Token counts and validation results are keyed by SHA-256 of file contents under .tokenfmt-cache/. bench and measure are cheap to re-run and don't waste API calls on the Anthropic tokenizer.

5. Atomic writes by default. --in-place and -o write to a sibling tmp path and rename(2). A crash mid-write leaves the original file untouched.

Installation

From npm

# Global CLI
npm install -g tokenfmt

# Or as a project dependency
npm install --save-dev tokenfmt

Requires Node >=18.17.0.

From source

git clone https://github.com/vnnkl/llm-format.git tokenfmt
cd tokenfmt
npm install
npm run build           # produces dist/cli.js with shebang
npm link                # exposes `tokenfmt` on $PATH

Or skip the build with npx tsx src/cli.ts <file>.

Python support (optional)

The Python pipeline shells out to a CPython helper. To format .py/.pyi files you need a Python 3 interpreter on PATH plus libcst:

pip install libcst        # or: pip3 install libcst

For .ts/.tsx/.js/.jsx files this step is unnecessary.

If you're working in the source repo and want a hermetic venv for the bench (which also pins black for the prettier-vs-tokenfmt comparison), run npm run setup:python once.

Quick start

# 1. Format a single file in place, conservatively
tokenfmt --in-place --level safe src/foo.ts

# 2. Diff what would change before committing
tokenfmt src/foo.ts | diff -u src/foo.ts -

# 3. Get the token cost before and after
npm run measure -- src/foo.ts                 # baseline
tokenfmt src/foo.ts | npm run measure -- -    # NB: measure currently expects a path
                                              # → in practice: write to a temp file first

# 4. Run the corpus benchmark to see what tokenfmt achieves on your own code
cp src/some-real-file.ts test/fixtures/corpus/
npm run bench
cat benchmarks/latest.md

Command reference

tokenfmt <file>

| Flag | Default | Behavior | | ------------------- | -------- | ----------------------------------------------------------------------- | | <file> | required | Source path, or - for stdin. | | -o, --output <p> | — | Write atomically to <p>. Mutually exclusive with --in-place. | | --in-place | false | Overwrite input atomically. Ignored when <file> is -. | | --level <tier> | safe | One of safe, balanced, aggressive. | | --keep-jsdoc | false | Preserve /** … */ JSDoc on exported bindings (TS/JS only). | | --debug-emit-broken-output <path> | — | If a post-emit gate (parse / equivalence / tsc) fails, write the suspect output to <path> before throwing. Use this to capture the broken intermediate for bug reports. |

Exit codes

| Code | Meaning | | ---- | -------------------------------------------------------------------- | | 0 | Success. | | 1 | Any error: missing file, parse failure, gate failure, conflict (-o + --in-place). |

npm run scripts

| Script | What it does | | --------------------- | ------------------------------------------------------------------------------ | | measure <file> | Print o200k=… cl100k=… anthropic=… bytes=…. Caches per content SHA. | | validate <file> | parseJs + tsc gates. Prints OK <name> or FAIL <name> and exits accordingly. | | bench | Re-runs the full corpus benchmark, writes benchmarks/latest.{md,json}. | | test | vitest run (278 tests). | | typecheck | tsc -p . --noEmit. | | build | Compile dist/, prepend shebang to dist/cli.js. | | setup:python | Create the bench-only Python venv with pinned black. |

Programmatic API

import { format, formatWithMap } from "tokenfmt";

const out = format(source, {
  filename: "Service.ts",   // controls TS vs JS vs Python dispatch
  level: "balanced",        // 'safe' | 'balanced' | 'aggressive'
  keepJsdoc: true,          // preserves JSDoc on exports (TS/JS)
});

// Same, but also returns the rename map applied to the AST.
const { code, renameMap } = formatWithMap(source, { level: "aggressive" });

FormatOptions:

type Level = "safe" | "balanced" | "aggressive";

interface FormatOptions {
  filename?: string;     // default: 'anonymous.ts'
  level?: Level;         // default: 'safe'
  rename?: boolean;      // deprecated; equivalent to level: 'balanced'
  keepJsdoc?: boolean;   // default: false
}

format throws Error on any gate failure. Error messages are prefixed with the filename and the failing gate, e.g. service.ts: gate tsc: Type 'string' is not assignable to type 'number'.

Architecture

                  ┌────────────────────────────────────────────────┐
                  │                  format(source)                │
                  └────────────────────────────────────────────────┘
                                       │
                          ext ∈ {.py, .pyi}?
                              ┌────────┴────────┐
                          yes │                 │ no
                              ▼                 ▼
              ┌──────────────────────┐    ┌─────────────────────────────┐
              │ Python pipeline      │    │ TS/JS pipeline (Babel)      │
              │                      │    │                             │
              │ ast.parse  (gate)    │    │ parseJs            (gate)   │
              │ helper.transform     │    │ stripComments               │
              │   (drop comments,    │    │   (--keep-jsdoc protects    │
              │    compact ws)       │    │    exported declarations)   │
              │ lineCap              │    │ renameIdentifiers           │
              │ ast.parse  (gate)    │    │   (level ≥ balanced)        │
              │ py_compile (gate)    │    │ rewriteExportAliases        │
              │ token-stream eq      │    │   (level = aggressive)      │
              │   (gate)             │    │ emit (whitespace)           │
              └──────────────────────┘    │ lineCap                     │
                                          │ parseJs            (gate)   │
                                          │ assertEquivalentModuloRenames│
                                          │   (gate)                    │
                                          │ tscCheck           (gate, .ts only) │
                                          └─────────────────────────────┘
                                       │
                                       ▼
                                 string output
                                       │
                              CLI: stdout | -o | --in-place
                                       │
                              io/atomicWrite (tmp + rename)

Caching:

.tokenfmt-cache/
├── tokens/
│   ├── openai/
│   │   ├── o200k_base/<sha>.json   ← integer count
│   │   └── cl100k_base/<sha>.json
│   └── anthropic/<sha>.json        ← integer count, or { skipped, reason }
├── validate/
│   ├── parse/<sha>.json
│   └── tsc/<sha>.json
└── …

Configuration

tokenfmt itself reads no config files. All behavior is controlled per-invocation via flags or the FormatOptions argument. The intent is that callers (CI, editors, agents) compose tokenfmt rather than configure it.

The bench corpus lives under test/fixtures/corpus/ (TS/JS) and test/fixtures/python/ (Python). Drop your own files in there and re-run npm run bench to see how tokenfmt does on your code.

Benchmarks

Run npm run bench to regenerate. Numbers below are copied verbatim from benchmarks/latest.md; benchmarks/latest.json is the structured source of truth.

TS/JS per-level (o200k_base, vs prettier)

| level | files | regressions | reduction | | ---------- | ----- | ----------- | --------- | | safe | 20 | 0 | +46.33% | | balanced | 20 | 0 | +47.53% | | aggressive | 20 | 0 | +46.48% |

Per-language summary (o200k_base)

| language | baseline | files | regressions | aggressive reduction | | -------- | -------- | ----- | ----------- | -------------------- | | TS/JS | prettier | 20 | 0 | +46.48% | | Python | black | 10 | 0 | +37.06% |

The Python row requires the npm run setup:python venv. Without it, bench fails loudly with a pointer back to setup.

The anthropic columns in benchmarks/latest.md show skipped when ANTHROPIC_API_KEY is unset at bench time; set the key and re-run to populate them. Token counts are cached by content SHA, so re-runs after a key change still need the cache cleared (see Troubleshooting).

Environment

| Variable | Required? | Effect | | -------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | ANTHROPIC_API_KEY | optional | Enables the Anthropic count_tokens call in measure and bench. When unset: prints anthropic=skipped(ANTHROPIC_API_KEY not set) and emits one stderr warning. OpenAI counts still succeed. |

Troubleshooting

| Symptom | Cause | Fix | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | <file>: gate parse: … | Input doesn't parse (Babel for TS/JS, ast for Python). Same failure mode as your build. | Fix the source. tokenfmt will not silently accept invalid code. | | <file>: gate parse: … (broken output written to /path/dump.ts) | A post-emit re-parse failed: tokenfmt's output isn't valid syntax. This is a tokenfmt bug. | The dump file at the indicated path holds the actual broken intermediate. Open an issue with that file plus the input attached. Use --debug-emit-broken-output <path> to capture it. | | <file>: gate tsc: … | Input parsed but doesn't type-check. | Fix the type error. The gate runs on the input, not the formatter's output, so this is not a tokenfmt bug. | | <file>: gate equivalence-modulo-renames: … | Internal bug: a transform changed AST shape beyond the declared rename map. | Open an issue with the input file attached. | | <file>: -o and --in-place are mutually exclusive | You passed both flags. | Pick one. | | anthropic=skipped(ANTHROPIC_API_KEY not set) | The Anthropic tokenizer needs an API key. | export ANTHROPIC_API_KEY=… and re-run. | | Anthropic counts still skipped after setting the key | Earlier offline runs cached { skipped, reason } objects under .tokenfmt-cache/tokens/anthropic/. | rm -rf .tokenfmt-cache/tokens/anthropic and re-run. | | bench fails with python helper missing / black not found | Python venv not set up. | npm run setup:python. | | tokenfmt: command not found after a project-local npm install | The CLI is exposed via bin, which only lands on $PATH for global installs. | Use npx tokenfmt <file> for project-local installs, or npm install -g tokenfmt. From source: npm run build && npm link. | | python helper not found at python3 (op=…) | No Python 3 on PATH, or libcst not installed. | Install Python 3, then pip install libcst. |

Limitations

• TS/JS aggressive aggregate is below the 60% target. Currently sits at +47.49% on o200k vs prettier. The bottleneck is export-alias framing overhead (export { internal as Original }) on already-short export names. See .gsd/milestones/M002/M002-SUMMARY.md for the cost-benefit analysis and known deviation. For some files, balanced actually wins over aggressive for this reason.
• Already-dense Python sees diminishing returns. large_oss_style.py in the bench corpus only loses 7.72%; black's output on that file is already close to floor.
• No source maps. tokenfmt is a one-way transformation. If you need to map a token offset in the formatted output back to a line in the original, you'll need to keep the original.
• JS files don't get a tsc gate. tscCheck short-circuits for .js/.jsx/.mjs/.cjs. The parse gate still runs.
• The tsc gate ignores module-resolution diagnostics. Codes 2307 (Cannot find module 'X'), 2792, 6053, and 7016 are filtered. tokenfmt is a formatter, not a build tool — it doesn't honor host tsconfig.json#paths or verify imports resolve. Real type errors still fail the gate. Honoring host tsconfig.json is on the roadmap.
• Python pipeline shells out to a CPython helper. Cold-start per file is ~hundreds of ms. Fine for batch jobs, noticeable in editor-on-save loops.
• No incremental / streaming mode. The whole file is parsed and transformed atomically.
• Anthropic tokenizer requires a network round-trip per unique content hash. Cache is content-addressed so subsequent runs are free.

FAQ

Is the formatted output safe to commit? Yes, every gate guarantees AST equivalence (modulo renames) and that the output type-checks or compiles. In practice you'd keep the human-readable version in your repo and only run tokenfmt when feeding code to an LLM. Treat its output as a transport format for prompts.

What does --level balanced actually rename? Local identifiers (parameters, locals, internal-only top-level bindings) are replaced with shorter names chosen to BPE-tokenize well under both o200k_base and cl100k_base. Anything reachable from an export is left alone unless you go to aggressive.

What does --level aggressive change beyond balanced? It also renames exported bindings, then rewrites the export site as export { newName as OriginalName }. The public API surface stays identical to importers; the internal name is shorter. The cost is the alias framing itself: for files with many short export names, the boilerplate can wipe out the savings.

Does --keep-jsdoc work at every level? Yes. JSDoc preservation is independent of --level. Only /** … */ blocks attached to exported declarations are protected; non-exported JSDoc and all // comments are still stripped.

Why does sample.ts only show +6.06%? It's two lines long. There's nothing to compress.

Can I use it as a library in another project? Yes. npm install --save-dev tokenfmt, then import { format } from "tokenfmt". The package is ESM-only.

What tokenizers do you measure against? gpt-tokenizer for o200k_base (GPT-4o / 4.1) and cl100k_base (GPT-4 / 3.5), and Anthropic's count_tokens HTTP API for Claude. All three are cached by content SHA.

License

MIT © Constantin Vennekel