@dwight-ship-it/context-cost

v0.9.0

Published

a month ago

Estimate the per-session token cost a Claude Code plugin adds to your context window.

0High
0Medium
0Low

andrfaria

claude-code claude plugin mcp tokens context-window cost cli

context-cost

What it is: a zero-dependency CLI that estimates the per-session token cost a Claude Code plugin adds to your context window — broken down by skills, MCP servers, hooks, and injected context — before you install it, and measures what's already installed from your own session logs.

Why it exists: installing plugins quietly inflates the context window of every new session. The unit you install is the plugin, but the cost hides inside it: MCP tool schemas and — most easily missed — hooks that inject context every turn. context-cost makes that cost visible, and is honest about what it can and can't know statically (estimated / variable / unknown are never dressed up as exact).

Install & run (15 seconds)

From npm (scoped package, context-cost command — published from v0.6.0):

npx @dwight-ship-it/context-cost inspect ./path/to/plugin
# or install it on your PATH:
npm install -g @dwight-ship-it/context-cost

Straight from GitHub — always works, no npm needed (used in CI too):

npx github:Dwight-ship-it/context-cost inspect ./path/to/plugin

Or clone for local/contributor use:

git clone https://github.com/Dwight-ship-it/context-cost
cd context-cost && npm install && npm run build
npm link        # optional — puts `context-cost` on your PATH

Without npm link, run node dist/cli/index.js … in place of context-cost. The npm package is scoped (@dwight-ship-it/context-cost) because the unscoped name context-cost belongs to an unrelated package — see installation paths.

Example output

Predict a plugin's cost before installing it — fully static, never runs its code:

$ context-cost inspect examples/bad-plugin
  bad-plugin  v0.9.0                      grade: D
  ─────────────────────────────────────────────────────────────
  confidence: partial · risk: high · 4 unmeasured (3 variable, 1 unknown)
  PER SESSION (fixed)   ~140 tok
  PER TURN (added)      ~0 tok
  ON-DEMAND (if used)   ~0 tok

  Breakdown
    skill       mega                     ~140 tok   per-session  metadata only; body loads on-demand
    mcp-server  everything               unknown    per-session  MCP tools not statically declared — run `--deep` or `audit` to measure
    hook        SessionStart (startup)   variable   per-session  injects context / runs code per-session — run `audit` to measure
    hook        UserPromptSubmit         variable   per-turn  injects context / runs code per-turn — run `audit` to measure
    hook        PreToolUse (*)           variable   per-tool-call  injects context / runs code per-tool-call — run `audit` to measure

  ⚠ Flags
    • 3 hook(s) inject context or run every turn — real cost not knowable without installing. Run `context-cost audit` after install to measure.
    • 2 hook(s) run every turn or tool call (e.g. UserPromptSubmit, Pre/PostToolUse) — cost recurs unbounded per turn; `audit` to measure the real total.
    • 1 MCP server(s) don't declare tools statically — run `--deep` to measure.

The confidence / risk / unmeasured line is the trust summary: how much of the grade rests on real numbers, and how much hidden cost the unmeasured parts could add. Here, four components can't be sized statically — so a D grade built from ~140 known tokens is flagged partial confidence and high risk, not taken at face value. See Trust signals.

Turn that into ranked, actionable advice:

$ context-cost optimize examples/bad-plugin
  bad-plugin  v0.9.0                      grade: D
  ...
  Recommendations  (4 total, 4 high)
    HIGH   INJECTING_HOOK    UserPromptSubmit   [cost unknown] Hook injects context every turn — real cost unknown until measured.
    HIGH   UNDECLARED_MCP    everything         [cost unknown] MCP server does not declare tools statically — opaque until run.
    ...

Measure what's already installed — this reads your real session transcripts and attributes the context injecting hooks actually added:

$ context-cost audit --plugin claude-mem
  claude-mem  v13.4.0                      grade: B
  ─────────────────────────────────────────────────────────────
  confidence: partial · risk: medium · 1 unmeasured (1 variable, 0 unknown)
  PER SESSION (fixed)   ~1098 tok
  PER TURN (added)      ~0 tok
  ON-DEMAND (if used)   ~0 tok
  ...

Diff two plugins (or two revisions) and fail CI on a regression:

$ context-cost compare examples/clean-plugin examples/bad-plugin --fail-on-grade-drop
  compare  clean-plugin v1.0.0  →  bad-plugin v0.9.0
  ─────────────────────────────────────────────────────────────
  PER SESSION (fixed)    ~16 → ~140 tok   (+124)  ▲
  GRADE                  A → D  ▼ worse
  ...
context-cost: gate failed — grade dropped A → D     # ← stderr
$ echo $?
1

Current coverage

What context-cost measures today, and what it doesn't yet. "Partial" means it contributes a number but with a caveat worth reading.

| Capability | Status | Notes | |------------|--------|-------| | skills/SKILL.md | ✅ Supported | Front-matter (name + description) estimated; the body loads on-demand and isn't counted as fixed cost. | | hooks/hooks.json | ✅ Supported | Events detected and classified; injecting events (SessionStart, UserPromptSubmit, Pre/PostToolUse) flagged variable. | | .mcp.json static detection | ✅ Supported | Launch config is read statically; tool schemas aren't declared there, so servers are flagged unknown until measured. | | Local stdio MCP deep measurement | ✅ Supported | inspect --deep --allow-exec runs each local stdio server, enumerates tools, reports estimated. | | Remote HTTP/SSE MCP | ❌ Not yet | Remote servers are never contacted and are not measured. | | Commands (commands/*.md) | ✅ Supported | Each command's full markdown is costed as on-demand (injected only when invoked) — estimated, never per-session fixed. | | Agents (agents/*.md) | ✅ Supported | Name + description costed as per-session metadata (mirrors skills); the body runs in a separate subagent context and isn't counted. | | Injected session-context attribution | ⚠️ Partial | audit measures injected tokens from transcripts, but attributes them by hook event, not by plugin — if two plugins inject on the same event the total is shared. Only hook_success transcript entries are attributed. | | Malformed config files | ✅ Supported | An unparseable .mcp.json / hooks.json is reported as an unknown blind spot (never crashes, never guessed), lowering confidence and capping the grade. |

When should I use this?

Use it if you:

install Claude Code plugins and want to know what each one costs your context window before committing to it;
maintain several plugins/MCP servers and suspect one is quietly bloating every session;
author a plugin and want a committable cost gate (manifest --fail-on) or a before/after check on a change (compare);
need a quick, honest order-of-magnitude read rather than exact token billing.

Skip it (for now) if you:

need exact, billable token counts — this uses a heuristic ~chars/4 tokenizer tuned for relative ranking, not Claude's exact tokenizer;
want costs for remote HTTP/SSE MCP servers — only local stdio servers can be measured today;
expect it to measure a plugin's cost without running it: undeclared MCP tool schemas are genuinely unknowable statically and are reported as unknown/ flagged, not guessed.

It is most valuable as a decision and regression tool, least valuable as a billing meter.

Safety model

context-cost is built so the default path can never run untrusted code. Here is exactly what each command does:

| Command | Reads | Executes plugin code? | |---------|-------|-----------------------| | inspect (default) | plugin manifest, SKILL.md front-matter, .mcp.json, hooks.json — as text | No. Never. | | optimize / manifest / badge / compare | the same static report as inspect | No. Pure consumers of the report. | | audit | your own session transcripts under ~/.claude/projects | No. It reads logs of sessions that already ran; it launches nothing. | | inspect --deep --allow-exec | the above, plus live MCP tool schemas | Yes — local stdio MCP servers only, and only with the explicit flag. |

The only command that executes anything is inspect --deep --allow-exec, and it is doubly gated:

--deep alone refuses and instead prints the exact commands it would run, so you can read them first:

$ context-cost inspect ./some-plugin --deep
context-cost: --deep would EXECUTE the following MCP server command(s) from this plugin:
  search: node server.js
This runs code from the plugin you are inspecting.
Re-run with --allow-exec to proceed.

Adding --allow-exec launches each declared local stdio MCP server, enumerates its tools, and shuts it down. Each server runs under a wall-clock timeout (--deep-timeout, default 10s) and a stdout byte cap (--deep-max-bytes, default 5MB); if it crashes, the tail of its stderr is surfaced in the error so you can see why. Failures are isolated per server. Remote (HTTP/SSE) servers are never contacted and are not deeply measured yet. Measured servers are reported estimated (real schemas, heuristic tokenizer) — never exact.
Inherited environment: a measured server inherits your full shell environment (process.env) with any per-server env from .mcp.json layered on top — exactly how Claude Code launches it, so the measurement is realistic. That also means the plugin's code sees your PATH, tokens, and other secrets, which is the main reason --deep --allow-exec should be run in a disposable environment.

All variable/unknown costs are reported separately from the token totals and are never converted into exact numbers — see the tier table under What it reports.

Recommended safe workflow

Static first. context-cost inspect <target> — no code runs. This is enough to decide whether to install.

Review the execution plan. If you want deep MCP numbers, see exactly what would run first — it executes nothing:

$ context-cost inspect <target> --print-exec-plan
context-cost: deep MCP measurement would EXECUTE these local stdio command(s):
  search: node server.js
Nothing was executed (--print-exec-plan). ...

--print-exec-plan is always safe — it overrides --allow-exec.

Only then, deep — in a disposable environment. Run inspect <target> --deep --allow-exec inside a sandbox/container you can throw away, never against untrusted code on your main machine.

So: deciding whether to install is always static and safe. Running code is an opt-in step you take only after reading what it will run.

inspect vs audit

inspect predicts cost from static files — use it before installing.
audit measures the real injected-context cost already paid, from your session transcripts, upgrading injecting hooks from variable to estimated where it can attribute them. Attribution is by hook event, not by plugin (if two plugins inject on the same event the measured total is shared), which is why it is reported as estimated rather than exact. Use it after installing.

Commands

context-cost inspect github:owner/repo        # predict cost before install
context-cost inspect github:owner/repo#v1.2.3  # pin to a tag or branch
context-cost inspect ./path/to/plugin         # predict from a local path
context-cost audit                       # measure installed plugins
context-cost audit --plugin claude-mem   # one plugin (prefix match)
context-cost optimize <target>           # ranked cost-reduction advice
context-cost manifest <target>           # committable cost-manifest.json
context-cost badge <target>              # Shields.io badge for your README
context-cost compare <before> <after>    # diff two plugins / two versions

context-cost inspect <target> --json     # raw PluginCostReport JSON

Flags:

--json                          emit raw JSON (report / comparison / manifest)
--plugin <name>                 audit: filter installed plugins by id prefix
--output <path>                 manifest: write the manifest JSON to a file
--fail-on <grade>               manifest: exit 1 if grade is WORSE than <grade>
--deep                          inspect: measure MCP servers by running them (needs --allow-exec)
--allow-exec                    permit --deep to execute the plugin's MCP server commands
--print-exec-plan               inspect: show what --deep would run, then exit (runs nothing)
--deep-timeout <ms>             inspect: per-server timeout for --deep (default 10000)
--deep-max-bytes <n>            inspect: per-server stdout byte cap for --deep (default 5000000)
--fail-on-grade-drop            compare: exit 1 if the grade got worse
--fail-on-unmeasured-increase   compare: exit 1 if variable/unknown component count rose
--fail-on-session-increase <n>  compare: exit 1 if per-session fixed tokens rose by > n

What it reports

Every plugin is broken into components, each with a cadence (per-session / per-turn / per-tool-call / on-demand) and a confidence tier:

| Tier | Meaning | |------|---------| | exact | a value known with certainty (e.g. a non-injecting hook that adds 0 tokens) | | estimated | a counted number with heuristic-tokenizer/attribution caveats (skill metadata; deep-measured MCP tools; hook tokens measured from session logs and attributed by event, not plugin) | | variable | injects context / runs every turn — not knowable without installing | | unknown | can't be sized statically (e.g. MCP tools not declared in .mcp.json) |

| Component | How it's costed | |-----------|-----------------| | Skills | metadata only (name + description); the body loads on-demand | | MCP servers | flagged unknown — .mcp.json declares launch config, not tool schemas | | Hooks | injecting events (SessionStart, UserPromptSubmit, Pre/PostToolUse) flagged variable; others 0; unrecognised events unknown | | Commands | commands/*.md body costed on-demand (estimated) — injected only when the command is invoked, so never a per-session cost | | Agents | agents/*.md name + description costed per-session (estimated); the body runs in a separate subagent context and isn't counted |

Totals are split three ways — per-session fixed (paid every session just by having it installed), per-turn added, and on-demand potential (only if used) — plus an A–F grade and flags for the things that hurt most (injecting hooks, undeclared MCP servers).

⚠️ The A–F grade is a decision aid, not a billable or safety score. It is derived from a heuristic token estimate (~chars/4), not Claude's exact tokenizer, and says nothing about a plugin's security, correctness, or quality. Don't use it for billing or as a safety gate.

How the grade is computed. The starting point is the static token total, but the grade is then capped by how much of the plugin is unmeasurable, so hidden cost can't hide behind a near-zero static count:

a per-turn / per-tool-call injecting hook, 3+ injecting hooks, or an injecting hook combined with an undeclared MCP server (i.e. risk: high) → at most D;
a single undeclared MCP server, a lone session-only injecting hook, or another isolated unmeasured surface (risk: medium) → at most C;
a report where everything found is unmeasurable (confidence: low) → at most C.

Caps only ever pull a grade down, never up, and unmeasured costs are still never converted into invented token numbers — they cap the letter grade and are called out in the flags and the unmeasured count instead.

Trust signals

So the grade is never read as more certain than it is, every report carries two extra axes alongside it (in the human output and in --json):

| Signal | Values | Answers | |--------|--------|---------| | confidence | full · partial · low | How much of the report is real numbers? full = nothing unmeasured; partial = a mix; low = everything found was unsizable. | | risk | low · medium · high | How much hidden context-bloat could the unmeasured parts add? high = a per-turn injecting hook, 3+ injecting hooks, or injecting-hook + undeclared MCP together. | | unmeasured | { variable, unknown, total } | How many components couldn't be sized statically, split by reason. |

confidence and risk are a separate axis from the token totals — they describe how much you can trust the numbers, not the numbers themselves. They also feed the grade caps above: a plugin that is nothing but an undeclared MCP server has a ~0 known-token total, but because its real cost is entirely unmeasured it lands at low confidence / medium risk and is capped at C, never read as a clean A. The totals themselves stay honest — only the letter grade is capped, and no unmeasured cost is ever turned into a fabricated token number.

optimize, manifest, badge & compare

These build on the same report — no new analysis, no new dependencies.

optimize <target> turns a report into ranked, actionable advice: which components cost the most and what to do about them (gate an injecting hook, audit an undeclared MCP server, trim heavy per-session metadata). Recommendations are ordered by token impact; anything whose cost is unknown or variable is flagged [cost unknown] and never ranked as if it were exact.
manifest <target> emits a stable, versioned cost-manifest.json a plugin author can commit to their repo — schemaVersion, totals, grade, honesty flags, and recommendation severity counts (not the full text, so minor optimizer tweaks don't churn the file). Write it with --output:
```
context-cost manifest ./my-plugin --output cost-manifest.json
```
Use --fail-on <grade> as a CI gate — it exits non-zero when the plugin grades worse than your threshold, so a regression fails the build:
```
context-cost manifest ./my-plugin --fail-on B   # exit 1 if C/D/F
```
badge <target> prints a Shields.io static badge URL plus a ready-to-paste Markdown snippet:
```
![context-cost](https://img.shields.io/badge/context--cost-A_%C2%B7_~16_tok-brightgreen)
```
Grade drives the colour (A→brightgreen … F→red); the ~ marks the count as approximate.
compare <before> <after> diffs two reports — two plugins, or the same plugin at two revisions. It shows the per-bucket token delta and the grade direction, and (crucially) reports any change in unmeasured (variable/ unknown) components separately, so adding an injecting hook never hides behind a +0 token delta:
```
context-cost compare examples/clean-plugin examples/bad-plugin
```
```
  compare  clean-plugin v1.0.0  →  bad-plugin v0.9.0
  ─────────────────────────────────────────────────────────────
  PER SESSION (fixed)    ~16 → ~140 tok   (+124)  ▲
  GRADE                  A → D  ▼ worse

  Unmeasured components (variable/unknown — cost not captured above)
    before: 0   after: 4   (+4)
    ⚠ numeric deltas above exclude variable/unknown costs — measure with `audit`.
```
For plugin authors: run it in CI against the last released revision to catch a cost regression in review, before it ships to users. Three optional gates make compare exit non-zero so a regression fails the build:
| Flag | Exits 1 when | |------|--------------| | --fail-on-grade-drop | the grade got worse (e.g. B → C) | | --fail-on-unmeasured-increase | the count of variable/unknown components rose — catches an added injecting hook or undeclared MCP server even at +0 tokens | | --fail-on-session-increase <tokens> | per-session fixed tokens rose by more than <tokens> (use 0 to fail on any increase) |
Gates combine with AND — any tripped gate fails the run, and each prints one reason to stderr:
```
$ context-cost compare /tmp/base . --fail-on-grade-drop --fail-on-session-increase 200
context-cost: gate failed — grade dropped A → C
context-cost: gate failed — per-session fixed cost increased by 240 tok (110 → 350), over the allowed +200
$ echo $?
1
```
The --fail-on-unmeasured-increase gate is the honest counterpart to the token gate: it reports a count of unmeasured components, never a fabricated token figure for costs that can't be sized statically.

CI for plugin authors

Catch a context-cost regression in pull requests. This workflow checks out the PR's base revision into a git worktree and diffs the working tree against it — no published action, just the CLI:

# .github/workflows/context-cost.yml
name: context-cost
on: pull_request
jobs:
  cost:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0          # need the base ref to diff against
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Fail on a cost regression
        run: |
          git worktree add /tmp/base "origin/${{ github.base_ref }}"
          npx --yes github:Dwight-ship-it/context-cost compare /tmp/base . \
            --fail-on-grade-drop \
            --fail-on-unmeasured-increase \
            --fail-on-session-increase 200

npx github:Dwight-ship-it/context-cost installs straight from GitHub. The repo ships a prepare script (npm run build) so that a git/npx install compiles dist/ automatically. You can equally use the published scoped npm package, npx @dwight-ship-it/context-cost (published from v0.6.0 — see installation paths). Pin to a tag for reproducible CI, e.g. github:Dwight-ship-it/context-cost#v0.6.0.

Exit codes (so CI behaves predictably):

| Code | Meaning | |------|---------| | 0 | all gates passed | | 1 | a gate failed (the reason is printed to stderr; stdout/JSON stays valid) | | 2 | invalid CLI usage/config (e.g. a non-numeric --fail-on-session-increase) |

Installation paths

Scoped npm (from v0.6.0): npx @dwight-ship-it/context-cost <command> or npm install -g @dwight-ship-it/context-cost. The package is scoped because the unscoped name context-cost on npm belongs to an unrelated package — this project is not it. The installed command stays context-cost (the bin name is independent of the scoped package name).
GitHub / npx (always works): npx github:Dwight-ship-it/context-cost <command>. Builds on install via the prepare script, so it works without going through npm at all. Pin a branch or tag for CI, e.g. …/context-cost#v0.6.0.
Clone + npm link (contributors): see Install & run.

Honesty

Numbers are approximate (a heuristic ~chars/4 tokenizer), tuned for relative ranking and order-of-magnitude truth, not exact billing. Every number is labelled measured / estimated / variable / unknown — the tool never silently guesses and presents it as fact. For a detailed explanation of the three tokenizer modes and why counts are not billing-exact, see docs/token-count-calibration.md.

Limitations

audit attributes injected content that appears as hook_success entries in transcripts; content injected through other mechanisms isn't attributed yet.
inspect --deep --allow-exec measures MCP tool schemas by running each stdio server locally and enumerating its tools. It executes code from the plugin under inspection, so it is gated behind --allow-exec; without that flag --deep prints the commands it would run and refuses. Remote (HTTP/SSE) servers aren't measured yet. Measured servers are reported estimated (real schemas, heuristic tokenizer).
The tokenizer is a pluggable approximation, not Claude's exact tokenizer.
github:owner/repo#ref targets resolve a branch, tag, or commit SHA. Branch/tag refs use a fast shallow clone (--depth 1); commit SHAs trigger a full clone + git checkout, which is slower. Pin to a tag for reproducibility and performance in CI.
SKILL.md front-matter is read by a minimal token-estimation parser. It handles plain values and block scalars (folded >/>-/>+, literal |/|-/|+), but is not a spec-complete YAML loader — flow collections and anchors aren't modelled.

Development

npm install
npm test          # run the vitest suite
npm run build     # compile to dist/

Pure cost engine under src/core/ (no CLI deps); thin CLI under src/cli/. The engine emits a PluginCostReport (see --json); optimize, manifest, badge, and compare are pure consumers of that report (src/core/optimizer/, src/core/manifest/, src/core/compare/), so the whole tool stays zero-dependency.

The examples/ directory holds small, self-contained plugin fixtures (clean / hook-heavy / mcp-heavy / command-heavy / agent-heavy / bad / malformed) used by both the docs above and tests/examples.test.ts. They are controlled fixtures — inspect never runs them; don't pass --deep --allow-exec against them.

Docs

docs/architecture.md — analyzer pipeline, confidence tiers, parser boundaries, static vs deep MCP, and why zero dependencies.
docs/report-schema.md — the JSON shapes for inspect, compare, and manifest, with stability notes.
docs/real-world-validation.md — results of running context-cost against real public plugins.
docs/token-count-calibration.md — the three tokenizer modes (heuristic, words, tiktoken), why counts are estimates rather than billing-exact, when to trust grade/risk signals vs raw numbers, and a worked example.