@codearchitects/ai-token-analyzer

Measures the token weight of instructions, MCP servers and MCP tools for AI agents (GitHub Copilot, Claude Code, etc.).

Installation

npm install -g @codearchitects/ai-token-analyzer

Or run directly with npx:

npx @codearchitects/ai-token-analyzer --path ./my-agent-dir

Usage

ai-token-analyzer --path <dir|file> [--path <...>] [--max-tokens <n>] [--json]

Examples

# Analyze an entire directory
ai-token-analyzer --path ./projects/my-mcp-server

# Analyze specific files
ai-token-analyzer --path ./CLAUDE.md --path ./.claude/mcp-ui-kit.md

# Analyze the current project
ai-token-analyzer --path .

# CI check — fail if total tokens exceed 10 000
ai-token-analyzer --path ./instructions --max-tokens 10000

# Machine-readable JSON output (useful for piping to jq or other scripts)
ai-token-analyzer --path ./instructions --json

# JSON output with CI threshold (exits 1 if exceeded)
ai-token-analyzer --path ./instructions --json --max-tokens 10000

CLI flags

| Flag | Description | |------|-------------| | --path <dir\|file> | Path to analyze (repeatable). Directories are walked recursively. | | --max-tokens <n> | Token budget cap. Exits with code 1 if the fixed total exceeds n. | | --json | Print a JSON summary to stdout instead of the human-readable report. |

Using in CI pipelines of other projects

Add a step to your GitHub Actions workflow (or any CI script) to catch token bloat early:

- name: Check AI context weight
  run: npx @codearchitects/ai-token-analyzer --path . --max-tokens 15000

Or capture structured data for custom gates:

- name: Check AI context weight (JSON)
  run: |
    npx @codearchitects/ai-token-analyzer --path . --json --max-tokens 15000 > token-report.json
    cat token-report.json

The JSON output schema:

{
  "tokenMethod": "tiktoken cl100k_base",
  "totalTokens": 4210,
  "autoInstrTokens": 1800,
  "onDemandTokens": 400,
  "serverTokens": 200,
  "toolTokens": 2210,
  "score": 14,                       // 0–100, normalized to GH Copilot safe budget
  "budgets": {
    "GitHub Copilot (GPT-4o)":  { "safeTokens": 30000,  "usedPct": 14.0, "status": "OK" },
    "Claude Code (Sonnet 4.6)": { "safeTokens": 100000, "usedPct": 4.2,  "status": "OK" }
  },
  "maxTokensThreshold": 15000,       // only present when --max-tokens is passed
  "passed": true,                    // only present when --max-tokens is passed
  "items": [
    { "filePath": "...", "type": "instruction", "loadMode": "auto", "label": "CLAUDE.md", "tokens": 1800 }
  ]
}

status values: "OK" (≤ 50 %), "WARNING" (50–80 %), "CRITICAL" (> 80 %) of the safe budget.

What it analyzes

| Type | Discovery | Load mode | |------|-----------|-----------| | CLAUDE.md, .claude/*.md | automatic | auto | | .github/copilot-instructions.md | automatic | auto | | .github/instructions/*.instructions.md | automatic | auto | | .github/prompts/*.prompt.md | automatic | on-demand | | Other .md, .txt | automatic | reference | | MCP JSON (mcp*.json, *schema*.json, *manifest*.json) | automatic | MCP server/tool | | .ts/.js files with registerTool / server.tool | automatic | MCP server/tool |

Output

╔══════════════════════════════════════════════════════╗
║        AI Context Weight Analyzer                    ║
╚══════════════════════════════════════════════════════╝

  ▸ MCP Tools — cost per call (schema injected on every invocation)
    [tool]   list_components              312 tok  ...
    [tool]   describe_component           280 tok  ...
    SUBTOTAL tools (1 call each)          592 tok

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  FIXED TOTAL (auto-instr + servers + tools×1)           592 tok
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Budget per target runtime:

  GitHub Copilot (GPT-4o)
  ~30k usable out of 128k (editor context takes the rest)
  Fixed: 592 tok / 30,000 tok safe  [█░░░░░░░░░]   2.0%  ✓  OK

Token estimator

If tiktoken is installed, it uses the cl100k_base model for an accurate count. Otherwise it estimates at ~4 characters per token.

Development

git clone ...
cd ai-token-analyzer
npm install
npm run build
node dist/index.js --path ./path/to/analyze

Note: avoid npm run start -- --path ... for local testing. npm intercepts flags like --path, --json, and --max-tokens as its own config options before passing them to the script. Use node dist/index.js directly after building, or npx tsx src/index.ts to run from source without a build step.

Build

npm run build   # compiles to dist/