TokenCap

Website · Docs · Downloads · Benchmarks · GitHub

AI coding agents waste tokens re-discovering your codebase. TokenCap stops that.

Run one command. Every AI session starts with full project context — ranked, budgeted, and ready.

npm install -g tokencap
tokencap make

The numbers

Benchmarked across 3 real open-source repositories, 5 tasks each, 15 data points total. Tokenizer: js-tiktoken exact counts. Model: gpt-4o ($2.50/1M).

Naive grep-and-read:  ████████████████████████  820,101 tokens  →  $2.0503 / query
TokenCap capsule:     █                          46,337 tokens  →  $0.1158 / query

                      12.2× fewer tokens.  94% cheaper.  Same answer.

| Repo | Size | Files | Naive avg | TokenCap | Ratio | | --------------------------- | ------ | ----- | ------------- | ---------- | ---------------- | | fastify-example-todo | small | 18 | 12,183 tok | 8,275 tok | 1.5× | | full-stack-fastapi-template | medium | 195 | 146,702 tok | 60,051 tok | 2.4× | | hoppscotch | large | 1,749 | 2,301,419 tok | 70,685 tok | 32.6× |

Full methodology and raw data: benchmark results
Reproduce: node benchmarks/run.js

What it does

Run tokencap make --full in any repo. This is what you see:

$ tokencap make --full

TokenCap: Full build requested.

TokenCap snapshot written:   ./.tokencap/snapshot.md
Included files:              47
Snapshot bytes:              198,432
Estimated tokens:            49,608
Profile:                     balanced

AI summary:                  .tokencap/graph/ai-graph-summary.md
Interactive graph:           .tokencap/graph/graph.html
Knowledge graph written:     .tokencap/graph/summary.md  (47 nodes, 83 edges)
Memory layer written:        .tokencap/memory/current.md
Security engine:             active (redaction engine v1)
Agent pointers written:      AGENTS.md, CLAUDE.md, .cursor/rules/tokencap.md,
                             .windsurf/rules/tokencap.md, .clinerules/tokencap.md,
                             .github/copilot-instructions.md, .kiro/steering/tokencap.md

── TokenCap Savings ─────────────────────────────────────
  Profile:  balanced
  Naive:    ████████████████████████  146,702 tok  $0.3668
  TokenCap: ████████████████           49,608 tok  $0.1240
  Saved:    97,094 tokens (66.2%)  $0.2428
  ─────────────────────────────────────────────────────────

Every major AI host (AGENTS.md, CLAUDE.md, .cursor/rules/, and 4 more) now points to .tokencap/agent/START_HERE.md. The next session starts with full project context — no re-discovery, no wasted tokens.

How it works

Your repo
    │
    ▼
tokencap make
    │  ├─ Ranks every file by git recency, importance score, and in-degree
    │  ├─ Enforces a token budget (default: 220 KB source)
    │  └─ Writes .tokencap/savings.json  ← exact before/after token counts
    ▼
`.tokencap/snapshot.md` ← the managed capsule
AGENTS.md             ← short start point for hosts that discover it automatically
.tokencap/savings.json  ← machine-readable savings data

The capsule is not a full dump. It's a graph-ranked selection of the files that matter most for the current state of the repo — so the model gets signal, not noise.

Features (v2.8.0)

1. Browser Companion Context Transfer (v2.8.0)

Bring local repository intelligence into browser-based AI chats without sending your code to a remote service:

• TokenCap Companion: Pair the Chrome companion with tokencap serve, select an indexed local project, and inject a bounded, redacted task pack into ChatGPT, Claude, or Gemini.
• Draft-Safe Injection: Existing composer text stays intact; TokenCap marks its inserted context and refuses duplicate injection. Every generated pack can also be previewed, copied, or downloaded.
• Fast, Bounded Local Bridge: tokencap serve pre-warms shared intelligence. Companion task packs stay in memory, include the project primer in their total budget, and do not create a context artifact for each browser request.
• Reviewable Capture Loop: Browser captures are validated, redacted, staged, and explicitly approved before becoming durable project memory.

2. Conversation-to-Repository Loop & Session Capture v1 (v2.7.0)

Turn finished AI coding sessions into small, redacted, reviewable handoffs — closing the conversation → repository loop without raw transcripts or telemetry:

• tokencap remember: Stage and review session summaries (--summary, --files, --decision, --next).
• Staged, Never Silent: Inbound captures land in .tokencap/memory/inbound/ with full provenance. Nothing modifies durable memory or ADRs until reviewed and approved via tokencap remember --approve <session-id> (or --reject / --note). Approved sessions archive to .tokencap/memory/sessions/.
• Graph-Grounded Quality Scoring: Captures are validated, bounded, redacted, and scored against the repository graph for path validity, groundedness, specificity, and novelty → confidence tier + evidence. Grounding evaluates references, not the truth of the summary.
• Handoff Surfaces: tokencap handoff exports the latest approved session as Markdown. Approved sessions surface in .tokencap/memory/current.md, agent-pack.md, and via MCP tools.
• Honest Host Support: Registry of 7 AI coding hosts (tokencap remember --hosts). No host is marked "auto" without a verified session-end hook. MCP process shutdown is never treated as a conversation-end signal.

3. Repository Build Health & Diagnostics (v2.7.0)

• tokencap health: Produces an explainable 0–100 repository health score evaluating intelligence freshness, git drift, symbol detection completeness, and test coverage.
• Actionable Remediation: Reports granular deduction factors and prints exact remediation commands (e.g. tokencap rebuild).
• MCP Integration: Build health is continuously surfaced in tokencap_overview and status diagnostics.

4. Evidence-Rich Risk Findings & Python Dependencies (v2.7.0)

• tokencap risk: Normalized risk findings (oversized-file, todo-density, possible-secret) with confidence tiers, structural evidence, scope, and recommended actions.
• Persistent Acknowledgment: Mark findings with --ack <id> or --dismiss <id>. Persistent state retains evidence without hiding findings from audits.
• Python pyproject.toml Parsing: Automatic dependency detection for PEP 621 standard declarations and Poetry dependencies.

5. Watcher Controls & Session ROI (v2.7.0)

• tokencap watch-config: Manage persistent file watching behavior (--enable, --disable, --debounce <ms>, --ignore <glob>).
• Session ROI Tracking: Computes observed per-session token savings and financial ROI aggregated across hosts in tokencap memory and MCP status.

6. High-Speed Loopback Bridge & Cold-Start Primer (v2.7.0)

Connect browser-based coding environments securely to local project intelligence:

• tokencap serve: Isolated loopback HTTP bridge (127.0.0.1) with pairing authentication (--token, --port, --daemon, --stop, --watch, --no-open).
• Project Primer (GET /primer): Provides a rich, human-written ~1.5k token orientation so a cold AI chat immediately understands the architecture, stack, and guidelines.
• Session Capture Endpoints: POST /capture stages captures from companions; POST /capture/approve and POST /capture/reject manage reviews over loopback.
• Warm Reuse (~3× Faster): Reuses query-independent graph, memory, and diff artifacts across queries within a short TTL, dropping repeat /generate response times from ~2.6s to ~0.8s.
• Model-Aware Dynamic Budgeting: Automatically scales task context to the target model's window (clamped between 4k and 24k tokens).
• Registry Hygiene & Zero Egress: Automatically prunes deleted directories from GET /projects. Strictly zero network egress.

7. Multi-Language Tier-1 AST Call Resolution

TokenCap parses code using pure Tree-sitter WebAssembly (zero native toolchain):

• Full Call Resolution: JavaScript, TypeScript, JSX/TSX, Python, Go, Rust, and Java.
• Go / Rust / Java Extraction: Resolves Go selector and method calls, Rust macro!/field/path calls, and Java constructor/method invocations.
• Precision Call Resolution: Resolves calls via import bindings, method receivers, and records unresolved symbols explicitly rather than guessing.

8. Static Taint & Security Flow Analysis

• tokencap analyze taint: Intra-procedural forward taint tracking from sources (req.body, query, params, cookies, process.argv, env) to sinks (db.query, exec, eval, fs.writeFile, etc.).
• Evidence-Based Risk: Clear HIGH/MEDIUM risk categorizations with sanitizer detection, without executing code.

9. Database & GraphQL Schema Graphs

• tokencap analyze schema: Parses SQL CREATE TABLE and GraphQL type schemas into a queryable relational graph, linking database models and types back to referencing application code.

10. Multi-Framework Component & SFC Graphs

• React & Next.js: AST-extracted RENDERS component trees, declared vs. passed props flow, App/Pages router maps.
• Vue & Svelte (.vue, .svelte): Single-file component (SFC) graphs with declared props and component render hierarchies.

11. Impact & Blast Radius Analysis

Know exactly what breaks before modifying or deleting a symbol:

$ tokencap impact src/auth/token.js:validateToken

  Direct callers      3 symbols · 2 files
  Transitive impact  17 symbols · 9 files
  Crosses boundary   src/api -> src/billing   ⚠
  Risk               HIGH — exported, 17 dependents

• tokencap impact --dead-code — lists uncalled, non-exported functions.
• tokencap impact --clusters — computes Louvain functional communities and cross-cluster coupling.

12. Guided Safe Refactoring

Modify code safely with deterministic, refuse-on-doubt transformations:

# Preview changes (dry-run by default)
tokencap refactor rename src/auth/token.js:validateToken checkToken
tokencap refactor rm-dead src/util/helpers.js:unusedHelper

# Apply atomically once verified
tokencap refactor rename src/auth/token.js:validateToken checkToken --write

• Safety by construction: Byte-exact outside the target edit ranges.
• Refuse-on-doubt: Aliased imports, shadowing, collisions, and computed accesses automatically block the plan.

13. Static Test Mapping & Gap Analysis

tokencap analyze tests                             # map symbols to exercising tests
tokencap analyze tests --gaps                      # list called-but-untested symbols
tokencap analyze tests --coverage coverage/lcov.info # ingest test reports (read-only)

14. Change Review & Local Ownership Signals

tokencap analyze review                    # review working-tree changes
tokencap analyze review --base main        # compare Git base ref against HEAD
tokencap analyze ownership --history       # opt-in local Git churn & author signals

Review packets include base-ref symbol evidence, caller candidates, test associations, and local Constitution/ADR context.

15. Evidence-Backed Code Simplification & Debt Ledger

tokencap analyze simplify                  # find unreachable code & duplicate artifacts
tokencap analyze debt                      # update TODO/FIXME/HACK/DEBT ledger
tokencap analyze compress AGENTS.md --write   # prose-only safe Markdown compression

Install & Quickstart

npm install -g tokencap

Requirements: Node.js 18+, Git.

tokencap make          # build + print savings summary
tokencap make --full   # full rebuild (graph + brain + agent + pointers)

Command Reference

7 Core Commands

TokenCap v3.0.0 keeps the seven canonical commands and expands them through focused subcommands and flags:

| Command | Description | | --- | --- | | tokencap make | Build or refresh repository intelligence (--full, --watch, --rebuild, --package, --max-memory-mb) | | tokencap ask <query> | Task-scoped context pack, ask brain <topic>, or ask impact <file>:<symbol> | | tokencap analyze <tool> | Audit and analyze: review, diff, ownership, team, plugins, compress, tests, taint, schema, components, constitution, agent, simplify, stats, debt, health, risk, security, refactor | | tokencap serve | Local loopback HTTP bridge for browser companions (--port, --token, --daemon, --stop) or MCP server (--mcp) | | tokencap update | Check for updates and upgrade TokenCap safely | | tokencap help | Built-in CLI reference and command guides | | tokencap version | Print current TokenCap version (-v, --version) |

[!NOTE] v3 removes standalone compatibility aliases. Running a former command returns its canonical replacement; run tokencap make --migrate once after upgrading a v2 workspace, then run tokencap make.

Model Context Protocol (MCP)

TokenCap includes a built-in MCP server exposing 18 specialized tools over stdio:

tokencap serve --mcp --init              # auto-configure all detected AI clients
tokencap serve --mcp --init --client claude   # configure for a specific host
tokencap serve --mcp --health            # test server diagnostics and tools

Available MCP Tools

• tokencap_overview — High-level summary, tech stack, and risk clusters.
• tokencap_files — Graph-ranked, query-relevant files.
• tokencap_cluster — Deep intelligence on a specific functional cluster.
• tokencap_dependencies — Upstream/downstream dependency trees.
• tokencap_constitution — Rule constraints and invariants before coding.
• tokencap_impact — Blast radius and broken invariants for proposed changes.
• tokencap_execution — Phase-by-phase execution contract guidance.
• tokencap_delta — Incremental changes since last build.
• tokencap_search — Cross-layer search across all intelligence files.
• tokencap_verify — Cluster-specific test, build, and lint commands.
• tokencap_manage_adr — Manage Architecture Decision Records (.tokencap/constitution/adr.json).
• tokencap_get_screen — UI Map screen details and components.
• tokencap_get_diagram — Local architecture, impact, or diff SVGs.
• tokencap_simplify — Evidence-backed simplification candidates.
• tokencap_refactor_plan — Dry-run preview of guided refactoring plans.
• tokencap_test_map — Static test-to-symbol mapping and untested gaps.
• tokencap_review — Advisory change review packets.
• tokencap_ownership — Opt-in local Git history ownership signals.

Agent Host Compatibility

tokencap serve --mcp --init --client <host> records your selected host. Builds maintain that host's pointer file without overwriting custom edits:

| File | Host | | --- | --- | | AGENTS.md | Codex, Antigravity, OpenCode, VS Code Codex | | CLAUDE.md | Claude Code | | .cursor/rules/tokencap.md | Cursor | | .windsurf/rules/tokencap.md | Windsurf | | .clinerules/tokencap.md | Cline / Roo Code | | .github/copilot-instructions.md | GitHub Copilot | | .kiro/steering/tokencap.md | Kiro |

VS Code & IDE Extensions

The VS Code extension provides commands to refresh Agent, Brain, Graph, and Memory intelligence independently, with automatic debounced capture on save.

code --install-extension .\tokencap-2.8.0.vsix

Codex and Claude Code Plugins

.\scripts\install-plugins.ps1

Security & Sovereignty

• Zero network egress — No telemetry, analytics, or remote calls. Only tokencap update touches the network. Enforced by test/local-only.test.js scanning all source files on every test run.
• Secret redaction — Every file read passes through a centralized redaction engine before becoming output. API keys, tokens, PEM blocks, high-entropy strings are replaced with [REDACTED:*] markers.
• Atomic writes — .tokencap/savings.json and all cache files use tmp-then-rename to prevent partial reads or corruptions.
• Zero native dependencies — Tree-sitter parsers are shipped as pure WebAssembly (WASM), keeping package size under 10 MB.

Further reading

| Document | Contents | | --- | --- | | Setup & Quickstart | Installation, IDE wiring, MCP server setup | | Documentation | Complete command manual, AST parsing, and frontend intelligence | | Benchmarks | Raw benchmark data, methodology, and reproduction steps | | Changelog | Full version history |

License

MIT — see LICENSE.