TokenCops

Token cost police for AI coding agents. TokenCops is a local-first CLI for context audits, secret-redacted output compression, durable handoffs, evidence-labeled accounting, and capability-based routing.

Provider-, model-, and agent-agnostic means that an arbitrary provider definition, model catalog row, or agent adapter can be supplied as data without changing TokenCops source. It does not mean TokenCops guesses undocumented APIs, tokenizers, cache rules, or session-store schemas.

TokenCops has zero runtime dependencies. Offline commands stay local. Network-capable commands and subprocess commands have the explicit limits described below.

Supported core

• Context inventory, CBOM generation, duplicate/stale/dead instruction checks, and instruction-file detection across 19 agent tools (29 file patterns).
• Safe and family-aware compression for common test runners, with a no-expansion guard, diagnostic summaries, and an explicit transform manifest.
• Secret redaction before output, telemetry, validation state, or receipt persistence. Receipts are sanitized copies, not raw evidence.
• Canonical task state, decisions, next actions, source-bound validation, handoffs, stale-snapshot detection, and restart packets.
• Evidence-labeled token and savings reports. Unsupported measurements and prices remain null or unknown.
• Caller-supplied provider definitions, model catalogs, pricing metadata, generic JSON/JSONL session mappings, and exact-identity estimator calibration.
• Safe manifest-based uninstall, a local MCP stdio server, and append-only hash-chained telemetry.

Optional built-in presets

Built-ins are conveniences, not the architecture boundary:

• Declarative live-HTTP provider presets for Anthropic, OpenAI, and Google. The generic renderer does not branch on provider or model names.
• Explicit session-parser presets for Claude Code, Codex CLI, and Gemini CLI, plus heuristic presets for a small number of other stores. Running tcop sessions without --source or --dir reads no store and prints selection guidance.
• A bundled USD-per-million-token price table dated 2026-08-07. It is an optional scenario, not live billing truth. tcop savings leaves USD null unless the caller explicitly supplies --input-per-m or selects a model scenario with --model; unknown or incomplete pricing stays unknown.
• Claude Code hooks, installed only with an explicit target such as --agent claude-code.
• Resume renderers for claude, agents, and prompt; configured agents can add data-only renderers.

Cache pricing, minimum prefixes, TTLs, and reuse scope are provider-, model-, deployment-, and account-specific. TokenCops advises only from caller-supplied metadata or an explicitly selected dated preset; absent fields stay unknown.

Preview surfaces

Firewall/gateway heuristics, provenance helpers, conformance checks, plugin discovery, experiments, the dashboard, and governed evolution are usable previews. They are not independently assured security or governance systems.

Plugins are currently manifest metadata only. TokenCops does not execute repository plugin code, every discovered plugin remains untrusted, and signature-shaped fields are reported but not cryptographically verified.

Install from source

Node.js 20 or newer is required.

npm ci --ignore-scripts
npm link
tcop audit /path/to/repository
tcop cbom /path/to/repository

init is optional. The first stateful command lazily creates a git-safe .tokencops/; tcop init adds the task, context, memory, receipt/report, and ownership scaffold used by the supported core. Preview subsystems create their own storage only when invoked. Initialization never overwrites an existing nested .gitignore.

Common workflows

tcop task start "Fix input validation" --acceptance "invalid input rejected;tests pass"
tcop next add "Implement the smallest safe fix"
tcop validate run --command "npm test"
tcop handoff .
tcop resume .
tcop status .

cat test-output.log | tcop compress --mode safe
tcop wrap -- npm test
tcop savings .
tcop benchmark --file test-output.log .

tcop sessions . --source codex
tcop sessions . --dir ./agent-sessions --format jsonl \
  --field-aliases '{"inputTokens":"usage.input","outputTokens":"usage.output","model":"engine.id"}'

tcop hooks show --agent claude-code
tcop hooks install --agent claude-code
tcop hooks install --agent claude-code --global

validate run auto-allows only a single recognized validation command. Shell composition, substitution, expansion, and redirection are not auto-allowed. --force is an explicit human-approval override; it does not create a sandbox.

Bring a fourth provider and model

This definition names neither a bundled provider nor a bundled model:

{
  "id": "acme-http",
  "url": "https://models.example.test/v2/deployments/{model}",
  "credential": {
    "env": "ACME_API_KEY",
    "header": "authorization",
    "prefix": "Bearer "
  },
  "headers": { "x-client": "tokencops" },
  "body": {
    "deployment": "$model",
    "conversation": "$messages"
  },
  "models": [
    { "id": "frontier-x", "tier": "frontier" }
  ]
}

Pass that JSON as DEF (shown here with POSIX-shell syntax), keep the credential only in its declared environment variable, and opt in to the network call:

DEF='{"id":"acme-http","url":"https://models.example.test/v2/deployments/{model}","credential":{"env":"ACME_API_KEY","header":"authorization","prefix":"Bearer "},"headers":{"x-client":"tokencops"},"body":{"deployment":"$model","conversation":"$messages"},"models":[{"id":"frontier-x","tier":"frontier"}]}'
tcop providers validate --definition "$DEF"
tcop invoke --definition "$DEF" --provider acme-http --model frontier-x \
  --prompt "Return a short health check" --allow-network

Provider definitions are bounded JSON data: no executable functions, embedded credentials, credential query parameters, or static authorization headers. HTTPS is required except for loopback HTTP. Private or loopback targets also require --allow-private-network; link-local and metadata targets remain forbidden.

Routing accepts an arbitrary caller catalog. Missing prices are not invented:

tcop route "review production credentials" \
  --models '[{"provider":"sovereign-runtime","id":"frontier-x","tier":"frontier"}]'

Same-named models from multiple providers must be provider-qualified. Pricing is used only when the caller supplies a complete applicable row; cache fields that are absent remain unknown.

Bring a fourth agent

Merge a data-only adapter into .tokencops/config.json:

{
  "version": 1,
  "agents": {
    "adapters": [
      {
        "id": "orbit-agent",
        "detect": [".orbit"],
        "instruction_files": [
          { "pattern": ".orbit/rules/**/*.prompt", "loading": "always" }
        ],
        "session": {
          "dir": "orbit-sessions",
          "format": "jsonl",
          "field_aliases": {
            "inputTokens": "usage.input",
            "outputTokens": "usage.output",
            "cacheReadTokens": "usage.cached",
            "model": "engine.id"
          }
        },
        "resume": {
          "file": "resume-ORBIT.md"
        }
      }
    ]
  }
}

tcop audit .
tcop sessions . --source orbit-agent
tcop resume . --emit orbit-agent

Configured globs and renderers use a bounded, non-executable data contract. Repository adapters cannot inject custom prose into resume artifacts; headings are generated from validated adapter IDs. A configured session directory must stay inside the repository; only an explicit CLI --dir may select an external store. Declared session mappings are labeled declared, not independently schema-verified, and do not imply pricing.

Calibrate without provider lock-in

Canonical JSONL samples can come from any provider/model/tokenizer. Each row contains text, outputTokens, provider, model, tokenizer, and scope:

{"text":"sample output","outputTokens":17,"provider":"acme-http","model":"frontier-x","tokenizer":"acme-v1","scope":"output"}

tcop calibrate . --source canonical --file ./samples.jsonl

Calibration requires at least 10 clean samples per exact provider/model/tokenizer/scope identity. A factor never falls back to another model, another repository, or an unknown identity.

Receipts and honest accounting

Compression savings compare estimated tokens before and after a local transformation; they are counterfactuals, not bill deltas. Cache effects are included only when the selected metadata supplies the relevant rates and usage. Reports distinguish provider-reported, agent-reported, exact-local, and estimated evidence.

compress, wrap, and validate run redact recognized secrets before writing receipts. A redacted secret's original bytes cannot be recovered from the TokenCops receipt. Redaction is pattern-based, so sensitive source logs still require normal access controls; child programs may also write their own files outside TokenCops.

Network and subprocess boundary

• Offline audit, compression, continuity, and reporting paths do not initiate network I/O.
• Live provider invocation, price checking, and ecosystem discovery require --allow-network for each invocation. Provider access to private/loopback addresses requires the additional --allow-private-network flag.
• These flags govern TokenCops-owned network clients only.
• tcop wrap -- ... executes the already-tokenized argument vector without a shell; Windows support is intentionally limited to direct executables plus the standard npm/npx shims. tcop validate run --command ... is the explicit shell-command surface and can execute approved repository code. Both inherit the caller's operating-system permissions and network access; TokenCops is not a process or network sandbox.

Verification and trust artifacts

npm run verify
npm pack --dry-run

npm run integrity recomputes SOURCE-MANIFEST.sha256 and fails on repository inventory drift. The manifest is not signed provenance, does not authenticate a checkout, and does not identify who produced it.

See docs/ARCHITECTURE.md, docs/PHASES.md, docs/spec/TOKENCOPS-OPEN-SPEC.md, the schemas in schemas/, and the security documents in the repository root.

Handoff correctness

Handoffs are historical snapshots and never override current Git state. tcop resume reconciles current HEAD, working tree, source-content fingerprint, validation freshness, required-context closure, task state, and next actions before producing a restart packet. See docs/HANDOFF-PROTOCOL.md.

License

Apache-2.0.