ai-token-reducer

ai-token-reducer is a production-oriented TypeScript/Node.js package for building a compact structural graph of a codebase so AI coding assistants can review code with fewer tokens than naive file-reading workflows.

It parses supported source files once, stores graph structure in SQLite, updates incrementally via file hashes, and exposes minimal-first APIs, a CLI, and an MCP server. The default behavior is intentionally compact: structure first, source last.

The graph now auto-syncs by default before read/query operations, so AI-facing tools see code changes without requiring a manual update step after each edit.

If you want the complete feature/settings reference, read docs/AI_REFERENCE.md. That file is intended to be the authoritative “AI got confused, read this first” document.

Reading Order

• Start here for overview and quick-start.
• Read docs/AI_REFERENCE.md for exhaustive feature, setting, CLI, MCP, and API behavior.
• Read src/types.ts if you need the exact exported TypeScript shapes.

Why This Reduces Tokens

Instead of asking an assistant to read many full files, this package answers review questions from a compact graph:

• symbol relationships instead of raw source dumps
• impacted files and top symbols instead of whole directories
• test gaps and risk hints instead of verbose analysis objects
• suggested next actions so the assistant pulls more context only when needed

The default response model is minimal-first. Full source is never returned unless another layer explicitly asks for it.

Features

• Structural graph for JavaScript, TypeScript, TSX, and initial Python
• Node kinds: File, Module, Function, Class, Test
• Edge kinds: CONTAINS, IMPORTS, CALLS, INHERITS, TESTED_BY
• SQLite-backed persistence with a clean storage adapter boundary
• Incremental rebuilds via file content hashing
• Automatic graph refresh before AI queries
• Live workspace watch mode for continuous updates
• Targeted queries:
  • callersOf
  • calleesOf
  • importsOf
  • testsFor
  • fileSummary
  • impactRadius
  • searchSymbols
• Minimal-first review context via getMinimalContext(task, changedFiles?)
• CLI for local workflows
• MCP server for LLM tool integration

AI Contract

If you are using this package from an AI system, these are the main rules:

• Start with getMinimalContext or a focused query, not source-code reads.
• Assume outputs are intentionally compact.
• Assume detailLevel defaults to "minimal".
• Assume graph reads auto-sync unless auto-update was explicitly disabled.
• Do not expect full file contents or snippets from this package by default.
• Use nextActions to decide what to query next.

Install

npm install
npm run build

Requirements:

• Node.js 20+
• npm

Quick Start

Build a graph for the current repo:

npx ai-token-reducer build --root .

Get compact review context:

npx ai-token-reducer minimal-context --task "Review auth changes" --changed src/auth.ts,src/session.ts

Inspect impact radius:

npx ai-token-reducer impact --file src/index.ts --depth 3 --detail standard

Start the MCP server over stdio:

npx ai-token-reducer serve-mcp --root .

Keep the graph hot in a local terminal:

npx ai-token-reducer watch --root .

CLI Usage

build

Parses the codebase and stores the initial graph.

npx ai-token-reducer build --root . --db ./.ai-token-reducer/graph.db

Example output:

{
  "mode": "build",
  "rootDir": "/repo",
  "dbPath": "/repo/.ai-token-reducer/graph.db",
  "scannedFiles": 7,
  "changedFiles": [
    "src/index.ts",
    "src/lib/helper.ts"
  ],
  "removedFiles": [],
  "stats": {
    "files": 7,
    "nodes": 23,
    "edges": 32,
    "tests": 1,
    "languages": {
      "typescript": 4,
      "tsx": 1,
      "javascript": 1,
      "python": 1
    }
  },
  "nextActions": [
    {
      "query": "minimal-context --task \"Review recent changes\" --changed src/index.ts,src/lib/helper.ts",
      "reason": "Get the compact review starting point."
    }
  ]
}

update

Re-parses only changed files and removes deleted files from the graph.

npx ai-token-reducer update --root .

status

Returns graph stats. This command auto-syncs the graph first.

npx ai-token-reducer status --detail minimal

query

Run focused structural queries. Queries auto-sync the graph first, so they reflect recent file edits.

npx ai-token-reducer query --type callersOf --symbol helper
npx ai-token-reducer query --type testsFor --symbol run
npx ai-token-reducer query --type fileSummary --file src/index.ts --detail standard

impact

Traverse structural neighbors around a changed file or symbol. This command auto-syncs first.

npx ai-token-reducer impact --file src/lib/helper.ts --depth 3
npx ai-token-reducer impact --symbol helper --detail standard

minimal-context

Return a strict token-efficient response with no source code by default. This command auto-syncs first.

npx ai-token-reducer minimal-context --task "Review helper changes" --changed src/lib/helper.ts

Example output:

{
  "detailLevel": "minimal",
  "data": {
    "task": "Review helper changes",
    "graphStatsSummary": {
      "files": 7,
      "symbols": 23,
      "edges": 32,
      "changedFiles": 1
    },
    "riskLevel": "medium",
    "topImpactedSymbols": [
      {
        "name": "helper",
        "kind": "Function",
        "path": "src/lib/helper.ts",
        "score": 6
      }
    ],
    "impactedFileCount": 3,
    "testGapCount": 1,
    "nextSuggestedQueries": [
      {
        "query": "impact --file src/lib/helper.ts",
        "reason": "Expand the structural radius only if the minimal view is insufficient."
      }
    ]
  },
  "nextActions": [
    {
      "query": "query testsFor --symbol helper",
      "reason": "Verify coverage around the highest-risk symbol."
    }
  ]
}

serve-mcp

Runs an MCP server exposing:

• buildOrUpdateGraph
• getAutoUpdateState
• getMinimalContext
• getImpactRadius
• queryGraph
• getReviewContext
• searchSymbols

The MCP server starts a live watcher automatically, so connected AI assistants receive fresh graph answers after local code edits.

watch

Starts a live local watcher that continuously updates the graph when supported source files change.

npx ai-token-reducer watch --root . --debounce 150

Library Usage

import { createCodeReviewGraph } from "ai-token-reducer";

const graph = createCodeReviewGraph({ rootDir: process.cwd() });

await graph.build();

// Queries auto-sync by default.
const context = await graph.getMinimalContext("Review router changes", [
  "src/router.ts",
  "src/routes/admin.ts"
]);

console.log(context.data);

Optional live watch mode:

await graph.startAutoUpdateWatcher();

MCP Usage

Point an MCP client at the stdio server:

npx ai-token-reducer serve-mcp --root .

Tool behavior:

• all tool responses are compact JSON strings
• all responses include suggested next actions
• all queries default to minimal detail
• graph data auto-refreshes before tool reads
• source code is not returned by default

Architecture Overview

Project structure:

• src/parser
  • TypeScript/TSX/JavaScript parsing via TypeScript compiler APIs
  • practical Python extraction via line-based heuristics for MVP
• src/graph
  • graph build/update pipeline
  • symbol/edge resolution
  • incremental update logic using file hashes
  • auto-sync coordination for AI reads
• src/storage
  • SQLite default backend
  • adapter interface for future backends
• src/queries
  • targeted graph queries
  • impact traversal
  • minimal-first context generation
• src/mcp
  • MCP stdio server
• src/cli
  • local command surface

Storage contents:

• file metadata
• file hashes
• graph nodes
• graph edges
• lightweight per-file summaries

Test Suite

Run:

npm test

Coverage includes:

• graph building and querying
• incremental update behavior
• minimal-context output shape

Fixtures live in tests/fixtures/sample-repo.

Design Notes

• compact responses are the default API contract
• graph queries prefer names, counts, and IDs over verbose payloads
• source is intentionally absent unless a caller chooses to fetch it separately
• SQLite is the default because local repeated queries are fast and easy to ship
• the storage boundary is intentionally clean so hosted backends can be added later

Limitations

• Python parsing is intentionally pragmatic for the MVP and does not implement a full Python AST
• call resolution is name-based and best-effort, so very dynamic patterns will not resolve perfectly
• import resolution focuses on relative local imports plus external module placeholders
• unchanged files are not reparsed on update, so some cross-file dynamic rename scenarios still require a broader rebuild
• the watch mode observes existing directories recursively; very unusual workflows that create entirely new nested source directories may need a manual rebuild
• source snippets are not yet provided as a first-class API because the MVP is optimized for minimal structural context

Deferred Features

• richer language-specific semantic resolution
• snippet retrieval with explicit opt-in APIs
• graph diffing across commits
• hosted backends and remote workspace syncing
• ranking models beyond current structural heuristics