📉 context-shrink

The AI-Native Codebase Compiler. Compress codebases into hyper-dense, token-efficient formats designed specifically for LLM attention heads.

🔴 The Problem (The Token Tax & Attention Dilution)

Modern LLMs have massive context windows (up to 2 million tokens), but feeding an entire raw repository to coding agents leads to major issues:

• High Token Costs: Sending thousands of lines of boilerplate code (imports, comments, redundant formatting, test setups) on every prompt is highly expensive.
• Latency Bloat: Processing large contexts increases LLM time-to-first-token (TTFT) and inference times.
• Attention Degradation: LLMs suffer from "lost in the middle" problems. If you dump 100 raw files into context, the model struggles to accurately retrieve functions, types, and logic structures.

Traditional code bases are optimized for human readability (white space, long naming, explicit typing, multiple directories). AI agents do not need formatting or boilerplate; they need raw AST mappings, semantic flow, and contract signatures.

🟢 The Solution

context-shrink compiles a standard codebase into a .agentcontext directory containing token-compressed, semantic maps of the system:

1. Strips Boilerplate: Removes comments, lint ignore statements, structural formatting, and standard import blocks.
2. Interface Isolation: Extracts only type signatures, class definitions, interface contracts, and exports, discarding the deep implementation details of modules that the agent doesn't need to see to understand the architecture.
3. Semantic Code Chunking: Uses a local parser to partition functions into semantic units, indexing them based on dependencies.
4. Token-Optimized Syntax: Compress symbols and structures using a specialized format that takes up 60-80% fewer tokens than raw code while retaining full semantic understanding for the LLM.

🛠️ Technical Architecture & Tech Stack

graph TD
    CodeFiles[Raw Source Code] --> Parser[Tree-Sitter AST Parser]
    Parser --> InterfaceExtractor[Interface & Signature Extractor]
    Parser --> CoreLogicExtractor[Clean Logic Extractor]
    InterfaceExtractor --> SemanticGraph[Dependency & Call Graph Builder]
    CoreLogicExtractor --> TokenCompressor[Token Optimizer & Boilerplate Stripper]
    SemanticGraph --> Compiler[Context compiler]
    TokenCompressor --> Compiler
    Compiler --> AgentContext[Output: .agentcontext maps]

The Tech Stack

• Core Compiler: Written in Rust (or fast TypeScript via Bun) for high-performance file processing.
• AST Parsing: Tree-sitter for robust, multi-language parsing of C, C++, JS/TS, Python, Rust, and Go.
• Tokenizer Tracking: tiktoken or tokenizers library (Hugging Face) to measure exact token sizes dynamically.
• Symbol Indexing: Custom dependency graph builder based on imports to determine which modules are core and which are secondary utilities.

🔬 Core Techniques & Algorithms

1. Interface-Only Stubs (.d.ts pattern for all languages)

For files that aren't the primary focus of the active task, context-shrink creates high-density "contracts":

• In Python: Keeps only class structures, function names, types, and docstrings. Removes function bodies.
• In TypeScript: Generates a single-line declaration of exports, types, and interfaces.
• In Go/Rust: Compiles down to struct definitions and public method signatures. This allows the agent to understand how to call modules without reading the hundreds of lines of code inside the method bodies.

2. Token-Compression Encoding

The compiler performs lossy compression for tokens:

• Symbol aliasing: Swaps long, descriptive local variable names that don't leak out of scope with single-character variables within function boundaries (e.g. mapping const databaseConnectionPoolInstance = ... to const c = ... internally inside the representation if it is not exported).
• Whitespace collapse: Removes all unnecessary newlines, tabs, and double spaces, representing the code in a single-line or condensed AST-like text structure that tokenizers encode much more efficiently.
• Comment-to-Metadata mapping: Parses docstrings into key architectural tags (e.g., #GET #auth #db), replacing long paragraphs with rich search metadata.

🗺️ Roadmap to MVP

• [ ] Phase 1: Code Scraper & Boilerplate Stripper
  • Build a file watcher that filters imports, comments, and white spaces.
  • Setup tokenizer integration to calculate token reduction ratios.
• [ ] Phase 2: Tree-Sitter AST Signature Extractor
  • Implement language parsers (starting with TypeScript and Python).
  • Write logic to extract function signatures and discard method bodies.
• [ ] Phase 3: Dependency Resolution Engine
  • Analyze code imports to construct an in-memory dependency tree.
  • Prioritize core files (high dependents) and compress secondary utilities aggressively.
• [ ] Phase 4: Agent Schema Generator
  • Create the .agentcontext/map.json and .agentcontext/interfaces.txt outputs.
  • Write an integration snippet for developer prompts (e.g., "how to feed this to Claude/Gemini CLI").
• [ ] Phase 5: CLI Packaging
  • Package as a lightweight binary with support for .contextignore configuration.

⭐️ Why it will get GitHub Stars

1. Solves the budget/latency issue: Running LLM queries over large repos is slow and expensive. Providing a tool that drops prompt costs by 75% gets instant community interest.
2. Improves Agent Reliability: By stripping noise, the LLM retrieves key definitions with significantly higher accuracy, directly reducing hallucinated API calls.
3. Slick CLI UX: Outputting clear compiler tables in the terminal showing: Original: 120,400 tokens -> Shrunk: 24,100 tokens (-80%) is highly satisfying.