Codebase Digest

A high-performance, enterprise-grade utility for transforming codebases into structured text digests optimized for Large Language Model (LLM) consumption, documentation generation, and code analysis.

Table of Contents

• Overview
• Key Features
• Quick Start
• Documentation
• Output Formats
• Common Use Cases
• Configuration
• Requirements
• Contributing
• License

Overview

Codebase Digest transforms source code repositories into structured text representations optimized for:

• LLM model ingestion
• Documentation generation
• Code analysis and metrics
• Security auditing

Key Features

• Multiple Output Formats: Text, JSON, Markdown, and Tree structure
• Intelligent Processing: Smart file handling, encoding detection, and metadata extraction
• High Performance: Parallel processing with configurable resource management
• Enterprise Ready: Comprehensive error handling, retry mechanisms, and detailed logging

Quick Start

Installation

Global installation

npm install -g codebase-digest

Local project installation

npm install --save-dev codebase-digest

Basic Usage

CLI usage

codebase-digest ./src -f json -o output.json

Node.js usage

import { processDirectory } from 'codebase-digest';
const result = await processDirectory('./src');

Documentation

Comprehensive documentation is available in:

• CLI Documentation - Complete command-line interface guide with examples
• API Documentation - Detailed programmatic API reference and usage patterns
• Examples - examples

Project Structure

codebase-digest/
├── src/
│   ├── index.js         # Main entry point
│   ├── cli/            # Command-line interface
│   ├── core/           # Core processing logic
│   ├── formatters/     # Output formatters
│   │   ├── json-formatter.js
│   │   ├── markdown-formatter.js
│   │   ├── text-formatter.js
│   │   └── tree-formatter.js
│   └── utils/          # Utility functions
│       ├── encoding.js
│       ├── file-detection.js
│       ├── git.js
│       ├── logger.js
│       └── metadata.js
├── docs/              # Documentation
│   ├── api.md         # API reference
│   ├── cli.md         # CLI usage guide
│   ├── configuration.md # Configuration options
│   └── plugins.md     # Plugin development guide
└── examples/          # Example implementations
    ├── api-usage/     # Programmatic usage examples
    ├── plugins/       # Plugin examples
    └── integrations/  # Third-party integrations

The project is organized to provide a clear separation of concerns:

• src/: Contains the core implementation
• docs/: Comprehensive documentation for all aspects of the tool
• examples/: Real-world usage examples and integrations

Output Formats

Text

Simple, readable text output with configurable formatting:

File: src/index.js
Size: 1.25 KB
Content: ...

JSON

Structured data format for programmatic usage:

{
  "path": "src/index.js",
  "size": 1280,
  "content": "..."
}

Markdown

Rich documentation with syntax highlighting:

## src/index.js
\`\`\`javascript
const code = 'here';
\`\`\`

Tree

Hierarchical view of codebase structure:

.
├── src/
│   ├── index.js (1.25 KB)
│   └── utils/

Common Use Cases

1. LLM Processing

codebase-digest ./src --llm-format --strip-comments

2. Documentation

codebase-digest ./src -f markdown --code-statistics

3. Code Analysis

codebase-digest ./src --code-metrics --git-stats

4. Security Audit

codebase-digest ./src --security-scan --dependency-check

See CLI Documentation for more advanced examples.

Configuration

Basic configuration via command line:

codebase-digest ./src --include "**/*.js" --exclude "**/*.test.js"

Advanced configuration via JSON:

{
  "outputFormat": "json",
  "maxFileSize": 1048576,
  "parallel": true,
  "maxParallelProcesses": 8
}

See CLI Documentation and API Documentation for complete configuration options.

Requirements

• Node.js 14.16.0 or higher
• NPM 6.14.0 or higher

Contributing

1. Fork the repository
2. Create your feature branch: git checkout -b feature/my-feature
3. Commit your changes: git commit -am 'Add my feature'
4. Push to the branch: git push origin feature/my-feature
5. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.