lingest

lingest (Local Ingest) is a command-line tool that generates a single Markdown file named lingest_output.md. It recursively finds all text files in the current directory and its subdirectories and concatenates their content. Each file's content is preceded by a header indicating its relative path.

This is particularly useful for creating a single context file for Large Language Models (LLMs) from your local projects, enabling them to understand the structure and content of your codebase or documentation.

Features

• Directory Tree Visualization: Automatically generates a visual directory tree structure at the beginning of the output, showing the project layout with standard tree characters.

• Recursive Scanning: Traverses directories deep to find all relevant files.

• Content Concatenation: Combines the content of all found text files into one output.

• Clear File Headers: Each file's content is demarcated with FILE: path/to/file.ext for easy identification.

• Customizable Output: Specify the name of the generated Markdown file.

• Flexible Filtering:

  • Ignore Patterns: Use glob patterns to exclude specific files or directories (e.g., build artifacts, temporary files).
  • Include Patterns: Focus the tool on specific file types or directories using glob patterns.

• Overwrite Protection: Prevents accidental overwriting of existing output files unless explicitly forced.

• Quiet Mode: Suppresses informational logs, ideal for scripting or CI/CD pipelines.

• Dry Run Mode: Preview which files would be included and where the output would be saved, without writing anything.

• Standard CLI Interface: Includes --help and --version flags.

Installation

The easiest way to use lingest is with npx:

npx lingest [options]

Alternatively, install it globally using npm, Yarn, or pnpm:

npm install -g lingest

# or

yarn global add lingest

# or

pnpm add -g lingest

Once installed globally, run it directly:

lingest [options]

Usage

Navigate to the root directory of the project you want to process, then run:

lingest [options]

Options

| Option | Alias | Description | Default | | ----------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | | --output | -o | Output file name for the generated Markdown. | lingest_output.md | | --ignore | -i | Comma-separated list of glob patterns to ignore (files or directories). These are added to default ignores. | "" | | --include | -n | Comma-separated list of glob patterns to include. If specified, only files matching these patterns will be processed (still respects ignore patterns). | "" (include all) | | --force | -f | Force overwrite of the output file if it already exists. | false | | --quiet | -q | Suppress informational messages; only errors will be shown. | false | | --dry-run | | List files that would be processed and the final output path, but don't actually write the file. | false | | --no-tree | | Skip generating the directory tree structure at the beginning of the output. | false | | --help | -h | Show this help message and exit. | | | --version | -v | Show the program's version number and exit. | |

Default Ignored Items

By default, lingest ignores common files and directories that are typically not needed in code analysis:

Directories:

• node_modules, .git, __pycache__, .pytest_cache, .tox, .nox, .mypy_cache, .ruff_cache, .hypothesis
• bower_components, .npm, .yarn, .pnpm-store, .gradle, build, .settings, .build, .bundle
• vendor/bundle, target, pkg, obj, packages, bin, .svn, .hg
• venv, .venv, env, .env, virtualenv, .idea, .vscode, .vs
• .cache, .sass-cache, dist, out, site-packages, .docusaurus, .next, .nuxt
• .terraform, vendor, xcuserdata

Files:

• Python: *.pyc, *.pyo, *.pyd, .coverage, poetry.lock, Pipfile.lock
• JavaScript/Node: package-lock.json, yarn.lock, bun.lock, bun.lockb
• Java: *.class, *.jar, *.war, *.ear, *.nar, .classpath, *.gradle, .project
• C/C++: *.o, *.obj, *.dll, *.dylib, *.exe, *.lib, *.out, *.a, *.pdb
• Swift/Xcode: *.xcodeproj, *.xcworkspace, *.xcuserstate, .swiftpm
• Ruby: *.gem, Gemfile.lock, .ruby-version, .rvmrc
• Rust: Cargo.lock, **/*.rs.bk
• .NET/C#: *.suo, *.user, *.nupkg
• Version control: .gitignore, .gitattributes, .gitmodules
• Media files: *.svg, *.png, *.jpg, *.jpeg, *.gif, *.ico, *.pdf, *.mov, *.mp4, *.mp3, *.wav
• Temporary/cache: *.log, *.bak, *.swp, *.tmp, *.temp, .DS_Store, Thumbs.db
• Build artifacts: *.egg-info, *.egg, *.whl, *.so
• Minified files: *.min.js, *.min.css, *.map
• Terraform: *.tfstate*
• The output file itself (e.g., lingest_output.md, or whatever is specified by --output)

Glob Patterns

lingest uses micromatch for glob pattern matching. Common examples:

• **/*.js: Matches all .js files in any directory.
• src/**: Matches all files and folders within the src directory.
• *.log: Matches all .log files in the current directory.
• !src/important.js: (Negation not supported directly but illustrates how to exclude files conceptually.)

Examples

1. Basic usage (generates lingest_output.md in the current directory):

   lingest

   Or using npx:

   npx lingest

2. Specify a custom output file name:

   lingest --output project_snapshot.md

   Or:

   lingest -o project_snapshot.md

3. Add custom ignore patterns (e.g., all dist folders and .log files):

   lingest --ignore "**/dist/**,**/*.log"

4. Only include JavaScript and TypeScript files:

   lingest --include "**/*.js,**/*.ts"

5. Include only .md files from the docs folder, excluding drafts:

   lingest --include "docs/**/*.md" --ignore "docs/drafts/**"

6. Force overwrite an existing output file:

   lingest --force

   Or:

   lingest -f

7. Dry run for Python files:

   lingest --dry-run --include "**/*.py"

8. Quiet mode for scripting:

   lingest -q -o context.md

9. Generate output without directory tree:

   lingest --no-tree

Output Format

Each file’s content in the generated Markdown file will be structured like this:

# FILE: path/to/your/file.ext

(Content of file.ext)

================================================

# FILE: another/path/file.js

================================================

(Content of file.js)

Handling Non-Text Files

If lingest encounters a non-UTF-8 or unreadable file, it will:

1. Log a warning (unless in quiet mode).
2. Include the file’s header.
3. Add [Content not included: Could not be read as UTF-8 text...].

This preserves file structure and alerts you about skipped files.

Contributing

Contributions are welcome!

1. Fork the repository.
2. Create a branch.
3. Make your changes.
4. Submit a pull request with details.

Issues and suggestions? Open one on GitHub (replace with actual link).

License

MIT License. See LICENSE file for details.