igrep

igrep is a fast, persistent, indexed regex search CLI. It stores no files in the searched project: indexes live under the operating system cache directory ($XDG_CACHE_HOME/niah/igrep on Linux and compatible systems), keyed by the canonical root path.

CLI

igrep index --root PATH [--force]
igrep status --root PATH
igrep search --root PATH --pattern PATTERN --json \
  [--fixed-strings] [--ignore-case] \
  [--include GLOB]... [--exclude GLOB]... [--max-results N]
igrep bench --root PATH [--pattern LITERAL]...

Search emits JSON Lines. Match records contain an absolute path, a 1-based line_number, and line. The final record is always a summary:

{"type":"summary","indexed":true,"candidate_files":12,"total_files":500,"elapsed_ms":4}

Exit status is 0 when at least one line matches, 1 when none match, and 2 for usage, regex, filesystem, or index errors. An absent index is built automatically. --include patterns are ORed; excludes take precedence. indexed reports whether sparse-term pruning was possible; fallback summaries also include fallback_reason. Summaries may include query_terms, while index and status report the term count, postings bytes, index format, and algorithm.

The index respects Git ignores, global Git ignores, .ignore, and hidden-file conventions. UTF-8 text files up to 16 MiB are indexed; NUL-containing and non-UTF-8 files are skipped. A changed Git HEAD rebuilds the index. Dirty and untracked Git files are indexed into a transient in-memory sparse overlay on every search (deleted paths are removed from baseline candidates). Non-Git roots compare indexed file metadata before searching.

How candidate pruning works

The index uses frequency-weighted sparse n-grams rather than exhaustive fixed trigrams. At index time, the linear-time BuildAllNgrams monotonic-stack algorithm emits at most roughly twice as many terms as input bytes. At query time, BuildCoveringNgrams emits a minimal covering subset with a maximum term length of 16 bytes. Index-time terms longer than that cap are discarded because no query can request them. Every query term is guaranteed to occur in the retained index terms of every document containing that query literal.

Rare byte pairs receive high priority and common pairs low priority, producing longer, more selective terms around rare source-code sequences. Equal buckets use a deterministic hash tie-break. ASCII is lowercased before both indexing and extraction. For Unicode ignore-case searches, files containing non-ASCII bytes are conservatively added back to the candidate set.

The on-disk layout matches Cursor's described design: a sorted hash→offset lookup table (terms.bin) and a contiguous postings file (postings.bin). Only the lookup table is memory-mapped; posting lists are read at the resolved offsets. Storing 64-bit content hashes instead of raw n-gram bytes keeps the table dense and is safe because collisions only widen candidates. Terms that occur in a single file store that file id directly in the lookup value, so singleton postings never touch postings.bin. Regex HIR is analyzed conservatively: mandatory concatenated literals are intersected and fully constrained alternation branches are unioned. Short literals, classes without a mandatory literal, optional-only expressions, and alternations containing an unconstrained branch fall back to direct scanning. Pruning admits false positives but not false negatives; Rust's regex engine performs final verification.

For Git repositories, the persisted index is tied to the current HEAD. Modified and untracked files form a dirty overlay that is removed from baseline candidates and re-indexed in memory with the same sparse n-gram algorithm; only overlay candidates that survive pruning are verified. Deleted paths are omitted.

npm

npm install @niah/igrep

The package installs the release binary and verifies it against SHA256SUMS. It exports:

import { binPathFor, igrepPath, resolveIgrepPath } from "@niah/igrep";

binPathFor({ os: "linux", arch: "x64" }); // .../bin/linux-x64/igrep
console.log(igrepPath); // current platform binary path

Binaries live under bin/<platform>-<arch>/igrep (same layout as @vscode/ripgrep-universal). Set IGREP_BINARY_PATH to force a local binary, or IGREP_SKIP_DOWNLOAD=1 to skip postinstall downloads. The package also exposes an igrep executable that forwards all arguments and exit status to the native CLI.

Releasing

Publishing is tag-driven via GitHub Actions, authenticated with the NPM_TOKEN repository secret.

1. Bump version in both package.json and Cargo.toml (keep them equal), and run a build so Cargo.lock picks up the new version — the release job builds with --locked and fails on a stale lockfile.
2. Commit and push to main.
3. Tag and push: git tag -a vX.Y.Z -m "vX.Y.Z" && git push origin vX.Y.Z
4. .github/workflows/release.yml builds all platform binaries, creates the GitHub release, then publishes @niah/igrep from the npm-publish environment.

Moving to npm Trusted Publishing (OIDC) means configuring a Trusted Publisher on npmjs.com listing workflow release.yml, repository ineedthis/igrep, and environment npm-publish, then dropping both registry-url from the setup-node step and NODE_AUTH_TOKEN from the publish step. Without the publisher registered, npm never starts the token exchange and the publish fails with ENEEDAUTH.

Development

Rust 1.91.1 is the supported toolchain.

ASDF_RUST_VERSION=1.91.1 cargo fmt --check
ASDF_RUST_VERSION=1.91.1 cargo clippy --all-targets -- -D warnings
ASDF_RUST_VERSION=1.91.1 cargo test
IGREP_BINARY_PATH=/path/to/igrep npm test

Frequency table

src/bigram_weights.bin is a checked-in 256x256 one-byte bucket table. It was generated from source-like UTF-8 files in the VS Code/Code OSS repository at commit 6452d926be4e943fd55958104d64674eb40df7a2:

python3 scripts/generate_bigram_weights.py \
  /Users/mcloutier/Sites/code src/bigram_weights.bin

The recorded corpus contained 24,579 files and 346,820,496 bytes. The table SHA-256 is 6bb77b6822edebf1adeb59455f0404d99c048104cae08ee34f90c3994ee9a6c7. The script deterministically filters generated/dependency directories, folds ASCII case, counts overlapping pairs, and logarithmically maps low frequency to high weight. End-user indexing never trains or changes these weights. A future table change must also change the index algorithm/version.

Benchmarking

The deterministic benchmark builds a legacy v0.1 trigram inverted index in the same hash→offset layout beside the current sparse index, then compares term count, index bytes, query term count, and baseline candidate count:

ASDF_RUST_VERSION=1.91.1 cargo run --release -- \
  bench --root /path/to/corpus --pattern function --pattern workbench

See THIRD_PARTY_NOTICES.md for attribution of the sparse n-gram algorithms.

Licensed under the MIT License.