code-search-mcp

@bglocation/code-search-mcp — a semantic code index (RAG) + MCP server for any codebase. Lets an AI agent (e.g. Claude Code) search a repository semantically — by meaning — instead of only by text (grep/find), cutting context tokens and missed matches on large projects.

It is project-agnostic: point it at any repo via a config file. Vectorization runs fully offline on a local model — your code never leaves the machine.

The CLI binaries keep the short rag-* names (rag-index, rag-init, rag-mcp, rag-usage).

Design and rationale: THEORY.md.

How it works

INDEXING (CLI)                          QUERY (MCP server)
  walk repo                               "fix the auth bug"
   → chunk (AST / markdown / lines)         → embed query (same model)
   → embed (local, offline)                 → kNN search in the vector store
   → store in SQLite + sqlite-vec           → top-K chunks (path + line range)

The embedding model only finds relevant chunks; the agent then reads the original source text of those chunks. Vectors never reach the agent.

For the concepts behind this — embeddings, the two-model relationship, what the 384-d vector actually is, and how to support other languages — see THEORY.md.

Status

Complete and in use. Indexing, the MCP server, the eval harness, auto-reindex git hooks, usage logging, and multi-language (tree-sitter) chunking are all built and tested.

| Component | Status | |---|---| | Config loader (rag.config.json) | ✅ | | File walker (.gitignore, segments, binaries) | ✅ | | Chunkers: line (fallback), TS/JS + Python + Go + Rust + Java + C/C++ + Kotlin + Swift + Dart/Flutter (AST via tree-sitter), Markdown (headings), YAML (top-level keys) | ✅ | | Local embedder (transformers.js, offline) | ✅ | | Vector store (SQLite + sqlite-vec, kNN) | ✅ | | Incremental reindex (file-hash change detection) | ✅ | | rag-index CLI + auto-reindex git hooks | ✅ | | MCP server (search_codebase, get_chunk, index_status, reindex) | ✅ | | Eval harness (file + symbol-level hit@5 / MRR) + usage logging | ✅ |

Acceptance: a full index of a ~1,150-file TypeScript/React codebase processes into ~5,500 chunks offline with no errors; a human-validated 50-query acceptance set scores hit@5 84% / MRR 0.69 with bge-small-en.

Quick start (published package)

cd /your-project
npx -p @bglocation/code-search-mcp rag-init

rag-init detects your project layout, writes rag.config.json + .mcp.json (with mcpServers.rag), patches .gitignore, and installs git hooks — then asks whether to build the index now. One command, no manual config required.

Flags: --dry (preview only), --yes (build index without asking), --no-index (skip index step — CI).

Using it across several repos? Install globally (see Install) and just run rag-init in each — no npx, no scope prefix.

Install

Requires Node ≥ 20.

Globally (recommended — it's a dev tool you point at many repos):

npm install -g @bglocation/code-search-mcp

This puts the four CLIs on your PATH, usable in any repository: rag-init, rag-index, rag-mcp, rag-usage. Then, in any project, just run the bin directly — no npx, no scope prefix:

cd /any-project
rag-init            # configure this repo (writes rag.config.json + .mcp.json)

In a single project (as a dev dependency), to pin the version per repo:

npm install -D @bglocation/code-search-mcp
npx rag-init        # npx finds the bin in node_modules/.bin

Without installing (one-off try):

npx -p @bglocation/code-search-mcp rag-init

From source (to develop the tool itself):

git clone https://gitlab.com/bglocation/code-search-mcp
cd code-search-mcp
npm install
npm run build      # → dist/
npm test           # fast unit tests (offline, no model download)

Configuration

Create a rag.config.json in the project you want to index:

{
  "segments": [
    { "name": "src", "root": "src", "include": ["**/*.{ts,tsx}"] },
    { "name": "docs", "root": ".", "include": ["*.md", "docs/**/*.md"] }
  ],
  "exclude": ["**/node_modules/**", "**/*.test.ts", "**/dist/**"],
  "embedder": { "provider": "local", "model": "Xenova/bge-small-en-v1.5" },
  "chunk": { "maxTokens": 512, "overlapLines": 8 },
  "store": { "path": ".rag/index.db" }
}

| Field | Meaning | |---|---| | segments[] | Independent index areas (e.g. separate gits in a monorepo). root is the directory; include are globs relative to it. name tags each chunk so queries can filter by segment. | | exclude | Globs dropped from every segment (on top of built-in binary/lockfile exclusions). .gitignore is always respected. | | embedder.provider | local only — the model runs on your machine (offline). | | embedder.model | Model id or short alias (see below). | | chunk.maxTokens | Target chunk size (approx. tokens). | | chunk.overlapLines | Lines of overlap between adjacent line-windowed chunks. | | store.path | Where the vector DB is written. Keep it under .rag/ and gitignored. |

All optional fields have defaults; only segments is required.

Embedder (offline)

Vectorization uses @huggingface/transformers (ONNX, CPU). The model is downloaded once on first use, then cached and run fully offline — no API keys, no network, no code leaving the machine.

Supported models (each 384-dim):

| Config value (or alias) | Pooling | Notes | |---|---|---| | Xenova/bge-small-en-v1.5 (bge-small) | CLS | Default. | | Xenova/all-MiniLM-L6-v2 (all-minilm) | mean | Alternative. | | Xenova/paraphrase-multilingual-MiniLM-L12-v2 (multilingual-minilm) | mean | Multilingual — regresses English; use only for mostly non-English code. | | Xenova/multilingual-e5-small (e5-small) | mean | Multilingual, asymmetric query:/passage: prefixes — regresses English. | | Xenova/bge-m3 (bge-m3) | CLS | Large multilingual (1024-d, q8, 560 MB) — no gain over e5-small. |

Changing embedder.model changes vector dimensions/semantics — rebuild the index with rag-index --full --reset afterwards.

Pooling is pinned per model (it's a modeling choice and can't be auto-detected); picking the wrong one silently degrades quality.

Model cache. Models are cached in ~/.cache/rag-mcp/models/ — a single user-wide location, so the model downloads once and is shared across every project and invocation (the model is identical everywhere). Override the location with the RAG_MODEL_CACHE env var (useful in CI or sandboxes).

Offline by default. The tool never downloads anything unless RAG_ALLOW_DOWNLOAD=1 is set: a model missing from the cache fails with an actionable error instead of a silent network fetch. Fetch a model once with the flag set (e.g. RAG_ALLOW_DOWNLOAD=1 rag-index --config rag.config.json); afterwards every run is fully offline — verify with HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1.

The model has a hard 512-token input limit. Every chunker keeps chunks within chunk.maxTokens (default 512): the line and markdown chunkers window by budget, and oversized AST symbols are split into sub-windows that repeat the symbol signature instead of being truncated. Chunk size is estimated (chars/4), so a window can still occasionally graze the model's exact tokenizer count.

Usage — indexing

After rag-init (or a hand-written rag.config.json), build and refresh the index with the rag-index binary.

Installed (global or via npx) — the package puts rag-index on PATH:

rag-index --config rag.config.json --full            # first full index
rag-index --config rag.config.json                   # incremental (skips unchanged files, default)
rag-index --config rag.config.json --segment src     # one segment only
rag-index --config rag.config.json --reset           # delete store & rebuild (after a model change)

Not installed globally? Run the bin through npx with -p (e.g. npx -p @bglocation/code-search-mcp rag-index --config rag.config.json --full). The -p is required because the bins are named rag-*, not after the package.

From a source checkout — the compiled binaries live under dist/:

node dist/cli/rag-index.js --config rag.config.json --full
node dist/cli/rag-index.js --config /path/to/other-project/rag.config.json

| Flag | Meaning | |---|---| | -c, --config <path> | Config file (default rag.config.json). Relative segment.root and store.path resolve against the config file's directory, not the shell's cwd. | | --changed | Only re-index files whose content hash changed (default). | | --full | Re-index everything, ignoring stored hashes. Use after changing chunking settings (chunk.maxTokens, include globs, etc.) to ensure all files are re-chunked. | | --reset | Delete the vector store and re-index from scratch. Use after changing embedder.model to avoid dimension mismatches (implies --full). | | -s, --segment <name> | Process only the named segment. | | -h, --help | Usage. |

Progress and the summary (added / skipped / removed / total-chunks / time) go to stderr; exit code is 0 on success, 1 on error — suitable for CI or a git hook. Incremental runs after a small edit re-embed only the changed files.

Auto-reindex on commit, checkout, and merge (git hooks)

Keep the index fresh automatically: three hooks run rag-index --changed in the background after the most common events that change working-tree content. All are non-fatal — commits/checkouts/merges never wait for embeddings and never fail because of a hook.

| Hook | When git fires it | Reindex condition | |---|---|---| | post-commit | after every local commit | always | | post-checkout | after git checkout / git switch | only on branch switch ($3=1); file restores are skipped | | post-merge | after git pull / git merge | always |

rag-init installs these automatically. To manage them separately — re-install after moving the checkout, opt into Husky, or remove them — run the hooks helper (dist/hooks/install-hooks.js) from a source checkout:

node dist/hooks/install-hooks.js --config rag.config.json              # install into every repo backing the config
node dist/hooks/install-hooks.js --config rag.config.json --husky      # opt in to the tracked .husky/<hook> in Husky repos
node dist/hooks/install-hooks.js --config rag.config.json --dry        # preview which repos would get the hooks
node dist/hooks/install-hooks.js --config rag.config.json --uninstall  # remove the managed blocks again

The installer reads the segments in rag.config.json, asks git which repository each segment lives in (git -C <root> rev-parse --show-toplevel) and where that repo runs hooks (git rev-parse --git-path hooks, which honors core.hooksPath), then writes three hooks per distinct repo. This makes it topology-agnostic with no flags (Husky repos are the one exception — see below):

| Topology | .git | Hooks installed | |---|---|---| | Single project | one, at the repo root | 3 | | Monorepo (many packages) | one, at the repo root | 3 | | Separate repos under a shared root | one per sub-repo; root is not a repo | 3×N (three per sub-repo) |

How it behaves:

• Idempotent. Re-running replaces the managed block in place (fenced by # >>> rag-mcp auto-reindex (managed) >>> markers). It never duplicates.

• Coexists with an existing hook. If a target hook file already exists, the block is appended and the rest of the file is left untouched; --uninstall strips only the managed block. (For Husky, this applies to .husky/<hook> — see the Husky section below for where the file lives and the opt-in.)

• Background + locked. Each hook detaches scripts/reindex-bg.sh, which takes a single-writer lock (.rag/reindex.lock) so concurrent triggers (e.g. a commit arriving while a checkout reindex is still running) don't stack, and logs to .rag/reindex.log. Build dist/ first (npm run build); if it's missing or node isn't on PATH, the run is skipped with a note in the log — the triggering git operation still succeeds.

• Safe alongside foreign hooks. A hook written in a non-sh language (a Python or Node shebang) is skipped with a warning rather than corrupted.

• Requires a POSIX shell (sh). On Windows use Git Bash / the shell git ships.

The hooks bake in absolute paths to the runner and the config at install time, so moving the checkout means re-running the hooks helper. Logic lives in reindex-bg.sh, so changing behaviour needs no reinstall.

Husky-managed repos (v9+)

Husky points git at its own hooks dir (core.hooksPath = .husky/_) and regenerates that dir on every npm/yarn install, so a block written there is wiped by the next install — and it is gitignored anyway. The installer detects this (git rev-parse --git-path hooks resolves to …/.husky/_) and instead writes the durable user hook one level up — .husky/post-commit, .husky/post-checkout, .husky/post-merge. Husky's .husky/_/… stub dispatches to it with arguments forwarded, so the post-checkout $3 branch-switch guard still works. Detection keys on the _ basename, so a custom Husky root (.config/husky/_) works too.

Unlike .git/hooks, .husky/<hook> is tracked by git — installing it is a committed change shared with your whole team. Because that is a bigger commitment than a local hook, it needs explicit consent:

• install-hooks.js --husky — opt in (scriptable, for CI/automation).
• On a TTY without the flag, the installer asks once per repo.
• With no flag and no TTY (CI, pipes), the repo is skipped with a hint (skipped-husky-optin) — it never silently modifies a tracked file or hangs on a prompt.

rag-init follows the same rule for its hook step: it prompts on a TTY and skips (with the same hint) otherwise.

--uninstall is never gated. If a managed block from a pre-Husky install is still sitting in .git/hooks, the installer warns that git now ignores it (it is dead — delete it from .git/hooks/<hook> manually).

Usage — programmatic (library API)

The package exposes its pipeline as a library — the same API the MCP server and CLIs are built on:

import { loadConfig, createEmbedder, reindex, VectorStore } from '@bglocation/code-search-mcp';

const config = loadConfig('rag.config.json');
const embedder = createEmbedder(config.embedder);
await reindex({ config, embedder, mode: 'incremental' });

const store = VectorStore.open(config.store.path, embedder.dimensions, embedder.modelId);
const [queryVec] = await embedder.embed(['where is auth handled?']);
const hits = store.search(queryVec, 8);          // [{ chunk, score }]
store.close();

MCP server

rag-init writes this automatically. If you need to hand-edit, the entry Claude Code looks for in .mcp.json:

// .mcp.json (place alongside rag.config.json)
{
  "mcpServers": {
    "rag": {
      "command": "rag-mcp",
      "args": ["--config", "rag.config.json"]
    }
  }
}

From a source checkout (without a global/npx install), point node at the compiled server:

{
  "mcpServers": {
    "rag": {
      "command": "node",
      "args": ["dist/server/server.js", "--config", "rag.config.json"]
    }
  }
}

The server flag:

| Flag | Meaning | |---|---| | -c, --config <path> | Config file (same rag.config.json used by rag-index). Relative paths inside it resolve from the config file's directory. |

On startup the server reads store_meta from the database to verify that the configured embedding model matches the one the index was built with. A mismatch produces a clear Dimension mismatch → rebuild error.

Tools

The server exposes four tools over the MCP protocol:

search_codebase

Semantic search: embeds the query with the same model the index was built with, runs kNN in the vector store, and returns the most relevant chunks.

{ "query": "auth token validation",        // required
  "k": 8,                                   // optional, default 8, max 100
  "segment": "src"                          // optional, restrict to one segment
}

Returns (structuredContent):

{ "results": [
    { "id": "<sha1>", "filePath": "...", "startLine": 5, "endLine": 42,
      "segment": "src", "kind": "function", "symbol": "validateToken",
      "score": 0.83 }
  ]
}

Each result also appears as a clickable path:line reference in the text content. score is cosine similarity (0–1; higher is more relevant). id can be passed directly to get_chunk.

get_chunk

Fetches the full, untruncated text of a single chunk by its id. Use it to expand a search_codebase hit when the snippet is not enough.

{ "id": "<sha1 from search_codebase>" }   // required

Returns (structuredContent):

{ "id": "...", "filePath": "...", "startLine": 5, "endLine": 42,
  "segment": "src", "kind": "function", "symbol": "validateToken",
  "language": "typescript", "fileHash": "...", "text": "<full source>" }

Unknown id → isError: true with a clear message.

index_status

Reports index health without requiring any arguments.

{}

Returns (structuredContent):

{ "chunks": 5713, "files": 1238, "modelId": "Xenova/bge-small-en-v1.5",
  "dimensions": 384, "lastIndexed": "2026-06-04T12:00:00.000Z",
  "segments": [
    { "segment": "src",  "chunks": 4194, "files": 1130 },
    { "segment": "docs", "chunks": 1519, "files": 108  }
  ]
}

Use this to check whether the index is built and roughly current before starting a session.

reindex

Refreshes the index without leaving the agent session. Runs incrementally (only changed/deleted files) unless the underlying files have all been re-written.

{ "paths": ["/abs/or/rel/path/to/file.ts"],  // optional — restrict to files
  "segment": "src"                             // optional — restrict to segment
}

Without arguments, re-indexes all segments incrementally (unchanged files are skipped by content hash).

Returns (structuredContent):

{ "added": 1, "skipped": 4, "removed": 0,
  "totalChunks": 5714, "durationMs": 320,
  "unmatchedPaths": []   // paths that matched no indexed file
}

Only one reindex call runs at a time; a concurrent call is rejected with already in progress.

⚠️ Model change requires a full rebuild. The vector dimensions are fixed in the database schema at index creation time. After switching embedder.model in the config, run rag-index --reset — it deletes the old store (including WAL sidecars) and rebuilds from scratch. Then restart the MCP server so it opens the new store (the running server holds an open file handle to the old database).

Typical session flow

1. rag-index --config rag.config.json --full   # first-time index (one-off)
2. Start Claude Code with .mcp.json pointing at this server
3. Agent calls search_codebase("your question") → gets ids
4. Agent calls get_chunk(id) for the most promising hits
5. Agent calls reindex() after you edit files → index stays current

Monitoring usage

The MCP server logs every tool call (query, result count, top score, latency) to .rag/usage.jsonl — one JSON line per call, appended in the background. At ~5 MB the file rotates to .rag/usage.jsonl.1 (replacing the previous generation), so it never grows unbounded. Logging is always non-fatal: an I/O error is printed to stderr but never propagates to the caller. Disable it with RAG_USAGE_LOG=0 in the server's environment. The log stores query text verbatim — keep .rag/ gitignored (the hook installer warns when it isn't).

Print a summary report with the rag-usage binary:

rag-usage --config rag.config.json

From a source checkout:

node dist/cli/rag-usage.js --config rag.config.json

Example output:

RAG-MCP Usage Report
====================

search_codebase: 24 call(s)
  Top score:  min 0.61  median 0.79  max 0.94
  Latency:    avg 38ms  p95 87ms
  Follow-up:  67% of searches followed by get_chunk (≤5 min)

get_chunk: 18 call(s)  (found 17, not-found 1)

reindex: 2 call(s)

Top queries:
    4×  "auth token validation"
    3×  "rate limiting middleware"

Top segments:
   14×  src
   10×  docs

The follow-up rate — fraction of searches where get_chunk is called within 5 minutes — is the primary usefulness signal: a high rate means the agent frequently digs deeper after finding relevant hits.

The log is in .rag/ which is already gitignored. Reset it by deleting .rag/usage.jsonl.

Limitations

• English-centric embedder. bge-small-en is trained on English, so heavily non-English identifiers and comments retrieve worse. Multilingual models (e5-small, bge-m3) were A/B-tested: they close the non-English gap but regress the dominant English case (file hit@5 87% → 40–60%), so bge-small-en stays the default. Switch embedder.model only if you index mostly non-English code.
• Brute-force kNN. sqlite-vec scans every vector per query, so latency grows linearly with index size — but stays interactive well past 100k files: ~5 ms mean over an 18k-chunk index (microsoft/TypeScript), ~45 ms over ~150k chunks (kubernetes/kubernetes). Approximate (ANN) indexing only becomes worthwhile around ~350k+ chunks; per-segment stores shard naturally past that.
• Reranking ships off. A local cross-encoder over the top-K is available behind RAG_RERANK=1, but the stock ms-marco-MiniLM web reranker regressed retrieval on code and Markdown (file hit@5 87% → 67%), so it is disabled by default. Point RAG_RERANK_MODEL at a code-aware cross-encoder before enabling.
• Symbol-level metrics are partial. The eval harness scores file- and symbol-level hit@5 / MRR, but the larger acceptance set is file-level only; a firmer multi-language, symbol-level verdict needs a bigger annotated corpus.

Roadmap (roughly in value order)

1. More languages — adding one is data, not code: a registry entry plus a vendored grammar .wasm. Shipped today: TS/JS, Python, Go, Rust, Java, C/C++, Kotlin, Swift, Dart/Flutter, plus Markdown and YAML. Next up: C#, Ruby, PHP.
2. Hybrid retrieval — fuse semantic kNN with a lexical signal (the eval harness already carries a grep baseline to measure any lift).
3. Scale & sharing — ANN tuning for very large monorepos, and a shared / CI-built index cache so developers don't each rebuild.
4. Richer chunk metadata — surface imports, callers, and doc links through get_chunk for structured context beyond raw text.

Development

• Tests use Vitest with the AAA pattern; unit tests run offline.

• E2E and real-model integration tests are opt-in:

  RAG_RUN_MODEL_TESTS=1 npm test

• After changing embedder.model, run rag-index --reset to delete the old store and rebuild with the new dimensions (vector dimensions are fixed in the store schema at creation time).

• The e2e test (src/server/server.e2e.test.ts) spawns the real server binary via StdioClientTransport to catch stdout leaks and protocol issues that unit tests cannot detect.

Design rationale and concepts live in THEORY.md.