Spanner CodeKnowledge

Spanner CodeKnowledge is an advanced, cloud-native code knowledge tool that transforms any codebase into a highly queryable Relational Property Graph and a Semantic Vector Search Engine backed entirely by Google Cloud Spanner and Vertex AI.

By combining the lightning-fast AST parsing of Tree-sitter (supporting 19 languages) with the massive scalability of Cloud Spanner, and seamlessly pairing it with highly concurrent compiled Go binaries, this tool allows AI assistants (like Claude, Cursor, and internal LLM agents) to instantly traverse complex project architectures, understand file dependencies, and perform deep semantic searches without ever hitting a token limit.

🚀 Features

• Universal Parsing (19 Languages): Natively parses C, C++, C#, Dart, Go, Java, JavaScript, TypeScript, Kotlin, Pascal, PHP, Python, Ruby, Rust, Scala, Swift, Vue, Svelte, and Liquid.
• Zero-Config Cloud Provisioning: The CLI automatically creates the required Spanner database and deploys the complex DDL schema on the fly if it doesn't exist.
• Relational Property Graphs: Maps out defines, calls, imports, and extends relationships using Spanner's native CREATE PROPERTY GRAPH capabilities.
• In-Database Semantic Vectors: Intelligently chunks massive files (using AST method truncation) and utilizes Spanner's ML.PREDICT with Vertex AI (gemini-embedding-001) to generate 768-dimensional float32 vector embeddings natively inside the database. Background processing runs in throttled, paginated batches to naturally respect Vertex AI API quotas (1500 QPM) and Spanner gRPC timeout deadlines.
• Lightning Fast Go Binaries: Search commands (search, semantic, trace, usage) are seamlessly delegated to OS-specific, compiled Go binaries (code-knowledge-cli) using a Node.js router, ensuring sub-millisecond graph query execution without loading V8 overhead.
• Agent Skills Injection: The init command can automatically generate a SKILL.md instruction manual directly in your workspace (.claude/skills/, .agents/skills/), teaching AI assistants how to efficiently use the structural Go tools instead of blindly relying on grep or cat.
• MCP Protocol Ready: Built-in Model Context Protocol (MCP) server integration, exposing tools directly to local IDEs and AI assistants.

📋 Prerequisites

Before running the CLI, you must have the following environment set up:

1. Node.js: v18.0.0 or higher.
2. Google Cloud Project: An active GCP project with a billing account attached.
3. Cloud Spanner Instance: An existing Spanner instance (e.g., my-instance). A free-trial instance works perfectly.
4. Vertex AI API Enabled: You must enable the Vertex AI API (aiplatform.googleapis.com) in your project to allow the database to generate semantic embeddings.
5. Google Cloud CLI (gcloud): You must authenticate your local terminal via Application Default Credentials (ADC).

💻 For End-Users: Getting Started

Note: Publishing the package to the npm registry is a TODO. Once published, you will not need to clone this repository to use the tool. You will be able to run it instantly via npx.

1. Authenticate your terminal

Run the following command to securely log in to Google Cloud. The CLI will automatically detect your active project!

gcloud auth application-default login

2. Index your codebase

Navigate to the root directory of the codebase you want to index (e.g., cd my-awesome-app) and run the CLI interactively:

npx code-knowledge init . --index

Alternatively, you can skip the interactive prompts by passing the flags directly:

npx code-knowledge init . \
  --index \
  --spanner-project-id=my-gcp-project \
  --spanner-instance-id=my-spanner-instance \
  --agent-skills=claude,antigravity,codex

What happens next?

1. The tool will auto-generate an isolated Spanner database for this repository (e.g., codeknowledge-db-my-app-1715421000) and apply the schema automatically.
2. It will scan all supported files in your directory and parse them via WASM Tree-sitter.
3. It will extract nodes, resolve cross-file references (using a streaming pagination architecture to prevent OOMs), and build the relational graph.
4. It will trigger a massive background job in Spanner to generate Gemini vector embeddings for your code chunks.
5. It will persist your database config into .codeknowledge/config.json for the Go binaries to read.
6. Once finished, it generates dynamic SKILL instructions (SKILL.md) for your AI assistants, allowing them to instantly query your massive codebase!

3. Querying the Graph (Go Binaries)

Once your codebase is indexed, you can execute blazing-fast commands via the OS-native Go binaries wrapped by the CLI:

• Keyword Search: npx code-knowledge search_symbol "Authenticator" --human
• Semantic Search: npx code-knowledge semantic_search "query evaluation logic" --human
• Call Tracing: npx code-knowledge trace_function_call "ProcessOrder" --direction outbound --depth 3 --human
• Find Usages: npx code-knowledge find_usage "SpannerServer" --human
• File Skeleton: npx code-knowledge skeleton "src/server.ts" (Returns a lightweight AST layout of the file)
• Read File: npx code-knowledge read_file "src/server.ts" --lines 10-20 (Reads specific lines with padding)

🛠 For Developers: Architecture & Contributing

If you want to modify, extend, or build Spanner CodeKnowledge locally, follow these steps.

Architecture Overview

• Frontend / CLI (src/bin): Built with clack for beautiful terminal UIs and commander for flag parsing. Intercepts Go commands and delegates them via the Node Router.
• Go Search CLI (code-knowledge-cli): High-performance Go application pre-compiled for macOS, Linux, and Windows. Exposes commands like search_symbol, semantic_search, and trace_function_call directly against the Spanner Property Graph.
• Parser (src/extraction): WebAssembly (WASM) implementations of Tree-sitter. AST extraction is heavily customized per language. Custom AST walkers strip the bodies of large methods while preserving signatures and docstrings.
• Database (src/db): Interacts heavily with @google-cloud/spanner.
  • Vector Pipeline (queries.ts): Code chunks are first uploaded with "dummy vectors" (zeros). A dedicated throttled while loop runs ML.PREDICT on Spanner to populate the embeddings in batches of 100, natively catching Quota Exceeded and Deadline errors.

Building Locally

1. Clone the repository:

   git clone https://github.com/your-username/code-knowledge.git
   cd code-knowledge

2. Install dependencies:

   npm install

3. Compile the Go Binaries & TypeScript: This command compiles the source, compiles the Go tools for your OS, moves the necessary WASM files to the dist/ folder, and sets the correct executable permissions.

   cd code-knowledge-cli
   GOOS=linux GOARCH=amd64 go build -o ../bin/linux-x64/codesearch .
   cd ..
   npm run build

4. Run the local binary: Instead of using npx, you can now run your local build directly:

   node dist/bin/codeknowledge.js init . --index

Running Tests

The project includes a robust test suite powered by Vitest.

npm run test

Note: The test suite is currently broken. It was originally written for a synchronous SQLite backend and is actively being rewritten to support the new asynchronous Cloud Spanner architecture.

⚖️ License & Attribution

This project is licensed under the MIT License.

This project ("Spanner CodeKnowledge") contains a substantial portion of code derived from the original open-source "CodeKnowledge" project created by Colby Mchenry. For strict compliance and detailed attribution, please refer to the THIRD_PARTY_NOTICES.txt and LICENSE files included in the root of this repository.