xdb

Intent-driven data collection management CLI for AI agents. Transparently combines LanceDB (vector) and SQLite (relational/FTS) behind a unified interface.

Features

• Dual-engine architecture: LanceDB for vector search, SQLite for metadata and full-text search
• Policy-based collections — declare intent, not implementation
• Automatic embedding via pai embed (no manual vector management)
• JSONL input/output for machine-to-machine workflows
• Upsert semantics with auto-generated UUIDs
• Embedding dimension tracking and consistency validation

Install

From npm

npm install -g @theclawlab/xdb

From source

npm install
npm run build
npm link

Requires [pai] installed for embedding support.

Quick Start

# Create a collection with hybrid (vector + FTS) policy
xdb col init my-docs --policy hybrid/knowledge-base

# Write data (embedding happens automatically via pai)
echo '{"id":"doc1","content":"How to compress files with tar"}' | xdb put my-docs

# Semantic search
xdb find my-docs "compress files" --similar

# Full-text search
xdb find my-docs "tar" --match

# SQL filtering
xdb find my-docs --where "json_extract(data, '$.category') = 'archive'"

# Batch write (JSONL via stdin)
cat records.jsonl | xdb put my-docs --batch

Commands

| Command | Description | |---------|-------------| | xdb put <collection> [json] | Write data (single JSON arg or JSONL via stdin) | | xdb find <collection> [query] | Search with --similar, --match, or --where | | xdb embed [text] | Generate text embeddings via configured pai provider | | xdb col init <name> --policy <p> | Create a collection with a policy | | xdb col list | List collections with stats | | xdb col info <name> | Show collection details | | xdb col rm <name> | Delete a collection | | xdb policy list | List available policies | | xdb config | Manage xdb configuration |

Policies

| Policy | Vector | FTS | Engine | |--------|--------|-----|--------| | hybrid/knowledge-base | content | yes | LanceDB + SQLite | | relational/structured-logs | — | — | SQLite | | relational/simple-kv | — | — | SQLite | | vector/feature-store | tensor | — | LanceDB |

Storage

~/.local/share/xdb/
└── collections/
    └── <name>/
        ├── collection_meta.json
        ├── vector.lance/
        └── relational.db

Documentation

• USAGE.md — Full usage guide with all providers, options, and examples