ydb-qdrant

v6.0.0

Published

8 days ago

Qdrant-compatible Node.js/TypeScript API that stores/searches embeddings in YDB using a global one-table layout with exact and approximate KNN search over serialized vectors.

YDB Qdrant-compatible Service

Qdrant-compatible Node.js/TypeScript service and npm library that stores and searches vectors in YDB using a global one-table layout (qdrant_all_points) with exact KNN search (single-phase over embedding) by default and an optional approximate mode (two‑phase bit-quantized over embedding_quantized + embedding). Topics: ydb, vector-search, qdrant-compatible, nodejs, typescript, express, yql, ann, semantic-search, rag.

Modes:

HTTP server: Qdrant-compatible REST API (/collections, /points/*) on top of YDB.
Node.js package: programmatic client via createYdbQdrantClient for direct YDB-backed vector search without running a separate service.

Promo site: ydb-qdrant.tech
Architecture diagrams: docs page

How it works

Architecture diagram

Documentation

Vector dimensions and embedding models: docs/vector-dimensions.md
Deployment and Docker options: docs/deployment-and-docker.md
Architecture, storage layout, and search modes: docs/architecture-and-storage.md
Evaluation, CI, and release process: docs/evaluation-and-ci.md

Requirements

Node.js 18+
A YDB endpoint and database path
One of the supported auth methods (via environment)

Install

As a dependency in another project (npm package):

npm install ydb-qdrant

For local development of this repo:

npm install

Configure credentials

The server uses getCredentialsFromEnv() and supports these env vars (first match wins):

Service account key file (recommended)
```
export YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS=/abs/path/sa-key.json
```
The JSON must contain: service_account_id, id, private_key.
Metadata service (Yandex Cloud VM/Functions)
```
export YDB_METADATA_CREDENTIALS=1
```

Static IAM token (short‑lived)

export YDB_ACCESS_TOKEN_CREDENTIALS=$(yc iam create-token)

Anonymous (local/dev only)
```
export YDB_ANONYMOUS_CREDENTIALS=1
```

Also set endpoint and database:

export YDB_ENDPOINT=grpcs://ydb.serverless.yandexcloud.net:2135
export YDB_DATABASE=/ru-central1/<cloud>/<db>

Optional env:

# Server
export PORT=8080
export LOG_LEVEL=info
# One-table search tuning (default is 'exact' when unset)
export YDB_QDRANT_SEARCH_MODE=approximate               # approximate or exact (default: exact)
export YDB_QDRANT_OVERFETCH_MULTIPLIER=10               # candidate multiplier in approximate mode

Use as a Node.js library (npm package)

The package entrypoint exports a programmatic API that mirrors the Qdrant HTTP semantics.

Import and initialize a client (reuses the same YDB env vars as the server):

import { createYdbQdrantClient } from "ydb-qdrant";

async function main() {
  // defaultTenant is optional; defaults to "default"
  const client = await createYdbQdrantClient({
    defaultTenant: "myapp",
    endpoint: "grpcs://lb.etn01g9tcilcon2mrt3h.ydb.mdb.yandexcloud.net:2135",
    database: "/ru-central1/b1ge4v9r1l3h1q4njclp/etn01g9tcilcon2mrt3h",
  });

  await client.createCollection("documents", {
    vectors: {
      size: 1536,
      distance: "Cosine",
      data_type: "float",
    },
  });

  await client.upsertPoints("documents", {
    points: [
      { id: "doc-1", vector: [/* embedding */], payload: { title: "Doc 1" } },
    ],
  });

  const result = await client.searchPoints("documents", {
    vector: [/* query embedding */],
    top: 10,
    with_payload: true,
  });

  console.log(result.points);
}

Multi-tenant usage with forTenant:

const client = await createYdbQdrantClient({
  endpoint: "grpcs://lb.etn01g9tcilcon2mrt3h.ydb.mdb.yandexcloud.net:2135",
  database: "/ru-central1/b1ge4v9r1l3h1q4njclp/etn01g9tcilcon2mrt3h",
});
const tenantClient = client.forTenant("tenant-a");

await tenantClient.upsertPoints("sessions", {
  points: [{ id: "s1", vector: [/* ... */] }],
});

The request/response shapes follow the same schemas as the HTTP API (CreateCollectionReq, UpsertPointsReq, SearchReq, DeletePointsReq), so code written against the REST API can usually be translated directly to the library calls.

Example: in-process points search with a shared client

In a typical server application you create a single ydb-qdrant client once and reuse it across requests. Then you can perform vector search (points search) directly in your business logic:

import {createYdbQdrantClient} from 'ydb-qdrant';

let clientPromise: ReturnType<typeof createYdbQdrantClient> | null = null;

async function getClient() {
  if (!clientPromise) {
    clientPromise = createYdbQdrantClient({
      defaultTenant: 'myapp',
      endpoint: 'grpcs://lb.etn01g9tcilcon2mrt3h.ydb.mdb.yandexcloud.net:2135',
      database: '/ru-central1/b1ge4v9r1l3h1q4njclp/etn01g9tcilcon2mrt3h',
    });
  }
  return clientPromise;
}

export async function searchDocuments(collection: string, queryEmbedding: number[], top: number) {
  const client = await getClient();

  const result = await client.searchPoints(collection, {
    vector: queryEmbedding,
    top,
    with_payload: true,
  });

  return result.points ?? [];
}

This pattern avoids running a separate HTTP service: vector search is executed directly against YDB via the shared createYdbQdrantClient instance, while the rest of your code works with plain TypeScript functions.

Recommended Vector Dimensions

For full tables of popular embedding models and their dimensions, see docs/vector-dimensions.md.

Quick Start

Use with IDE agents (Roo Code, Cline)

Option 1: Public Demo (No setup required)

Set Qdrant URL to http://ydb-qdrant.tech:8080
No API key needed
Free to use for testing and development
Shared instance - use X-Tenant-Id header for isolation

Option 2: Self-hosted (Local)

Set Qdrant URL to http://localhost:8080
API key optional (not enforced)
Full control and privacy

Option 3: All-in-one local YDB + ydb-qdrant (Docker)

Run a single container that includes both YDB and ydb-qdrant (no external YDB required):

docker run -d --name ydb-qdrant-local \
  -p 8080:8080 \
  -p 8765:8765 \
  ghcr.io/astandrik/ydb-qdrant-local:latest

HTTP API available at http://localhost:8080
YDB Embedded UI at http://localhost:8765
No credentials or env vars needed for local dev

For detailed configuration and env tuning, see docs/deployment-and-docker.md.

cURL smoke test (Self-hosted)

# 1) Install & run
npm install
npm run dev

# 2) Create a collection (384-d Cosine floats)
curl -X PUT http://localhost:8080/collections/mycol \
  -H 'Content-Type: application/json' \
  -d '{
    "vectors": { "size": 384, "distance": "Cosine", "data_type": "float" }
  }'

# 3) Upsert points
curl -X POST http://localhost:8080/collections/mycol/points/upsert \
  -H 'Content-Type: application/json' \
  -d '{
    "points": [
      {"id": "1", "vector": [0.1,0.2, ... 384 vals ...], "payload": {"repo":"demo"}}
    ]
  }'

# 4) Semantic search (query)
curl -X POST http://localhost:8080/collections/mycol/points/query \
  -H 'Content-Type: application/json' \
  -d '{
    "vector": [0.1,0.2, ... 384 vals ...],
    "limit": 5,
    "with_payload": true,
    "score_threshold": 0.4
  }'

Run

npm run build
npm start
# or during development
npm run dev

Health check:

curl -s http://localhost:8080/health

Docker (self-hosted HTTP server)

Published container: ghcr.io/astandrik/ydb-qdrant

Basic example:

docker run -d --name ydb-qdrant \
  -p 8080:8080 \
  -e YDB_ENDPOINT=grpcs://ydb.serverless.yandexcloud.net:2135 \
  -e YDB_DATABASE=/ru-central1/<cloud>/<db> \
  -e YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS=/sa-key.json \
  -v /abs/path/sa-key.json:/sa-key.json:ro \
  ghcr.io/astandrik/ydb-qdrant:latest

For full deployment options (local builds, all-in-one image, Docker Compose, Apple Silicon notes), see docs/deployment-and-docker.md — including an optional “Persistence across restarts” example for the ydb-qdrant-local image when you want embedded YDB data to survive container restarts.

API Reference

Create collection (PUT /collections/{collection}):

curl -X PUT http://localhost:8080/collections/mycol \
  -H 'Content-Type: application/json' \
  -d '{
    "vectors": {
      "size": 384,
      "distance": "Cosine",
      "data_type": "float"
    }
  }'

Upsert points (POST /collections/{collection}/points/upsert):

curl -X POST http://localhost:8080/collections/mycol/points/upsert \
  -H 'Content-Type: application/json' \
  -d '{
    "points": [
      {"id": "1", "vector": [0.1,0.2, ... 384 vals ...], "payload": {"repo":"demo"}}
    ]
  }'

Search (POST /collections/{collection}/points/search):

curl -X POST http://localhost:8080/collections/mycol/points/search \
  -H 'Content-Type: application/json' \
  -d '{
    "vector": [0.1,0.2, ... 384 vals ...],
    "top": 10,
    "with_payload": true
  }'

Delete points (POST /collections/{collection}/points/delete):

curl -X POST http://localhost:8080/collections/mycol/points/delete \
  -H 'Content-Type: application/json' \
  -d '{"points": ["1"]}'

Architecture and Storage

For details on the YDB one-table storage layout, vector serialization (full-precision and bit‑quantized), approximate vs exact search modes, request normalization, and Qdrant compatibility semantics, see docs/architecture-and-storage.md.

Evaluation, CI, and Release

Badges at the top of this README link to build, test, integration, and recall/F1 workflows. For a deeper explanation of how recall is measured and how publishing to npm works, see docs/evaluation-and-ci.md.