@memberjunction/ai-vectordb

A provider-agnostic abstraction layer for vector databases in MemberJunction. This package defines the VectorDBBase abstract class and all associated types that concrete vector database implementations must fulfill. It contains no provider-specific logic -- only the contracts, data structures, and query interfaces shared across all implementations.

Architecture

graph TD
    subgraph Abstraction["@memberjunction/ai-vectordb"]
        VDBB["VectorDBBase (abstract)"]
        VR["VectorRecord"]
        QO["QueryOptions / HybridQueryOptions"]
        QR["QueryResponse / ScoredRecord"]
        IDX["IndexDescription / IndexList"]
        BR["BaseResponse"]
    end

    subgraph Implementations["Provider Implementations"]
        PC["PineconeDatabase"]
        CUSTOM["Custom Provider"]
    end

    subgraph Consumers["Consuming Packages"]
        SYNC["ai-vector-sync"]
        DUPE["ai-vector-dupe"]
        CORE["ai-vectors (VectorBase)"]
    end

    PC -->|extends| VDBB
    CUSTOM -->|extends| VDBB
    SYNC --> VDBB
    DUPE --> VDBB
    CORE --> VDBB

    style Abstraction fill:#2d6a9f,stroke:#1a4971,color:#fff
    style Implementations fill:#2d8659,stroke:#1a5c3a,color:#fff
    style Consumers fill:#7c5295,stroke:#563a6b,color:#fff

All vector database providers extend VectorDBBase and implement its abstract methods. Consumer code programs against the base class, making it straightforward to swap providers without changing application logic.

Installation

npm install @memberjunction/ai-vectordb

Quick Start

import { VectorDBBase, VectorRecord } from '@memberjunction/ai-vectordb';

// Use any concrete provider that extends VectorDBBase
const db: VectorDBBase = new PineconeDatabase(apiKey);

// Insert a vector record
const record: VectorRecord = {
    id: 'doc-001',
    values: [0.1, 0.2, 0.3 /* ... */],
    metadata: { entity: 'Products', recordId: '12345' }
};
await db.createRecord(record);

// Query for similar vectors
const result = await db.queryIndex({
    vector: [0.1, 0.2, 0.3 /* ... */],
    topK: 10,
    includeMetadata: true
});

if (result.success) {
    for (const match of result.data.matches) {
        console.log(`Match: ${match.id} (score: ${match.score})`);
    }
}

VectorDBBase Abstract Class

All vector database providers must extend this class. The constructor requires an API key, which is validated (must be non-empty) and stored for subclass access via a protected getter.

classDiagram
    class VectorDBBase {
        <<abstract>>
        #apiKey : string
        +constructor(apiKey: string)
        +SupportsHybridSearch : boolean
        +listIndexes()* IndexList
        +getIndex(params)* BaseResponse
        +createIndex(params)* BaseResponse
        +deleteIndex(params)* BaseResponse
        +editIndex(params)* BaseResponse
        +queryIndex(params)* BaseResponse
        +HybridQuery(params) BaseResponse
        +createRecord(record)* BaseResponse
        +createRecords(records)* BaseResponse
        +getRecord(params)* BaseResponse
        +getRecords(params)* BaseResponse
        +updateRecord(record)* BaseResponse
        +updateRecords(records)* BaseResponse
        +deleteRecord(record)* BaseResponse
        +deleteRecords(records)* BaseResponse
    }

    class PineconeDatabase {
        +listIndexes() IndexList
        +queryIndex(params) BaseResponse
    }

    class CustomProvider {
        +SupportsHybridSearch : boolean
        +HybridQuery(params) BaseResponse
    }

    VectorDBBase <|-- PineconeDatabase
    VectorDBBase <|-- CustomProvider

    style VectorDBBase fill:#2d6a9f,stroke:#1a4971,color:#fff
    style PineconeDatabase fill:#2d8659,stroke:#1a5c3a,color:#fff
    style CustomProvider fill:#2d8659,stroke:#1a5c3a,color:#fff

All abstract methods use union return types (BaseResponse | Promise<BaseResponse>) so subclasses can implement them as either synchronous or asynchronous.

Abstract Methods Reference

Index Operations

| Method | Parameters | Returns | Description | |---|---|---|---| | listIndexes() | none | IndexList | List all indexes in the project. | | getIndex(params) | BaseRequestParams | BaseResponse | Retrieve configuration details for a single index by ID. | | createIndex(params) | CreateIndexParams | BaseResponse | Create a new vector index with specified dimension, metric, and optional provider-specific params. | | deleteIndex(params) | BaseRequestParams | BaseResponse | Permanently delete an index and all its records. | | editIndex(params) | EditIndexParams | BaseResponse | Modify an existing index's configuration. | | queryIndex(params) | QueryOptions | BaseResponse | Execute a similarity search against an index by vector values or by record ID. |

Record Operations

| Method | Parameters | Returns | Description | |---|---|---|---| | createRecord(record) | VectorRecord | BaseResponse | Insert a single vector record into an index. | | createRecords(records) | VectorRecord[] | BaseResponse | Insert multiple vector records in batch. | | getRecord(params) | BaseRequestParams | BaseResponse | Fetch a single record by ID. | | getRecords(params) | BaseRequestParams | BaseResponse | Fetch multiple records (IDs passed via data). | | updateRecord(record) | UpdateOptions | BaseResponse | Update a single record's vector values and/or metadata. | | updateRecords(records) | UpdateOptions | BaseResponse | Update multiple records in batch. | | deleteRecord(record) | VectorRecord | BaseResponse | Delete a single record from an index. | | deleteRecords(records) | VectorRecord[] | BaseResponse | Delete multiple records in batch. |

Virtual Methods (Non-Abstract)

| Method | Parameters | Returns | Description | |---|---|---|---| | SupportsHybridSearch (getter) | none | boolean | Returns false by default. Override to return true in providers that implement HybridQuery(). | | HybridQuery(params) | HybridQueryOptions | BaseResponse | Perform a hybrid search combining vector similarity with keyword matching. Default implementation falls back to queryIndex(). |

Hybrid Search

Hybrid search combines dense vector similarity (semantic search) with sparse keyword matching (BM25) to produce more relevant results than either technique alone. This is particularly useful when queries contain specific terms (product names, codes, identifiers) that benefit from exact text matching alongside semantic understanding.

How It Works

1. Check support: Read the SupportsHybridSearch getter before calling HybridQuery().
2. Call HybridQuery: Pass both a vector embedding and an optional keyword query string.
3. Fallback behavior: If the provider does not override HybridQuery(), the base class automatically falls back to a pure vector queryIndex() call, ignoring keyword-specific parameters (KeywordQuery, Alpha, FusionMethod). This means consumer code does not need to branch on provider capability -- it is always safe to call HybridQuery().

HybridQueryOptions

HybridQueryOptions extends QueryParamsBase with the following additional fields:

| Property | Type | Required | Description | |---|---|---|---| | vector | RecordValues | Yes | Embedding vector for the semantic similarity component. | | KeywordQuery | string | No | Keyword query string for BM25/text search. | | Alpha | number | No | Balance between vector and keyword results. 0.0 = pure keyword, 1.0 = pure vector. Default: 0.7. | | FusionMethod | 'rrf' \| 'weighted' | No | Strategy for fusing vector and keyword result lists. Default: 'rrf' (Reciprocal Rank Fusion). |

Plus the inherited QueryParamsBase fields: topK, includeValues, includeMetadata, filter.

Implementing Hybrid Search in a Provider

import { VectorDBBase, HybridQueryOptions, BaseResponse } from '@memberjunction/ai-vectordb';

export class WeaviateDB extends VectorDBBase {
    public override get SupportsHybridSearch(): boolean {
        return true;
    }

    public override async HybridQuery(params: HybridQueryOptions): Promise<BaseResponse> {
        const results = await this.client.hybridSearch({
            vector: params.vector,
            query: params.KeywordQuery,
            alpha: params.Alpha ?? 0.7,
            fusionType: params.FusionMethod ?? 'rrf',
            limit: params.topK,
            includeMetadata: params.includeMetadata
        });
        return { success: true, message: 'Hybrid query complete', data: results };
    }

    // ... remaining abstract method implementations
}

Consumer Usage

const db: VectorDBBase = getVectorProvider();

// Safe to call regardless of provider -- falls back to pure vector search
const results = await db.HybridQuery({
    vector: embedding,
    KeywordQuery: 'quarterly revenue report',
    Alpha: 0.6,
    topK: 20,
    includeMetadata: true
});

// Or check support for conditional UI behavior
if (db.SupportsHybridSearch) {
    showKeywordSearchInput();
}

Implementing a Custom Provider

Follow these steps to add support for a new vector database (e.g., Weaviate, Qdrant, Milvus).

Step 1: Create the Provider Class

import {
    VectorDBBase,
    BaseRequestParams,
    BaseResponse,
    CreateIndexParams,
    EditIndexParams,
    IndexList,
    QueryOptions,
    UpdateOptions,
    VectorRecord
} from '@memberjunction/ai-vectordb';

export class QdrantDB extends VectorDBBase {
    private client: QdrantClient;

    constructor(apiKey: string, host: string) {
        super(apiKey); // Validates the API key
        this.client = new QdrantClient({ apiKey: this.apiKey, host });
    }

    async listIndexes(): Promise<IndexList> {
        const collections = await this.client.getCollections();
        return {
            indexes: collections.map(c => ({
                name: c.name,
                dimension: c.vectors.size,
                metric: c.vectors.distance,
                host: this.host
            }))
        };
    }

    async getIndex(params: BaseRequestParams): Promise<BaseResponse> {
        const info = await this.client.getCollection(params.id);
        return { success: true, message: 'Index retrieved', data: info };
    }

    async createIndex(params: CreateIndexParams): Promise<BaseResponse> {
        await this.client.createCollection(params.id, {
            vectors: { size: params.dimension, distance: params.metric }
        });
        return { success: true, message: 'Index created', data: null };
    }

    async deleteIndex(params: BaseRequestParams): Promise<BaseResponse> {
        await this.client.deleteCollection(params.id);
        return { success: true, message: 'Index deleted', data: null };
    }

    async editIndex(params: EditIndexParams): Promise<BaseResponse> {
        await this.client.updateCollection(params.id, params.data);
        return { success: true, message: 'Index updated', data: null };
    }

    async queryIndex(params: QueryOptions): Promise<BaseResponse> {
        const results = await this.client.search(params);
        return { success: true, message: 'Query complete', data: results };
    }

    async createRecord(record: VectorRecord): Promise<BaseResponse> {
        await this.client.upsert([record]);
        return { success: true, message: 'Record created', data: null };
    }

    async createRecords(records: VectorRecord[]): Promise<BaseResponse> {
        await this.client.upsert(records);
        return { success: true, message: `${records.length} records created`, data: null };
    }

    async getRecord(params: BaseRequestParams): Promise<BaseResponse> {
        const record = await this.client.retrieve(params.id);
        return { success: true, message: 'Record retrieved', data: record };
    }

    async getRecords(params: BaseRequestParams): Promise<BaseResponse> {
        const records = await this.client.retrieveMany(params.data);
        return { success: true, message: 'Records retrieved', data: records };
    }

    async updateRecord(record: UpdateOptions): Promise<BaseResponse> {
        await this.client.update(record);
        return { success: true, message: 'Record updated', data: null };
    }

    async updateRecords(records: UpdateOptions): Promise<BaseResponse> {
        await this.client.updateMany(records);
        return { success: true, message: 'Records updated', data: null };
    }

    async deleteRecord(record: VectorRecord): Promise<BaseResponse> {
        await this.client.delete(record.id);
        return { success: true, message: 'Record deleted', data: null };
    }

    async deleteRecords(records: VectorRecord[]): Promise<BaseResponse> {
        await this.client.deleteMany(records.map(r => r.id));
        return { success: true, message: 'Records deleted', data: null };
    }
}

Step 2: Register with MemberJunction Class Factory (Optional)

If you want MemberJunction's class factory to discover your provider automatically:

import { RegisterClass } from '@memberjunction/global';

@RegisterClass(VectorDBBase, 'QdrantDB')
export class QdrantDB extends VectorDBBase {
    // ... implementation
}

Step 3: Add Hybrid Search Support (Optional)

If your vector database supports hybrid search natively, override the getter and method:

export class QdrantDB extends VectorDBBase {
    public override get SupportsHybridSearch(): boolean {
        return true;
    }

    public override async HybridQuery(params: HybridQueryOptions): Promise<BaseResponse> {
        // Use your provider's native hybrid search API
        const results = await this.client.hybridSearch({
            vector: params.vector,
            keyword: params.KeywordQuery,
            alpha: params.Alpha ?? 0.7,
            limit: params.topK
        });
        return { success: true, message: 'Hybrid query complete', data: results };
    }
}

QueryOptions Reference

QueryOptions is a union type supporting two query strategies:

graph LR
    QPB["QueryParamsBase<br/>topK, includeValues,<br/>includeMetadata, filter"]
    QBV["QueryByVectorValues<br/>+ vector"]
    QBID["QueryByRecordId<br/>+ id"]
    QO["QueryOptions"]

    QPB --> QBV
    QPB --> QBID
    QBV --> QO
    QBID --> QO

    style QPB fill:#2d6a9f,stroke:#1a4971,color:#fff
    style QBV fill:#2d8659,stroke:#1a5c3a,color:#fff
    style QBID fill:#2d8659,stroke:#1a5c3a,color:#fff
    style QO fill:#b8762f,stroke:#8a5722,color:#fff

QueryParamsBase

Shared query parameters inherited by both query strategies.

| Property | Type | Required | Description | |---|---|---|---| | topK | number | Yes | Number of results to return. | | includeValues | boolean | No | Whether to include embedding vectors in results. Default: false. | | includeMetadata | boolean | No | Whether to include metadata in results. Default: false. | | filter | object | No | Metadata filter to narrow the search scope. |

QueryByVectorValues

Query by providing a vector directly. Extends QueryParamsBase.

| Property | Type | Required | Description | |---|---|---|---| | vector | RecordValues | Yes | Vector values output from an embedding model. |

QueryByRecordId

Query using an existing record's stored vector. Extends QueryParamsBase.

| Property | Type | Required | Description | |---|---|---|---| | id | string | Yes | ID of an existing record whose vector to use for the query. |

Usage Examples

// Query by vector values
const resultByVector = await db.queryIndex({
    vector: [0.1, 0.2, 0.3],
    topK: 5,
    includeMetadata: true,
    filter: { category: 'finance' }
});

// Query by record ID (find similar to an existing record)
const resultById = await db.queryIndex({
    id: 'doc-001',
    topK: 10,
    includeMetadata: true
});

Type Reference

Core Types

| Type | Description | |---|---| | VectorRecord<T> | A vector record with id, values (dense embedding), optional sparseValues, and optional metadata. Generic over metadata type. | | RecordValues | Array<number> -- a dense embedding vector. | | RecordSparseValues | Sparse vector representation with parallel indices and values arrays for non-zero positions. | | RecordMetadata | Record<string, RecordMetadataValue> -- arbitrary key-value metadata attached to records. | | RecordMetadataValue | string \| boolean \| number \| Array<string> -- valid metadata value types. | | BaseResponse | Standard response envelope: success (boolean), message (string), data (result payload). |

Index Types

| Type | Description | |---|---| | IndexDescription | Describes an index: name (max 45 chars), dimension, metric, and host URL. | | IndexList | Container with an optional indexes array of IndexDescription. | | IndexModelMetricEnum | 'cosine' \| 'euclidean' \| 'dotproduct' distance metrics for similarity comparison. |

Request/Options Types

| Type | Description | |---|---| | BaseRequestParams | Base request with id (string) and optional data. | | CreateIndexParams | Extends BaseRequestParams with dimension (number), metric (IndexModelMetricEnum), and optional additionalParams. | | EditIndexParams | Extends BaseRequestParams for index configuration changes. | | UpdateOptions<T> | Record update with id, optional values, optional sparseValues, and optional metadata. Values are optional to support metadata-only updates. |

Query Types

| Type | Description | |---|---| | QueryParamsBase | Shared query parameters: topK, includeValues, includeMetadata, filter. | | QueryByRecordId | Extends QueryParamsBase with id -- query using an existing record's vector. | | QueryByVectorValues | Extends QueryParamsBase with vector -- query using a raw embedding array. | | QueryOptions | Union of QueryByRecordId \| QueryByVectorValues. | | HybridQueryOptions | Extends QueryParamsBase with vector, KeywordQuery, Alpha, and FusionMethod for combined vector + keyword search. | | QueryResponse<T> | Query result containing matches (sorted ScoredRecord array), namespace, and optional usage. | | ScoredRecord<T> | Extends VectorRecord with a score field. Score interpretation depends on the index metric. | | OperationUsage | Usage metrics with optional readUnits count. |

Distance Metrics

The IndexModelMetricEnum supports three metrics for similarity comparison:

| Metric | Value | Description | Score Interpretation | |---|---|---|---| | Cosine | 'cosine' | Measures angle between vectors (direction similarity) | Higher = more similar | | Euclidean | 'euclidean' | Straight-line distance between vector endpoints | Lower = more similar | | Dot Product | 'dotproduct' | Measures both direction and magnitude alignment | Higher = more similar |

Available Implementations

| Package | Provider | Hybrid Search | |---|---|---| | @memberjunction/ai-vectors-pinecone | Pinecone | No |

Create additional implementations by extending VectorDBBase and optionally registering with MemberJunction's class factory.

Dependencies

| Package | Purpose | |---|---| | @memberjunction/core | Core MemberJunction functionality | | @memberjunction/global | Global utilities and class registration |

Development

# Build
cd packages/AI/Vectors/Database
npm run build

# Run tests
npm run test

# Watch mode
npm run test:watch

License

ISC