@matteuccimarco/slim-vector-db

SLIM compression for vector database metadata. Reduce metadata storage by 40-50% in Pinecone, Chroma, and other vector databases.

Installation

npm install @matteuccimarco/slim-vector-db

Quick Start

Pinecone

import { Pinecone } from '@pinecone-database/pinecone';
import { SlimPinecone } from '@matteuccimarco/slim-vector-db/pinecone';

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });

const slim = new SlimPinecone({
  client: pinecone,
  indexName: 'my-index',
  preserveFields: ['category'], // Keep for filtering
});

// Upsert with SLIM-encoded metadata
await slim.upsert([
  {
    id: 'doc-1',
    values: [0.1, 0.2, 0.3, ...],
    metadata: {
      category: 'tech',           // Preserved for filtering
      content: 'Long text...',    // SLIM-encoded
      attributes: {               // SLIM-encoded
        author: 'Alice',
        tags: ['ai', 'ml'],
      },
    },
  },
]);

// Query with automatic decoding
const results = await slim.query({
  vector: [0.1, 0.2, 0.3, ...],
  topK: 10,
  filter: { category: 'tech' },
});

console.log(results[0].metadata);
// { category: 'tech', content: 'Long text...', attributes: { author: 'Alice', tags: ['ai', 'ml'] } }

ChromaDB

import { ChromaClient } from 'chromadb';
import { SlimChroma } from '@matteuccimarco/slim-vector-db/chroma';

const chroma = new ChromaClient();

const slim = new SlimChroma({
  client: chroma,
  preserveFields: ['source'],
});

const collection = await slim.getOrCreateCollection('documents');

// Add with SLIM-encoded metadata
await collection.add({
  ids: ['doc-1', 'doc-2'],
  embeddings: [[0.1, 0.2], [0.3, 0.4]],
  metadatas: [
    { source: 'file.pdf', content: { page: 1, text: '...' } },
    { source: 'file.pdf', content: { page: 2, text: '...' } },
  ],
});

// Query with automatic decoding
const results = await collection.query({
  queryEmbeddings: [[0.1, 0.2]],
  nResults: 5,
});

How It Works

Metadata is stored with SLIM encoding:

// Original metadata
{
  category: 'tech',
  attributes: { author: 'Alice', tags: ['ai', 'ml'], rating: 4.5 }
}

// Stored in vector DB (with SLIM)
{
  category: 'tech',  // Preserved for filtering
  attributes: '__slim__{author:Alice,tags:@[ai,ml],rating:#4.5}'  // ~40% smaller
}

Options

const slim = new SlimPinecone({
  client: pinecone,
  indexName: 'my-index',

  // Fields to compress (default: all object fields)
  slimFields: ['content', 'attributes'],

  // Fields to keep uncompressed (for filtering)
  preserveFields: ['category', 'timestamp'],

  // Disable SLIM (pass-through mode)
  enabled: true,
});

API Reference

SlimPinecone

class SlimPinecone {
  upsert(vectors: VectorRecord[], namespace?: string): Promise<void>;
  query(options: QueryOptions): Promise<QueryResult[]>;
  fetch(ids: string[], namespace?: string): Promise<Record<string, VectorRecord>>;
  delete(ids: string[], namespace?: string): Promise<void>;
  deleteAll(namespace?: string): Promise<void>;
}

SlimChroma

class SlimChroma {
  getOrCreateCollection(name: string): Promise<SlimCollection>;
  getCollection(name: string): Promise<SlimCollection>;
  deleteCollection(name: string): Promise<void>;
  listCollections(): Promise<string[]>;
}

class SlimCollection {
  add(params: AddParams): Promise<void>;
  query(params: QueryParams): Promise<QueryResults>;
  get(params: GetParams): Promise<GetResults>;
  update(params: UpdateParams): Promise<void>;
  upsert(params: UpsertParams): Promise<void>;
  delete(params: DeleteParams): Promise<void>;
  count(): Promise<number>;
}

Benchmarks

| Metadata Type | JSON Size | SLIM Size | Savings | |---------------|-----------|-----------|---------| | Document chunk | 500 bytes | 290 bytes | 42% | | Product data | 2 KB | 1.1 KB | 45% | | User profile | 1.5 KB | 850 bytes | 43% |

Use Cases

• RAG Applications: Compress document metadata
• E-commerce: Compact product attributes
• Recommendation Systems: Reduce user profile storage
• Knowledge Bases: Efficient entity metadata

License

MIT