@torquedev/ext-embeddings

Vector embedding indexing and semantic search extension for the Torque framework.

Features

• Provider/store abstraction — embeddingService.search() stays stable regardless of backend
• LocalProvider — deterministic hash-based embeddings for development/testing (not semantic)
• SqliteStore — persists embeddings in a better-sqlite3 database with cosine similarity search
• Event wiring — automatically indexes entities when *.created / *.updated events fire

Installation

npm install @torquedev/ext-embeddings

Requires @torquedev/core as a peer dependency.

For SQLite storage, also install:

npm install better-sqlite3

Quick Start

import Database from 'better-sqlite3';
import { EmbeddingService } from '@torquedev/ext-embeddings';

const db = new Database('app.db');
const service = EmbeddingService.create({ provider: 'local', store: 'sqlite' }, db);

// Index some text
await service.index('myBundle', 'articles', 'article-1', 'title', 'Hello world');

// Search
const results = await service.search('myBundle', 'hello', { limit: 10, threshold: 0.5 });

API

EmbeddingService

new EmbeddingService({ provider, store })

Direct constructor — accepts any provider and store that conform to the interface.

EmbeddingService.create(config, db)

Factory method. config fields:

| Field | Default | Description | |-------|---------|-------------| | provider | 'local' | Provider name ('local') | | store | 'sqlite' | Store name ('sqlite') | | providerOptions | {} | Options forwarded to provider |

db is required for the sqlite store.

service.index(bundle, table, entityId, field, text)

Generate an embedding for text and store it.

service.search(bundle, query, { limit, threshold })

Embed query and return matching rows sorted by cosine similarity (descending).

service.wireEvents(eventBus, bundles, contextDefs)

Subscribe to <bundle>.<table>.created and <bundle>.<table>.updated events for every table that has at least one field with vectorize: true. On event, indexes all vectorized fields from the payload.

LocalProvider

import { LocalProvider } from '@torquedev/ext-embeddings/providers/local';

const provider = new LocalProvider({ dimensions: 128 });
const vector = provider.embed('some text'); // Float32Array

SqliteStore

import { SqliteStore } from '@torquedev/ext-embeddings/stores/sqlite';

const store = new SqliteStore(db);
store.ensureTable();
store.upsert({ bundle, table, entityId, field, vector });
const results = store.search({ vector, bundle, limit: 5, threshold: 0.3 });

Architecture

EmbeddingService
├── provider (LocalProvider | custom)
│   └── embed(text) → Float32Array
└── store (SqliteStore | custom)
    ├── ensureTable()
    ├── upsert({ bundle, table, entityId, field, vector })
    └── search({ vector, bundle, limit, threshold })

Implement this interface to plug in OpenAI, Cohere, or any other backend.

License

MIT