vecito

Tiny hybrid semantic search for Node and the browser — dense embeddings (@huggingface/transformers; default Xenova/all-MiniLM-L6-v2, but any feature-extraction model works) fused with BM25 sparse lexical scoring over an altor-vec WASM HNSW index. No server, no API keys.

Where to build the index. Building a snapshot means embedding every document and constructing the HNSW graph — the expensive part. It's usually best to do this once on the server or with the CLI (vecito index), then serve the resulting .vecito file and load it in the browser with Vecito.loadFromUrl(), which restores the pre-built graph in milliseconds. Building, indexing, and adding documents directly in the browser is fully supported too — it just runs that same per-document embedding client-side, which is slow for large corpora.

Install

npm install vecito

The first run downloads the embedding model to the Hugging Face cache. By default it loads quantized weights (dtype: 'q8') — about 22 MB for the default model. Pass dtype: 'fp32' for full precision (~87 MB) if you need maximum quality.

Library usage

The core library indexes any data — plain strings or raw JSON objects — with no pre-formatting. By default the searchable text is the item's flattened string values, and the whole object comes back as metadata on each hit.

import { Vecito } from 'vecito';

const v = new Vecito();
await v.addDocuments([
  { id: 'a', title: 'Animals', body: 'The quick brown fox jumps over the lazy dog.' },
  { id: 'b', title: 'Search', body: 'BM25 ranks documents by term frequency and rarity.' },
  { id: 'c', title: 'Botany', body: 'Photosynthesis converts sunlight into chemical energy.' },
]);

const hits = await v.search('how do plants make food?', { mode: 'hybrid', top: 3 });
// → [{ score, metadata: { id, title, body }, dense_rank?, sparse_rank? }, ...]

Plain strings work too (addDocuments(['some text', ...])). For custom shapes, pass extractor functions:

await v.addDocuments(rows, {
  text: r => `${r.headline}\n${r.summary}`,  // what gets embedded + BM25-scored
  metadata: r => ({ id: r.id }),             // what's returned with hits
});

mode is 'hybrid' (default, reciprocal-rank fusion), 'dense' (vectors only), or 'sparse' (BM25-weighted). All modes support a filter predicate to post-filter results by metadata. If the query has no in-vocabulary terms, hybrid/sparse automatically fall back to dense.

// `filter` is a JS predicate over each hit's metadata — works in any mode.
// Filtering happens after ranking, so vecito over-fetches and grows the candidate
// set adaptively (up to the whole index) to still return `top` matches when they exist.
const hits = await v.search('how do plants make food?', {
  mode: 'hybrid',
  top: 3,
  filter: m => m.title === 'Botany',
});

Options & models

new Vecito({ model, dtype, k1, b });

Pass any transformers.js feature-extraction model as model — the embedding width is detected automatically (e.g. 384 for MiniLM/BGE-small, 768 for MPNet/GTE-base). dtype picks the weight precision: 'q8' (quantized, the default, ~4× smaller download) or 'fp32' (full precision); the chosen dtype is stored in the snapshot so loads stay consistent. k1 / b tune BM25. v.model, v.dtype, v.dimensions, and v.count expose index state.

addDocuments fits BM25 on the first call, then freezes it, so you can add more documents later (including to a loaded snapshot — see below). Dense search covers new documents fully; sparse scoring only sees terms already in the frozen vocabulary, so pass your whole corpus up front for best lexical recall.

Highlighting

Pass matchedTerms: true to get the query terms each hit matched, then render an excerpt with the exported Highlighter. snippet() extracts a relevant window centred on the first match; highlight() wraps matches in <mark> tags. Both are stem-aware (the term run matches running/ran) and case-insensitive, and highlight() HTML-escapes everything else.

import { Vecito, Highlighter } from 'vecito';

const hits = await v.search('how do plants make food?', { matchedTerms: true });
for (const h of hits) {
  const excerpt = Highlighter.snippet(h.metadata.body, h.matchedTerms); // plain-text window (≤220 chars)
  const html = Highlighter.highlight(excerpt, h.matchedTerms);          // '…<mark>Photosynthesis</mark>…'
}

In dense mode (no BM25 terms available) matchedTerms falls back to the query's own tokens, so highlighting still works. You can also call Highlighter.tokenize(query) to derive terms yourself.

Persistence

// Node — single self-contained file (vectors + metadata + sparse + BM25 + model)
await v.save('data.vecito');
const loaded = await Vecito.load('data.vecito');

// Universal — in-memory bytes (use in the browser, or to store anywhere)
const bytes = v.exportBytes();             // Uint8Array
const fromBytes = await Vecito.loadFromBytes(bytes);
const fromUrl = await Vecito.loadFromUrl('https://example.com/data.vecito');

A snapshot is self-describing — it stores the model it was built with, so load always searches with the right embedder. You can keep extending a loaded index and re-save it:

const loaded = await Vecito.load('data.vecito');
await loaded.addDocuments(moreItems);
await loaded.save('data.vecito');

The primitives are exported too if you want to wire them yourself: import { Embedder, BM25, VecStore, Highlighter } from 'vecito'.

File indexing (vecito/file)

A thin Node-only layer on top of the core turns files and directories into indexed documents. It's a separate subpath import, so the core stays browser-safe.

import { indexDirectory } from 'vecito/file';

const v = await indexDirectory('./docs');   // → a ready Vecito
await v.save('docs.vecito');
const hits = await v.search('renewable energy sources', { top: 5 });

Each file becomes one document: .json/.jsonl are parsed to objects (then flattened), everything else is indexed as raw text; metadata is { path, name }. Options: ext (extension allowlist), hidden (include dotfiles, off by default), limit, model. Also exported: indexFiles(paths, opts), walk(dir, opts), and DEFAULT_EXTENSIONS.

Browser

The core (Embedder + BM25 + VecStore + Vecito) runs in the browser unchanged — it only needs a dev server that resolves bare imports and serves the two WASM deps. A ready-to-run smoke test lives in browser/index.html:

pnpm install
pnpm dev:browser   # vite — opens browser/index.html

It imports vecito, loads pre-built snapshots or embeds documents from scratch in the altor-vec WASM HNSW store, and runs hybrid search entirely client-side (the only network call is the one-time ~22 MB quantized model download from the Hugging Face CDN). Don't import vecito/file in the browser — it's the Node-only layer.

The included vite.config.js keeps @huggingface/transformers and altor-vec out of dependency pre-bundling (optimizeDeps.exclude) so their import.meta.url-relative .wasm URLs resolve correctly. Any bundler works as long as it does the same.

CLI

Install globally to get the vecito command on your PATH:

pnpm add -g vecito --config.onlyBuiltDependencies='["onnxruntime-node","protobufjs","sharp"]'

Or run it without installing via pnpm dlx vecito ….

# Index the current directory into ./data.vecito
vecito index

# …or a specific directory / output file
vecito index ./docs -o docs.vecito

# Search (path is optional; defaults to data.vecito in the current directory)
vecito search "renewable energy sources" --mode hybrid --top 5
vecito search "renewable energy sources" docs.vecito --top 5

# Filter by metadata — a JS expression with the hit's metadata bound to `meta`
vecito search "renewable energy" --filter 'meta.name.endsWith(".md")'

# Machine-readable output — score, ranks, and full metadata as JSON (pipeable)
vecito search "renewable energy" --json | jq '.[].metadata'

index recursively walks the directory, indexing a broad set of text/data/code extensions (.md, .txt, .json, .yaml, .js, .py, … — override with --ext .md,.txt). Dotfiles and dot-directories are skipped by default (pass --hidden to include them). .json files are flattened to their string values before indexing.

The trailing path is optional and defaults to the current directory — index scans cwd, and search loads data.vecito from cwd (a directory path resolves to data.vecito inside it).

vecito index [dir] [-o data.vecito] [--ext .md,.txt,...] [--hidden] [--limit N]
vecito search <query> [path] [--mode dense|sparse|hybrid] [--top N] [--filter <expr>] [--json]

Sample data

The pre-built snapshots in sampledata/ are derived from the Books Dataset on Kaggle.

License

MIT © Jeka Kiselyov