lucivy-wasm 4.3.0

One index answers every question, and every answer is checked. The default index answers exact substrings, matches across separators, typos across token boundaries, regular expressions and two-character needles — with BM25 and the exact bytes of every match — and nothing to configure per question; the ground-truth harness compares every answer to a scan of the files. In the browser — the same engine as the native bindings, built with emscripten: threads (pthreads over SharedArrayBuffer), OPFS persistence, snapshot import. Runs in a Web Worker. MIT.

Try the live playground — it clones lucivy's own source from GitHub and indexes it in your browser.

What's new in 4.3

• Indexing 48 → 35 s on the same index. The whole kernel (Linux 7.2, 101 141 files, 941 MB) on a 24-core machine, the same 308 segments, the same ids minted, the same answers. Four pieces, none of them an option: a derived group index next to every dictionary generation (.pidx, 2.5 % of the .sfx, rebuilt in RAM when an older index lacks it) that reaches the right parent group by binary search instead of reading every header before it; the commit no longer stops the world to forget its pending texts (they live by commit epoch and are dropped whole — 6.7 s of serial path gone); the suffix collector allocates per token instead of per occurrence; and a text minted since the last fold is found in the pending table before every FST part is walked (FST walks 68 → 38 s of CPU).
• Two exactness fixes, present since 4.0.0. A segment holding few of a candidate list's ids kept one item per id and lost the repeated occurrences of a needle inside one token (0xdedede00 for de) — whether a segment took that path depended on its size, so the same corpus was exact at 16 writer threads and lost 3 spans of 7.9 M at 24. And relaxed separators could report a match ending inside a multi-byte character (D\n,,“underscan): a word's content overlap was taken from the word after the next when the next one began with a three-byte character, and an anchored word candidate was accepted at any suffix. Both fixed on the query side (an existing index answers exactly with the new reader), the second at indexing too; tests red on the old code.
• The segment's dictionary pair is named for what it is: <uuid>.<field>.minted.termtexts and .minted.sfx (the texts a segment minted first and the FST over them, folded into the next generation), instead of .newtexts / .newsfx. Every reader still opens the old names.
• Compatibility: 4.2 opens a 4.3 index and 4.3 opens 3.0.x to 4.2 indexes as before; the only thing 4.2 cannot do is fold the pairs a 4.3 writer left pending after a crash (a cleanly closed index has none). Every binding already had update (delete by _node_id and add) — editing one file costs that file, not a rebuild.

What's new in 4.2

• search(query, { fields: false }) no longer reads any document: the id comes from the fast field; with fields: true the documents are read through the scheduler (verified on 10 000 kernel files in Chrome: 968 hits with their content in 260 ms, 500 without in 12 ms).
• Spans only when highlights: true — they were built for every search.
• Indexing: the dictionary's compaction is three times faster natively and runs the same stages in sequence here; the single indexing thread and its 15 MB heap are unchanged, and so is the index on disk — 4.1 opens a 4.2 index.

What's new in 4.1

• positions: false in IndexConfig — about 40 % less OPFS (typed in lucivy.d.ts). The postings keep each token's documents and frequencies instead of its positions, and no position sidecar is written; every match is verified on the stored text — same documents, spans and scores. Measured in Chrome on Linux kernel files: 2 000 files 268 → 174 MB, 10 000 files 1 052 → 638 MB, indexed in 37 s instead of 43, the tab's memory peak unchanged (1.5 GB); the playground's 21-query parity panel returns the same counts, top 10, scores and spans on both indexes (the differences: equal scores ordered otherwise, and spans the default index of 4.0.2 returned twice — fixed in 4.1). Natively the whole kernel goes 5 289 → 2 598 MB. Every text field must be stored (the default); excludes derived_in_ram; fixed at creation. The playground builds its index this way with ?nopos.
• Fix: one occurrence, one span. In relaxed mode a needle that ends a word cut into chunks — lock in superblock — came back twice, and counted twice in the score (542 duplicated spans for lock over 10 000 kernel files, since the v3 engine, 4.0.2 included); the ground-truth harness now counts duplicates too

What's new in 4.0.0

• The index is 3.7× smaller, and a tab holds a whole repository: the playground's prompt indexes MDN Web Docs (14 611 pages, 14 s), the entire Linux 2.6.0 kernel (14 032 files, 28 s, 1.1 GB held in memory), Go, Godot, TypeScript (39 044 files, 33 s), PostgreSQL, CPython, Redis, Git, curl, SQLite, nginx — index <name>, kept in OPFS, reopened in seconds. The ceiling of a tab is about 200 MB of text. Browser against native on the 2.6.0 kernel: same counts and spans, 41 s to index against 23, substrings 10-18 ms against 2 (README).
• shared_dictionary and derived_in_ram in IndexConfig (typed in lucivy.d.ts): the shard dictionary is the playground's default for big corpora (23-25 % smaller); derived_in_ram stays an option here — it saves a quarter of OPFS but raises the tab's memory peak.
• dictionary_wait in IndexConfig (typed): shared dictionary only — natively a commit returns before the shard's new texts are merged and a search waits for that merge by default; in the browser the merge stays inside the commit (few threads, and building the segments' dictionary pieces in parallel raised the memory high-water mark for no time gained), so the option changes nothing there yet
• Lucivy.dropIndex(path) — delete an index directory through the worker: WASMFS caches what it mounted, so a directory removed from the main thread still exists for it and the next create at that path fails.
• A commit every 8 MB of text in the playground's indexing loop (?commitmb=M), not only every 2 000 files: a segment's size, not its document count, sets the memory peak (Godot: 3.3 GB → 1.8 GB).
• Compatibility contract — 4.0 opens a 3.0.x index and returns what 3.0.x returned; 3.0.x does not open a 4.0 index; the first commit converts for good.

Against Elasticsearch and tantivy — one corpus, one truth

Same 93 983 Linux kernel files, 857 MB of text. Each engine is configured at its best for substring search, not at its default: Elasticsearch 8.19 with a trigram analyzer plus a wildcard field for regexes, tantivy 0.25 with its NgramTokenizer. The truth of every row is the same byte-by-byte scan of the files; a lucivy count is right only when its documents and its byte spans match it. On the substring itself all three agree to the document; where they part:

| asked | truth | lucivy | Elasticsearch | tantivy | |---|---|---|---|---| | spin_lock, separators relaxed (also spin lock, spin-lock, spinlock) | 9 552 | 9 552, 23 ms | 6 577 — not with this analyzer: its trigrams carry the underscore | 6 601 — relaxed is the only mode it has: the separator never enters its index | | spinlokc, two edits, across the token boundary | 10 034 | 10 034, 148 ms | 3 549 — fuzziness compares whole terms | 6 557 — same | | spin_lock_[a-z]+, a regex, case folded | 5 510 | 5 510, 219 ms | 5 510 on the wildcard field, 1 ms warm — as [a-zA-Z]+: Lucene's case_insensitive folds a pattern's literals, not its character classes | 0 — terms are already cut | | de, two characters | 93 009 | 93 009, 7.7 M spans, 561 ms | 0, silently | 0, silently | | retur -ENOMEM, a fuzzy phrase | 14 449 | 14 449, 30 ms | 14 446 (span_near), 24 ms — it does this well | — | | where it matched: mutex_lock, 5 145 documents | 20 797 spans | all 20 797, 15 ms | highlight on the top 200: 179 ms | verifying 5 145 stored texts: 96 ms | | your index in your transaction | — | yes: pluggable store, one commit for your rows and the index, rollback included | no: a server next to your database, a synchronisation to write | no: its own directory, its own commit | | shards and nodes scoring as one index, as a library | — | yes, asserted by test_federated_search | yes, as a cluster | no: one index, one scale of scores |

Where a cell says "not with this analyzer", a purpose-built analyzer or plugin may get closer, at the price of designing it, configuring it and reindexing. Every question in the table is answered by lucivy's default index with nothing to configure, and each answer is checked. Sizes, indexing times and the full generated report: docs/compare-engines-2026-09-05.md; benches/compare_engines.sh replays it.

What 3.0.0 brought

Measured on 10 000 Linux kernel files, in the browser, 8 threads:

• Indexing in 55 s (was ~25 min) and ~1.5x the native query time (124-133 ms per query, median 69-92 ms; native 79 / 49) — the engine now runs on mimalloc; emscripten's default allocator serialised every thread on one lock.
• SFX v3 index format with denser sidecars (−22 %), exact highlights on every query mode, parse boolean syntax, queryWarnings, fuzzy by Levenshtein or Jaro-Winkler (fuzzy_metric, min_similarity).
• Memory made explicit: memoryStatus() says whether the index is held in memory (up to 3 GB by default) or streamed from OPFS, and preload() loads it once, after every background merge is done. A page that has just indexed several GB cannot also serve them (4 GB address space): persist, reload, open.
• Bounded by construction: SFX segment budget, two segment builds at a time, merges capped at 800 documents (48 small segments fill eight threads where 19 large ones fed one), at most 512 documents queued.

From 2.x

• SFX-only engine — all queries route through the Suffix FST, no legacy code paths
• Distributed search — merge_stats for multi-node BM25
• Correct BM25 cross-shard — identical scores whether 1 shard or 4
• 5 bindings — Python, Node.js, C++, WASM, Rust

Install

npm install lucivy-wasm

Requirements

• COOP/COEP headers required for SharedArrayBuffer (threading):
  • Cross-Origin-Opener-Policy: same-origin
  • Cross-Origin-Embedder-Policy: require-corp
• For GitHub Pages or environments without header control, use coi-serviceworker.

Quick start

import { Lucivy } from 'lucivy-wasm';

const lucivy = new Lucivy('./lucivy-worker.js');
await lucivy.ready;

const index = await lucivy.create('/my-index', {
    fields: [
        { name: 'title', type: 'text' },
        { name: 'body', type: 'text' },
    ],
    shards: 4,
});

await index.add(1, { title: 'Rust Programming', body: 'Systems programming with memory safety' });
await index.add(2, { title: 'Python Guide', body: 'Data science and web development' });
await index.commit();

const results = await index.search(
    { type: 'contains', field: 'body', value: 'program' },
    { highlights: true, fields: true }
);
for (const r of results) {
    console.log(r.docId, r.score, r.fields.title);
}

lucivy.terminate();

API

Lucivy (main class)

import { Lucivy } from 'lucivy-wasm';

const lucivy = new Lucivy('./lucivy-worker.js');
await lucivy.ready;

// Create a new index (config object with fields and optional shards)
const index = await lucivy.create('/my-index', {
    fields: [
        { name: 'title', type: 'text' },
        { name: 'body', type: 'text' },
    ],
    shards: 4,
    // shared_dictionary: true — one dictionary per shard instead of one per
    // segment: about 20 % less OPFS and memory, queries slightly slower at
    // cold cache (roughly x1.2 to x1.6 on exact queries, fuzzy ones faster),
    // same answers. Fixed at creation. The default since 4.0.0 (false keeps
    // a suffix FST per segment).
    // derived_in_ram: true — the three derived sidecars of each segment
    // (about a third of the index) are rebuilt in memory when the index
    // opens instead of written to OPFS. Same answers. Off by default.
    // positions: false (4.1) — documents and frequencies instead of
    // positions, no position sidecar: about 40 % less OPFS; every match is
    // verified on the stored text, same answers. Text fields must be stored
    // (the default); excludes derived_in_ram. On by default.
});

// Open an existing index from OPFS
const index2 = await lucivy.open('/my-index');

// Import from a LUCE snapshot (Uint8Array)
const index3 = await lucivy.importSnapshot(snapshotData, '/restored');

// Terminate the worker (frees all WASM memory)
lucivy.terminate();

Startup options

new Lucivy(workerUrl, options) — every option maps to a module argument read before the engine starts; the defaults are the measured ones:

| option | default | effect | |---|---|---| | noOpfs | false | in-memory filesystem, the index lives for the session | | verbose | false | engine diagnostics (LUCIVY_VERBOSE, V3_PROFILE) | | fileCacheMb | index size | whole-file cache; pins the budget when set | | ramIndexMaxMb | 3072 | above this the index is streamed from OPFS instead of held | | schedulerThreads | min(cores, 8) | luciole scheduler pool (12 gains nothing over 8) | | writerThreads | 1 | indexer threads; the writer heap follows | | maxMergedDocs | 800 | largest segment a background merge may produce | | maxBuilds | 2 | segment builds at once (four exhaust the address space) |

Memory

const st = await index.memoryStatus();
// { index_bytes, in_memory, num_docs, warnings: [..], shards: [..] }
if (st.in_memory) {
    const pre = await index.preload();   // { bytes, files, ms, skipped }
}

preload() waits for background merges to be quiet, then reads every file of the index into memory once — a single substring query opens nearly every sidecar of every segment, so lazy reading only hides that cost in the first search. It does nothing when the index is streamed. warnings carries the sentence to show a user when the index does not fit.

LucivyIndex

Add / update / delete

await index.add(1, { title: 'Hello', body: 'World' });

await index.addMany([
    { docId: 2, title: 'Foo', body: 'Bar' },
    { docId: 3, title: 'Baz', body: 'Qux' },
]);

await index.update(1, { title: 'Updated', body: 'Content' });
await index.remove(2);
await index.commit();
await index.drainMerges();  // wait for background segment merges

Search

All substring queries are cross-token: they match across token boundaries.

// Substring — matches "programming", "programmer", "getProgramHandle", etc.
const results = await index.search(
    { type: 'contains', field: 'body', value: 'program' },
    { highlights: true }
);

// Fuzzy substring (Levenshtein distance)
await index.search({ type: 'contains', field: 'body', value: 'mutx', distance: 1 });

// Regex substring — cross-token regex matching
await index.search({ type: 'contains', field: 'body', value: 'lock.*mutex', regex: true });

// Prefix / startsWith
await index.search({ type: 'startsWith', field: 'body', value: 'prog' });

// Multi-word search — each word as contains, combined with OR
await index.search({ type: 'contains_split', field: 'body', value: 'rust safety' });

// Multi-word with fuzzy distance
await index.search({ type: 'contains_split', field: 'body', value: 'memry safty', distance: 1 });

// Phrase — adjacent tokens in order
await index.search({ type: 'phrase', field: 'body', value: 'mutex lock' });

// Boolean
await index.search({
    type: 'boolean',
    must: [{ type: 'contains', field: 'body', value: 'rust' }],
    must_not: [{ type: 'contains', field: 'body', value: 'deprecated' }],
});

// Retrieve stored fields with results
const results2 = await index.search(
    { type: 'contains', field: 'body', value: 'rust' },
    { fields: true, limit: 10 }
);

// Pre-filtered by doc IDs
const results3 = await index.searchFiltered(
    { type: 'contains', field: 'body', value: 'rust' },
    [1, 3, 5],
    { highlights: true, fields: true }
);

Filtering on non-text fields:

await index.search({
    type: 'contains', field: 'body', value: 'lock',
    filters: [
        { field: 'category', op: 'eq', value: 'kernel' },
        { field: 'score', op: 'gte', value: 0.5 },
    ]
});

Filter ops: eq, ne, lt, lte, gt, gte, in, not_in, between, starts_with, contains.

Metadata

const count = await index.numDocs();
const schema = await index.schema();

Snapshots

// Export to Uint8Array (.luce format)
const snapshot = await index.exportSnapshot();

// Import from Uint8Array
const restored = await lucivy.importSnapshot(snapshot, '/restored');

Cleanup

await index.close();    // remove from tracking (OPFS files kept)
await index.destroy();  // remove from tracking + delete OPFS files
lucivy.terminate();     // kill worker, free all WASM memory

Note: Always call lucivy.terminate() when done. Individual close()/destroy() are instant and non-blocking. Actual WASM memory is reclaimed on terminate().

License

MIT