minisearch-wasm

A Rust + WebAssembly full-text search engine built against the MiniSearch behavior contract. It computes identical rankings to MiniSearch (same ids, same BM25 scores) and is faster than the JavaScript original on the real workload — verified continuously by the benchmark in the sibling keyword-search project.

This is an independent WebAssembly reimplementation that is API-compatible with MiniSearch. It is not affiliated with or endorsed by the original MiniSearch project.

The original JavaScript conformance tests are copied under reference-tests/, and Rust integration tests in tests/ translate them feature by feature. There are no JavaScript callbacks in the engine — everything runs in Wasm.

Design: keep the boundary thin

A Wasm search engine lives or dies at the JS↔Wasm boundary. Two rules:

1. Do all the work in Wasm. Tokenization, the radix tree, BM25, prefix and fuzzy traversal, scoring, and serialization all run in Rust.
2. Cross the boundary as little as possible. A query enters as a string plus one orMode flag — no per-call options object to deserialize. The result set leaves as just three values: scores (a Float64Array, one bulk copy) and ids / terms as single newline-joined strings the host splits natively. No per-hit JS object is created on the Rust side.

That is what searchJoined does, and it is the recommended path for embedding apps. search() is also provided for drop-in MiniSearch compatibility (it returns the full nested per-hit { id, score, terms, queryTerms, match, … } objects) — but it is slower than JS by design: rebuilding MiniSearch's nested objects across the boundary is inherently expensive, and real consumers (e.g. the jobboard worker) only ever read id, score, and terms.

API

import init, { MiniSearchWasm } from "minisearch-wasm";
await init();

// Build (or load a prebuilt index — far cheaper, see below).
const mini = new MiniSearchWasm({
  idField: "id",
  fields: ["title", "description", "company", "location", "org"],
  tokenizer: "jobboard",                 // or "default"
  searchOptions: { boost: { title: 4 }, prefix: true, fuzzy: 0.2, combineWith: "AND" },
});
mini.addAllJSON(rawJobsJsonText);        // index straight from JSON text, in Wasm

// Fast path — everything in Wasm, minimal boundary crossing:
const r = mini.searchJoined("software engineer", /* orMode */ false);
// r = { count, ids: "id0\nid1\n…", scores: Float64Array, terms: "a b\nc\n…" }
const ids   = r.count ? r.ids.split("\n")   : [];
const terms = r.count ? r.terms.split("\n") : [];
for (let i = 0; i < r.count; i++) {
  use(ids[i], r.scores[i], terms[i] ? terms[i].split(" ") : []);
}

// …with per-call option overrides (partial, like search()):
const exact = mini.searchJoinedOpts("java", { prefix: false, fuzzy: false });

// Fastest path — everything numeric. Fetch the id table once, then per query
// only typed arrays + one small interned term table cross the boundary:
const idTable = mini.docIdTable().split("\n");   // row n = internal doc id n
const raw = mini.searchRaw("software engineer"); // optional options 2nd arg
// { count, docIds: Uint32Array, scores: Float64Array, termTable: "a\nb\n…",
//   termOffsets: Uint32Array, termIds: Uint32Array }
const termTable = raw.termTable ? raw.termTable.split("\n") : [];
for (let i = 0; i < raw.count; i++) {
  const terms = [];
  for (let k = raw.termOffsets[i]; k < raw.termOffsets[i + 1]; k++)
    terms.push(termTable[raw.termIds[k]]);
  use(idTable[raw.docIds[i]], raw.scores[i], terms);
}

// Compatibility path — MiniSearch-shaped result objects (slower, see above):
const full = mini.search("software engineer", { combineWith: "AND" });

// Query-expression trees and wildcard, like JS MiniSearch: subqueries combine
// with AND / OR / AND_NOT and nest arbitrarily; a node's other keys override
// the search options for its subtree. `MiniSearchWasm.wildcard` matches every
// document (e.g. "everything except…" via AND_NOT).
const advanced = mini.search({
  combineWith: "AND_NOT",
  queries: [
    { combineWith: "OR", queries: ["designer", { prefix: true, queries: ["develop"] }] },
    "senior",
  ],
});
const everything = mini.search(MiniSearchWasm.wildcard);

// Auto-suggest (search-as-you-type), MiniSearch-compatible: AND + prefix on
// the last term by default; defaults configurable via the constructor's
// `autoSuggestOptions`, overridable per call.
const suggestions = mini.autoSuggest("softw eng", { fuzzy: 0.2 });
// => [{ suggestion: "software engineer", terms: ["software", "engineer"], score: … }, …]

// Non-blocking batch indexing, yielding between chunks:
await mini.addAllAsync(documents, { chunkSize: 100 });

// Discard leaves stale postings. Auto-vacuum is enabled by default and can be
// tuned or disabled in the constructor; manual vacuuming is also asynchronous.
mini.discard(oldDocumentId);
await mini.vacuum({ batchSize: 1000, batchWait: 10 });
console.log(mini.dirtCount, mini.dirtFactor, mini.isVacuuming);

// Fast path: one string + one Float64Array across the boundary; a suggestion
// row IS its space-joined terms.
const s = mini.autoSuggestJoined("softw eng");
const phrases = s.count ? s.suggestions.split("\n") : [];
for (let i = 0; i < s.count; i++) use(phrases[i], s.scores[i], phrases[i].split(" "));

// Persistence: compact binary snapshot — smaller on the wire than JSON and
// faster to load.
const bytes = mini.toBytes();            // Uint8Array
const loaded = MiniSearchWasm.loadBytes(bytes);

Differences from MiniSearch

This is a from-scratch reimplementation. It matches MiniSearch's ranking — identical BM25 scores, verified to float epsilon on a 21k-document corpus (max delta ≈ 1e-14) and on a 3k-document differential suite covering search + autoSuggest before and after removes/discards (max delta ≈ 5e-16, i.e. last-ulp Math.log vs ln rounding; per-document matched-term order identical, since the radix tree replicates JS MiniSearch's key order and traversal exactly) — but deliberately diverges from its API and internals:

• No JavaScript callbacks. MiniSearch takes user functions for extractField, tokenize, processTerm, and search-time filter / boostDocument. To keep the JS↔Wasm boundary coarse, this port has none: tokenization and term processing are built-in Rust modes ("default", "jobboard"), fields are read by name from JSON, and nothing calls back into JS per token/document. Custom tokenizers/filters must be added Rust-side.
• Serialization is not interoperable. toJSON/toBytes use this crate's own formats (a Rust-struct JSON and a compact varint binary snapshot). You cannot load a MiniSearch v1/v2 index here, nor load one of these indexes into JS MiniSearch. Persist and reload with the same engine.
• Equal-score ties order differently. Scores and the result set are identical; results with exactly equal scores are ordered by internal document id, whereas MiniSearch keeps them in first-scored order. This affects only ties (≈1 query in 30 on the test corpus).
• Term length is counted in code points. Prefix/fuzzy weighting uses chars().count(), which equals JS String.length for all BMP text; it differs only for astral-plane characters (emoji, etc.), where JS counts UTF-16 code units.
• Search never mutates the index. MiniSearch lazily removes stale postings mid-query when it meets a discarded document; this port just skips them (and skips the liveness check entirely on a clean index). Explicit and automatic vacuuming remove those stale postings in asynchronous Wasm-side batches.
• Added beyond MiniSearch. searchJoined / autoSuggestJoined (compact columnar results for a thin Wasm boundary), addAllJSON (index straight from a raw JSON string), toBytes/loadBytes (compact binary snapshot), the "jobboard" tokenizer, and — internally, with identical results — a bit-parallel (Myers) fuzzy traversal and HashMap-backed postings.

Benchmark results

Measured by keyword-search/scripts/bench-minisearch-vs-wasm.mjs on the real jobboard corpus (~22k documents, 30 representative queries), comparing against JS MiniSearch. Search ratios are reported by median (robust to system noise); expect some run-to-run variance.

| Category | Rust vs JS MiniSearch | |---|---| | Search — app workload ({id, score, terms}, end to end) | ~12× faster (median; mean ~7×) | | Index download (prebuilt, brotli) | ~0.73× — smaller on the wire than JS | | Load prebuilt index — loadBytes vs loadJSON | ~13× faster | | Serialize index — toBytes vs JSON.stringify | ~12× faster | | Build index — addAllJSON vs addAll | ~3.4× faster | | Full compat search() vs JS search() | ~0.7× (slower by design — see Design) |

Search-as-you-type re-issues the same committed terms every keystroke; the engine memoizes prefix/fuzzy tree expansions (bit-identical replay, invalidated on any mutation), which is where most of the search speedup comes from on repeated queries. Cold first-run queries are ~2-3× faster than JS.

The synthetic maintenance benchmark (npm run bench:maintenance, 5,000 documents, 500-document async chunks) measured addAllAsync at 2.89× the synchronous Wasm indexing time, reflecting its deliberate event-loop yields. Vacuuming 1,250 discarded documents took 122 ms and reduced the binary snapshot by 20%. These timings are workload- and machine-dependent; the benchmark always verifies post-maintenance search parity and dirt cleanup.

The compact binary snapshot is delta+varint encoded, so it is both smaller than the JSON index and low-entropy enough to compress well; loadBytes rebuilds the term tree from the sorted terms and still loads ~8× faster than JS loadJSON.

Truthfulness: the benchmark verifies the two engines return identical results — set-identical 30/30, max score delta ≈ 1e-14 (float epsilon), 0 real ranking bugs. The only ordering differences are between results with exactly equal scores (a tie-break nuance: JS breaks ties by scoring-encounter order, this port by document id).

Install

npm install minisearch-wasm

The package is a wasm-pack --target web build: it loads with a static import in every environment — Vite, webpack, Turbopack (dev and build), plain ESM, Node, and Web Workers — with one explicit await init() to load the WebAssembly module up front:

import init, { MiniSearchWasm } from "minisearch-wasm";

await init();                              // load the .wasm once, before first use

const ms = new MiniSearchWasm({ fields: ["title", "text"] });
ms.addAll(documents);
const results = ms.search("query");

In a Web Worker the same static import works — just await init() before the first call. (The --target web build is what makes worker and Turbopack use friction-free; a bundler-target build would require a dynamic import() there.)

Build

cargo fmt -- --check
cargo test
rustup target add wasm32-unknown-unknown
npm run build   # wasm-pack build --target web --release + metadata patch

Install wasm-pack with cargo install wasm-pack if needed.

Publishing

npm run publish:pkg   # rebuilds, then `npm publish ./pkg`

You must npm login first. The publishable artifact is the generated pkg/ directory; its package.json metadata is injected by scripts/finalize-pkg.mjs on every build.

See PORTING.md for conformance status and the next test slices.