convex-improved-search
v0.6.1
Published
An improved search component for Convex.
Readme
convex-improved-search
Transactional, reactive, exact substring search for Convex — built for the languages the built-in search index can't handle.
Convex's built-in full text search
tokenizes on whitespace and punctuation and "works best with English or other
Latin-script languages". For Chinese, Japanese, Korean, Thai — or simply for
LIKE '%…%'-style substring matching in any language — it structurally cannot
help: a run of CJK text becomes a single token, and terms over 32 bytes (≈11 CJK
characters) are silently dropped from the index entirely.
This component fills that gap with a bigram inverted index that lives in ordinary Convex tables:
- Exact substring semantics. The result set is provably identical to "some field of the document contains the query" (after folding) — the index only accelerates, it never approximates. Verified by a randomized differential test suite against a naive-scan oracle.
- Any language. Bigrams over folded codepoints — no tokenizer, no dictionary, no language configuration. A two-character Chinese query is a single index point-lookup.
- Multi-field. Index named fields (
note,cargo, …); a match never spans a field boundary, and every hit reportsmatchedFields— which fields contained the query — for free. - Transactional. One table, one mapper:
tableIndexerkeeps the index in the same transaction as your writes, with backfill and triggers driven by the same mapper. No drift window, ever. - Reactive. Search from a normal Convex query and results update live.
- Unicode-sane. NFKC + lowercase folding: full-width
ABC123matchesabc123, decomposed and precomposed accents match each other, surrogate pairs are handled per-codepoint. Exported highlight utilities map folded matches back to original-text offsets.
Installation
npm install convex-improved-search// convex/convex.config.ts
import { defineApp } from "convex/server";
import improvedSearch from "convex-improved-search/convex.config.js";
const app = defineApp();
app.use(improvedSearch);
export default app;Usage
One SearchIndex per logical collection, one tableIndexer binding it to a
table with one mapper:
// convex/search.ts
import { SearchIndex } from "convex-improved-search";
import { components } from "./_generated/api";
import type { Doc } from "./_generated/dataModel";
export const notesSearch = new SearchIndex(components.improvedSearch, {
name: "notes",
// Typing-only: misspelled filter names in set/search become compile errors.
filterFields: ["category"],
});
export const notesIndexer = notesSearch.tableIndexer<Doc<"notes">>({
table: "notes",
map: (ctx, note) =>
note.archived
? null // null = keep this document out of the index
: {
text: { title: note.title, body: note.body }, // named fields (or one string)
sortKey: note.createdAt, // results are ordered by sortKey descending
filters: { category: note.category },
},
});The mapper receives a query ctx, so derived text is first-class — join in related rows, aggregate, whatever; the index still updates atomically.
Writing
Call sync after any write, in the same mutation — it reads the row and upserts
or removes the entry, hard deletes included:
export const addNote = mutation({
args: { title: v.string(), body: v.string() },
handler: async (ctx, args) => {
const noteId = await ctx.db.insert("notes", {
...args,
createdAt: Date.now(),
});
await notesIndexer.sync(ctx, noteId); // same transaction — the index can never drift
return noteId;
},
});
export const deleteNote = mutation({
args: { noteId: v.id("notes") },
handler: async (ctx, args) => {
await ctx.db.delete("notes", args.noteId);
await notesIndexer.sync(ctx, args.noteId); // missing row → entry removed
},
});Updates pay only the gram diff — re-syncing a document with slightly changed
text writes a handful of rows, not thousands. syncMany(ctx, ids) covers writes
that touch a batch of rows at once.
(You can also call notesSearch.set / notesSearch.remove directly — the
indexer is sugar over them, not a requirement.)
Searching
searchDocs searches and hydrates from your table in one call:
export const searchNotes = query({
args: {
query: v.string(),
cursor: v.optional(v.union(v.string(), v.null())),
},
handler: async (ctx, args) => {
const { page, cursor, isDone } = await notesSearch.searchDocs<Doc<"notes">>(
ctx,
{
table: "notes",
query: args.query, // "停車" — any substring, any language
filters: { category: "work" }, // optional exact-match filters
cursor: args.cursor,
limit: 50,
},
);
// page: [{ doc, sortKey, matchedFields }] newest-first
return { page, cursor, isDone };
},
});matchedFields names the fields whose text contained the query (for a
single-string entry it is ["text"]) — exact, straight from the verifier.
Plain search returns { key, sortKey, matchedFields } hits without hydration,
for keys that are not document ids. searchPaginated / searchDocsPaginated
speak Convex's standard paginationOpts / { page, isDone, continueCursor }
shape for paginated-query clients.
Pagination contract: a non-null
cursorwith a short — or even empty —pagemeans the scan budget ran out before the page filled (e.g. a very selective filter over a common substring). Keep paging; matches keep arriving. OnlyisDone: truemeans the search is exhausted. If you render pages incrementally, auto-continue while the page is empty.
To search only some fields, pass fields. Hits whose query occurrence lies only
in other fields are dropped inside the component, and matchedFields lists only
the named fields. Prefer it over filtering matchedFields yourself. Each
posting row records which fields hold its gram, so the component drops most
other-field candidates before reading their document. Those candidates cost no
index range and no scan budget.
await reimbursementsSearch.search(ctx, {
query: "TR-260720-006",
fields: ["no"],
});To get "up to N matches" in one query without writing that loop, pass
fill: true. The call keeps paging until the page holds limit hits, the
search is done, or 768 candidates have been scanned in total (budget lowers
that cap). searchDocs counts a hit whose document is gone as a miss and keeps
going. The cursor contract is unchanged, so a filled page can still come back
short with a non-null cursor:
const { page } = await notesSearch.searchDocs<Doc<"notes">>(ctx, {
table: "notes",
query: args.query,
limit: 20,
fill: true,
});Highlighting
The same folding the index uses, mapped back to original offsets — so a half-width query highlights its full-width occurrence:
import { matchSegments } from "convex-improved-search";
matchSegments("停車場月租 ABC大樓", "abc");
// [{ text: "停車場月租 ", match: false }, { text: "ABC", match: true }, { text: "大樓", match: false }]
// React:
{
matchSegments(note.title, query).map((seg, i) =>
seg.match ? (
<mark key={i}>{seg.text}</mark>
) : (
<span key={i}>{seg.text}</span>
),
);
}findFoldedRange(text, query) returns the raw [start, end) UTF-16 range (or
null when the occurrence cannot be located — render unhighlighted). Use
matchedFields to decide which field to highlight.
Keeping the index in sync
Triggers (zero drift)
With convex-helpers/server/triggers every
write to the table syncs automatically, driven by the same mapper:
import { Triggers } from "convex-helpers/server/triggers";
import {
customMutation,
customCtx,
} from "convex-helpers/server/customFunctions";
const triggers = new Triggers<DataModel>();
triggers.register("notes", notesIndexer.trigger());
export const mutation = customMutation(rawMutation, customCtx(triggers.wrapDB));Caveat (inherited from triggers itself): writes from the dashboard and
npx convex import bypass wrapped mutations. If you use those, run the backfill
recipe afterwards.
Backfilling existing data
Use @convex-dev/migrations —
syncDoc is idempotent and runs the same mapper as live writes, so the
migration doubles as a drift-repair reconcile:
export const backfillSearch = migrations.define({
table: "notes",
migrateOne: async (ctx, doc) => {
await notesIndexer.syncDoc(ctx, doc);
},
});To inspect what the index believes about one key,
notesSearch.get(ctx, { key }) returns its folded fields, sortKey and
filters. To drop a whole namespace:
let cursor: string | null = null;
do {
({ cursor } = await notesSearch.clear(ctx, { cursor }));
} while (cursor !== null);notesSearch.retag(ctx, { cursor }) loops the same way and tags postings
written before 0.5.0 (see Upgrading to 0.5.0).
Scaling out: time shards
Past ~100k entries per namespace, partition by time with ShardedSearchIndex.
Each shard is its own namespace (receipts/2026Q3). It is created on its first
write, can be indexed ahead of a cutover with the ordinary backfill recipe, and
can be dropped with clear.
With a period ("month", "quarter" or "year"), the shard comes from the
sortKey, which must be a timestamp in ms. Periods are in UTC.
export const receipts = new ShardedSearchIndex(components.improvedSearch, {
name: "receipts",
period: "quarter",
filterFields: ["source"],
});
// Writes (or receipts.trigger((ctx, doc) => …) with the same shape):
await receipts.set(ctx, {
key: doc._id,
text: doc.note,
sortKey: doc.createdAt,
});
// Every shard that holds entries, newest first:
const page = await receipts.search(ctx, { query: "停車" });
// Only the last 90 days (pass `now` in; queries must not read the clock):
const recent = await receipts.search(ctx, {
query: "停車",
shards: receipts.shardsBetween(args.now, args.now - 90 * 24 * 3600 * 1000),
});Without shards, the search finds the next shard with one index range, so empty
periods cost nothing. shardsBetween works at shard grain: the oldest and
newest shard can return hits a little outside the range.
Sharding bounds the cost when a search is limited to recent shards. A search
over every shard reads about what one large namespace would. A query whose every
gram is in most entries (shared prefixes such as TR-26) still reads every
shard to finish, so limit such searches by time.
Without a period, name the shard yourself and pass the shard list, newest
first:
await receipts.set(ctx, { ...entry, shard: "2026Q3" });
await receipts.search(ctx, {
shards: ["2026Q3", "2026Q2", "2026Q1"],
query: "停車",
});The contract: derive the shard from the sortKey, so shard ranges are disjoint
and newest-first shard order makes concatenated results globally sorted. Paging
continues into older shards with the same cursor contract as
SearchIndex.search.
How it works
- Each field's text is folded (
NFKC+toLowerCase) and split into overlapping bigrams of codepoints, with an end-of-text sentinel per field so every character starts exactly one gram and no gram spans a field boundary. One posting row per distinct gram of the entry, tagged with the fields that hold the gram. - A query folds identically. Two-codepoint queries stream one gram. Longer
queries stream the scarcest gram (chosen by probing) and merge-join up to
three others against it in index order. A candidate reaches its document only
if it holds those grams and some wanted field holds all of them.
Single-codepoint queries use a first-codepoint index. Longer queries walk the
documents by a (sortKey, key) index in step with the candidates; one-gram and
single-codepoint queries read each candidate's document by key, in parallel
rounds. When most documents are candidates anyway, the call walks the
documents alone. Every candidate is then verified per field with
folded.includes(query)— the index is a pure candidate filter, so correctness never depends on the gram approximation (the same structural argument SQLite's FTS5 trigram tokenizer relies on), andmatchedFieldsis the verifier's own answer. - Design lineage: SQLite FTS5 trigram tokenizer (spec), pg_bigm (gram size 2 for
CJK), Lucene
TestNGramTokenizerand SQLancer NoREC (differential testing methodology).
Limits
- Text: up to 4096 codepoints per entry after folding, shared across its
fields in field order. Longer text is truncated by default (the overflowing
field is cut, later fields dropped; substrings beyond the limit are
unfindable); pass
onOverflow: "error"to throw instead. The write path is engineered so a full-size entry indexes within one Convex transaction — but avoid writing several max-size entries in a single mutation. - Deletes are lazy:
remove(and large shrinking edits) make the entry unfindable immediately, but posting cleanup beyond a small inline budget runs in a component-internal scheduled mutation moments later. Until it runs, leftover postings are only rejected candidates — never wrong results. (Convex counts deletes as reads, 4096/transaction, so a max-size entry cannot be unindexed synchronously.) - Writes:
setcompares the new entry with the stored one first. When the folded text andsortKeyare unchanged it skips the posting diff, so re-syncing an unchanged document or changing only its filters costs two row reads (the entry and one of its postings, to check its Field tags) and at most one row write. - Ordering:
sortKeydescending only. Changing an entry'ssortKeyrewrites all its posting rows synchronously; a change touching more than 3000 surviving grams is refused. Prefer stable sort keys (creation time). - Filters: exact-match only, checked on the document. Documents are read in
the same order as candidates, so a filter that keeps 1 in 100 over a common
substring costs about 100 streamed rows per hit and few index ranges. Each
rejected candidate still counts against
budget, so such a page often comes back short with a cursor. When one filter value selects a small slice you search often (a tenant, an archive flag), give it its own namespace. - Read cost: one
searchcall probes up to 16 of the query's grams, streams the sparsest, and merge-joins the next three against it, so a rejected candidate costs streamed rows, not index ranges. Surviving candidates are checked against their documents, read by walking the documents in the same order: dense hits cost rows, and a far-apart hit costs one index range. When the postings and documents read come to as much per document as reading every document would (a dense query, or grams split across fields), the call walks the documents alone instead: about one row, text included, per document passed. It switches back once the grams thin out. The cursor carries the chosen grams, or that it was walking documents, so the next page skips probing.budgetcaps the documents checked per call (default 256, max 768). Every call in one transaction shares Convex's limits (4096 index ranges, 32000 rows, 16 MiB read, and 15 s of total database time) with your own reads and with every othersearchin it. Each call checks what the transaction has left, keeps 512 ranges, 4096 rows and 2 MiB for you, and otherwise returns a short page with a cursor. If your reads have already reached the reserve, search reads nothing and returns a cursor to retry in a fresh transaction. The 15 s limit cannot be measured from inside the query, so the reserve does not cover it. Callingsearchin a loop inside one query therefore adds up; usefill: trueandfieldsinstead, and page further from the client. - Exact identifiers: to look up a whole document number, an equality index on your own table answers in one index range. Substring search is for partial numbers and free text.
- Scale: designed for up to ~100k documents per namespace. Storage is
roughly one small row per character of indexed text. Beyond that, partition by
time with
ShardedSearchIndex(see above). - Traditional vs Simplified Chinese: NFKC folds width and compatibility
forms but does NOT map Simplified↔Traditional —
台will not match臺. If you need that, pre-fold your text (and queries) with a kVariant table (e.g. the IRG kVariants data) before callingset/search; an opt-in fold option is on the roadmap.
Upgrading to 0.5.0
Posting rows gain an optional
fieldstag. Rows written by earlier versions have none. Search still returns the same results for them, but a query withfieldscannot drop their other-field candidates before reading their docs, and that can cost several times more: on one 3650-entry index, a query whose text sat only in an unselected field read 575 KB over 2 calls untagged and 76 KB in 1 call tagged. Retag right after deploying, one loop per index:let cursor: string | null = null; do { ({ cursor } = await notesSearch.retag(ctx, { cursor })); } while (cursor !== null);retagreads the stored fields, so it needs no source table, and skips entries that are already tagged. For aShardedSearchIndex, run it on each shard (receipts.shard("2026Q3").retag(...)). For large namespaces, run the loop across scheduled mutations. An unchangedsetorsyncDocalso tags its entry, so for a table-backed index the backfill migration does the same job under@convex-dev/migrations: run it once after deploying. Entries you write after the upgrade are tagged as they are written.docsgains the indexby_ns_sort. Convex builds it on deploy; there is nothing to run.The component's
searchresult gainsverified: candidates checked against their documents, at mostbudget.scannedstill counts distinct candidates examined, but most of them are now rejected by their postings alone, so it can exceedbudget. A loop that caps its cost should countverified. Preferfill: trueandfieldsover such loops.
Upgrading from 0.1.x
- Storage shape changed (single folded text → named fields), and Convex
validates existing rows against the component schema at push time — a
deployment still holding 0.1.x index data will refuse the push. Clear every
namespace on 0.1.x first (the
clearloop above), then upgrade and re-run your backfill (syncDoc/setare idempotent). SearchIndex.trigger(mapper)is gone — bind atableIndexer({ table, map })and useindexer.trigger(). The mapper now receives(ctx, doc); so doesShardedSearchIndex.trigger's.getreturns{ fields, sortKey, filters }instead offoldedText.
Development
npm install
npm run test # randomized differential suites included
CIS_BENCH=1 npx vitest run src/component/cost.bench.test.ts # read-cost benchmark
npm run typecheck
npm run lint
npm run format:checkA runnable demo lives in example/ — a Vite + React app.
Releasing
Conventional commits drive the pipeline: feat / fix (and breaking !)
commits on main make
release-please open or refresh a
release PR with the version bump and changelog. Merging that PR tags the
release; the workflow then rebuilds, re-runs the full test gate, and publishes
to npm via trusted publishing
(GitHub OIDC — no tokens, provenance included). A failed publish should first be
rerun from GitHub Actions. If the rerun window has expired, manually dispatch
the same Release workflow with the existing immutable tag; the job verifies
that the tag and package version agree before publishing. Caveat: the release PR
itself doesn't trigger the PR test workflow (GitHub doesn't run workflows on
GITHUB_TOKEN-created PRs); the publish job's own gate covers it.
License
Apache-2.0
