munja (문자 / 文字)

A tiny, WASM-first full-text search index for CJK (Chinese, Japanese, Korean — scripts that don't put spaces between words) and Latin text.

munja is inspired by microsoft/docfind (FST keyword index + FSST-compressed documents), but replaces docfind's whitespace/RAKE tokenizer — which cannot segment CJK — with a pluggable tokenization strategy that works for space-less scripts.

Status: v0.1 (first cut). Core index + search + n-gram tokenizer + CLI + tests. Not yet build-verified in CI, and the WASM/Cloudflare Worker bindings are on the roadmap below.

The idea

The one hard rule of a keyword index: index keys and query tokens must be produced by the same tokenizer, or CJK queries silently match nothing. munja makes tokenization a strategy that both sides share, and records which strategy built an index so a mismatched query is rejected instead of returning garbage.

| Strategy | How | Data | Where it runs | |---|---|---|---| | NgramTokenizer | CJK runs → overlapping bigrams; Latin → whitespace | none | build and runtime (wasm-safe, tiny) | | CharabiaTokenizer (feature charabia) | morphological analysis via charabia (jieba / lindera) | heavy dictionaries | build time |

Because the n-gram strategy carries no dictionary, it compiles to a small wasm32 module you can run inside a Cloudflare Worker. When you can afford the dictionary size, build and query with charabia for higher precision.

Usage (library)

use munja::{build_index, search, Document, NgramTokenizer};

let docs = vec![Document {
    title: "全文検索エンジン".into(),
    category: "docs".into(),
    href: "/ja".into(),
    body: "日本語の検索に対応します".into(),
    keywords: None,
}];

let tok = NgramTokenizer;
let index = build_index(&docs, &tok)?;
let hits = search(&index, "検索", &tok, 10)?; // finds "/ja"
std::fs::write("index.bin", index.to_bytes()?)?;

Usage (CLI)

# dictionary-free bigram index (wasm-safe runtime)
cargo run --features cli -- build docs.json index.bin
cargo run --features cli -- search index.bin "검색"

# high-precision morphological index (heavy, build-time)
cargo run --features "cli charabia" -- build docs.json index.bin

docs.json is an array of { title, category, href, body, keywords? }.

Install (npm)

munja ships to npm as the wasm package built by wasm-pack — no Rust toolchain needed to consume it.

npm install @pickhealer/munja

// Astro island / any ESM. Build index.bin at build time, ship it as an asset.
import init, { MunjaIndex } from "@pickhealer/munja";

await init(); // loads the bundled wasm (pass a URL to override)
const bytes = new Uint8Array(await (await fetch("/index.bin")).arrayBuffer());

// Parse the index once, then query it per keystroke — the constructor does the
// O(index size) deserialization, `search` does not.
const index = new MunjaIndex(bytes);
const hits = JSON.parse(index.search("검색", 10)); // [{ title, href, ... }]

Exports: build_index_bytes(docsJson: string) -> Uint8Array and the MunjaIndex class — new MunjaIndex(indexBytes: Uint8Array) then .search(query: string, max: number) -> string (JSON array of hits). The old free function search_json(indexBytes, query, max) -> string still works but re-parses the whole index on every call; prefer MunjaIndex for repeated queries. Errors surface as thrown JS exceptions.

A runnable, pagefind-style Astro island (<SearchUI/>, usable from .astro and .mdx, with build-time MDX indexing) lives in examples/astro/.

Publishing uses npm Trusted Publishing (OIDC provenance) — no NPM_TOKEN. The Publication workflow builds with wasm-pack and runs npm publish --provenance on a GitHub Release. One-time setup: on npmjs.com add a Trusted Publisher for this repo + publication.yml + a GitHub Environment named npm, and create that npm environment in repo settings. The package publishes under the scoped name @pickhealer/munja (npm rejects the bare munja as too similar to an existing package), so the very first publish must be done manually to bootstrap the name before Trusted Publishing can take over. To release: bump version in Cargo.toml, then cut a release tagged vX.Y.Z (the workflow fails if the tag and package version disagree). Build the package locally with wasm-pack build --target web --scope pickhealer -- --features wasm (→ pkg/).

The published module is --target web (call init() once, no bundler wasm plugin required). Rebuild with --target bundler if you prefer zero-init imports at the cost of a vite-plugin-wasm in consumers.

Index storage & build caching

munja deliberately keeps the index separate from the wasm module (unlike docfind, which patches the index into the .wasm binary):

• Cloudflare Worker: ship the small code wasm as the Worker and load index.bin from a Static Asset / R2 / KV. This keeps the index out of the Worker size limit, keeps the code wasm a stable cacheable artifact, and lets you re-index without redeploying code.
• Build caching (MDX / CI): two cache layers —
  1. code wasm keyed by lib version + features + toolchain (rarely changes);
  2. index.bin keyed by a hash of documents + tokenizer config. A content edit only busts layer 2. Per-file token caching (cache the expensive tokenization keyed by file hash) enables incremental re-indexing of large document trees.

A docfind-style single self-contained .wasm (index embedded) is planned as an opt-in --target static output for backend-less static sites.

Roadmap

• [ ] CI build/test verification (this first cut is not yet compiled in CI).
• [x] wasm-bindgen runtime bindings (--features wasm). workers-rs example Worker still TODO.
• [ ] Measure real gzipped wasm size per language to finalize the per-language runtime strategy (charabia vs bigram).
• [x] MDX / Astro integration example (examples/astro/): docs → index.bin → island.
• [x] Search precision: IDF weighting + score cutoff (see tests/precision.rs).
• [ ] Optional --target static (docfind-style index-in-wasm) output.
• [ ] Ranking quality: fuzzy/prefix tuning, field weighting, mixed CJK+Latin.

License

MIT © 2026 SimYunSup. Design inspired by microsoft/docfind (MIT).