Albex

Local full-text search for documents. Runs entirely in the browser — no server, no upload, no network request after the initial load.

Drop a DOCX, PDF, XLSX, HTML, Markdown, JSON, CSV, EML, RTF, TXT, or XML file, start typing, and search it locally without shipping the content anywhere.

Install

npm install albex

import { AlbexEngine } from 'albex';

const engine = new AlbexEngine();
await engine.init();

That's it. The main WASM core is embedded in the package (base64, ~19 KB gzipped over the wire, decoded once at startup), so the default path has nothing to serve and nothing to configure. It works in the common bundlers people use for browser apps — Vite, Webpack, Rollup, Parcel, esbuild and Angular included — and in Node ≥ 18. No assets entries, no copying the main binary, no wasmUrl.

Matrix-tested in CI today: Vite and Node (via the test suite). Other bundlers and runtimes (Next SSR, Bun, Deno) ride the same embedded path; if you hit a problem, open an issue.

PDFs load a separate engine (~118 KB, ~58 KB gzipped) on demand the first time you index a .pdf — it isn't embedded in the core. In bundlers that resolve new URL('…', import.meta.url) (Vite, Webpack 5+, Rollup, Parcel) it just works; on esbuild/Angular, hand it the bytes once via pdfWasmBytes (or a URL via pdfWasmUrl). See Advanced. It's a lightweight best-effort extractor — see PDF extraction — coverage & limitations.

Quick start

import { AlbexEngine } from 'albex';

const engine = new AlbexEngine();
await engine.init();

// Index a file from <input type="file"> or drag-and-drop.
const file = inputElement.files[0];
const doc  = await engine.indexFile(file);
console.log(`Indexed ${doc.chunks} chunks in ${doc.indexTimeMs.toFixed(0)} ms`);

// Search.
const results = engine.search('contrato marco');
for (const r of results) {
  console.log(`[${r.score}] ${r.documentName} — ${r.snippet}`);
}

Cooperative search — yields to the scheduler between slices so the UI thread keeps a chance to paint while a long search runs:

for await (const r of engine.searchCooperative('contrato', { frameBudgetMs: 8 })) {
  renderResult(r);
}

searchCooperative returns the same shape as search. The "stream" is not incremental yet — results arrive in one batch after the search completes, but the work is split into frame-budget slices that yield to the scheduler. Real incremental streaming is on the backlog.

That's the entire onboarding. Read on for what else the engine can do.

Features

• Zero server — all text stays on the user's machine.
• Install and use — the main WASM core ships embedded in the JS, so new AlbexEngine() runs in normal bundler setups and in Node with nothing to serve or copy. SIMD and CDN hosting stay available as opt-in options (see Advanced).
• Fuzzy matching — finds "contrato" even if you type "conttrato" (Bitap with adaptive edit distance). Sound under a two-stage pre-filter (character Bloom for exact tokens, a 256-bit trigram q-gram signature for everything) that prunes the candidate set ~10× on prose without ever dropping a real approximate match.
• Accent-insensitive — "accion" matches "acción", "espana" matches "España", plus Latin Extended (Polish, Czech, Slovak, Turkish…).
• 11 formats with varying depth — DOCX · XLSX · PDF · HTML · MD · JSON · CSV · EML · RTF · TXT · XML. See the support table below; several formats are deliberately "lite" (CSV is RFC-4180-lite, EML is MIME-lite, RTF is regex-stripped, etc.).
• Phrase + OR queries — "contrato marco" and contrato | acuerdo work out of the box.
• Cooperative search — searchCooperative(query, { frameBudgetMs }) yields to the scheduler between slices. Results land in one batch (real incremental streaming is on the backlog).
• Persistence — snapshot the searchable index to OPFS / IndexedDB and restore it on the next visit. Snapshots store the indexed text, chunk metadata, document names and content hashes; they do not upload anything and they are scoped to the current browser origin.
• Incremental updates — removeDocument, replaceDocument, compact. Content-hash dedup is automatic.
• Resource aware — pauses speculative work in background tabs, shrinks workers on low battery, defers PDF download on slow networks.
• Off-main-thread — AlbexEngineWorker mirror or AlbexPool shard across N workers (map-reduce search).
• WebGPU pre-filter — experimental, opt-in (gpu: 'auto'). Implemented for corpora over 20 k chunks; no reproducible speedup number yet — the bench in this repo runs on a 200-document synthetic corpus only.
• SIMD opportunistic — picks a SIMD-accelerated variant when the host supports v128.
• Tiered storage — TieredStore keeps recent docs hot, evicts cold ones to OPFS, promotes on demand.
• Runtime capacity — one binary, pools sized at init: capacity: 'std' (default, 128 docs / 100k chunks / 16 MB text), 'large' (1 024 docs / 800k chunks / 128 MB text) or a custom { maxDocs, maxChunks, textPoolBytes, namePoolBytes }.
• Capacity-safe — when a pool fills (docs/chunks/text/names), indexFile throws AlbexCapacityError with limit (which pool) and max (the runtime limit) instead of silently truncating the corpus.
• Re-entrancy-safe — async operations on one engine serialize; sync search/compact/reset refuse to run mid-operation (AlbexError kind busy) rather than corrupting the shared WASM state. Use searchCooperative for overlapping search-as-you-type.
• Typed errors — AlbexParseError, AlbexUnsupportedFormatError, AlbexCapacityError, AlbexInitError. All extend AlbexError.
• Tiny core — main WASM ~47 KB (~19 KB gzipped); the SIMD build is ~54 KB (~21 KB gzipped). PDF module ~118 KB (~58 KB gzipped) loads on demand. The OCR companion (@albex/ocr) is a separate package and pulls Tesseract.js (~3.5 MB) only when you call enableOcr().

Supported formats

| Extension | How text is extracted | |--------------------|-----------------------| | .docx | Native Rust/WASM XML parser — streams word/document.xml | | .xlsx | Native Rust/WASM XML parser — shared strings + inline strings | | .pdf | Separate albex_pdf.wasm (pure Rust, loaded on demand) — lightweight best-effort; see limitations | | .md / .markdown| TS parser — strips CommonMark marks | | .html / .htm | TS parser — strips <script> / <style>, paragraphs at block boundaries | | .json | TS parser — recursive walk over keys + string leaves | | .csv | TS parser — RFC 4180 lite; one row per chunk | | .eml | TS parser — MIME-lite: From/To/Subject + text/plain body | | .rtf | TS parser — strips control words / groups | | .txt | Plain text split on double newlines | | .xml | Tag-stripped, entity-decoded |

PDF extraction — coverage & limitations

The PDF engine (albex_pdf.wasm, ~118 KB / ~58 KB gzipped) is a lightweight, best-effort text extractor purpose-built for search indexing — it recovers the words on each page, not the page's layout. It is intentionally not a spec-complete PDF reader: it skips the font-rasterization, reading-order and encryption machinery a full renderer needs, which is what keeps it ~8× smaller than a general-purpose extractor.

How good is it? Measured by token-level parity against a full extractor (pdf-extract) over 240 real-world PDFs:

| Corpus | Recall | Precision | F1 | |--------|-------:|----------:|---:| | Real-world documents (159 files) | 98.65 % | 95.44 % | 96.47 % | | Incl. pathological stress corpus (224 files) | 91.94 % | 86.52 % | 88.46 % |

Hostile/malformed input is contained: the module enforces input-size, decompression, predictor, text-output and image budgets, so a decompression bomb or a malformed file degrades to a bounded failure instead of hanging the tab.

Unsupported by design (these yield no text and fall through to the OCR path when @albex/ocr is wired, or register the document with no searchable chunks):

• AES-encrypted PDFs. Only the common empty user password RC4 case is opened (the "restrict printing/copying" protection on a file you already have locally); password-protected PDFs are not unlocked.
• Type0/CID fonts without a /ToUnicode map or a standard CMap — e.g. some CJK PDFs whose glyph→Unicode mapping lives only in the embedded font program.
• Reading order in complex layouts. Multi-column, dense tables and heavily designed pages that draw glyphs out of visual order may serialize in the wrong order. Words are still recovered (search by word works); snippet/chunk order can suffer. This is the stress corpus above.

The underlying parser is tolerant rather than spec-correct (it recovers from broken cross-reference tables by scanning the file; objects are keyed by number, not number+generation). For indexing this is a feature; for byte-exact round-tripping it is not the right tool.

Query syntax

| Input | Behaviour | |----------------------|-----------| | contrato | Fuzzy match, accent-insensitive | | contrato marco | Both words must appear in the same chunk | | "contrato marco" | Both words AND they must be adjacent (phrase) | | contrato \| acuerdo | OR: union of results matching either branch |

Up to 4 space-separated tokens per simple/phrase query. OR branches are unlimited.

API at a glance

// Construct
const engine = new AlbexEngine();
await engine.init();

// Indexing
const doc = await engine.indexFile(file);

// Search (synchronous fast path)
const results = engine.search('contrato', { windowed: true });

// Cooperative search (yields to the scheduler between slices)
for await (const r of engine.searchCooperative('contrato', { frameBudgetMs: 8 })) {
  /* … */
}

// Incremental updates
engine.removeDocument('contract.pdf');
await engine.replaceDocument('contract.pdf', newFile);
engine.compact();

// Persistence (OPFS or IndexedDB)
await engine.save('my-corpus');
await engine.loadOrInit('my-corpus');

// Tuning
engine.setMaxErrors(2);
engine.setThreshold(400);
engine.setMaxResults(50);
engine.setLanguage('es');

// Introspection
const stats   = engine.getStats();
const lastRun = engine.getLastSearchStats();

Full API reference and types: bdovenbird.com/albex/docs.

Off the main thread

For interactive search UIs, run the engine inside a Web Worker:

import { AlbexEngineWorker } from 'albex/worker';

const engine = new AlbexEngineWorker({
  workerUrl: new URL('albex/worker-runtime', import.meta.url),
});
await engine.init();

Same surface as AlbexEngine; everything returns a Promise. The new URL('albex/worker-runtime', import.meta.url) form works in Vite / Webpack / Rollup / Parcel. On esbuild / Angular the worker entry must be pre-bundled — see the Angular recipe.

Sharding across cores

For large corpora, an AlbexPool shards documents across N workers:

import { AlbexPool } from 'albex/pool';

const pool = new AlbexPool({
  workerUrl: new URL('albex/worker-runtime', import.meta.url),
  workers:   'auto',   // = cores / 2, clamped [1, 8]
});
await pool.init();

await pool.indexFile(fileA);    // sharded round-robin
const results = await pool.search('contrato');  // map-reduce

Persistence

Albex can save the current search index locally so users do not need to re-index the same documents on every visit:

const engine = new AlbexEngine();
await engine.init();

const restored = await engine.loadOrInit('my-library');

if (!restored) {
  await engine.indexFile(fileA);
  await engine.indexFile(fileB);
  await engine.save('my-library');
}

const results = engine.search('contrato');

save(name) stores a snapshot of the index: document names, chunk metadata, indexed text, ranking data and content hashes. It does not upload anything and it does not require a server. By default Albex uses OPFS when the browser supports it and falls back to IndexedDB otherwise. The data belongs to the current origin, so it can be removed if the user clears site data.

Use save / load / loadOrInit when you want to restore the searchable index. Use TieredStore when you also want Albex to keep the original files in OPFS so cold documents can be brought back into the hot in-memory index later.

| API | What it does | |-----|--------------| | save(name) | Writes a snapshot of the current index. | | load(name) | Restores an existing snapshot and returns whether it loaded. | | loadOrInit(name) | Loads a snapshot if present; otherwise leaves the engine empty. | | deleteSnapshot(name) | Removes a saved snapshot. | | listSnapshots() | Lists snapshot names for the current origin. | | TieredStore | Stores original blobs as a warm tier for larger corpora. |

Snapshots are admitted by content: a snapshot saved with capacity: 'large' can load into a default engine whenever its actual document/chunk/text counts fit. If a snapshot is incompatible or does not fit, load() returns false and the previous in-memory index remains usable.

Big corpora — tiered storage

For workloads that exceed the engine's RAM capacity:

import { AlbexEngine, TieredStore } from 'albex';

const engine = new AlbexEngine();
await engine.init();

const store = new TieredStore(engine, { evictThreshold: 0.85 });
await store.init();

await store.indexFile(file);              // persists original blob in OPFS
await store.promote('older-doc.pdf');     // brings warm doc back

Hot tier = the live AlbexEngine index. Warm tier = original files stored in OPFS. LRU eviction is automatic, and promotion re-indexes a warm file when the user needs it again.

Advanced configuration

new AlbexEngine() covers the default case. The options below address specific deployment needs:

Capacity (runtime, single binary)

Capacity is a runtime parameter — there is one engine binary (plus its SIMD variant) and the pools are heap-allocated at init() to the size you ask for. The old compile-time tiers (and the tier option) are gone:

const engine = new AlbexEngine({
  capacity: 'large',               // or 'std' (default), or a custom object
  simd: 'auto',                    // picks baseline/simd by WASM probe
  gpu:  'auto',                    // engages WebGPU when corpus > 20k chunks
});

Presets and cost (≈ maxChunks × 64 B + textPool + namePool; WASM memory never shrinks, so the largest capacity initialised stays committed):

| Capacity | Max docs | Max chunks | Max text | Working set | |-----------|---------:|-----------:|---------:|------------:| | 'std' | 128 | 100 000 | 16 MB | ~22 MB | | 'large' | 1 024 | 800 000 | 128 MB | ~180 MB | | custom | ≤ 65 536 | ≤ 4 M | ≤ 1 GiB | as configured |

Custom objects may be partial — missing fields are completed from the std ratios (maxChunks = maxDocs × 782, textPoolBytes = maxChunks × 168 B, namePoolBytes = maxDocs × 256 B, with sane floors):

const tiny = new AlbexEngine({ capacity: { maxDocs: 16 } });

Snapshots are admitted by content: a snapshot saved with 'large' loads into a 'std' engine whenever its counters fit, and fails cleanly (previous index intact) when they don't.

WASM source — SIMD, CDN, custom bytes

The default embeds the baseline core, which is all most apps need. Three opt-in overrides, none required:

// SIMD on capable hosts: serve both binaries and let Albex probe.
new AlbexEngine({ wasmBaseUrl: '/albex' }); // dir with albex_wasm.wasm + albex_wasm_simd.wasm

// A single core from your CDN.
new AlbexEngine({ wasmUrl: 'https://cdn.example.com/albex_wasm.wasm' });

// Bytes you loaded yourself (skips all fetching).
new AlbexEngine({ wasmBytes: myArrayBuffer });

Setting any of these bypasses the embedded core. If a fetch fails, the thrown AlbexInitError names the URL it tried and tells you the one-line exit (drop the option to fall back to the embedded core) — no guessing.

PDFs on esbuild / Angular

The PDF engine (~118 KB) loads on demand and isn't embedded. Bundlers that resolve new URL('…', import.meta.url) (Vite, Webpack 5+, Rollup, Parcel) serve it automatically. esbuild and Angular don't, so you point Albex at the binary yourself. The file ships in the package at albex/wasm/pkg/albex_pdf.wasm.

Vite / Webpack — let the bundler emit the asset URL:

import pdfWasmUrl from 'albex/wasm/pkg/albex_pdf.wasm?url';   // Vite (?url)
// Webpack 5: import pdfWasmUrl from 'albex/wasm/pkg/albex_pdf.wasm';

const engine = new AlbexEngine({ pdfWasmUrl });

esbuild / Angular — import … from '*.wasm' does not give a URL here. Serve the file yourself and pass its path (see the verified Angular recipe below):

const engine = new AlbexEngine({ pdfWasmUrl: 'albex_pdf.wasm' });
// or, if you already have the bytes:
// new AlbexEngine({ pdfWasmBytes: await fetch('/albex_pdf.wasm').then(r => r.arrayBuffer()) })

Angular (verified, Angular 19 application builder)

The embedded search core needs no setup — new AlbexEngine() works in an Angular component out of the box. Two extras only if you need them: PDFs and the Web Worker.

1. PDFs. Copy the PDF binary into the build output via angular.json assets, then point pdfWasmUrl at it:

// angular.json → projects.<app>.architect.build.options.assets
{ "glob": "albex_pdf.wasm", "input": "node_modules/albex/wasm/pkg", "output": "." }

// albex.service.ts
import { AlbexEngine } from 'albex';

const engine = new AlbexEngine({ pdfWasmUrl: 'albex_pdf.wasm' });
await engine.init();
const doc = await engine.indexFile(pdfFile);   // real text via the PDF core

2. Web Worker (optional). Angular's esbuild can't analyse new URL(…, import.meta.url) when the new Worker() call lives inside the library, so the worker entry needs to be pre-bundled into a self-contained file and the core wasm shipped as an asset (the worker entry does not embed the core). One build step + one asset:

// package.json scripts
"build:albex-worker": "esbuild node_modules/albex/dist/worker-runtime.js --bundle --format=esm --outfile=public/albex-worker.js"

// angular.json assets — also copy the core wasm for the worker
{ "glob": "albex_wasm_bg.wasm", "input": "node_modules/albex/wasm/pkg", "output": "." }

import { AlbexEngineWorker } from 'albex/worker';

const engine = new AlbexEngineWorker({
  workerUrl: 'albex-worker.js',      // pre-bundled, served from public/
  wasmUrl:   'albex_wasm_bg.wasm',   // worker entry isn't embedded — give it the core
});
await engine.init();

Run npm run build:albex-worker before ng build (wire it into your build script). If you don't need to keep the UI thread free during bulk indexing, skip the worker entirely — main-thread search is sub-millisecond on typical corpora.

Errors

All errors thrown by Albex extend AlbexError:

import {
  AlbexError, AlbexInitError, AlbexParseError,
  AlbexUnsupportedFormatError, AlbexCapacityError,
} from 'albex';

try {
  await engine.indexFile(file);
} catch (e) {
  if (e instanceof AlbexUnsupportedFormatError) {
    console.warn(`Skipped .${e.ext} (unsupported)`);
  } else if (e instanceof AlbexParseError) {
    console.warn(`Parse failed for ${e.format}:`, e.message);
  } else throw e;
}

Each error carries a kind field that survives structuredClone across worker boundaries.

Stability policy

The public API is the set of package exports declared in package.json:

• albex
• albex/worker
• albex/worker-runtime
• albex/pool
• albex/gpu
• albex/tiered
• albex/wasm/pkg/*.wasm

albex/inline is also exported, but only as a backward-compatibility alias for the default entry — the embedded core is the default since 0.7.0, so import { AlbexEngine } from 'albex' and from 'albex/inline' are identical. Prefer the bare albex import in new code.

The package also ships src/ so declaration maps and source maps remain useful when debugging. Treat those files as source reference, not as a supported import surface: code should import through the package exports above.

Snapshot compatibility is content-based. Newer Albex releases should continue loading supported older snapshots when their contents fit the live engine capacity. If a snapshot is corrupt, incompatible or too large for the current capacity, load() returns false and leaves the existing in-memory index usable.

Deprecated aliases are removed before 1.0 rather than carried into the stable surface. Use searchCooperative, not the old searchStream name.

Browser requirements

• WebAssembly (every browser since 2017)
• DecompressionStream for DOCX/XLSX (Chrome 80+, Firefox 113+, Safari 16.4+)
• OPFS for fastest persistence (Chrome 102+, Safari 15.2+, Firefox 111+); IndexedDB fallback works everywhere
• WebGPU is optional; without it the CPU path is the default

PDF support requires albex_pdf.wasm to be served with MIME type application/wasm.

Building from source

rustup target add wasm32-unknown-unknown

npm install
npm run build:all         # main (baseline + SIMD) + PDF + TypeScript

Partial builds:

npm run build:wasm        # main module (baseline + SIMD)
npm run build:pdf-wasm    # PDF module
npm run build             # TypeScript only

Tests

# Rust unit tests
cargo test --manifest-path core/Cargo.toml
cargo test --manifest-path ingest/Cargo.toml

# TypeScript + WASM integration tests
npm test

# Micro-benchmarks
npm run bench

About the benchmark. The included bench is a regression tool for Albex itself: it covers raw WASM operations, engine-level search, snapshotting and a larger synthetic corpus. It is not a universal performance promise and it is not a comparison against hosted search engines or other libraries. Use it to spot changes in this implementation, then test with your own documents and queries before making product decisions.

CI runs every check on every push to main.

Privacy

Albex never transmits document content. Text extraction, indexing, search and persistence all happen inside the browser. The only network requests are the initial fetches for the .wasm binaries (and the lazy PDF module on first PDF). Persisted snapshots live in OPFS / IndexedDB, scoped to your origin.

License

MIT