okf-search-native

Search Open Knowledge Format (OKF) Markdown collections at native speed from Node.js, powered by Rust and Tantivy. Get the best matching section from each document, with its source path, line numbers, and snippet.

Install

npm install okf-search-native

Requires Node.js >=22.19.0. Includes TypeScript declarations and native binaries for macOS x64/arm64 and Linux x64 (glibc >= 2.17); Windows x64 is experimental. Browsers and Alpine/musl are not supported. See the full platform list.

Search a collection

Given a directory of OKF Markdown files at ./knowledge:

import { openOkf } from "okf-search-native";

const index = await openOkf("./knowledge");
const hits = index.search("rollback deployment");

for (const hit of hits) {
  console.log(hit.path, hit.headingPath, hit.snippet);
}

To try this with one document, save the following as knowledge/runbooks/deployment.md before running the example:

---
type: runbook
---
# Deployment

## Rollback

To rollback a deployment, restore the previous release and check service health.

The result points to the rollback section (selected fields shown):

{
  path: "runbooks/deployment.md",
  headingPath: "Deployment > Rollback",
  startLine: 6,
  endLine: 8,
  snippet: "To rollback a deployment, restore the previous release and check service health."
}

Use the path and line numbers to open the source, and the heading and snippet to display a preview. Results contain at most one hit per document, ordered by relevance. See the complete result shape.

Open and reuse an index

Without options, openOkf reads and indexes the collection in memory. The handle does not watch files or write source files. Without a cache, call openOkf again to pick up source filesystem changes.

To reuse a native snapshot across processes, pass a filesystem cachePath:

import { openOkf } from "okf-search-native";

const cachePath = "./.cache/knowledge.okf";
const index = await openOkf("./knowledge", { cachePath });

index.ingest({
  path: "runbooks/new.md",
  markdown: "---\ntype: runbook\n---\nNew material.\n",
});
await index.save(cachePath);

• First open: a missing cache is built from root; parent directories are created, and the complete cache is published before openOkf resolves.
• Later open: an existing cache is loaded without reading or requiring root; saved document paths stay unchanged.
• Failures: an existing directory, dangling link, unreadable, corrupt, or incompatible cache destination rejects instead of silently rebuilding from root. Damaged contents report ERR_OKF_CACHE_INVALID; unsupported cache metadata reports ERR_OKF_CACHE_INCOMPATIBLE.
• No cache: without cachePath, the handle stays in memory and creates no cache artifacts. cachePath is a cache-file path, not a Markdown identity.

openOkf recursively reads lowercase .md files, excluding files named exactly index.md or log.md. See the persistence contract for writer exclusion, snapshot timing, and filesystem caveats.

Already have Markdown strings?

Use createOkfSearch instead of reading a directory. It builds the same kind of handle synchronously:

import { createOkfSearch } from "okf-search-native";

const index = createOkfSearch([{
  path: "runbooks/deployment.md",
  markdown: "---\ntype: runbook\n---\nTo rollback a deployment, restore the previous release.\n",
}]);

const hits = index.search("rollback deployment");

Refine a search

Require all query terms and restrict results to runbooks:

index.search("rollback deployment", {
  match: "all",
  where: { types: ["runbook"] },
});

Enable typo tolerance:

index.search("deploymnt", { fuzzy: true });

By default, searches return up to ten documents, match any query term across all searchable fields, and disable fuzzy matching. The final term still matches prefixes when it has at least three characters: "deploy" can match "deployment", even with fuzzy: false.

Use limit to change the result count and fields to restrict where terms match. Filters also support tags, status, trust tier, staleness, and conformance. See search options and defaults for field boosts, filter combinations, and detailed matching rules.

Update the in-memory index

ingest adds a document or replaces the document with the same path. remove returns whether the document was present. Neither operation changes files:

index.ingest({
  path: "runbooks/restart.md",
  markdown: "---\ntype: runbook\n---\nRestart the service after draining active requests.\n",
});

index.remove("runbooks/restart.md");

Use relative .md paths, such as runbooks/restart.md. If preparation of a replacement fails, the existing document remains searchable. See update results and path rules.

Save a snapshot

save(path) explicitly writes the current handle state. It is available on handles from both openOkf and createOkfSearch:

import { createOkfSearch } from "okf-search-native";

const index = createOkfSearch([
  { path: "notes/one.md", markdown: "---\ntype: note\n---\nOne.\n" },
]);

await index.save("./.cache/notes.okf");

save captures one consistent snapshot before returning its promise and resolves after atomic publication. Mutations made after capture require another save. Concurrent writers to one destination reject with ERR_OKF_CACHE_BUSY; independent handles are not merged. A failed save does not replace a previous complete cache or poison a healthy handle. See the full persistence contract for locking, reader visibility, and filesystem caveats.

Check documents and handle failures

Documents with valid OKF metadata are strict. Some metadata problems make a document degraded: it remains searchable, with diagnostics explaining what needs repair. Fatal problems, such as missing required type metadata, prevent indexing.

Constructors and ingest validate automatically. To inspect diagnostics before indexing, use validateOkfDocument:

import { validateOkfDocument } from "okf-search-native";

const input = {
  path: "runbooks/draft.md",
  markdown: "---\ntype: runbook\nstatus: not-a-status\n---\nDraft deployment instructions.\n",
};
const validation = validateOkfDocument(input);

for (const { path, field, message } of validation.errors) {
  console.warn(path, field, message);
}

if (validation.isIndexable) {
  index.ingest(input); // Degraded documents can still be indexed.
}

Validation returns expected document problems as diagnostics. Indexing rejects fatal document problems with OkfError; openOkf also rejects unreadable files. Invalid search options throw TypeError. An ERR_OKF_INDEX_UNUSABLE error means the handle must be rebuilt, not retried. See validation outcomes and error handling.

To inspect an existing collection:

console.log(index.indexStats().logical.documents.total);
console.log(index.listTypes());
console.log(index.listDegradedDocuments());

Performance benchmarks

Measured on 13,692 Markdown documents (59.57 MiB of source text), using the public openOkf and search APIs.

Default search options

Fuzzy matching is disabled by default; final-term prefix matching remains enabled.

| Metric | okf-minisearch | okf-search-native | | --- | ---: | ---: | | Median openOkf time | 20.40 s | 1.57 s | | Warm query p50 | 10.45 ms | 1.38 ms | | Warm query p95 | 86.81 ms | 2.38 ms | | Warm query p99 | 95.64 ms | 2.45 ms | | Reported index storage¹ | 200.37 MiB | 80.75 MiB | | Median post-open RSS | 2,235 MiB | 527 MiB |

Fuzzy matching enabled

The same queries and defaults, changing only the search call to:

index.search(query, { fuzzy: true });

| Metric | okf-minisearch | okf-search-native | | --- | ---: | ---: | | Warm query p50 | 14.36 ms | 7.96 ms | | Warm query p95 | 87.36 ms | 14.12 ms | | Warm query p99 | 96.71 ms | 15.00 ms |

¹ MiniSearch reports serialized JSON bytes; native reports in-memory Tantivy index-file bytes. These are different storage representations, not equivalent RAM measurements. Native storage can vary with background merges.

Reference and development

• API reference: options, return values, errors, and index statistics.
• Prepared API: for applications that already produce prepared documents.
• Backend differences: Tantivy ranking differs from okf-minisearch; autoSuggest is unsupported.
• Development: local builds, tests, and release artifacts.

License

MIT