@galdor/memory-okf

An Open Knowledge Format (OKF) knowledge backend for galdor: a memory.Store over OKF bundles — knowledge as Markdown + YAML frontmatter in a git-versioned directory tree. One .md file is one concept; markdown links between concepts form a directed graph; the only required frontmatter field is type.

It implements the full OKF v0.1 spec: a lexical (BM25) store over galdor's native code-aware index — customer_id is findable whole and by its parts — plus the knowledge layer on top: the link graph (with graph-expanded retrieval), per-directory index.md progressive disclosure, log.md change history, numbered citations, strict conformance validation, and bundle writing — with okf_search and okf_browse tools for agents.

Documentation: rag · Quickstart

Install

bun add @galdor/memory-okf   # or: npm install @galdor/memory-okf

Usage

import { open, newSearchTool } from "@galdor/memory-okf";

// Load + chunk + index in one call.
const store = await open("./bundle");

const hits = await store.retrieve({ text: "monthly recurring revenue", k: 3 });
for (const h of hits) console.log(h.score, h.chunk.metadata?.concept_id);

await store.close();

Give an agent the bundle as a tool:

import { Registry } from "@galdor/core/tool";
import { run } from "@galdor/core/agent";

const answer = await run(
  { provider, model: "claude-haiku-4-5", tools: new Registry(newSearchTool(store)) },
  "What is MRR and which table models it?",
);

Filtering

Every query.filter entry is an exact metadata match pushed down to the store — except the reserved keys, which are post-filters the core key/value contract can't express: FilterTag (tag membership — OKF tags are a list) and FilterSince / FilterUntil (ISO-8601 timestamp bounds):

import { FilterSince, FilterTag, MetaSection, MetaType } from "@galdor/memory-okf";

await store.retrieve({ text: "revenue", filter: { [MetaType]: "Metric" } });      // by concept type
await store.retrieve({ text: "mrr", filter: { [FilterTag]: "billing" } });        // by tag membership
await store.retrieve({ text: "mrr", filter: { [FilterSince]: "2026-06-01" } });   // by timestamp
await store.retrieve({ text: "columns", filter: { [MetaSection]: "schema" } });   // by body section

The whole bundle

load reads only the concepts; loadBundle returns everything the spec layers on top — and it writes and validates too:

import { GraphExpander, hasErrors, loadBundle, newBrowseTool, writeBundle } from "@galdor/memory-okf";

const bundle = loadBundle("./bundle");
bundle.version;                          // okf_version from the root index.md
bundle.outlinks("metrics/mrr");          // the link graph, navigable
bundle.indexFor("tables");               // real index.md, or synthesized on the fly
bundle.logs.get("");                     // date-grouped change history
bundle.citations("metrics/mrr");         // numbered [n] [text](url) citations
if (hasErrors(bundle.validate())) throw new Error("bundle not conformant"); // CI gate
writeBundle("./out", bundle);            // producer side: render it back to disk

const expander = new GraphExpander({ inner: store, bundle }); // hits + their linked concepts
const browse = newBrowseTool(bundle);    // okf_browse: explore before searching

Hybrid retrieval

This backend is the lexical half. Fuse it with a dense (vector) source using HybridRetriever from @galdor/core/memory — Reciprocal Rank Fusion (k=60) works on ranks, so no score normalization is needed:

import { HybridRetriever, Retriever, HashingEmbedder } from "@galdor/core/memory";
import { load, chunkConcepts, newStore } from "@galdor/memory-okf";
import { openSqlite } from "@galdor/memory-sqlite";

const chunks = chunkConcepts(load("./bundle").documents);
const bm25 = await newStore(chunks);

const embedder = new HashingEmbedder(256); // swap for a provider embedder in prod
const vectors = await embedder.embed(chunks.map((c) => c.text));
const vec = openSqlite(":memory:");
await vec.add(chunks.map((c, i) => ({ ...c, embedding: vectors[i] })));

const hybrid = new HybridRetriever({ sources: [bm25, new Retriever({ store: vec, embedder })] });
const hits = await hybrid.retrieve({ text: "how is recurring revenue measured", k: 3 });

See packages/examples/okf-rag.ts for the runnable version (--mode bm25|hybrid).

Pieces

open() is the one-liner. Use the parts when you also need the documents (e.g. to build a second, vector-backed source):

| Export | What it does | |---|---| | load(root) / loadFiles(files) | Read a bundle's concepts → { documents, warnings } | | loadBundle(root) / loadBundleFiles(files) | The whole bundle: graph, indexes, logs, version, citations, validate | | chunkConcepts(docs) | Concept-first chunks; splits large bodies by top-level # headings | | newStore(chunks) / open(root) | Native BM25 store from chunks / from a directory | | newSearchTool(store) | okf_search tool (query, type, tag, since, until, section, k) | | newBrowseTool(bundle) | okf_browse tool (directory → index, subdirs, concepts) | | GraphExpander | Store decorator that appends a hit's graph neighbors, score-decayed | | marshal(doc) / writeBundle(root, bundle) | Producer side: render concepts / whole bundles back to disk | | hasErrors(bundle.validate()) | Strict conformance gate (errors block, warnings inform) | | Filter*, Meta* | The reserved filter keys and metadata key constants |

Behavior

• Reserved files — index.md and log.md are never concepts; loadBundle parses them into the navigation layer (indexes, logs, version).
• Chunking — one chunk per concept; bodies over 1200 chars split by top-level # headings. Each chunk's indexed text is prefixed with title. description tags: … resource: … so BM25 matches those fields; chunks from conventional # Schema / # Examples / # Citations sections carry section metadata.
• Links — markdown links resolve to concept ids (bundle-absolute /x.md and relative ../x.md); resolved ids land in the outlinks metadata (the graph's edges), broken ones become warnings.
• Permissive — only type is required; unknown fields, unknown types and broken links are tolerated, not rejected. Producer-defined frontmatter keys are preserved under the fm. metadata prefix and written back by marshal. bundle.validate() is the strict counterpart when you need conformance.
• Tokenization — galdor's code-aware tokenizer keeps compound identifiers whole and by parts: customer_id matches customer_id, customer or id, and the literal carrier outranks coincidental mentions.

License

Apache-2.0