okf-toolkit

Parse, validate, and chunk Open Knowledge Format (OKF) bundles — and turn them into RAG-ready chunks for any vector store or embedding API.

OKF (announced by Google Cloud, June 2026) represents knowledge as a directory of markdown files with YAML frontmatter — table schemas, runbooks, metric definitions, anything an AI agent needs context on. It's a great storage and authoring format, but going from "a folder of markdown" to "rows in a vector database" still takes real work: splitting documents sensibly, carrying metadata through, resolving cross-links, and catching malformed bundles before they corrupt your index. That's what this package does.

npm install okf-toolkit

Quick start

import { parseBundle, validateBundle, chunkBundle } from "okf-toolkit";

const bundle = parseBundle("./my-knowledge-bundle");

const validation = validateBundle(bundle);
if (!validation.valid) {
  console.error(validation.issues);
  // handle/fix before indexing — see "Validation" below
}

const chunks = chunkBundle(bundle, { maxChunkChars: 1500 }); // override the 1800 default
// chunks is now an array of { chunkId, text, metadata, linksTo, ... }
// ready to embed and upsert into any vector store.

Or from the command line. The installed binary is okf:

okf validate ./my-bundle
okf chunk ./my-bundle -o chunks.json --pretty

…or run it without installing via npx (this still invokes the okf binary):

npx okf-toolkit validate ./my-bundle
npx okf-toolkit chunk ./my-bundle -o chunks.json --pretty

Why chunking an OKF bundle isn't trivial

A naive approach — embed each .md file as one chunk — breaks down fast:

• A table concept with a # Schema and a # Joins section conflates two different questions into one vector.
• Long concepts blow past most embedding models' useful context window, diluting the vector.
• Frontmatter (type, tags, resource, timestamp) is exactly the metadata you want for filtering at query time (e.g. "only BigQuery Table concepts tagged sales") — it shouldn't get lost in the chunking step.
• Cross-links ([customers](/tables/customers.md)) are part of what makes OKF richer than flat text; a good RAG pipeline should be able to say "this chunk references tables/customers," for example to fetch a linked concept when a retrieved chunk alone is insufficient.

okf-toolkit handles all of this: it splits at heading boundaries first (so each chunk stays topically coherent), further splits oversized sections on sentence/paragraph boundaries with configurable overlap, folds frontmatter into per-chunk metadata, and attributes each cross-link only to the chunk whose text actually contains it.

API

parseBundle(rootDir: string): OkfBundle

Walks a directory on disk, reading every .md file. Files named index.md or log.md (the spec's reserved filenames) are collected as specialFiles; everything else becomes a concept, with frontmatter parsed, body extracted, and markdown links resolved to concept IDs where possible.

parseConcept(conceptId: string, raw: string): OkfConcept

Parses a single document from a string, without touching the filesystem — useful if you're fetching OKF content from an API, a database export, or generating it on the fly.

validateBundle(bundle: OkfBundle): ValidationResult

Checks the bundle against the OKF v0.1 conformance rules:

| Code | Severity | What it means | |---|---|---| | MISSING_TYPE | error | A concept has no type frontmatter field — required by spec. | | INVALID_TAGS_FIELD | error | tags is present but isn't an array. | | DUPLICATE_CONCEPT_ID | error | Two documents resolve to the same concept ID. | | BROKEN_LINK | warning | An internal markdown link doesn't resolve to any concept in the bundle. The spec explicitly allows partial/growing bundles, so this is non-fatal. | | MISSING_INDEX | warning | A directory holding concepts has no index.md for progressive disclosure. | | EMPTY_BODY | warning | A concept has frontmatter but no body content. | | UNPARSEABLE_TIMESTAMP | warning | The timestamp field isn't a parseable date. |

result.valid is true only when there are zero errors (warnings don't affect it). Good as a CI gate: okf validate ./bundle exits non-zero on any error.

chunkConcept(concept: OkfConcept, options?): OkfChunk[] / chunkBundle(bundle: OkfBundle, options?): OkfChunk[]

interface ChunkOptions {
  maxChunkChars?: number;       // default 1800
  overlapChars?: number;        // default 150
  includeHeadingContext?: boolean; // default true — prepends "Section > Subsection" to chunk text
}

Each OkfChunk looks like:

{
  chunkId: "tables/orders#1",       // stable, deterministic: `${conceptId}#${index}`
  conceptId: "tables/orders",
  headingPath: ["Joins"],
  text: "Joins\n\nJoined with [customers](/tables/customers.md) on `customer_id`...",
  metadata: {
    type: "BigQuery Table",
    title: "Orders",
    tags: ["sales", "revenue"],
    resource: "https://...",
    sourcePath: "tables/orders.md",
    headingPath: ["Joins"],
    chunkIndex: 1,
    totalChunksInConcept: 3,
  },
  linksTo: ["tables/customers"],     // concept IDs referenced in *this* chunk specifically
}

Wiring it into a real RAG pipeline

The output is intentionally vector-store-agnostic. A minimal example with any embeddings API and any vector DB's upsert call:

import { parseBundle, validateBundle, chunkBundle } from "okf-toolkit";

const bundle = parseBundle("./knowledge");
const { valid, issues } = validateBundle(bundle);
if (!valid) throw new Error(`Bundle has errors: ${JSON.stringify(issues)}`);

const chunks = chunkBundle(bundle);

for (const chunk of chunks) {
  const embedding = await embed(chunk.text); // your embedding model of choice

  await vectorStore.upsert({
    id: chunk.chunkId,
    vector: embedding,
    metadata: {
      ...chunk.metadata,
      conceptId: chunk.conceptId,
      linksTo: chunk.linksTo,
    },
  });
}

At query time, once you've retrieved top-k chunks by similarity, linksTo lets you optionally pull in directly-linked concepts too (e.g. if a retrieved "Orders" chunk links to "Customers" and the question seems to need both) — a cheap way to get some of the benefit of graph-aware retrieval without a graph database.

A note on what this package is not

It doesn't fetch embeddings, talk to any vector database, or implement retrieval/ranking — those choices are yours, and deliberately out of scope so this stays a small, dependency-light building block rather than a framework. It also doesn't produce OKF bundles from arbitrary source systems (Google's reference enrichment agent does that for BigQuery); this package starts from an existing bundle.

Example output (the bundled sales fixture)

The repo ships a small example bundle at test/fixtures/sales (table and metric concepts, plus an intentionally malformed stub). Running the CLI against it shows the toolkit end to end.

okf validate ./test/fixtures/sales — exits non-zero because of the one seeded error:

ERROR MISSING_TYPE [metrics/untyped_stub]: Concept is missing the required "type" frontmatter field.
WARN  BROKEN_LINK [metrics/weekly_active_users]: Link "/reference/timezones.md" does not resolve to any concept in the bundle (resolved to "reference/timezones").
WARN  MISSING_INDEX: Directory "metrics" has concept documents but no index.md.
WARN  MISSING_INDEX: Directory "tables" has concept documents but no index.md.

4 concept(s) checked — 1 error(s), 3 warning(s).

okf chunk ./test/fixtures/sales — turns 4 concepts into 8 heading-aware chunks, with cross-links attributed to the specific chunk whose text contains them:

| chunkId | headingPath | linksTo | |---|---|---| | metrics/untyped_stub#0 | — | — | | metrics/weekly_active_users#0 | Definition | tables/orders | | metrics/weekly_active_users#1 | Caveats | reference/timezones | | tables/customers#0 | Schema | — | | tables/customers#1 | Joins | tables/orders | | tables/orders#0 | Schema | tables/customers | | tables/orders#1 | Joins | tables/customers | | tables/orders#2 | Notes | — |

Note how tables/orders splits into separate Schema / Joins / Notes chunks, and each link is carried only by the chunk it actually appears in — not smeared across the whole concept.

Spec conformance

Implements OKF v0.1: concept identity via file path, required type frontmatter field, reserved index.md/log.md filenames, and markdown links as the cross-reference graph. OKF is a draft spec under active development — this package will track it as it evolves.

Changelog

This project follows semantic versioning. OKF itself is a draft spec, so minor versions may track spec changes until OKF stabilizes.

0.1.0

• Initial release.
• parseBundle / parseConcept — read OKF bundles from disk or a raw string, with frontmatter parsing and cross-link resolution.
• validateBundle — OKF v0.1 conformance checks (MISSING_TYPE, INVALID_TAGS_FIELD, DUPLICATE_CONCEPT_ID, BROKEN_LINK, MISSING_INDEX, EMPTY_BODY, UNPARSEABLE_TIMESTAMP).
• chunkConcept / chunkBundle — heading-aware chunking with configurable size/overlap, per-chunk metadata, and per-chunk link attribution.
• okf CLI with validate and chunk commands.

License

Apache-2.0