@promptowl/contextnest-engine
v3.7.0
Published
Core engine for governed, versioned, AI-ready context — selectors, hash-chain versioning, graph traversal, and integrity verification for LLMs and agents
Readme
@promptowl/contextnest-engine
The governed, versioned context engine for AI agents — not a memory store.
The core engine behind Context Nest. It turns a folder of markdown into a typed, queryable document graph where every change is hash-chained and auditable. Where a memory store appends opaque blobs and hopes for recall, this engine gives agents a deterministic query grammar, graph traversal, and a byte-level audit trail — the same vault onboards one developer in ten minutes and passes a SOC 2 review when that day comes.
Install
npm install @promptowl/contextnest-engine7 packages at depth 2, down from 76. The markdown AST, glob and frontmatter dependencies were removed outright rather than swapped, and no install script runs. Every package is listed with its version and licence in DEPENDENCIES.md.
Quickstart
import { NestStorage, GraphQueryEngine } from "@promptowl/contextnest-engine";
const storage = new NestStorage("/path/to/vault");
const engine = new GraphQueryEngine(storage);
// Deterministic selector + graph traversal (default: 2 hops).
// Selectors match document metadata first — no file bodies loaded —
// then BFS over relationship edges, loading bodies only for reached nodes.
const result = await engine.query("#engineering + type:document", { hops: 3 });
for (const doc of result.documents) {
console.log(`${doc.id}: ${doc.frontmatter.title}`);
}Worked Example
Read a vault, query its engineering skills, then verify the whole vault's integrity.
import {
NestStorage,
GraphQueryEngine,
CheckpointManager,
} from "@promptowl/contextnest-engine";
const storage = new NestStorage("./my-vault");
const engine = new GraphQueryEngine(storage);
// 1. Pull every skill node tagged #engineering, 2 hops of related context.
const skills = await engine.query("type:skill + #engineering", { hops: 2 });
console.log(`Found ${skills.documents.length} engineering skills`);
for (const skill of skills.documents) {
const { title } = skill.frontmatter;
const trigger = skill.frontmatter.skill?.trigger ?? "(no trigger)";
console.log(`- ${title} — triggers ${trigger}`);
}
// 2. Verify the hash chain across the entire vault before trusting it.
const checkpoints = new CheckpointManager(storage);
const report = await checkpoints.verify();
console.log(report.valid ? "Integrity OK" : `Tampering: ${report.errors}`);What It Does
- Selector Grammar — Deterministic query language: select by tag, type, URI, pack, status, and boolean combinations (
type:skill + #engineering) - Graph Traversal — Hop-based BFS over
context.yamlas a lightweight graph index, with priority-weighted edges - Skill Nodes — First-class
type: skillnodes with trigger, inputs, tools_required, output_format, and guard_rails - Versioning — Hash-chained version history with keyframe + diff reconstruction; each non-keyframe version's change log is a standalone
v{N}.diffunified diff beside the keyframes - Operation Catalog —
@promptowl/contextnest-engine/api: one canonical, schema-described set of 24 operations (context_get,context_query,context_create, …) that CLI, MCP, and REST surfaces bind to instead of hand-rolling their own. As of 2.0 thecorenamespace is complete and every surface actually runs on it - Integrity — SHA-256 content hashes, chain hashes, and checkpoint verification down to the byte
- URI Resolution — Resolve
contextnest://URIs to documents, tags, folders, or search results - Storage — Read/write documents, version histories, checkpoints, and config from the vault file system; discovery is folder-scoped, and
listFoldersreturns the vault's shape from directory entries without parsing a document - Parsing & Validation — Markdown + YAML frontmatter, validated against the spec (skill and source node rules)
- Index Generation — Generate
context.yaml(document graph) andINDEX.md - Agent Config Generation — Auto-generate CLAUDE.md, GEMINI.md, .cursorrules, etc. so AI tools discover the vault
Graph Traversal
The engine evaluates selectors against document metadata (no bodies loaded), then traverses relationship edges via BFS for N hops, loading bodies only for reached nodes. [[wikilinks]] in document bodies ([[Title]], [[Title|alias]], [[nodes/id]]) are resolved at index time into reference edges alongside contextnest:// links, so a wiki-style vault traverses the same way. Both link forms only produce an edge when they resolve to a published document — a [[Title]] or contextnest://nodes/… pointing at a draft, a missing node, or a tag/folder/search URI counts as unresolved (ctx index reports the counts) until it does. The one exception is a cross-namespace contextnest:// link, which names a node in another nest and keeps its full URI as the edge target.
depends_onedges and edges to hub nodes are free (always traversed)referenceedges cost 1 hop- Edges with explicit
priority: 0in frontmatter are free - Adaptive expansion retries with more hops if too few results
Key Exports
| Export | Description |
|--------|-------------|
| NestStorage | File system abstraction for vault operations |
| GraphQueryEngine | Graph-aware query orchestrator (recommended) |
| GraphTraverser | BFS traversal with priority-weighted edge costs |
| Resolver | URI resolution against an in-memory document set |
| ContextInjector | Legacy full-load query orchestrator |
| VersionManager | Document version history management |
| CheckpointManager | Vault-wide checkpoint management |
| generateContextYaml | Generate the context.yaml graph index |
| generateAgentConfigs | Generate AI tool config files |
| parseSelector | Parse selector query strings into AST |
| evaluateFromIndex | Evaluate selectors against the lightweight index (no bodies) |
| publishDocument | Publish a document (bump version, checkpoint) |
| publishDocuments | Bulk publish — one checkpoint and one index pass for the whole batch, with an onProgress(done, total) sink |
| sealWrites / writePublishedVersion | sealWrites seals deferred writes in one checkpoint and one index update. writePublishedVersion publishes a version without its checkpoint. See Host Options. |
| manifestEntry / ManifestEntry | Describes one document's index inputs, for a host that keeps the vault manifest itself. |
| markOnboarding | Records onboarded_at or skipped_at in the registry. Bare ctx writes this after first-run setup. |
| filterDocuments | The one type / status / tag filter, for surfaces that filter a list they already hold |
| setVaultDescription | Set or clear a registry alias's description |
| parseStewards / serializeStewards | Canonical stewards.yaml marshalling (format only) |
| traverseWikiGraph | Body-link seed resolution and hop traversal — follows [[wikilinks]] and contextnest:// links (pinned @N / #anchor forms included) alike |
| extractLinkedIds | Every node a body links to, both link forms, resolved to ids |
| archiveDocument / unarchiveDocument | Take a node out of lists and retrieval with its history kept (spec §1.13), and put it back in the status it left |
| moveDocument | Give a node a new id, carrying its version history and any PDF with it. Refuses an occupied or forgotten destination |
| readVersionSource / restoreSealedVersion | Read a past version as stored (plus its PDF), and put a sealed version back as the live file without cutting a new one, for hosts that keep their own records beside the vault |
| findDanglingReferences | Every [[wikilink]], contextnest:// link and depends_on that points at no document, with its document and line |
Errors all carry a code: InvalidSelectorError, CorruptHistoryError, VersionArtifactExistsError, and the rest are exported from the package root.
Operation Catalog
@promptowl/contextnest-engine/api is a second entry point holding the canonical operation set — the same names, input/output schemas, and error codes that the CLI, the MCP server, and REST surfaces bind to, so an agent config written against one works unchanged against another.
import {
NestStorage,
GraphQueryEngine,
VersionManager,
} from "@promptowl/contextnest-engine";
import { createEngineApi } from "@promptowl/contextnest-engine/api";
const storage = new NestStorage("./my-vault");
const ctx = {
storage,
query: new GraphQueryEngine(storage),
versions: new VersionManager(storage),
};
const api = createEngineApi();
const doc = await api.run("context_get", { id: "nodes/api-design" }, ctx);run(name, input, ctx) resolves the operation (canonical name or legacy alias), validates the input, runs every extension's authorize gate, executes, and notifies onResult. The context is identity-agnostic and per-vault — the same primitives every surface already builds.
Each operation exposes a Zod schema plus a draft-07 inputJsonSchema / outputJsonSchema for tool manifests. EngineExtension lets a consumer register extra operations and wrap every call with authorize / onResult without forking the engine — governance policy stays out of the AGPL core.
OperationContext.onProgress(done, total) is an optional sink for long-running operations such as context_import. It lives on the context rather than in an operation's input because inputs must stay JSON-serializable for the MCP/REST wire; in-process callers supply it, wire transports leave it undefined.
context_nests is the catalog's first registry-scoped operation — it reads ~/.contextnest/config.yaml rather than one vault, so it ignores its OperationContext.
Browsing Without Reading
Discovery's cost is parsing every markdown file it finds, so a caller that only needs the vault's shape should not pay it. Two operations narrow the crawl rather than the result:
context_folders(andNestStorage.listFolders) returns each folder's path and its document count, read from directory entries without opening a single document. Folders are read rather than inferred from the documents inside them, so a folder holding only subfolders still appears.context_listandNestStorage.discoverDocumentstakefolder— a path relative to the vault root, i.e. the id prefix ("nodes/gtm", not"gtm") — andrecursive. Withrecursive: falsea folder's subfolders are never opened, so a lazily-expanded document tree pays only for the level it shows.
const { folders } = await api.run("context_folders", { recursive: false }, ctx);
// → [{ path: "nodes", count: 0 }, …]
const { documents } = await api.run(
"context_list",
{ folder: "nodes/gtm", recursive: false },
ctx,
);Previously the only way to browse one folder was to read and parse every document in the vault and filter afterwards, which costs the same as not filtering — painful on a large vault, and worse on a network-backed mount where each document is a round trip.
Importing an Existing Folder
context_import bulk-publishes in one pass — one checkpoint and one index regeneration for the whole batch, with failures reported per-document rather than aborting the rest. It takes four kinds of input:
| Input | What it does |
|---|---|
| documents | New nodes synthesized from title + content |
| ids | Documents already written into the vault, published as-is |
| files | An existing vault's files written verbatim at their own relative paths |
| discover | The engine scans the vault itself and decides what to publish |
files synthesizes nothing, so the source's own frontmatter survives (version, checksum, custom keys a generated draft would drop) and non-document files travel too — .versions/<doc>/history.yaml included, which is what lets an imported version chain still reconstruct. Paths are guarded: .. and absolute paths are rejected per-file rather than aborting the import.
publish: false stages files without publishing them, so an upload arriving in several batches can stage every batch and make one final discover call — the import seals one checkpoint rather than one per batch.
// Stage each batch as it arrives…
await api.run("context_import", { files: batch, publish: false }, ctx);
// …then let the engine take responsibility for the whole vault, once.
const result = await api.run(
"context_import",
{ discover: true, author: "alice", exclude_ids: alreadyImported },
ctx,
);
// result.documents: { id, title, version, status, tags, content } per document —
// enough to record the import without re-reading the vault.
// result.checkpoint: the single checkpoint sealing the batch.Publishing is opt-in. Only a document whose frontmatter explicitly says published or approved is published. Everything else is held as a draft — including a document that states no status at all, because saying nothing is not consent. A vault of hand-authored notes carries no governance state, and importing it should not decide on the author's behalf that every note is fit to serve to an AI. Held is recoverable; published-by-default is not, since the exposure has already happened by the time anyone reviews it. A held document is stamped with an explicit status: draft so a file read outside the engine is never ambiguous — except where the author already wrote pending_review or rejected, which is theirs to keep.
exclude_ids skips what an earlier run already took, so a re-import is idempotent. parser.explicitStatus(node) exposes the same distinction the importer relies on: the status the author actually wrote, or null where parseDocument would have defaulted to draft.
Host Options
For a host that serves a vault (the hosted Community server). The CLI and MCP server set none of these, so local vaults behave as before.
new NestStorage(root, { agentConfigs: false }): index regeneration stops rewritingCLAUDE.md,GEMINI.md,.cursorrules,.windsurfrulesand.github/copilot-instructions.md. The default istrue. Either way, the files are now written in parallel.new NestStorage(root, { manifest }):manifest()returns every document'smanifestEntryfrom a record the host updates with each write (returningnullfalls back to the crawl). The index update and checkpoint then take the vault's documents from it, so there is no vault crawl, no folder re-read and no manifest file on disk. The host is responsible for keeping it fresh: a document it misses is not indexed or sealed untilctx indexcrawls the vault.OperationContext.deferSeal: true:context_create,context_update,context_publish,context_deleteandcontext_import_pdfstop at the document and its version entry, and a publish answerscheckpoint: null. The host then callssealWrites(storage, { changed, published, triggeredBy })to seal every published document in one checkpoint and update the index once, under the vault lock.CONTEXTNEST_MANIFEST=on(experimental, off by default): the engine keeps its own manifest in.context/manifest/, so a write reads only the changed folder instead of crawling the vault, andcontext_verifywarns (stale_index) about documents changed outside the engine. When it is off, no manifest files are written.
Upgrading to 2.0
Vault files are unchanged. The API breaks in five places:
context_overviewis removed.context_initreturns everything it did plus the vault'sCONTEXT.mdinstructions, configuration and path — one call to open a vault instead of two. The node list is opt-in behindinclude_nodes(withlimit).- Ids are taken exactly as stored.
resolveIdno longer re-roots a bare id undernodes/, which is what made every id from a flat-layout vault resolve to a document that does not exist. Callers migrating fromread_documentmust pass the full id. A trailing.mdand leading slashes are still stripped. context_importoutput:created→published,checkpointadded, andfailedentries carryid(for theidspath) ortitle(for thedocumentspath).context_update:titlesets a new title instead of selecting the node — select byid.tagsreplaces rather than merges. Anullmetadata value clears that key. Publishing defaults to false whenstatusnames a non-published lifecycle value, since those are metadata transitions rather than content releases.reconstructVersionrefuses a version the history does not contain, where it previously returned the nearest keyframe's content as though it were the version asked for.
Widening, not breaking: agent, artifact and table join NODE_TYPES, and TAG_PATTERN accepts : so namespaced tags (#dept:engineering) validate. The selector lexer does not tokenize : inside a tag yet, so namespaced tags are not addressable in a query.
Also fixed — vault aliases matching __proto__, constructor or prototype are rejected at every mutating registry entry point, and alias lookup uses an own-property check. Reported by CodeQL (js/prototype-polluting-assignment).
Part of Context Nest
The engine is the library layer. Most users reach it through one of these:
| Surface | What it is |
|---|---|
| @promptowl/contextnest-cli | The ctx command — ctx init, ctx query, ctx verify |
| @promptowl/contextnest-mcp-server | MCP server exposing 44 vault tools to Claude, Cursor, Gemini, and Copilot |
| Claude integration | Drop-in MCP config for Claude Code and Claude Desktop |
Links
License
AGPL-3.0. If your organization restricts AGPL, or you need to embed this in a proprietary product, a commercial license is available from PromptOwl.
