mind-palace-graph

v0.3.5

Published

7 days ago

Node-centric context retrieval CLI for LLM harnesses. Token-budgeted search with effort presets, mind palace (stash/drop/compose/relationships), pagination, MCP server, multi-tool schemas for Claude & Gemini.

Downloads

1,026

0High
0Medium
0Low

jadeoni-az

llm context retrieval ripgrep cli ai harness rag node-search

mpg — node-centric context retrieval for LLM harnesses

mpg is a CLI for retrieving token-budgeted context nodes from files, command output, URLs, and stdin, designed to be consumed directly by LLM harnesses.

The differentiator: a search returns nodes (a match + sized pre/post context), not files or lines. Each node is sized in tokens, and you can cap the number of nodes and the total token budget independently. Depth of context is set by effort, not by blindly loading more text.

Plus a persistent mind palace — named, addressable stashes of search results that compose, intersect, prune, and form a graph. mpg is how agents browse and trim long-term memory, not just how they grep.

New in recent versions: warm-process server (mpg --serve stdio / --serve-http) eliminates the ~1.1 s Node cold-start for harnesses making many calls. --format agent-json adds a structured envelope (status, n_literal_matches, warning, errors) so agent loops can detect bad patterns without paying an LLM round-trip. mpg tool-spec --format <openai|anthropic|gemini> prints provider-shaped tool descriptors at install time — no hand-written JSON schemas.

Headline numbers vs alternatives

Pulled from the in-repo benchmark suite (bench/ + BENCHMARKS.md).

| workload | mpg config | result vs alternatives | | :--- | :--- | :--- | | Literal recall on a markdown/JSON memory corpus | --effort scan --clip 30 | 100% / 100% / 377 tokens vs ripgrep's 1,197 — 3.2× cheaper than rg | | Typo-tolerant search (drop / insert / swap / sub) | --fuzzy | 100% recall vs ripgrep's 0%, ~12× cheaper than per-file embedding retrieval | | Topic compaction at fixed token budget | --effort scan --clip 30 --max-tokens N | matches LLM summarization on downstream-Q&A pass-rate at the same 2k budget — at zero LLM input tokens vs ~21k for the LLM path (n=3 topics, claude-haiku-4-5; run-to-run variance high) | | Macro agent task lift | mpg_search + mpg_stash | +20% pass-rate / -27% wall-clock on a 5-task code+specs corpus (claude-haiku-4-5). 4% fewer turns, 7% less output reasoning at convergence. | | Long-horizon memory (palace-as-framework) | per-session --mp-stash + agent-driven palace_search / palace_compose tools | 0.44 vs 0.32 on LongMemEval oracle, N=500, GPT-4o judge — cheap-model+palace beats frontier-model+flat-log. Lift concentrates in temporal-reasoning (+19pp), knowledge-update (+37pp), multi-session (+20pp). See BENCHMARKS.md. | | Mind palace set semantics | --mp-compose / --mp-intersect / --mp-except | 17/17 micro assertions pass |

BENCHMARKS.md has the full scorecard and the cases where mpg loses (single-keyword greps, CLI cold start), reported honestly.

Install

Requires Node 20+ and ripgrep.

npm install -g mind-palace-graph
# or from source:
git clone https://github.com/JadeZaher/mind-palace-graph.git
cd mind-palace-graph && npm install && npm run build && npm link

For Claude / Gemini / coding agents: load skills/mpg-context/SKILL.md into your system prompt or tool descriptions — it has the decision tree for effort levels, mind palace patterns, pagination, and error recovery.

Command reference

The shape of every command is:

mpg [<pattern>] [options]

<pattern> is a ripgrep regex (or a literal with -F). It is required for searches and stash-producing operations, omitted for pure palace operations (--mp-list, --mp-get, --mp-drop, --mp-link, --mp-related, --mp-graph, --mp-prune-*, --ls).

Sources — where to search

| Flag | Example | What it does | | :--- | :--- | :--- | | -i, --in <path>... | mpg "TODO" --in src/ test/ | One or more files, dirs, or globs. Greedy: consumes non-flag args. Dirs recurse. | | --in @<file> | mpg "TODO" --in @files.txt | Read path list from a file (one per line, # comments). | | --in @- | ls *.ts \| mpg "TODO" --in @- | Read path list from stdin. | | --in a,b,c | mpg "TODO" --in src/,test/ | Comma-separated path list. | | trailing paths | mpg "TODO" src/ test/ | rg-style positionals; equivalent to --in. | | --in '**/*.ts' | mpg "TODO" --in '**/*.ts' | Glob (single or multi-wildcard). | | --cmd <cmd> | mpg "error" --cmd "git log --oneline -100" | Search the stdout of a shell command. | | --stdin | cat README.md \| mpg "install" | Search piped stdin (auto-detected when piped). | | -u, --url <url> | mpg "deprecated" -u https://example.com/docs | Fetch URL body and search it. |

To pass a path that starts with -, prefix it with ./ (e.g. ./-weird-name) or use @file syntax.

Node sizing — context width and density

Output

Token estimation uses a fast chars/4 heuristic — accurate enough for budgeting, not a substitute for a real tokenizer when billing matters. The tokens field is always approximate and prefixed with ~ in llm format.

The agent-json format wraps the standard result in a structured envelope designed for agent loops:

{
  "status": "ok",
  "pattern": "TODO",
  "n_literal_matches": 12,
  "n_fuzzy_matches": 0,
  "fallback_used": false,
  "warning": null,
  "nodes": [ ... ],
  "next_suggestion": null,
  "errors": []
}

status is "ok" / "no_matches" / "partial" / "error". warning and next_suggestion carry actionable hints when the pattern quality is poor. Use --no-fill alongside agent-json to disable fill-padding and force the agent to iterate on a bad pattern rather than silently accepting padded context.

Search options (forwarded to ripgrep)

Pagination

Mind palace — instantiable short-term memory

A palace is a JSON file (default ./.mpg/mind-palace.json) of named stashes: search results addressable by name. Future searches can use stashes as inputs (--mp-from), compose them across sets (--mp-compose / --mp-intersect / --mp-except), tag them, and link them into a graph. Stashes default to merge on duplicate name (dedup by file:line); pass --mp-replace to overwrite. Override the palace file at runtime with --mp-path or MPG_MIND_PALACE=<file> — useful for one-palace-per-task isolation.

Pruning — keep the palace from growing unbounded

Expired-TTL stashes also auto-prune silently on every --mp-list / --mp-get.

Stashes record source_mtime_ms and match_line_hash at write time. Retrieved nodes carry a stale field (false, "unknown", "mtime_advanced_content_intact", "mtime_advanced", "content_drifted", "file_missing"). Re-fetch on content_drifted or file_missing — drop the stash and re-run the original search. See skills/mpg-context/SKILL.md for recovery patterns.

Relationships — make the graph in mind-palace-graph real

Warm-process server — eliminate cold-start

Every CLI invocation pays ~1.1 s of Node startup. For a harness making many mpg calls per task this compounds fast. Run mpg once as a long-lived server instead:

# Stdio NDJSON (best when the agent spawns mpg as its own child)
mpg --serve

# HTTP JSON (best for a shared local daemon)
mpg --serve --serve-http --port 17317

Both transports expose the same methods: search, palace.* (list, get, stash, drop, compose, intersect, except, link, graph, prune_expired, prune_tag, prune_older_than, prune_keep), tool_spec, health. Full wiring instructions and the NDJSON spawn snippet are in INSTALL.md under "Warm-process mode".

Tool-spec bootstrap — generate provider schemas

mpg tool-spec --format anthropic > tools/mpg-anthropic.json
mpg tool-spec --format openai   > tools/mpg-openai.json
mpg tool-spec --format gemini   > tools/mpg-gemini.json

Generates provider-shaped tool descriptors at install time. Re-run after upgrading. See INSTALL.md for per-provider wiring into a function-calling harness.

Discovery & meta

| Flag | Example | What it does | | :--- | :--- | :--- | | --ls / --tree | mpg --ls --in src/ | List/tree all searchable files and exit. | | -h, --help | mpg --help | Show inline help. | | -v, --version | mpg --version | Print version. | | --print-entry | mpg --print-entry | Print the resolved JS entry path (dist/index.js) and exit. For Node subprocess callers — see Calling mpg from another process. | | --pattern-file <path> | mpg --pattern-file /tmp/p --in src/ | Read the regex from a file (trailing newline stripped). Mutually exclusive with the positional pattern. Keeps exotic regexes off argv. | | --serve | mpg --serve | Start a warm-process stdio NDJSON server. Eliminates ~1.1 s cold-start for harnesses making many calls. | | --serve-http --port <n> | mpg --serve --serve-http --port 17317 | HTTP JSON mode. POST / with { method, params }. GET /health returns status. |

Environment variables

Exit codes

| Code | Meaning | | ---: | :--- | | 0 | Matches found (or palace operation succeeded) | | 1 | No matches (matches ripgrep's convention) | | 2 | Bad arguments | | 3 | ripgrep not installed | | 4 | Mind palace error (unknown stash, etc.) | | 99 | Unexpected error |

Recipes

# Cheapest first-touch index — "browse my recent memory"
# 3.2x cheaper than rg at 100/100 recall/precision on memory-system content.
mpg "JWT|Bearer|ProviderContext" --in . \
  --effort scan --clip 30 --sort recent --page 1 --page-size 10

# Typo-tolerant search (catches drop/insert/substitute/swap, edit dist <= 2)
mpg "PrvderiContext" --in . --fuzzy --effort scan --clip 30

# Topic compaction at a hard token budget (zero LLM cost)
mpg "auth|JWT|Bearer|ProviderContext" --in conductor/tracks \
  --effort scan --clip 30 --sort recent --window-curve log \
  --max-tokens 2000 --format llm > auth-compaction.md

# Quick recon
mpg "auth" --in . --effort quick --max-nodes 5

# Deep grounding for a final answer
mpg "session" --in src/auth/ --effort deep --max-tokens 16000

# Search the output of a command
mpg "error" --cmd "git log --oneline -100"

# Pipe content in
cat README.md | mpg "install"

# Stash + tag + TTL
mpg "TODO" --in src/auth/ --mp-stash auth-todos "Auth TODOs" \
  --mp-stash-tag auth --mp-stash-tag p0 --mp-ttl 7d

# Compose two stashes, re-search across their union
mpg "error" --mp-compose auth-todos perf-hotspots

# Re-search scoped to one stash's files
mpg "rate.limit" --mp-from auth-todos

# Link stashes into a graph, then traverse it
mpg --mp-link auth-todos perf-hotspots depends-on "shared db layer"
mpg --mp-graph auth-todos 3

# Prune safely
mpg --mp-prune-older-than 7d --mp-prune-dry-run     # preview
mpg --mp-prune-older-than 7d                        # commit

# One palace per task (isolation)
MPG_MIND_PALACE=./.mpg/task-42.json mpg "TODO" --in src/ --mp-stash t42 "..."

# Programmatic JSON for a harness
mpg "TODO" --in src/ --format json --page 1 --page-size 5

# Agent-loop envelope — detect bad patterns before paying an LLM round-trip
mpg "TODO" --in src/ --format agent-json --no-fill

# Warm-process server (stdio) — spawn once, write NDJSON requests over stdin
mpg --serve

# Generate provider tool schemas at install time
mpg tool-spec --format anthropic > tools/mpg-anthropic.json

Output format: `llm`

The default. Designed to be both human-readable and directly consumed by an LLM harness:

<mpg result pattern="TODO" nodes=4 tokens=~566 effort=normal strategy=fill>

--- NODE 1 of 4 | src/auth/login.ts:8 | ~196 tokens ---
  5    // Authentication flow for the public API.
  6    // Validates the user credentials, then issues a short-lived session token.
  7    export async function login(user: User, password: string) {
  8 >>   // **TODO**: add rate limiting per IP+user to prevent brute force
  9      const valid = await db.users.verifyPassword(user.id, password);
 10      if (!valid) {
 11        logger.warn(`failed login for ${user.id}`);
 ...

--- TOTAL ---
4 nodes | ~566 tokens | 3 sources | 30ms
</mpg result>

The result block carries the pattern, total token cost, source attribution per snippet, and match highlighting (>> + **bold**). The JSON format includes a pagination block with total_items, total_pages, has_next, has_prev.

Programmatic API

For TS/Node harnesses that prefer to embed mpg rather than shell out:

import { search, stash, listStashes, toolDefinition } from "mind-palace-graph";

const result = await search({
  pattern: "TODO",
  in: ["src/"],
  effort: "quick",
  page: 1,
  pageSize: 5,
});

await stash(result, {
  name: "auth-issues",
  note: "Auth TODOs to review",
  tags: ["auth", "p0"],
});

// Expose to OpenAI / Anthropic function calling:
openai.tools.create({ name: "mpg", ...toolDefinition });

The API mirrors the CLI 1:1. Pre-built tool schemas are exported as claudeTools and geminiTools.

Calling mpg from another process

If you spawn mpg from Node child_process.spawn rather than from a shell, Windows needs care. The npm-installed mpg on Windows is a .cmd shim, and the obvious approaches both fail:

| Approach | What happens on Windows | | :--- | :--- | | spawn("mpg", args) | EINVAL — Node won't directly execute a .cmd. | | spawn("mpg", args, { shell: true }) | cmd.exe parses the line and splits on &/\|/^/etc.; flags get corrupted before mpg sees them. Typical symptom: '--stdin' is not recognized as an internal or external command followed by mpg: Unknown argument: --git. |

Spawn node directly on mpg's resolved JS entry instead:

import { spawn } from "node:child_process";
import { entryPath } from "mind-palace-graph/entry";

spawn(process.execPath, [
  entryPath, "TODO", "--in", "src/", "--json",
], { stdio: ["ignore", "pipe", "pipe"] });

entryPath is exported by the side-effect-free mind-palace-graph/entry subpath (importing does NOT execute the CLI). For non-Node callers, mpg --print-entry prints the same path on stdout.

For regexes with shell metacharacters or untrusted input, write the pattern to a temp file and pass --pattern-file <path>:

import { writeFileSync, mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

const dir = mkdtempSync(join(tmpdir(), "mpg-"));
const patternFile = join(dir, "p");
writeFileSync(patternFile, exoticRegex);

spawn(process.execPath, [
  entryPath, "--pattern-file", patternFile, "--in", "src/", "--json",
]);

Quick picker:

| You are building... | Use | | :--- | :--- | | MCP host (Claude Desktop, Cline, Windsurf) | The MCP server. No subprocess concerns. | | Node agent that imports the SDK directly | import { search } from "mind-palace-graph". | | Custom Node subprocess wrapper (Pi extension, agent runner) | spawn(process.execPath, [entryPath, ...args]). | | Shell-only agent (bash one-liners) | Plain mpg .... Shells handle the shim correctly; the footgun only hits when spawning from another program. |

Architecture

src/
  cli.ts           hand-rolled arg parser + effort preset resolution
  types.ts         shared types (Node, Source, Result, etc.)
  tokens.ts        token estimation + line trimming to budget
  rg.ts            ripgrep wrapper (rg --json)
  sources.ts       source resolution: file / glob / command / stdin / url
  nodes.ts         match → context node construction
  format.ts        llm / markdown / json / text output
  mind-palace.ts   stash / drop / list / compose / except / intersect
  palace-format.ts llm-friendly formatters for palace output
  pagination.ts    page-through-the-results utility
  api.ts           programmatic API (search, stash, toolDefinition)
  entry.ts         side-effect-free JS entry path export
  index.ts         orchestrator (CLI entry point)

mpg shells out to rg --json for the actual search — fastest regex engine, structured match data. Everything else (node building, context sizing, output formatting, mind palace) is in-process TypeScript.

Development

npm run dev     # run with tsx (no build step)
npm run build   # compile to dist/
npm test        # run smoke tests

License

MIT.