memory-token

MCP server plus Cursor hooks for governed workspace memory with token-budget packs: agents propose facts, you (or policy) confirm, then a bounded summary loads at session start—with preCompact nudges, post-tool reminders, and optional user-message triggers so long sessions still persist durable context.

The problem this solves

Coding agents (Cursor / Claude / Copilot) forget everything between sessions. Every new chat means:

• The agent re-reads the same files to understand your project (5–20k wasted tokens per session).
• You re-explain the same architectural decisions, conventions, build commands, and known bugs.
• Long-running sessions silently lose context to summarization and the agent starts repeating mistakes you already fixed.
• "Just put it in CLAUDE.md / AGENTS.md" doesn't scale: the file grows unbounded, eats context every turn, and has no provenance — you can't tell which lines are stale, who proposed them, or what actually got applied.

The shortcut of dumping everything into a markdown rules file works until it doesn't; you end up paying token rent on context the agent doesn't even need this turn.

How memory-token solves it

A two-tier memory with a strict token budget and a human-in-the-loop gate:

1. Confirmed store (store.json) — small, curated, typed facts (decisions, fixes, APIs, build commands, conventions). Loaded at session start as a bounded markdown pack (default ≤1500 tokens). The agent proposes, you (or policy) confirm — nothing gets in by accident.
2. RAG layer (rag.sqlite) — large chunks (chat distills, session capsules, design docs) stored compressed with embeddings. Pulled on demand via rag_query, ranked by hybrid score (embeddings + BM25-style lexical), returned already compressed within a token budget. Verbatim text only when explicitly fetched via rag_get_full.

Both layers respect a token budget on every read — you never pay for context the agent isn't using right now.

Benefits (measured on this repo)

| Benefit | Mechanism | Example impact | |---|---|---| | Cross-session memory | sessionStart hook injects confirmed pack automatically | Critical facts (build cmd, security fix, race condition root cause) survive across all future Cursor windows. | | Big token savings on context loads | RAG returns compressed, ranked snippets instead of file reads | A query that would cost ~3000–5000 tokens in Read calls cost 364 tokens of compressed RAG hits in this session (~85% reduction). | | Bounded session-start cost | Pack hard-capped at max_tokens, deterministic ordering | This repo's pack: 397 tok / 1500 budget = 26%, fits in any sessionStart without crowding out user prompt. | | Compression for free | compress_candidate shrinks bodies to ~1.6× ratio | 37.8% size reduction on stored bodies; same retrieval quality. | | No silent drift | propose → confirm gate, dedupe by hash + similarity, audit log | Every fact has a status, timestamp, and reason; you can prune with confidence. | | Causal graph | Typed links (SOLVES, CAUSES, BUILDS_ON, …) + traverse BFS | Ask "what fixed X?" and get back the commit that solved it, the build that depends on it, in one hop. | | Local-first, private | Embeddings via Transformers.js (ONNX, ~25 MB model, downloaded once) | No API keys required. Optional: Ollama or OpenAI for embeddings. Your facts never leave the repo. | | Workspace-scoped | Each project gets its own .memory-token/ directory | No bleeding of facts between unrelated repos; gitignored by default but can be versioned. | | Agent steering, not vibes | Hooks inject skill + policy + pack; skill is a decision tree the agent follows | Agents actually call propose/confirm/rag_query consistently instead of "maybe sometimes if they remember". | | Prevents the CLAUDE.md bloat trap | Token-budget pack + RAG fallback | Old facts move to RAG (compressed, on-demand) instead of permanently inflating the rules file. |

When to use it

• ✅ Multi-week projects where you reopen Cursor often and don't want to re-explain context.
• ✅ Codebases big enough that re-reading "to understand structure" costs noticeable tokens per session.
• ✅ Teams that want provenance and control over what the agent "remembers".
• ✅ Long debugging sessions where you want the root cause + fix to survive into next week's session.

Skip if your project is a one-off script or you don't run more than 1–2 chat sessions on the same codebase.

What it does

| Layer | Role | | --- | --- | | Store (store.json) | Typed memories (decision, fix, api, …), statuses (proposed → confirmed / rejected), links between memories, audit log. | | RAG (rag.sqlite) | Chunked text with hybrid retrieval (embeddings + lexical). Compressed snippets in query results; verbatim text only via memory_token_rag_get_full. | | MCP | Eighteen tools: pack, propose/confirm/reject, search, prune, link graph, export/import, RAG ingest/query/delete, stats, audit. | | Hooks | Inject skill path + policy + pack on sessionStart; hints on beforeSubmitPrompt / preCompact / postToolUse. Hooks do not call the MCP (shell → Node CLIs only). Flow: hook → skill → MCP. |

Workspace root comes from MEMORY_TOKEN_WORKSPACE, CURSOR_PROJECT_DIR, or CLAUDE_PROJECT_DIR, else process.cwd() (see src/workspace.ts).

Requirements

• Node.js ≥ 22.5.0 (uses node:sqlite experimental API)

Install

Option A — npx (zero-install, recommended)

No clone, no global install. Cursor pulls the package on demand:

// ~/.cursor/mcp.json
{
  "mcpServers": {
    "memory-token": {
      "command": "npx",
      "args": ["-y", "@barbozaa/memory-token"],
      "env": { "MEMORY_TOKEN_WORKSPACE": "${workspaceFolder}" }
    }
  }
}

Restart Cursor. The first call downloads the package (~52 KB tarball). The first semantic embedding call additionally downloads the local ONNX model (~25 MB) into <workspace>/.memory-token/transformers-cache/.

To use the hooks or the skill with this install method, copy them once from the installed package into your ~/.cursor/:

PKG=$(npm root -g 2>/dev/null)/@barbozaa/memory-token   # or use `npm pack` to extract locally
cp -r "$PKG/.cursor/hooks"  ~/.cursor/
cp    "$PKG/.cursor/hooks.json" ~/.cursor/
cp -r "$PKG/.cursor/skills" ~/.cursor/
chmod +x ~/.cursor/hooks/*.sh

Option B — Global install

npm install -g @barbozaa/memory-token

{
  "mcpServers": {
    "memory-token": {
      "command": "memory-token-mcp",
      "env": { "MEMORY_TOKEN_WORKSPACE": "${workspaceFolder}" }
    }
  }
}

Option C — From source (development)

git clone https://github.com/barbozaa/memory-token.git
cd memory-token
npm install
npm run build
chmod +x .cursor/hooks/*.sh
npm run smoke

Point Cursor at dist/mcp/index.js with MEMORY_TOKEN_WORKSPACE=${workspaceFolder} (see Cursor MCP). Merge .cursor/hooks.json into your project if you use hooks elsewhere.

Global Cursor install (all new windows / workspaces)

Use user-level config so you do not copy hooks into every repo:

1. ~/.cursor/mcp.json — add the memory-token server with MEMORY_TOKEN_WORKSPACE: ${workspaceFolder} (each project still gets its own .memory-token/).
2. ~/.cursor/hooks.json + ~/.cursor/hooks/memory-token/*.sh — wrappers that set MEMORY_TOKEN_CLI_ROOT to your clone and call dist/hook-*.js. See ~/.cursor/hooks/memory-token/README.md after install.
3. ~/.cursor/skills/memory-token/SKILL.md — global skill; session hook sets MEMORY_TOKEN_SKILL_PATH so the banner points here.
4. User Rules — Cursor reads them from Settings → Rules, not from ~/.cursor/rules/. Use ~/.cursor/rules/memory-token.mdc only as a reference to paste into User Rules, or add the same rule under each repo’s .cursor/rules/.

Restart Cursor after changing MCP or hooks. Re-run npm run build in the memory-token clone when you pull updates.

Data locations

All under <workspace>/.memory-token/ (typically gitignored):

| Path | Contents | | --- | --- | | store.json | Memories, links[], audit[] | | rag.sqlite | RAG chunks + vectors | | transformers-cache/ | Downloaded ONNX model (first semantic embed run) |

Remove .memory-token/ from .gitignore if you want the store versioned.

Repository layout

src/
  mcp/index.ts          # MCP server + tool handlers
  store.ts              # JSON store + lockfile (O_EXCL) for concurrent writes
  pack.ts               # Token-budget pack builder (used by MCP + session hook)
  types.ts              # Memory types, links, audit shapes
  session-policy.ts     # Injected policy text for hooks
  compress-lite.ts      # Lightweight compression helpers
  memory-dedupe.ts      # Near-duplicate detection on propose
  workspace.ts          # Resolve workspace root from env
  hook-session-start.ts # sessionStart payload
  hook-post-tool-nudge.ts
  hook-user-message.ts
  hook-git-commit.ts    # Optional git post-commit integration
  mcp-nudge-messages.ts
  rag/                  # db, query, embed, embed-local, lexical, paths, compress-default
scripts/
  smoke-mcp.mjs         # MCP smoke (listTools, pack, propose, RAG, …)
  metrics.mjs         # Store/pack/link/RAG health report
  install-git-hook.sh # Install post-commit hook in another repo
.cursor/
  hooks.json            # Cursor hook wiring
  hooks/*.sh            # Shell wrappers → dist/*.js
  skills/memory-token/SKILL.md   # Agent workflow (read at session start)
  rules/memory-token.mdc         # Optional Cursor rules

Cursor MCP

For npx or global installs see Install. The block below is for the from-source workflow:

"memory-token": {
  "command": "node",
  "args": [
    "--disable-warning=ExperimentalWarning",
    "/ABS/PATH/TO/memory-token/dist/mcp/index.js"
  ],
  "env": {
    "MEMORY_TOKEN_WORKSPACE": "${workspaceFolder}"
  }
}

Replace /ABS/PATH/TO/memory-token with this repo’s path. Restart Cursor after edits.

CLI / other clients: npm run mcp runs the server with cwd as workspace unless env overrides. memory-token-mcp is the package bin entry (same script).

Cursor hooks

Shipped in .cursor/hooks.json:

| Event | Script | Purpose | | --- | --- | --- | | sessionStart | .cursor/hooks/session-memory.sh | Skill banner (MEMORY_TOKEN_SKILL_PATH or .cursor/skills/memory-token/SKILL.md), policy cheat sheet, token-capped confirmed pack. | | preCompact | .cursor/hooks/precompact-memory-hint.sh | Reminder to rag_ingest / propose / confirm before context compaction. | | postToolUse | .cursor/hooks/post-tool-mcp-nudge.sh | Every MEMORY_TOKEN_NUDGE_EVERY tool calls (default 10), short [memory-token] nudge (skips tools already named like memory_token_*). | | beforeSubmitPrompt | .cursor/hooks/user-message-triggers.sh (matcher: UserPromptSubmit) | Same as legacy “user send” nudge: phrases like “remember this”, “root cause”, long paste → additional_context for propose / rag_ingest. |

Hook env (optional)

| Variable | Default | Meaning | | --- | --- | --- | | MEMORY_TOKEN_SESSION_MAX_TOKENS | 2400 | Total rough budget for policy + pack in session hook (char/4 estimate). | | MEMORY_TOKEN_NUDGE_EVERY | 10 | Post-tool nudge interval. | | MEMORY_TOKEN_CLI_ROOT | auto | Absolute path to this repo when hooks live in another project (so dist/hook-*.js resolve). | | MEMORY_TOKEN_SKILL_PATH | — | Custom SKILL.md path. |

MCP tools (full list)

| Tool | Purpose | | --- | --- | | memory_token_get_context_pack | Confirmed memories within max_tokens; optional query / tags rerank. | | memory_token_propose | Create proposed memory (type, importance, optional body_compressed, force to bypass dedupe). | | memory_token_confirm / memory_token_reject | Promote or drop proposals. | | memory_token_compress_candidate | Deterministic squeeze for body_compressed. | | memory_token_search | Substring search over memories. | | memory_token_list_audit | Recent audit entries. | | memory_token_prune | Remove old/matching proposed (and optionally confirmed) rows; dry_run preview. | | memory_token_link / memory_token_unlink | Directed edges between memories (relation_type, optional bidirectional). | | memory_token_traverse | BFS from a memory id over the link graph. | | memory_token_stats | Workspace counts / health-style summary. | | memory_token_export / memory_token_import | JSON backup / import (include_rag on export). | | memory_token_rag_ingest | Store chunk (text_raw, optional text_compressed, source); dedupe by hash of raw. | | memory_token_rag_query | Ranked compressed chunks within token budget (hybrid score). | | memory_token_rag_get_full | Verbatim text_raw for an id (use before quoting or patching from RAG hits). | | memory_token_rag_delete | Remove a chunk. |

Recommended link relation strings include SOLVES, CAUSES, BUILDS_ON, CONTRADICTS, SUPERSEDES, BLOCKS, REQUIRES, ALTERNATIVE_TO, RELATED_TO (see src/types.ts).

RAG embeddings (local first)

Default: local embeddings via @xenova/transformers + SQLite vectors (model Xenova/all-MiniLM-L6-v2). First run downloads into .memory-token/transformers-cache/ (needs network once).

| Env | Effect | | --- | --- | | MEMORY_TOKEN_LOCAL_EMBED_MODEL | Override Hugging Face model id (ONNX / Transformers.js). | | MEMORY_TOKEN_LOCAL_EMBED_THREADS | ONNX WASM threads (default 2). | | MEMORY_TOKEN_NO_LOCAL_EMBED=1 | Skip local embedder → lexical unless Ollama/OpenAI configured. | | MEMORY_TOKEN_OLLAMA_EMBED_MODEL | Ollama embeddings. | | MEMORY_TOKEN_OLLAMA_URL | Ollama base URL (default http://127.0.0.1:11434). | | OPENAI_API_KEY / MEMORY_TOKEN_OPENAI_API_KEY | Optional cloud embeddings. |

This is not ChromaDB; the DB format is project-specific.

npm scripts

| Script | Command | Purpose | | --- | --- | --- | | build | tsc | Compile src/ → dist/. | | mcp | node … dist/mcp/index.js | Run MCP stdio server. | | smoke | node … scripts/smoke-mcp.mjs | End-to-end MCP smoke against repo root. | | metrics | node … scripts/metrics.mjs | Human or --json report: store, pack usage, links, RAG, audit. Optional --root /path/to/workspace. |

Optional git post-commit hook

Install into another git repo:

./scripts/install-git-hook.sh /path/to/target/repo

Uses MEMORY_TOKEN_CLI_ROOT if the memory-token repo is not next to the script. On each commit, runs dist/hook-git-commit.js with MEMORY_TOKEN_WORKSPACE set to the target repo root (see scripts/install-git-hook.sh).

VS Code and Kiro

Stock VS Code has no Cursor-style sessionStart / postToolUse hooks. Options: run node dist/hook-session-start.js with MEMORY_TOKEN_WORKSPACE set, use a task, or an extension that exposes similar lifecycle events.

Kiro IDE: full install (MCP, steering, hooks, spec-task ideas, Cursor parity table) is in KIRO_SETUP.md. Example hook YAML lives under .kiro/hooks/.

Agent workflow

1. Read .cursor/skills/memory-token/SKILL.md (injected path on session start).
2. Call memory_token_get_context_pack early; use memory_token_rag_query before reading many files or huge pasted context.
3. Treat RAG snippets as non-verbatim until memory_token_rag_get_full.
4. Persist stable facts with propose → confirm; link related memories when useful; prune junk or stale proposals.

License

MIT © 2026 barbozaa