levi-cortex

Optional Cortex memory layer for Levi: ingest local agent chat history, search it, summarize it, and synthesize it into reusable context.

What This Package Is

@arborship/context-engine is the shared Cortex package used by Levi desktop and Cortex-facing tooling.

It is intentionally not runtime-critical for the core Levi control plane:

• levi-hostagent does not depend on it for VM lifecycle or template operations
• leviathan does not depend on it for job queueing or orchestration

If Cortex is offline, core Levi VM and orchestration flows can still run.

What It Does Today

The implemented surface is strongest around agent conversation ingestion and synthesis.

Built-in ingestion sources

• Claude Code local history
• Codex local session history
• Cursor agent chat history
• OpenCode session history

These are normalized into conversation sources with titles, tags, timestamps, and, where available, structured turns and trajectory steps.

Retrieval and memory

• hybrid full-text plus optional vector search
• conversation browsing and filtering
• source browsing
• continuation prompt generation for resuming prior work
• trajectory search for similar prior situations

Synthesis

• per-conversation summaries
• conversation bullets and reports
• daily rollups
• shard-level synthesis artifacts
• structured synthesis sections such as summary, constraints, spec hints, decisions, risks, and open questions

Interfaces

• MCP server for agents
• HTTP server for CLI and desktop flows
• library exports for ingestion, retrieval, synthesis, exchange, and plugins

Runtime Modes

levi-cortex can run in two modes:

• stdio MCP mode
• HTTP server mode

Behavior is chosen like this:

• --stdio forces stdio mode
• --http forces HTTP mode
• setting CORTEX_PORT also forces HTTP mode
• otherwise, non-TTY stdin defaults to stdio MCP mode

Quick Start

1. Start PostgreSQL

From the repo root, there is a helper compose file for the Cortex database:

docker compose -f docker-compose.context-engine.yml up -d

That exposes PostgreSQL on host port 5434, so a typical local DB URL is:

export CORTEX_DB_URL=postgresql://cortex:cortex@localhost:5434/cortex

2. Build the package

cd packages/protocols/context-engine
yarn build

3. Start Cortex over HTTP

CORTEX_DB_URL=postgresql://cortex:cortex@localhost:5434/cortex \
CORTEX_PORT=4119 \
node dist/index.js

The HTTP server initializes the schema on startup.

4. Or run as an MCP server over stdio

CORTEX_DB_URL=postgresql://cortex:cortex@localhost:5434/cortex \
node dist/index.js --stdio

Common Local Workflows

Ingest recent local conversations

From the Levi CLI:

levi cortex sync
levi cortex sync --plugins claude-code,codex
levi cortex sync --since-days 7

Built-in plugin names currently exposed by the Cortex server are:

• claude-code
• codex
• cursor-agent
• opencode

Ingest local files and directories

levi cortex ingest is the document lane: it walks a path, admits the text files it finds as kind: file sources, and reports exactly one typed outcome for every path it saw — accepted, updated, skipped, pending (preview only), excluded, non-text, unsupported (a symlink or other non-regular file, reported rather than dropped), or failed. A path that cannot be read — a missing argument, an unlistable directory, a file the engine rejects — is one failed line; it never aborts the batch.

levi cortex ingest ./docs --shard my-project              # previews, delivers nothing
levi cortex ingest ./docs --shard my-project --yes
levi cortex ingest ./docs README.md --shard my-project --tag design --yes
levi cortex ingest ./corpus --shard my-project --exclude 'vendor/**' --yes

Preview-by-default, exactly like levi cortex import. Identity is <prefix>:<relative-path>, never the absolute local path — so the same corpus ingested from a different checkout converges instead of duplicating. The prefix is the corpus root's basename, and only the root decides it: identity must not depend on which other paths shared a command line, or the same corpus would fork by invocation. Two roots in one command that share a basename are therefore refused, not renamed — --prefix names one corpus and cannot disambiguate two, so run them as separate commands.

Re-ingest is safe and named: unchanged files come back skipped without re-chunking or re-embedding, changed files updated, new files accepted. Content identity is the bytes as submitted — a whitespace-only edit is a real change and reports updated. Annotations are not part of that identity: adding --tag reviewed to an unchanged file still applies, and reports updated without touching a chunk or a vector.

.git/, node_modules/, .cortex/ and dotfiles are excluded by default.

Embeddings are reported honestly, and the advice matches the situation — the response names the regime in embedding:

• fts-only — no embedding provider is configured: @huggingface/transformers is not installed, or CORTEX_EMBED_LOCAL=off declares the local model disabled, and no gateway embedding binding exists. Documents are stored and remain findable by text search; the revision converges without vectors, because re-running could never produce any and refusing to converge would re-chunk the whole corpus on every run. The command says so, and does not tell you to retry.
• unconverged — a provider is configured but the revision came up short. The engine withholds that revision's content hash, so the next ingest re-embeds instead of reporting skipped forever. Re-running does repair it.

The distinction is drawn from configuration, never from a live probe or from a failed load. A configured gateway that is briefly unreachable, and a local model that has not warmed its cache yet, must not be mistaken for an engine that has neither — a transient outage classified as the steady state freezes the document permanently and, worse, tells the operator that retrying cannot help.

Because the local model is the default provider, an engine with it installed reports unconverged until embeddings actually work, and re-ingest keeps retrying rather than converging. That is deliberate — it costs CPU visibly instead of losing recall invisibly. An operator who genuinely wants text search only should declare it with CORTEX_EMBED_LOCAL=off, which turns the local provider off and returns the engine to the converging fts-only regime.

Search and read prior material

levi cortex search "authentication flow"
levi cortex search "egress design" --kind file --since 7d
levi cortex conversations --search "refactor"
levi cortex sources list --kind conversation
levi cortex sources show <source-id>
levi cortex conversations show <conversation-id> --tail 20

Search hits carry their owning source's title, kind, last-updated time and tags, so a result is readable without a second round trip. --kind and --since are honored by the engine, not trimmed locally; --since takes a duration (7d, 24h, 30m, 2w) or a date (2026-08-01, or any ISO-8601 instant).

One-shot local ingestion script

For quick local ingestion there is also:

node scripts/ops/context-engine-ingest-local.mjs

That script currently targets Claude Code local history and defaults to a recent lookback window.

HTTP Surface Used By Levi

The Levi CLI and desktop use the Cortex HTTP server for flows such as:

• POST /api/sync
• GET /api/sync/status
• GET /api/search — accepts kind and since (epoch ms); hits carry the owning source's title, kind, updatedAt and tags
• GET /api/conversations — conversation-kind sources only
• GET /api/conversations/:id — conversation-kind sources only (a file source id is a typed 404 here); accepts tail=<n>, which returns the newest N units and says whether they came from stored turns or from chunk order
• GET /api/sources
• GET /api/sources/:id — metadata plus content
• POST /api/sources/ingest — names its outcome: accepted | updated | skipped, and reports embeddedChunks plus an embedding regime (complete | unconverged | fts-only) so a partly-embedded revision is visible, and so a caller can tell whether re-running would help. A same-content call that changes only title/tags/url/project writes those annotations and returns updated, touching no chunk, no vector, and no stored turns
• GET /api/shards
• POST /api/shards/:id/rebuild
• GET /api/shards/:id/synthesis
• POST /api/shards/:id/synthesis/run

There are also endpoints for URL ingestion, MCP gateway management, summaries, and remote import/push flows.

Security And Auth Notes

• local mode effectively trusts the local operator
• distributed mode can require x-cortex-admin-key for REST access
• MCP sessions can verify Levi job tokens with LEVI_TOKEN_SIGNING_KEY

Important current boundary:

• Cortex is optional infrastructure
• it should not be positioned as the canonical source of truth for orchestration state

Dependencies

Required

• PostgreSQL for chunk, source, shard, and synthesis storage

Optional

• @huggingface/transformers for embedding-backed vector retrieval
• Exa and SearXNG integrations for research/search extensions

Full-text search works without embeddings. The system degrades gracefully when optional vector or external search dependencies are missing.

Tests

Run the package tests with:

npx vitest run

The test suite is designed to be self-contained; external services are mocked where practical.

Pointers

• src/index.ts for library exports and mode selection
• src/server.ts for HTTP and MCP server behavior
• src/plugins for built-in conversation ingestors
• src/lib/synthesis.ts for conversation and daily synthesis
• ../../../docs/strategy/CONTEXT_ENGINE_CHAT_INGESTION_AND_SYNTHESIS_VALUE_PROP.md for product framing