@blackridder22/sonomem

A built-in, provider-neutral hybrid memory engine for AI chat/agent apps. Built for Sonoa Search, designed to drop into any host that can push messages with stable IDs.

The model proposes; the database disposes. And memory must never be able to break the chat.

Spec: DESIGN.md · Agent skill: skills/sonomem/SKILL.md · Prior design reviews: chats/

What works

• Layer 2 — conversation recall: retrieval units per exchange, edit/regen lineage, rolling chat summaries (optional summarizer port), hybrid retrieval (pgvector HNSW + Postgres FTS simple, RRF fusion, recency decay, dense similarity floor).
• Layer 1 — learned memory: versioned assertions with categories, fact keys, sensitivity, quoted evidence spans, revisions, supersede-never- overwrite, hash dedup, hot profile projection with snapshots + rollback.
• Write path: ingestTurn persists units and enqueues extraction in the same transaction (outbox); deterministic secret pre-filter (EN/FR patterns + entropy); ONE structured extractor call; policy gate (user-authored quotes only, sensitivity opt-in, confidence threshold); resolver ops ADD/CONFIRM/SUPERSEDE/MERGE/CONTRADICT/REVIEW/IGNORE; shadow mode; per-chat debounce; batched embed backfill.
• Manual Layer 1: remember / forget / pin / correct + activity feed (listEvents) with undo for saves, forgets, supersessions.
• Deletion semantics: onChatDeleted (cascade + solely-evidenced memory forgetting), onMessageEdited (stale units/evidence), onUserDeleted, nightly consolidation (expiry, stale-evidence review) + orphan sweep via existsCheck.
• Files: ingestFile → chunks (searchable, source: 'file') + an event memory; contents never auto-become facts.
• Jobs: transactional outbox in Postgres — runPendingJobs() (serverless after()/cron) or startWorker() (long-lived), retries with backoff, dead letters in the activity feed.
• Guarantees: fail-open buildContext (timeout + circuit breaker, never throws), scope required on every call, doctor() health checks, migrate() idempotent migrations, versioned embeddings.
• AI SDK glue (/ai-sdk): createAiSdkEmbedder, createAiSdkExtractor (generateObject + repair retry), createSonomemTools (remember/forget/ recall), formatContextForPrompt (quoted-data injection template).

Verified by 51 tests including an end-to-end suite against real Postgres + pgvector (extraction, supersession + undo, shadow mode, secret redaction, two-user isolation, deletion cascades, fail-open).

Quick start

import { createSonoMem } from '@blackridder22/sonomem'
import { createAiSdkEmbedder, createAiSdkExtractor, createSonomemTools,
         formatContextForPrompt } from '@blackridder22/sonomem/ai-sdk'
import { fromUIMessages } from '@blackridder22/sonomem/converters'
import { openai } from '@ai-sdk/openai'

const sonomem = createSonoMem({
  db, // your Drizzle Postgres instance (pgvector enabled)
  embedder: createAiSdkEmbedder({ model: openai.textEmbeddingModel('text-embedding-3-small'), dimensions: 1536 }),
  extractor: createAiSdkExtractor({ model: openai('gpt-4.1-mini') }),
  config: { shadowMode: true },
})
await sonomem.migrate()
await sonomem.doctor()

// per turn:
const scope = { userId }
const bundle = await sonomem.buildContext({ scope, query })   // fail-open, ≤300ms
const memoryBlock = formatContextForPrompt(bundle)            // → system prompt
// ...generate with tools: createSonomemTools({ sonomem, scope, chatId })
await sonomem.ingestTurn({ scope, chatId, messages: fromUIMessages(uiMessages) })
await sonomem.runPendingJobs()                                // or startWorker()

See the skill for the full integration protocol (lifecycle calls, shadow-mode launch, debugging).

Development

pnpm install
pnpm test        # 51 tests; integration suite needs local Postgres
                 # (SONOMEM_TEST_DB=postgres://... to override, skips if absent)
pnpm typecheck
pnpm build       # tsup → dist/ (esm + d.ts, 5 entries)

Notes: typescript pinned to 5.x (rollup-plugin-dts can't drive the TS 7 native compiler yet). FTS uses the simple config so French/English/Kreyòl all work unstemmed; semantic weight comes from the (multilingual) embedder.