@demystify/memory — agent memory you can actually erase

Store, recall and erase durable context for agentic products, in your own Postgres. Zero runtime dependencies. Zero-config in-memory default; a Postgres migration ships alongside for production.

npm install @demystify/memory

Why this exists

A memory store holding financial conversation history without an erasure path is a legal exposure, not a missing feature. India's DPDP Act 2023 gives a Data Principal the right to erasure; the GDPR calls it Article 17. Most agent-memory libraries have remember and recall and nothing else.

The prior art this package was extracted from — @dmstfy/jarvis-memory, private and unpublished — had four memory layers, pgvector search, Mem0-style reconciliation, and no way to delete a person's data at all. This package is that contract, rebuilt around the delete.

The thing everyone gets wrong

Erasing a memory row does not erase the personal data in it. By the time you delete it, the data has already propagated into everything derived from it:

memory: "Ramesh, 9876543210, disputed invoice INV-42 on Tuesday" → weekly summary: "Ramesh at 9876543210 disputed INV-42" → monthly digest: "one dispute, from Ramesh" → an embedding of each of the above

Delete the memory and all three derivatives survive, still naming him. So:

• every derived row must record its provenance — which rows produced it. The SQL migration refuses one that doesn't, with a deferred constraint trigger;
• forget() cascades transitively to a fixpoint. A summary of a summary of an erased memory is erased. The cascade is an AFTER DELETE trigger in the database, not a query in the adapter, so a host's own retention job, a psql session and a future adapter all get it. A rule enforced only by the code path in front of it is not enforced at all;
• the headline test asserts exactly this: write a memory, derive a summary from it, forget the subject, assert the summary is gone (test/erasure.test.ts), on both adapters.

import { Memory, InMemoryStore } from "@demystify/memory";

const memory = new Memory({ store: new InMemoryStore() });

const { memory: row } = await memory.remember({
  tenantKey: "org_abc",              // opaque; never parsed
  subjectKey: "customer:ramesh",     // the Data Principal. What forget() erases by.
  text: "Ramesh at 9876543210 disputed invoice INV-42",
});
// row.text === "Ramesh at [PHONE:****3210] disputed invoice INV-42"

const { derived } = await memory.derive({
  tenantKey: "org_abc",
  kind: "summary",
  text: "one dispute this week",
  sources: [{ sourceKind: "memory", sourceId: row.memoryId }],   // NOT optional
});

const receipt = await memory.forget({
  tenantKey: "org_abc",
  subjectKey: "customer:ramesh",
});
// { counts: { memories: 1, derived: 1, provenanceEdges: 1 }, cascadeDepth: 1, … }

await memory.getDerived("org_abc", derived.derivedId);   // → null

Over-erasure is deliberate

A weekly digest built from five customers is removed when any one of them is erased. The alternative leaves the erased person's data inside it. Re-derive the digest from what remains — that is the cheap half of the trade.

Erasure vs audit: the tombstone

The tension is real. "Delete everything about this person" and "prove you deleted it" pull in opposite directions. The resolution:

Erasure removes CONTENT. A content-free tombstone records that an erasure happened.

The tombstone holds an id, the tenant, a hash of the subject key, a reason code, per-table counts and a timestamp. Nothing else — and that is enforced by the schema, not promised by this README:

| guarantee | how | |---|---| | no raw identifier can be stored | subject_hash is CHECK-constrained to ^[0-9a-f]{64}$. A phone number cannot be written there even by a host doing its own inserts. | | no free-text field to leak into | reason is a closed code list (subject_request / retention_policy / operator). Free text is where somebody eventually writes "erased Ramesh, 9876543210, on request". | | the record cannot be quietly edited | UPDATE/DELETE on the tombstone table are refused by a trigger. Evidence the erasing party can edit is not evidence. | | there is no content column at all | verify_install() asserts it. |

forget() returns the receipt so a compliance flow can show a Data Principal what was removed, per table, rather than asserting that something was. A tombstone is written even when nothing matched — "we looked and there was nothing" is an answer they are owed, and an erasure request that leaves no trace cannot be shown to have been honoured.

Set a pepper. subjectHashPepper is mixed into the hash and should live in your secret manager, not in the database beside the tombstones. Subject keys are often phone numbers, emails or small integers — an enumerable space, so an unpeppered hash is a pseudonym rather than an erasure. The package will not pretend a default is a secret.

What the tombstone cannot do

It does not reach your backups, your WAL archive, your read replicas' retention, or a logical replication sink. Nothing in a library can. That is precisely why redaction happens at ingest — see below — and why your erasure runbook has to name those systems too.

PII redaction at ingest

Redaction happens before the write. Redacting at read time means the raw identifier is already in the table, the indexes, the WAL and the backups — the places an erasure has the most trouble reaching.

Indian identifiers, checksum-gated where a checksum exists:

| kind | rule | |---|---| | aadhaar | 12 digits and a valid Verhoeff check digit and a leading 2–9 | | pan | [A-Z]{5}[0-9]{4}[A-Z] with a real holder-type character | | gstin | 15 chars, valid state code, valid embedded PAN, mod-36 check char | | ifsc | [A-Z]{4}0[A-Z0-9]{6} | | phone_in | +91 / leading-zero / bare 10-digit mobile, spaced or not | | email | conventional | | bank_account | 9–18 digits next to an account cue — see below |

Why the Verhoeff check matters. A detector that fires on every 12-digit run redacts invoice totals in paise, order ids and challan numbers. A team whose real data keeps getting mangled switches redaction off, which is strictly worse than a slightly leaky redactor. redact("invoice total 234567890123 paise") returns the line unchanged, because that number fails Verhoeff and is therefore not an Aadhaar number.

Redaction is reported, not silent — you can tell a user "3 identifiers were redacted before storing":

const { redactions } = await memory.remember({ … });
// [{ kind: "phone_in", start: 10, end: 20, replacement: "[PHONE:****3210]" }]

The finding deliberately carries no value field: these reports end up in logs and audit trails, and putting the identifier in the report of its own redaction is how it lives forever in a log aggregator.

Relationship to @demystify/ai-guardrails

The detectors, masks, ordering and checksum tables here are mirrored from @demystify/ai-guardrails (src/pii.ts, src/checksums.ts). They are copied rather than imported because this package has zero runtime dependencies — and because two @demystify packages must never disagree about what an Aadhaar number is. If you already depend on ai-guardrails, run one implementation:

import { redactPii } from "@demystify/ai-guardrails";
new Memory({ store, redactor: { redact: redactPii } });   // structurally compatible

One deliberate divergence, and it is a default, not a pattern. ai-guardrails masks any bare 9–18 digit run as an account number. That is right for its contract: its output is a transient prompt, and masking an order id costs a little answer quality on one call. Here the write is durable and lossy — the original is never stored, so an over-mask is permanent. So the same pattern and the same mask are gated on an account cue nearby (A/c, account, IFSC, …). Pass { bareAccountNumbers: true }, or use the exported strictRedactor, for ai-guardrails' exact behaviour.

Recall, and its honest failure mode

This package never calls an embedding provider. It ships none, imports none, and has no key handling. You supply an EmbedderPort; it stores and compares the vectors it is given.

With no embedder configured, recall does not fail and does not quietly return worse results under the same shape as a semantic search. It degrades to deterministic lexical matching and says so:

const out = await memory.recall({ tenantKey: "org_abc", text: "invoice" });
out.strategy;  // "lexical"
out.note;      // "no embedder is configured and no embedding was supplied, so results
               //  are ranked by exact term overlap — no stemming, no synonyms, no
               //  semantic match. Supply an EmbedderPort for semantic recall."

The lexical scorer is the fraction of distinct query terms present in the row. "invoices" does not match "invoice". It is a floor, not a search engine, and that is asserted in the test suite so this paragraph cannot quietly go stale.

strategy is "vector" whenever a vector was available, and note says where the ranking happened — in Postgres via pgvector's <=>, or in this process over a bounded candidate pool.

The embedder is the one honest hole in "no transport"

An EmbedderPort is, in most hosts, somebody else's HTTP endpoint. What the package guarantees is narrower and checkable:

• there is no default embedder and no provider SDK, so the zero-config path is offline and keyless;
• the only text handed to an embedder is text that has already been redacted — including the query text on recall(). Proved by a recording embedder in test/remember.test.ts, not by this sentence;
• forget() never touches it. An erasure must not depend on a third party being reachable.

pgvector is optional

Detected, never assumed. If the extension is absent — or your role cannot create it — the migration still applies, verify_install() still passes, and ranking happens in-process over a bounded, recency-ordered candidate pool. Be clear about what that is: an exact search over the most recent N rows, not an ANN index. If a tenant holds a million memories and the match is two years old, it will not be found. capabilities() tells you which one you have.

With pgvector present, the adapter casts the stored real[] and uses the <=> cosine operator. An ivfflat/HNSW index needs a fixed-dimension vector column; that dimensionality belongs to your embedder, so adding one is a documented host-side optimisation rather than something this migration guesses at.

Rows embedded with a different model (a different vector length) are skipped rather than compared — in both adapters. Two models' vectors are not a worse match, they are a meaningless one. Changing embedding model means re-embedding.

Postgres

psql "$DATABASE_URL" -f node_modules/@demystify/memory/migrations/0001_memory.sql
select * from demystify_memory.verify_install();   -- every row ok = true

Own schema, demystify_memory. Never public — jarvis-memory put episodes, memories, entities and session_state in public, which in a shared database is a collision waiting to happen and an RLS story you cannot tell. verify_install() asserts thirteen properties, so a partially-applied migration fails loudly instead of silently.

import { Memory, PgMemoryStore } from "@demystify/memory";
import { Pool } from "pg";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const memory = new Memory({ store: new PgMemoryStore(pool) });

PgMemoryStore takes any { query(text, params) } — pg.Pool, a pooled transaction, PGlite, or a wrapper over Supabase's connection. The package imports no driver; pg is an optional peer.

The erasure is one statement: a recursive CTE computes the cascade, three DELETEs and the tombstone INSERT all run inside it. There is no window in which the memories are gone and the summaries derived from them are not, and none in which content is gone with no tombstone to show for it.

Tenant isolation, and the pooling hazard

RLS is deny-by-default (enable + force) on all four tables, scoped on app.current_tenant, which the host sets — the package owns no identity model. It joins the app.* namespace jarvis-memory already uses (app.current_company, app.current_owner) so one routing context sets the tenant once and every package that scopes on it agrees.

-- dedicated connection
select set_config('app.current_tenant', 'org_abc', false);

-- pooled connection: transaction-scoped, so the key cannot leak to the next borrower
begin;
  select set_config('app.current_tenant', 'org_abc', true);
  -- ... your queries ...
commit;

Mind the third argument — the two mistakes fail in OPPOSITE directions.

| mistake | what happens | direction | |---|---|---| | true outside a transaction | a driver that sends statements separately puts set_config and your query in different implicit transactions; the key is discarded and every row is denied | fails safe | | false on a pooled connection | session scope outlives the checkout; the next borrower of that connection inherits the previous tenant's key | fails toward a cross-tenant READ |

The second is the dangerous one, and it is the one that looks like it is working. It was learned the hard way in @demystify/agent-kernel 0.2.1.

Rule: if the connection is pooled, use true inside an explicit transaction. Only use false when the connection is genuinely dedicated for the life of the request.

Two more things, stated rather than buried:

• every policy reads its GUC with missing_ok = true. Without it, a session that never set the GUC makes the policy expression throw "unrecognized configuration parameter" instead of returning nothing. In jarvis-memory that killed a whole nightly pipeline — 55 episodes distilled into 0 memories — while the read path's abstain made it look like an honest empty memory;
• a superuser bypasses RLS unconditionally. An RLS test that runs as one proves nothing, so test/rls.sqlreal.test.ts runs as an unprivileged role, and your application must connect as a non-BYPASSRLS role. force row level security covers the table owner; it does not cover a superuser, and nothing does.

API

| call | what it does | |---|---| | remember(input) | redact → (optionally) embed → store. Returns the row and the redaction report. | | derive(input) | store an artifact with its provenance. Redacts the derived text too — a model asked to summarise redacted turns will happily restate an identifier. | | recall(query) | vector or lexical, always saying which. | | forget(input) | erase a subject, cascade transitively, write the tombstone, return the receipt. | | erasureHistory(tenantKey) | the tombstones, newest first. Content-free, so it is safe in an admin surface. | | subjectFingerprint(t, s) | the hash this package would write, so you can find a tombstone later. | | capabilities() | { vectorSearch: "pgvector" \| "in-process" }. Never guessed. | | getMemory / getDerived | plain reads. |

Also exported: InMemoryStore, PgMemoryStore, the MemoryStore / EmbedderPort / RedactorPort types, redact / scanPii / defaultRedactor / strictRedactor / noRedactor, the four checksum validators, cosineSimilarity / lexicalScore, sha256Hex / subjectHash, and cascadeDerived.

No transport

Doctrine D7: everything ships as a library that runs on the adopter's Postgres. We operate nothing. No hosted service, no pods, no phone-home.

test/no-transport.test.ts — ported from @demystify/agent-kernel — fails the build if anyone adds fetch, a socket, child_process, a dynamic import, a runtime dependency, a driver import, a provider SDK, a callback-accepting export, or a port method whose name implies an outbound effect. jarvis-memory shipped llm-ollama.ts, a fetch client for embeddings and summaries, inside the memory package. That is the thing being designed out.

What this does NOT do

• No working memory, no session windows, no MemGPT-style compaction. jarvis-memory had them; they are a host concern and they are not erasure-shaped.
• No LLM distillation. No extraction pass, no fact reconciliation, no ADD/UPDATE/INVALIDATE. That needs a model, and a model needs a transport. Do it in your host and write the result through derive() with its provenance.
• No decay or ranking beyond cosine and term overlap. No recency half-life, no access-frequency boost.
• No entity graph.
• metadata is not redacted. The package stores your JSON and never interprets it. Put an identifier in there and it will be stored verbatim — it is erased with the row, but it was never masked.
• It cannot reach your backups, WAL archive or replicas. See above.
• The lexical fallback is not a search engine, and without pgvector the vector path is not an index. Both are stated in the result, not hidden.
• A hash is not anonymisation. Without a pepper, a tombstone's subject_hash is enumerable over a small subject space.

Testing

pnpm test   # 206 tests

Coverage 97.8% statements, 92% branches. Every behavioural test runs against both adapters — the in-memory default and real Postgres via pglite — because an offline default that erased less thoroughly than production would be worse than no offline default at all. The SQL-real suites additionally prove the migration's triggers and constraints directly, including that the cascade fires on a plain DELETE that never touched this library.

MIT.