Semantic Memory

A persistent memory backend for AI agents. Stores text chunks with vector embeddings for semantic search, FTS5 for keyword search, and a cross-encoder reranker for relevance fusion. Supports EAV metadata (scope, sources, memory type, etc.) and chunk-to-concept relationships.

Architecture

                  ┌─────────────────────────────┐
                  │      Tool Wrappers          │
                  │  (Base / Coding / Assistant) │
                  └──────────────┬──────────────┘
                                 │ extends
                  ┌──────────────┴──────────────┐
                  │          DB Class            │
                  │  insert / search / merge     │
                  │  EAV properties / concepts   │
                  └───┬──────┬──────┬────────────┘
                      │      │      │
              ┌───────┘      │      └───────────┐
              ▼              ▼                  ▼
        better-sqlite3   USearch         @huggingface/transformers
        (SQLite + FTS5)  (HNSW vec idx)   (embeddings + reranker)

Quick Start

import { Embedder, Reranker, CodingWrapper } from '@malaclyde/knowledge-base';

const embedder = new Embedder('Xenova/all-MiniLM-L6-v2', 384);
const reranker = new Reranker('Xenova/bge-reranker-base', 'Xenova/bge-reranker-base');
const memory = new CodingWrapper(embedder, reranker, { dbPath: './memory.db' });

// Store knowledge
await memory.store('Crawl4AI is an open-source LLM-friendly web crawler.', 
  [{ name: 'Crawl4AI', description: 'Web crawler framework' }],
  [], ['https://github.com/unclecode/crawl4ai']);

// Search with semantic + keyword + reranker fusion
const results = await memory.search('web crawler', 5);
// results[0] → { id, text, rerankerScore, sources }

Core Concepts

Chunks

The fundamental unit of memory. Each chunk has:

• Text — the stored content
• Embedding — vector for semantic search (auto-generated)
• created_at — timestamp (auto-set)
• access_count — retrieval counter (auto-incremented)
• outdated — soft-delete flag

Concepts

Named entities that can be linked to multiple chunks. Each concept has a name and optional description. Concepts are searchable via FTS5 and vector embeddings.

EAV Properties (Entity-Attribute-Value)

Flexible metadata attached to chunks. Properties are free-form name-value pairs stored in a separate table. Common properties used by tool wrappers:

| Property | Type | Used by | Purpose | |----------|------|---------|---------| | scope | string | CodingWrapper | Filter by project area (backend, frontend) | | sources | JSON string | Both | Source URLs backing the fact | | memory_type | "working" / "archival" | AssistantWrapper | Memory tier | | created_at | ISO timestamp | Auto-set | Creation time | | access_count | string (number) | Auto-incremented | Times retrieved |

API Reference

Low-Level DB (src/kb/db.ts)

| Method | Description | |--------|-------------| | insertChunk(text, concepts?, existingConceptIds?, properties?) | Store a chunk | | semanticSearch(text, limit) | Pure vector search | | keywordSearch(text, limit) | FTS5 keyword search | | combinedSearch(text, limit, filters?) | Vector + FTS + reranker fusion | | conceptCombinedSearch(name, description?) | Find concepts by name or description | | getChunksByIds(ids) | Batch fetch chunks | | getChunksByProperty(name, value) | Filter chunks by EAV property | | getConceptChunks(conceptId, maxLen?) | Get linked chunks for a concept | | getConceptsByIds(ids) | Batch fetch concepts | | setChunkProperties(id, props) | Set EAV properties | | getChunkProperties(id) | Get all EAV properties | | deleteChunkProperty(id, name) | Remove a property | | setChunkOutdated(id) | Soft-delete a chunk | | editConcept(id, name, desc) | Update a concept | | mergeChunks(sourceIds, targetText) | Consolidate chunks |

Wrappers

BaseWrapper — safe core API for any agent: search, store, getChunks, getPreviews, findConcept, setOutdated, editConcept, setProps, getProps, delProp, merge

CodingWrapper — extends BaseWrapper, adds scope filtering and sources tracking. The store() method accepts an optional sources parameter.

AssistantWrapper — extends BaseWrapper, adds memory_type filtering. The store() method accepts optional sources and memoryType parameters. Also provides promoteToWorking(id) and demoteToArchival(id).

SearchOptions

All wrappers accept a SearchOptions object on search():

interface SearchOptions {
  filters?: { propertyName: string; value: string; required: boolean }[];
  scope?: string;       // CodingWrapper: filter by scope
  memoryType?: string;   // AssistantWrapper: filter by memory type
}

Database Schema

The library uses SQLite with the following tables:

• chunks — text storage with native created_at, access_count, outdated columns
• concepts — named entities with name/description
• edges — many-to-many chunk↔concept links
• properties — EAV property name registry
• chunk_properties — EAV values linked to chunks
• chunks_fts — FTS5 full-text index on chunk text
• concepts_fts — FTS5 index on concept name + description

OpenCode Plugins

This library is distributed with two OpenCode plugins:

• @malaclyde/coding-memory-oc — memory tools for coding agents (opencode/coding-memory/)
• @malaclyde/assistant-memory-oc — memory tools for personal assistant agents (opencode/assistant-memory/)

See their respective README files for usage and configuration.

Testing

npm test                # Run all 109 tests
npm run test:watch      # Watch mode
npm run seed            # Populate test DB with sample data

License

MIT