Memorix Corpus Layer

Client-facing specification for storing extracted documents and web pages as Memorix knowledge.

Purpose

The corpus layer standardizes how external textual sources — web pages, PDFs, Markdown files, wiki pages, and uploads — are ingested, extracted, chunked, and exposed as reusable knowledge.

It is built on existing Memorix primitives. It does not add a fourth storage target.

Mental model

| Target | Purpose | |--------|---------| | entity | Stable operational objects (assets, subnets, groups) | | event | Findings, detections, timeline observations | | knowledge | Curated reference material, documents, playbooks, extracted corpora |

Extracted documents use:

target: knowledge
database: memorix-knowledge
identity field: knowledgeId
object type: content-documents

Why not target: "content"?

Memorix already uses “content” for:

• content types — collection slices such as snapshots, chunks
• content objects — large Markdown/text bodies stored externally

A fourth target named content would collide with that vocabulary. Corpus records are knowledge-target object types with corpus-specific conventions.

Package

@x12i/memorix-corpus

| Package | Role | |---------|------| | @x12i/memorix-corpus | Conventions, ids, hashing, chunking, record builders | | @x12i/memorix-descriptors | Object type / list / item descriptors | | @x12i/memorix-writer | Descriptor-driven writes + content-object storage | | @x12i/memorix-retrieval | Reads for Explorer, APIs, agents |

Object type: content-documents

| Content type | Collection | Cardinality | Purpose | |--------------|------------|-------------|---------| | snapshots | content-documents-snapshots | 1:1 | Canonical metadata + content pointers | | extractions | content-documents-extractions | 1:n | Fetch/extract run history | | chunks | content-documents-chunks | 1:n | Retrieval-ready text segments |

Record shape rule

Keep data clean. Only put useful payload there.

| Block | Holds | |-------|--------| | data | Title, summary, language, topics, chunk text, extracted facts | | source | Origin URI, kind, canonical URI, source id | | content | Pointers to stored bodies (contentKey, format, preview) | | provenance | Fetch/extract/hash/pipeline metadata | | parent | Parent document reference on chunks |

Do not put URLs, hashes, extractor versions, or storage keys inside data unless they are genuine domain payload.

Identity

Stable knowledgeId from source identity (not body content):

doc:<source-kind>:<stable-hash>

Examples:

doc:web_page:8f4c9a1b2c3d4e5f
doc:pdf:12a7be9f0e1d2c3b

Body changes update contentHash, extraction records, and chunks — not knowledgeId.

Document snapshot example

{
  "knowledgeId": "doc:web_page:8f4c9a...",
  "recordId": "doc:web_page:8f4c9a...",
  "capturedAt": "2026-06-24T10:00:00.000Z",
  "source": {
    "kind": "web_page",
    "uri": "https://example.com/security/networking",
    "canonicalUri": "https://example.com/security/networking"
  },
  "content": {
    "cleanMarkdown": {
      "contentKey": "content-documents/doc-web_page-8f4c9a/sha256-abcd/clean.md",
      "format": "markdown",
      "preview": "This guide explains..."
    }
  },
  "provenance": {
    "fetchedAt": "2026-06-24T09:59:00.000Z",
    "extractedAt": "2026-06-24T10:00:00.000Z",
    "extractor": "memorix-corpus-html",
    "extractorVersion": "1.0.0",
    "contentHash": "sha256:abcd..."
  },
  "data": {
    "title": "Networking Security Guide",
    "summary": "Reference guide for secure networking.",
    "language": "en",
    "topics": ["networking", "security"]
  }
}

Chunk example

{
  "knowledgeId": "doc:web_page:8f4c9a...",
  "recordId": "chunk:doc:web_page:8f4c9a:00012",
  "capturedAt": "2026-06-24T10:01:00.000Z",
  "parent": {
    "knowledgeId": "doc:web_page:8f4c9a...",
    "contentHash": "sha256:abcd..."
  },
  "provenance": {
    "chunkingStrategy": "markdown-heading-window-v1",
    "chunkIndex": 12
  },
  "data": {
    "text": "The subnet must not expose administrative ports...",
    "sectionTitle": "Subnet Exposure Rules",
    "tokenCount": 214
  }
}

Client integration flow

1. Configure MONGO_URI and optionally MEMORIX_KNOWLEDGE_DB.
2. Apply corpus descriptor seeds (content-documents object type).
3. Fetch or receive a source document.
4. Normalize source identity with normalizeCorpusSource.
5. Generate knowledgeId with createDocumentKnowledgeId.
6. Extract clean Markdown or plain text.
7. Hash body with hashCorpusContent.
8. Build snapshot with buildDocumentSnapshotRecord.
9. Write via @x12i/memorix-writer (content-documents-snapshot-write) or direct Mongo insert.
10. Build extraction record with buildExtractionRecord.
11. Chunk with buildChunkRecords.
12. Write chunks via content-documents-chunk-write.

Quick start

import {
  normalizeCorpusSource,
  createDocumentKnowledgeId,
  hashCorpusContent,
  buildDocumentSnapshotRecord,
  buildExtractionRecord,
  buildChunkRecords,
} from "@x12i/memorix-corpus";

const source = normalizeCorpusSource({
  kind: "web_page",
  uri: "https://example.com/docs/security",
});

const knowledgeId = createDocumentKnowledgeId(source);
const markdown = "# Security Guide\n\nBody text...";
const contentHash = hashCorpusContent(markdown);

const snapshot = buildDocumentSnapshotRecord({
  knowledgeId,
  source,
  contentHash,
  title: "Security Guide",
  language: "en",
  markdown,
});

const extraction = buildExtractionRecord({
  knowledgeId,
  source,
  contentHash,
  title: "Security Guide",
});

const chunks = buildChunkRecords({
  knowledgeId,
  contentHash,
  markdown,
});

Environment

MONGO_URI=mongodb://localhost:27017
MEMORIX_KNOWLEDGE_DB=memorix-knowledge

Optional content-object storage:

MEMORIX_CONTENT_BUCKET=memorix-content
MEMORIX_CONTENT_PREFIX=content-documents/

Write descriptors

Seeded in @x12i/memorix-writer:

| Id | Content type | Operation | |----|--------------|-----------| | content-documents-snapshot-write | snapshots | upsert | | content-documents-chunk-write | chunks | add |

Related docs

• docs/CORPUS-LAYER.md — extended specification
• memorix-descriptors/docs/MEMORIX-OBJECT-TYPES-AND-TARGETS.md