@x12i/memorix-retrieval

@x12i/memorix-retrieval is the read-side runtime data tier for Memorix.

It loads Catalox descriptors, inventories Mongo databases, resolves collection bindings, reads mapped and unmapped records, executes list/slice descriptors, composes item views, builds graph/inventory views, and returns Explorer/API-ready JSON.

Positioning

| Package | Role | |---------|------| | @x12i/memorix-retrieval | Runtime reads/views — Catalox descriptors + Mongo reads → lists, items, inventory, slices, graph, health | | @x12i/memorix-descriptors | Descriptor writes, slice schema, inventory ignore policy | | @x12i/memorix-completion | Payload writes |

Retrieval does not write descriptors and does not write Memorix payload documents.

Quick start

Set MONGO_URI. Database names, entity/event/knowledge routing, and collection resolution are handled internally.

MONGO_URI=mongodb://localhost:27017

Host apps (Catalox + Mongo)

import {
  createMemorixRetrievalStackFromEnv,
  fetchMemorixListForEntity,
  fetchMemorixItemForEntity,
} from "@x12i/memorix-retrieval";

const { client, appId } = await createMemorixRetrievalStackFromEnv({
  cataloxContext: { superAdmin: true, actor: { type: "service", id: "my-api" } },
});

const list = await fetchMemorixListForEntity(client, {
  entityName: "assets",
  page: { limit: 50, offset: 0 },
});

const item = await fetchMemorixItemForEntity(client, {
  entityName: "assets",
  entityId: "10.150.68.31",
});

await client.close?.();

Targets: entity, event, knowledge

| Target | Default DB | Identity field | Env override | |--------|------------|----------------|--------------| | entity | memorix-entities | entityId | MEMORIX_ENTITIES_DB | | event | memorix-events | eventId | MEMORIX_EVENTS_DB | | knowledge | memorix-knowledge | knowledgeId | MEMORIX_KNOWLEDGE_DB |

Shipped knowledge-target object types in retrieval seeds: knowledge-playbooks, content-documents (extracted documents — see @x12i/memorix-corpus).

import {
  resolveMemorixDatabaseName,
  resolveMemorixIdentityField,
  resolveMemorixTargetFromDescriptor,
} from "@x12i/memorix-retrieval";

Unified inventory

One API powers Catalox-first and DB-first Explorer inventory modes:

import { buildMemorixUnifiedInventory } from "@x12i/memorix-retrieval";

const inventory = await buildMemorixUnifiedInventory(client, {
  sourceLens: "catalox-first", // or "db-first"
  targets: ["entity", "event", "knowledge"],
  includeExactCounts: false,
  includeSamples: true,
});

Returns normalized rows, issues, per-target summaries, and suggested actions.

Every inventory result includes source (MemorixRuntimeSourceRef), targetBindings, and per-row sources explaining descriptor and Mongo provenance.

Source-aware runtime provenance

Retrieval never returns anonymous data. Every Explorer-facing result can explain where it came from:

import {
  getSourceAwareMemorixRetrievalHealth,
  type MemorixRuntimeSourceRef,
  type RuntimeSourceAware,
} from "@x12i/memorix-retrieval";

const health = await getSourceAwareMemorixRetrievalHealth(client, {
  includeInventory: true,
});

// health.source.sourceClass === "runtime-projection"
// health.mongo.targets.knowledge.status may be "not-configured" or "empty" (non-fatal)

Key types: MemorixRuntimeSourceRef, RuntimeSourceAware<T>, MemorixTargetDbBinding, MemorixRequiredUserInput.

Legacy Catalox catalogs (memorix-entities, memorix_entity_content_types, knowledge) are not used for descriptor-driven reads.

Catalox-first vs DB-first

• Catalox-first — start from entity descriptors and declared content types; append orphan Mongo collections.
• DB-first — start from Mongo collections (parsed with last-dash rule); append missing Catalox-declared collections.

Both lenses use the same MemorixUnifiedInventory shape.

Graph-run filters (_graphRuns) — MRX-FRS-001

Planning queries for Exellix jobs over reserved _graphRuns.<graphId> paths:

import {
  fetchMemorixListForEntity,
  compileGraphRunFilter,
} from "@x12i/memorix-retrieval";

const list = await fetchMemorixListForEntity(client, {
  entityName: "assets",
  graphRunFilter: { graphRunNotDone: { graphId: "graph-qcrbz6t" } },
  includeGraphRuns: true, // optional; default false on lists
  page: { limit: 100, offset: 0 },
});

| Facet | Use case | |-------|----------| | graphRunNotDone | Records needing processing | | graphRunStatus | Filter by in_progress / done / failed | | graphRunCompletedBefore | Stale results | | graphRunFailed | Retry candidates | | graphRunStaleVersion | Graph version invalidation |

Item reads include _graphRuns by default (includeGraphRuns defaults to true). Writer APIs: @x12i/memorix-writer — see memorix-writer/docs/GRAPH-RUNS.md.

Graph-run counts and processing overview

Count documents by graph-run filter without fetching bodies (countDocuments only — never paginated reads):

import {
  countByGraphRunFilter,
  getProcessingOverview,
} from "@x12i/memorix-retrieval";

const pending = await countByGraphRunFilter(client, {
  target: "entity",
  collection: "assets-snapshots",
  graphRunFilter: { graphRunNotDone: { graphId: "graph-qcrbz6t" } },
});

const overview = await getProcessingOverview(client, {
  appId: "memorix",
  objectTypes: ["assets", "subnets"],
  graphIds: ["graph-qcrbz6t"], // omit to discover graphIds from _graphRuns data
  estimated: true, // approximate recordTotal only; bucket counts stay exact
});

Returns byTarget (storage-tier rollups), byEntity (per object type), and byGraph (per graphId) without paginating documents.

Index requirement on each canonical collection:

db.<canonicalCollection>.createIndex({ _graphRuns: 1 }, { sparse: true, name: "idx_graphRuns" });

Nested path filters (_graphRuns.<graphId>.status) use the sparse top-level index as baseline; per-graph compound indexes remain opt-in via @x12i/memorix-writer ensureGraphRunIndexes.

Job-type run stamps (_jobTypeRuns) — MRX-CR-003

Parallel to _graphRuns, eligibility and reprocessing can use _jobTypeRuns.<jobTypeId> (sanitized map keys). Read helper:

import { getJobTypeRun } from "@x12i/memorix-mongo";

await getJobTypeRun(retrieval, "assets", recordId, jobTypeId, "core");

Writer stamp APIs: @x12i/memorix-writer — see memorix-writer/docs/JOB-TYPE-RUNS.md.

Baseline sparse index:

db.<canonicalCollection>.createIndex({ _jobTypeRuns: 1 }, { sparse: true, name: "idx_jobTypeRuns" });

Per-jobType compound indexes: @x12i/memorix-writer ensureJobTypeRunIndexes creates:

• { "_jobTypeRuns.<key>.status": 1, <idField>: 1 }
• { "_jobTypeRuns.<key>.completedAt": -1 }
• { "_jobTypeRuns.<key>.status": 1, "_jobTypeRuns.<key>.completedAt": -1 } (reprocessing filters)

Linked record resolution — MRX-CR-004

import { resolveLinkedRecordIds } from "@x12i/memorix-mongo";

const { entity, recordIds } = await resolveLinkedRecordIds(
  retrieval,
  sourceEntity,
  sourceDoc,
  relationshipKey,
);

Estimated counts: Pass estimated: true for large unfiltered totals (estimatedDocumentCount / $collStats). Graph-run bucket counts always use exact countDocuments even when estimated: true.

Entity identity field

import {
  resolveDefaultListDescriptorForEntity,
  resolveEntityIdentityField,
} from "@x12i/memorix-retrieval";

const { entity, list, identityField } = await resolveDefaultListDescriptorForEntity(
  client,
  "subnets",
);
// identityField === "subnetId" when declared on descriptor; else target default

Last-dash collection parsing

Collection names follow <objectName>-<contentType> where the last dash is the separator (object names may contain dashes):

import { parseMemorixCollectionName } from "@x12i/memorix-retrieval";

parseMemorixCollectionName("topology-cidr-graphs-scoped");
// objectName = "topology-cidr-graphs", contentType = "scoped"

Raw collection records

Read Mongo directly without a Catalox descriptor:

import { fetchMemorixRawCollectionRecords } from "@x12i/memorix-retrieval";

const raw = await fetchMemorixRawCollectionRecords(client, {
  target: "entity",
  collectionName: "assets-snapshots",
  page: { limit: 50, offset: 0 },
});

Open records from inventory rows with descriptor or raw fallback:

import { fetchMemorixCollectionRecords } from "@x12i/memorix-retrieval";

const records = await fetchMemorixCollectionRecords(client, {
  entityName: "assets",
  contentType: "snapshots",
  mode: "auto",
});

Slice execution

List descriptors with kind: "slice" are executable saved views:

Slice descriptors use the canonical shape from @x12i/memorix-descriptors (entityName, target, contentType, filter, ui.group). Create slices with createSliceDescriptor in descriptors; execute them here with fetchMemorixSliceRecords.

import {
  fetchMemorixSlices,
  fetchMemorixSliceRecords,
  fetchMemorixSliceItem,
  loadMemorixSliceDescriptor,
} from "@x12i/memorix-retrieval";

const slices = await fetchMemorixSlices(client, { includeHealth: true });
const page = await fetchMemorixSliceRecords(client, {
  sliceId: "critical-vulnerabilities",
  page: { limit: 50, offset: 0 },
});

Record display names

Human-readable titles for operators come from record.concept.name on stored documents (see MEMORIX-DATABASE-CONVENTIONS.md). Memorix read-tier APIs resolve display names centrally — hosts must not scan top-level id, name, or data.* fields.

import { resolveMemorixRecordDisplayName } from "@x12i/memorix-retrieval";

const displayName = resolveMemorixRecordDisplayName(record, {
  preferredLocale: "en", // optional; for localized concept.name objects
});
// string | undefined — never falls back to record id or identity fields

Supported concept.name shapes: plain string, string array (first non-empty entry), localized object ({ en: "...", he: "..." }).

List and item responses include optional top-level displayName (and list rows also expose recordId when resolvable):

• fetchMemorixList / fetchMemorixListForEntity → rows[].displayName
• fetchMemorixItem / fetchMemorixItemForEntity → displayName
• listMemorixEntityContentTypeDocuments → documents[].displayName

When concept.name is absent, displayName is omitted. Show the record id separately for disambiguation — do not substitute it as the title.

Easy retrieval helpers

Use these when you need identity + display name without re-implementing Memorix rules:

import {
  fetchMemorixRecordInfoForEntity,
  resolveMemorixRecordInfo,
  resolveMemorixRecordInfoFromRow,
  resolveMemorixRecordInfoFromItem,
  formatMemorixRecordLabel,
} from "@x12i/memorix-retrieval";

// One call: lookup by entity + recordId / entityId / eventId / knowledgeId
const info = await fetchMemorixRecordInfoForEntity(client, {
  entityName: "vulnerability-groups",
  recordId: "69d216ed358778e93215788d",
});
// → { recordId, displayName?, entityId?, identityField?, identityValue? }

// From list rows (already stamped with displayName)
const fromRow = resolveMemorixRecordInfoFromRow(listRow);

// From item responses
const fromItem = resolveMemorixRecordInfoFromItem(itemResponse);

// Picker / journal label — name when present, else id
const label = formatMemorixRecordLabel(info);
const withSuffix = formatMemorixRecordLabel(info, { includeIdSuffix: true });

Search: memorixLabelSearchFilter (used by raw entity reads) includes concept.name for string-valued names. Localized-object search is best-effort on the concept.name path only.

Record identity

import { resolveMemorixRecordIdentity } from "@x12i/memorix-retrieval";

const { id, identityField, source } = resolveMemorixRecordIdentity("knowledge", record);

Resolution order: descriptor identity field → target default (entityId / eventId / knowledgeId) → id → Mongo _id.

Health model

import { getMemorixRetrievalHealth } from "@x12i/memorix-retrieval";

const health = await getMemorixRetrievalHealth(client, { includeInventory: true });
// health.catalox, health.mongo.targets, health.descriptors, health.inventory

Logging

Retrieval uses @x12i/logxer ≥ 5.1.0 (dual ESM/CJS; use @x12i/logxer/package-levels or @x12i/logxer/diagnostics in browser-safe code — never the main entry). Pass host logging through constructors; retrieval forwards it to logxer as { stack } and does not manage host-wide registries.

Environment (this package only)

# Default when unset: warn
MEMORIX_RETRIEVAL_LOGS_LEVEL=info

# Silence retrieval logs
MEMORIX_RETRIEVAL_LOGS_LEVEL=off

Allowed values: off (or none / silent), error, warn, info, debug, verbose.

Downstream usage

Pass the same StackLoggingOptions object you use for other x12i packages:

import {
  createMemorixRetrieval,
  createMemorixRetrievalStackFromEnv,
  resolveMemorixRetrievalLogLevel,
  type StackLoggingOptions,
} from "@x12i/memorix-retrieval";

const logging: StackLoggingOptions = {
  packageLevels: {
    MEMORIX_RETRIEVAL: "debug",
    CATALOX: "warn",
  },
};

// Preferred: pass logging when creating the client / stack
const { client } = await createMemorixRetrievalStackFromEnv({
  logging,
  cataloxContext: { superAdmin: true, actor: { type: "service", id: "my-api" } },
});

// Or shorthand for this package only
await createMemorixRetrievalStackFromEnv({ logLevel: "info", ... });

// Introspection (optional stack argument; defaults to bound stack)
resolveMemorixRetrievalLogLevel(logging); // "debug"

Precedence for this package: parent logging stack → MEMORIX_RETRIEVAL_LOGS_LEVEL env → logxer default warn. Host apps configure bulk env / process registries via @x12i/logxer directly — not through memorix-retrieval.

Exports: createMemorixRetrievalLogger, bindMemorixRetrievalLogging, memorixRetrievalLogger, toMemorixRetrievalStackLogging, MEMORIX_RETRIEVAL_LOGS_LEVEL_ENV_KEY.

Documentation

| Doc | Purpose | |-----|---------| | DATA-TIER-CONTRACT.md | Public API surface (what hosts may call) | | XRONOX-DATA-TIER-REQUIREMENTS.md | Xronox API/env requirements for Memorix data tier | | MEMORIX-CATALOX-CONTRACTS.md | Descriptor catalogs, JSON formats, cross-component sync | | MEMORIX-DATABASE-CONVENTIONS.md | Mongo DB names, collections, env vars | | EXPLORER-HOST-APIS.md | Graph, raw reads, inventory, slices (Explorer hosts) |

Advanced (optional)

| Option / env | Purpose | |--------------|---------| | MEMORIX_ENTITIES_DB / MEMORIX_EVENTS_DB / MEMORIX_KNOWLEDGE_DB | Non-default database names | | MEMORIX_ENTITIES_COLLECTION_* / MEMORIX_EVENTS_COLLECTION_* / MEMORIX_KNOWLEDGE_COLLECTION_* | Per-type collection overrides | | CATALOX_APP_ID / MEMORIX_APP_ID | Catalox app namespace (default: memorix) | | MEMORIX_RETRIEVAL_LOGS_LEVEL | Logxer verbosity for this package (warn default; off to silence, info/debug for more detail) | | memorixDb | Code overrides for entity/event/knowledge DB names | | xronox | Inject a pre-configured Xronox client instead of built-in Mongo reads |

Entity descriptors declare target: "entity" | "event" | "knowledge" (default entity).

Catalox seeds

Source of truth: catalox-seeds/inputs/

npm run catalox:seed:build
npm run catalox:seed:memorix-retrieval:validate
npm run catalox:seed:memorix-retrieval:apply
npm run catalox:seed:ensure

Lists catalog (lists)

The list descriptor catalog id is lists (legacy: memorix-list-descriptors). Migrate existing Catalox data with memorix-descriptors/scripts/migrate-list-descriptors-catalog.mjs.

Compound sort

fetchMemorixList accepts sort: [{ property, direction }, ...]. Secondary sort keys must resolve to the same content type as the primary key. Ensure a matching compound index exists on the driver collection — there is no automatic index provisioning.

Analytics fields on lists

List descriptors may declare analytics[] with { field, collection, joinBy, metric }. Values are merged into rows per page via scoped aggregation (groupMemorixRecords).

Full record fetch

import { fetchMemorixFullRecord } from "@x12i/memorix-retrieval";

const full = await fetchMemorixFullRecord(client, {
  entityName: "assets",
  entityId: "abc-123",
  // contentTypes: ["snapshots", "analysis"], // optional subset
});
// full.contentTypes — raw document(s) per content type

Unlike fetchMemorixItemForEntity, this bypasses item descriptors and returns uncurated documents.

List filters

Request filters use MemorixFilterInput[]. A property is authorized when it is filterable: true on the entity descriptor and belongs to the list's leading content type. The list descriptor's filters[] array supplies pinned defaults only — it is not an allow-list for ad-hoc filters.

Build, test, and live smoke

npm run build
npm test

Live checks (require .env with MONGO_URI and Mongo Catalox credentials):

npm run mongo:check-collections
npm run mongo:inventory -- --sourceLens db-first --targets entity,event,knowledge
npm run mongo:inventory -- --sourceLens catalox-first --targets all
npm run mongo:inventory -- --includeExactCounts
npm run smoke:retrieval -- --health --source-aware
npm run smoke:retrieval -- --inventory --sourceLens db-first
npm run smoke:retrieval -- --raw-records --target entity --collection assets-snapshots --limit 5
npm run smoke:retrieval -- --collection-records --entity assets --contentType snapshots --mode auto
npm run smoke:retrieval -- --slice critical-vulnerabilities-list --limit 5
npm run smoke:retrieval -- --graph --sourceLens db-first --includeInventory

Publish

npm run build && npm test && npm publish