@x12i/memorix-completion

Folder name: this package lives in memorix-compeletion/ (typo). The npm package is @x12i/memorix-completion.

Fill missing fields in Memorix entity, event, and knowledge records by looking up data in source MongoDB databases, using JSON completion mappings or Catalox (--source catalox).

Mappings are plain JSON files (arrays of mapping objects). Use {{ENV.variableName}} tokens in strings; they resolve via @x12i/env-tokens at load time.

Install

npm install @x12i/memorix-completion

Quick start

Set MONGO_URI and run. Database names and entity/event routing are resolved internally (defaults: memorix-entities, memorix-events).

MONGO_URI=mongodb://localhost:27017
npx memorix-complete --mappings ./my-mappings.json

Source database for enrichment: set MEMORIX_SOURCE_DB or per-mapping source.databaseName.

See Memorix Database Conventions for shared naming rules across x12i peers.

Databases (advanced overrides)

Each mapping declares a target — either entity or event — which determines which Memorix database receives updates:

| Target | Env variable | Default database | |--------|--------------|------------------| | entity | MEMORIX_ENTITIES_DB | memorix-entities | | event | MEMORIX_EVENTS_DB | memorix-events |

Source records are read from a separate database configured per mapping (or via env defaults):

| Env variable | Purpose | |--------------|---------| | MEMORIX_SOURCE_DB / SOURCE_DATABASE_NAME | Default source database when not set in mapping | | mapping.source.databaseName | Per-mapping source DB (supports {{ENV.*}}) |

Mapping JSON

Each mapping file is a JSON array. Example entry:

{
  "name": "complete-vulnerability-entity-missing-data",
  "entityType": "vulnerability",
  "target": "entity",
  "version": "1.0.0",
  "enabled": true,
  "targetCollection": "vulnerabilities",
  "targetSelector": {
    "sourceCollection": "vulnerabilities-information"
  },
  "source": {
    "databaseName": "{{ENV.sourceDatabaseName}}",
    "collection": "vulnerabilities-information"
  },
  "match": {
    "targetPath": "entityId",
    "sourcePath": "vulnerabilityId"
  },
  "writeRoot": "data",
  "mode": {
    "onlyFillMissing": true,
    "overwriteExisting": false,
    "createNestedObjects": true
  },
  "ignore": {
    "exactProperties": ["_id", "metadata"],
    "idProperties": true
  },
  "fields": [
    { "sourcePath": "enrichment", "targetPath": "enrichment" }
  ]
}

| Field | Description | |-------|-------------| | target | entity or event — selects Memorix database and record type | | targetCollection | Memorix Mongo collection for target records (recommended) | | targetCollectionCandidates | Extra names to try if targetCollection is omitted | | targetSelector.sourceCollection | Source lineage selector; matches _system.provenance.source.collection | | source.databaseName / collection | Source DB lookup (supports {{ENV.*}}) | | match.targetPath / sourcePath | Join key between Memorix record and source document | | fields | Paths to copy from source → target data |

For event mappings, set "target": "event" and point targetCollection at the appropriate collection in memorix-events.

CLI

After npm run build:

# Dry-run (default)
npx memorix-complete --mappings ./my-mappings.json

# Write to Mongo
npx memorix-complete --mappings ./my-mappings.json --write

# Filters
npx memorix-complete -m ./my-mappings.json --entity vulnerability --target entity --limit 100 --write

Environment:

• MONGO_URI — required Mongo connection (simple mode)
• MEMORIX_SOURCE_DB / SOURCE_DATABASE_NAME — source database when not in mapping
• MEMORIX_ENTITIES_DB / MEMORIX_EVENTS_DB — optional overrides (defaults shown below)
• MEMORIX_ENTITIES_COLLECTION_<ENTITY_TYPE> — optional entity collection override
• MEMORIX_EVENTS_COLLECTION_<ENTITY_TYPE> — optional event collection override
• MEMORIX_COMPLETION_MAPPINGS — default mappings file path for CLI
• MEMORIX_COMPLETION_TARGET — default target filter (entity, event, or all)

Programmatic API

Full Mongo pipeline (mappings file → read → complete → write)

import {
  runMongoCompletion,
  summarizeCompletionResults,
  loadCompletionMappingsFromFile
} from "@x12i/memorix-completion";

const report = await runMongoCompletion({
  mappings: "./completion-mappings.json", // or pre-loaded array
  dryRun: true,
  limit: 0, // 0 = all records per mapping
  filter: { entityType: "vulnerability", target: "entity" }
});

console.log(summarizeCompletionResults(report.results));

Host-provided I/O (any database)

import {
  runMemorixCompletion,
  loadCompletionMappingsFromFile
} from "@x12i/memorix-completion";

const mappings = loadCompletionMappingsFromFile("./completion-mappings.json");

const results = await runMemorixCompletion({
  sourceDatabaseName: mappings[0].source.databaseName,
  records,
  mappings,
  dryRun: false,
  sourceReader: async ({ databaseName, collection, matchPath, matchValue }) => {
    return findOne({ db: databaseName, collection, filter: { [matchPath]: matchValue } });
  },
  recordWriter: async ({ record, updatedData }) => {
    await updateRecord(record.recordId, { data: updatedData });
  }
});

Load mappings only

import { loadCompletionMappingsFromFile } from "@x12i/memorix-completion";

const mappings = loadCompletionMappingsFromFile("./maps.json", {
  env: process.env // default; resolves {{ENV.*}}
});

Mongo adapters (custom orchestration)

import {
  connectMongoCompletion,
  createMongoSourceReader,
  createMongoRecordWriter,
  loadRecordsFromCollection,
  resolveTargetCollectionName,
  resolveMemorixDbName
} from "@x12i/memorix-completion";

const entitiesDb = resolveMemorixDbName("entity");
const eventsDb = resolveMemorixDbName("event");

Core exports

| Export | Description | |--------|-------------| | runMongoCompletion | End-to-end runner from mapping JSON file(s) | | runMemorixCompletion | Batch runner with injected readers/writers | | completeMemorixRecord | Single-record completion | | loadCompletionMappingsFromFile | Load + resolve {{ENV.*}} in mapping JSON | | resolveJsonEnvTokens | Deep token resolution on any JSON value | | filterCompletionMappings | Filter by entity, target, name, enabled, fields | | resolveMemorixDbName | Resolve entity/event Memorix database from env | | resolveDefaultSourceDbName | Resolve default source database from env |

Scripts (this repo)

npm run build
npm test
npm run poc:live -- --dry-run --entity vulnerability --target entity --limit 10
npm run complete -- --mappings test/poc-data-fix.json --write

Publish

From the x12i workspace root:

./scripts/publish-memorix-packages.sh