@x12i/memorix-snapshot-completion

Fill missing snapshot.data fields from source MongoDB records using JSON completion mappings.

Mappings are plain JSON files (arrays of mapping objects). Use {{ENV.variableName}} tokens in strings; they resolve via @x12i/env-tokens at load time.

Install

npm install @x12i/memorix-snapshot-completion

Requires MONGO_URI (or MONGO_CONNECTION_STRING) when using the built-in Mongo runner.

Mapping JSON

Each mapping file is a JSON array. Example entry:

{
  "name": "complete-vulnerability-snapshot-missing-data",
  "entityType": "vulnerability",
  "version": "1.0.0",
  "enabled": true,
  "snapshotCollection": "vulnerabilities-snapshots",
  "snapshotSelector": {
    "sourceCollection": "vulnerabilities-information",
    "idField": "vulnerabilityId"
  },
  "source": {
    "databaseName": "{{ENV.sourceDatabaseName}}",
    "collection": "vulnerabilities-information"
  },
  "match": {
    "snapshotPath": "entityId",
    "sourcePath": "vulnerabilityId"
  },
  "writeRoot": "data",
  "mode": {
    "onlyFillMissing": true,
    "overwriteExisting": false,
    "createNestedObjects": true
  },
  "ignore": {
    "exactProperties": ["_id", "metadata"],
    "idProperties": true
  },
  "fields": [
    { "sourcePath": "enrichment", "targetPath": "enrichment" }
  ]
}

| Field | Description | |-------|-------------| | snapshotCollection | Memorix Mongo collection for snapshots (recommended) | | snapshotCollectionCandidates | Extra names to try if snapshotCollection is omitted | | snapshotSelector.sourceCollection | Tag on snapshots for mapping selection | | source.databaseName / collection | Source DB lookup (supports {{ENV.*}}) | | fields | Paths to copy from source → snapshot data |

CLI

After npm run build:

# Dry-run (default)
npx memorix-complete-snapshots --mappings ./my-mappings.json

# Write to Mongo
npx memorix-complete-snapshots --mappings ./my-mappings.json --write

# Filters
npx memorix-complete-snapshots -m ./my-mappings.json --entity vulnerability --limit 100 --write

Environment:

• MONGO_URI — Mongo connection
• MEMORIX_DB — Memorix database (default: memorix)
• MEMORIX_SNAPSHOT_COLLECTION_<ENTITY_TYPE> — override snapshot collection per entity
• MEMORIX_COMPLETION_MAPPINGS — default mappings file path for CLI

Programmatic API

Full Mongo pipeline (mappings file → read → complete → write)

import {
  runMongoSnapshotCompletion,
  summarizeCompletionResults,
  loadCompletionMappingsFromFile
} from "@x12i/memorix-snapshot-completion";

const report = await runMongoSnapshotCompletion({
  mappings: "./completion-mappings.json", // or pre-loaded array
  dryRun: true,
  limit: 0, // 0 = all snapshots per mapping
  filter: { entityType: "vulnerability" }
});

console.log(summarizeCompletionResults(report.results));

Host-provided I/O (any database)

import {
  runMemorixSnapshotCompletion,
  loadCompletionMappingsFromFile
} from "@x12i/memorix-snapshot-completion";

const mappings = loadCompletionMappingsFromFile("./completion-mappings.json");

const results = await runMemorixSnapshotCompletion({
  sourceDatabaseName: mappings[0].source.databaseName,
  snapshots,
  mappings,
  dryRun: false,
  sourceReader: async ({ databaseName, collection, matchPath, matchValue }) => {
    return findOne({ db: databaseName, collection, filter: { [matchPath]: matchValue } });
  },
  snapshotWriter: async ({ snapshot, updatedData }) => {
    await updateSnapshot(snapshot.snapshotId, { data: updatedData });
  }
});

Load mappings only

import { loadCompletionMappingsFromFile } from "@x12i/memorix-snapshot-completion";

const mappings = loadCompletionMappingsFromFile("./maps.json", {
  env: process.env // default; resolves {{ENV.*}}
});

Mongo adapters (custom orchestration)

import {
  connectMongoCompletion,
  createMongoSourceReader,
  createMongoSnapshotWriter,
  loadSnapshotsFromCollection,
  resolveSnapshotCollectionName
} from "@x12i/memorix-snapshot-completion";

Core exports

| Export | Description | |--------|-------------| | runMongoSnapshotCompletion | End-to-end runner from mapping JSON file(s) | | runMemorixSnapshotCompletion | Batch runner with injected readers/writers | | completeMemorixSnapshotRecord | Single-snapshot completion | | loadCompletionMappingsFromFile | Load + resolve {{ENV.*}} in mapping JSON | | resolveJsonEnvTokens | Deep token resolution on any JSON value | | filterCompletionMappings | Filter by entity, name, enabled, fields |

Scripts (this repo)

npm run build
npm test
npm run poc:live -- --dry-run --entity vulnerability --limit 10
npm run complete -- --mappings test/poc-data-fix.json --write