@x12i/memorix-pipeline

Descriptor-driven entity processing pipeline for Memorix. This package orchestrates single-record stages (resolution, enrichment, association roll-down, narrative sync). Cross-record associated* joins are handled by @x12i/memorix-associator and wired into the full enrichment flow below.

Documentation

Start here: docs/README.md

| Guide | Topic | |-------|-------| | Association format | v2 naming, payload forms, _system.association, matching rules | | Running enrichment | Full orchestration (recommended) | | Running associations | Plan → review → apply, presets, registry | | Running resolvers | Join normalization in pipeline and associator | | Running narrative sync | Per-record narratives.{key} tags | | Explorer API — consuming data | HTTP read model for apps and agents | | Explorer API — advanced | HTTP writes, pipeline, associations | | Pipeline Manager UI | Visual association designer and plan/apply |

Stack overview

Entity descriptors (processing + associatedProperties + narratives)
        │
        ▼
┌───────────────────────────────────────────────────────────────────┐
│  @x12i/memorix-pipeline                                           │
│  entity flow graph · stages · lifecycle · parts catalog            │
└───────────────────────────────────────────────────────────────────┘
        │                    │                    │
        ▼                    ▼                    ▼
 memorix-resolvers    memorix-insights     association roll-down
 (per-record joins)   (staged facts)       (parent routes + postfix)
        │                    │                    │
        └────────────────────┴────────────────────┘
                             │
                             ▼
              @x12i/memorix-associator (cross-record joins)
              plan → review → apply on associated* fields
                             │
                             ▼
              narrative-sync (flat doc.narratives.{key} tags)

| Package | Role in enrichment | Deep dive | |---------|-------------------|-----------| | memorix-pipeline (this) | Stage engine, Mongo data access, status filters, parts catalog | this README | | memorix-resolvers | Normalize join keys, CIDR/topology/zone logic on data | README | | memorix-insights | Multi-stage fact → judgment processors | README | | memorix-associator | Materialize cross-record associated* arrays | README | | memorix-descriptors | Descriptor schema, narrative signals, job-type runs | README | | memorix-explorer-api | HTTP surface for run/status, parts catalog, registry | README |

Registry: .operational/metadata/associated-properties-registry.json

Two kinds of “association”

| Mechanism | Scope | When it runs | |-----------|-------|--------------| | Association roll-down (memorix-pipeline) | Same record or parent→child via processing.entityRoutes | Pipeline association stage | | Associator presets (memorix-associator) | Cross-record joins (e.g. assets → subnets by CIDR) | Enrichment orchestration between pipeline association and narrative-sync |

Roll-down stamps parent context into associatedData and re-attaches same-record side content by postfix. The associator scans collections and appends linked records' data payloads onto root-level associated* fields — it never writes back into linked collections.

End-to-end enrichment flow

Per object type, in entity-flow-graph order (parents before children):

1. resolution      — resolvers on data.* join fields
2. enrichment      — insight stages (facts → judgments)
3. association     — parent route roll-down + postfix buckets
4. associator      — all presets for this object type (plan → apply)
5. narrative-sync  — derive narratives.{key} from signals + associations

The recommended entrypoint runs all five steps:

node memorix-pipeline/scripts/run-enrichment-pipeline.mjs --dry-run   # from monorepo root
npm run enrichment:dry-run -w @x12i/memorix-pipeline

Pipeline-only (steps 1–3 and optionally 5, no associator):

node memorix-pipeline/scripts/run-entity-pipeline.mjs --dry-run

Pipeline stages

| Stage | Job type id | Primary lifecycle | Driven by | |-------|-------------|-------------------|-----------| | resolution | entity-pipeline.resolution | entity-resolved | processing.resolvers | | enrichment | entity-pipeline.enrichment | entity-insights-processed | processing.insightStages | | association | entity-pipeline.association | entity-associated | processing.entityRoutes, associatedContent | | narrative-sync | entity-pipeline.narrative-sync | entity-narratives-synced | narratives catalog + relation/association signals |

Association may also stamp route outcomes: entity-parent-linked, entity-side-content-associated, entity-route-unlinked, entity-route-skipped, entity-route-deferred, entity-prerequisite-missing.

Status is persisted on records under:

• _system.processing.entityPipeline — lifecycle + route statuses
• _system._jobTypeRuns.<jobTypeId> — run status (in_progress, done, failed)

Sub-packages (what runs in each stage)

Resolvers (memorix-resolvers)

Assigned in descriptor processing.resolvers. Built-in types used in production:

| Type | Purpose | Typical stage | |------|---------|---------------| | exact-match | Copy/normalize join field values | resolution | | multi-value | Split delimited strings into arrays | resolution | | bridge-index | Map keys through a bridge collection | resolution | | subnet-cidr | Normalize CIDR / resolve asset IPs to subnet CIDRs | resolution | | subnet-key | Derive legacy subnetIp / subnetId | resolution | | topology-extractor | Build associatedTopology from associated data or subnet lookup | resolution | | zone-from-topology | Remediate UNKNOWN subnet zones from topology-raw | enrichment |

Remote resolvers can be registered via Explorer /pipeline/registry/parts and are merged at run time.

Insights (memorix-insights)

Assigned in descriptor processing.insightStages. Built-in processors:

| Processor id | Input → output | Purpose | |--------------|----------------|---------| | network.cidr-topology-path-resolver | cidr → cidr-path-facts | Structural topology facts from topology-raw | | network.cidr-zone-path-insights | cidr-path-facts → insights | Zone flags and judgment insights |

Remote insight processors use the same registry mechanism as resolvers.

Associator (memorix-associator)

Not a pipeline stage — orchestrated by run-enrichment-pipeline.mjs after association roll-down.

• Presets: committed under memorix-associator/presets/ (manifest lists refreshRuleKey → preset file).
• Registry: .operational/metadata/associated-properties-registry.json is the source of truth for managed joins.
• CLI: memorix-associator plan --refresh-rule-key assets.subnets.snapshots.by-cidr builds the request from registry.
• Workflow: plan (dry-run) → review planFingerprint → apply with expectedPlanFingerprint → re-plan to confirm zero pending updates.

Single preset (outside full enrichment):

memorix-associator plan --refresh-rule-key assets.subnets.snapshots.by-cidr --output temp/plan.json
memorix-associator apply --request memorix-associator/presets/assets.subnets.snapshots.by-cidr.json \
  --reviewed-report temp/plan.json

Regenerate presets after registry changes:

node memorix-associator/scripts/generate-presets-from-registry.mjs
node memorix-associator/scripts/verify-presets-parity.mjs

See Running associations for the full operational guide.

Prerequisites

• Node.js ≥ 18
• MongoDB via MONGO_URI (CLI scripts load memorix-explorer-api/.env when unset)
• Optional: MEMORIX_ENTITIES_DB, MEMORIX_EVENTS_DB, MEMORIX_KNOWLEDGE_DB
• Build order:

cd memorix-resolvers && npm run build
cd ../memorix-insights && npm run build
cd ../memorix-associator && npm run build
cd ../memorix-pipeline && npm run build

Package API

import {
  runPipelineJob,
  processEntityPipeline,
  getPipelineRecordStatus,
  listPipelineRecordsByStatus,
  compilePipelineStatusFilter,
  createMongoPipelineDataAccess,
  buildEntityFlowGraph,
  loadPipelinePartsCatalog,
  PIPELINE_STAGES,
} from "@x12i/memorix-pipeline";

const result = await runPipelineJob({
  descriptors,
  dataAccess: createMongoPipelineDataAccess({ client, descriptors, env: process.env }),
  stages: ["resolution", "enrichment", "association", "narrative-sync"],
  objectTypes: ["assets"],
  dryRun: true,
  rerun: false,
  graphId: "entity-pipeline",
  resolverRegistry,   // optional — Explorer merges builtin + remote
  insightRegistry,
});

Also exported: stage runners, association roll-down, entity flow graph, narrative sync (evaluateHavingNarratives, applyNarrativeSync), and parts-catalog loaders.

Status filters

compilePipelineStatusFilter({ stageNotDone: { stage: "resolution" } });
compilePipelineStatusFilter({ stageDone: { stage: "association" } });
compilePipelineStatusFilter({ stageJobStatus: { stage: "association", status: "failed" } });
compilePipelineStatusFilter({ routeStatus: { status: "deferred" } });
compilePipelineStatusFilter({ lifecycleHas: { process: "entity-resolved" } });

CLI

Run from monorepo root (xmemory-mono-repo).

Full enrichment (recommended)

node memorix-pipeline/scripts/run-enrichment-pipeline.mjs --dry-run
node memorix-pipeline/scripts/run-enrichment-pipeline.mjs --apply

# Subset
node memorix-pipeline/scripts/run-enrichment-pipeline.mjs --dry-run --object-types subnets,assets

# Resume (first object type runs only the given phase; later types run full pipeline)
node memorix-pipeline/scripts/run-enrichment-pipeline.mjs --dry-run \
  --from-object-type assets --only-phase narrative-sync

--only-phase: resolution | associator | narrative-sync

Report: temp/enrichment-pipeline-<timestamp>/summary.json

Pipeline only

node memorix-pipeline/scripts/run-entity-pipeline.mjs --dry-run
node memorix-pipeline/scripts/run-entity-pipeline.mjs --apply

Report: temp/entity-pipeline-run-<timestamp>/summary.json

Backup / catalog snapshot

node memorix-pipeline/scripts/backup-collections.mjs [--output-dir path]
node memorix-pipeline/scripts/generate-parts-catalog-snapshot.mjs
# → .operational/metadata/pipeline-parts-catalog.json

Explorer HTTP API

Base: /api/explorer (see memorix-explorer-api).

Pipeline run and status

| Method | Route | Purpose | |--------|-------|---------| | POST | /pipeline/run | Run stages (default dry-run) | | POST | /pipeline/rerun | Re-run with rerun: true | | GET | /pipeline/summary?objectType= | Stage/route counts + available parts | | GET | /pipeline/status?objectType=&recordId= | One record's status | | GET | /pipeline/records?objectType=&stage=&status= | Filter by stage job status | | GET | /pipeline/records?objectType=&routeStatus= | Filter by route status |

Writes are enabled by default; set MEMORIX_EXPLORER_ENABLE_PIPELINE_WRITES=false to block apply. Body: stages, objectTypes, recordIds, filter, apply / dryRun, rerun, graphId, jobRunId.

Parts catalog and playbooks

| Method | Route | Purpose | |--------|-------|---------| | GET | /pipeline/parts-catalog | Builtin + remote parts (?kind=, ?objectType=) | | GET | /pipeline/parts-catalog/:partId | Part detail + associator request template | | GET | /pipeline/enrichment-playbooks | Declared playbooks | | GET | /pipeline/enrichment-playbooks/:id | Playbook with resolved steps |

Associator (HTTP)

| Method | Route | Purpose | |--------|-------|---------| | POST | /associations/plan | Dry-run association plan | | POST | /associations/apply | Apply (requires expectedPlanFingerprint when gated) | | POST | /associations/verify | Verify associated field shapes | | GET | /associations/presets | Presets from parts catalog | | GET | /associations/presets/:refreshRuleKey/request | Request template for a preset | | GET | /operational/associated-properties-registry | Live registry JSON |

Plugin registry (remote resolvers / insights)

| Method | Route | Purpose | |--------|-------|---------| | GET | /pipeline/registry/parts | Builtin + registered remote parts | | POST | /pipeline/registry/parts | Register provider (MEMORIX_EXPLORER_ENABLE_REGISTRY_WRITES=true) | | DELETE | /pipeline/registry/parts/:partId | Deregister part | | DELETE | /pipeline/registry/providers/:providerId | Deregister provider |

Persisted in the operational DB as pipeline-registry-parts (not Catalox, not knowledge). See Explorer advanced APIs.

Descriptor configuration

| Metadata path | Controls | |---------------|----------| | processing.resolvers | Resolution stage assignments | | processing.insightStages | Enrichment stage assignments | | processing.entityRoutes | Parent/child association roll-down | | processing.associatedContent | Same-record postfix → associated* buckets | | associatedProperties | Managed cross-record joins (refreshRuleKey → associator preset) | | narratives | Authored catalog; narrative-sync derives per-record tags |

Descriptor seeds: memorix-retrieval/catalox-seeds/inputs/entity-descriptors/

Narrative sync needs no separate processing array — it runs when discoverHavingNarrativeSignals() finds eligible catalog entries, relations, or association common-property rules.

npm scripts

| Script | Command | |--------|---------| | build | Compile TypeScript → dist/ | | test | Vitest | | enrichment:dry-run | Full enrichment dry-run | | enrichment:apply | Full enrichment apply |