@x12i/catalox

Catalox is platform infrastructure for catalogs: a reusable, interoperable way to manage catalog definitions once, expose them consistently to apps, and avoid rebuilding catalog plumbing in every product. It can power a catalogs-as-a-service layer without being a hosted service itself.

It is a data-tier package for managing app-scoped catalogs in Firebase Firestore, including:

• Catalog discovery for an appId (what catalogs are available + access)
• App-agnostic catalog discovery (list all catalogs, text search, AI match by name/description; respects descriptor visibility)
• First-class catalog descriptors (capabilities, query metadata, identity metadata, field metadata)
• Descriptor creation/modification rules (optional governance blocks for AI-assisted create/modify)
• Native catalogs (items stored in Firestore)
• Mapped catalogs (items normalized from MongoDB or APIs)
• Catalog item search (within a catalog: text search + AI match)
• AI-assisted item create/modify (calls aifunctions-js createItem / modifyItem; optional autoPersist for native catalogs)
• References + validation contracts (standardized cross-catalog shapes)
• Seed/import/export + batch upsert workflows for native catalogs (library + catalox seed apply, catalox seed validate, catalox items … — see docs/cli-items.md)
• Optional per-record native history — NDJSON payloads in GCS plus a Firestore catalogItemHistory index; list/show/restore/replay APIs and catalox history … (docs/record-history.md)
• Optional catalog lifecycle — hard delete (with manifest), restore from manifest, and hard rename across collections; APIs + catalox catalog … (docs/catalog-crud.md)
• Optional GCS tools — NDJSON export/import of Firestore collections to a bucket, compare live data vs bucket snapshots, backupData mode gcs (Catalox-shaped NDJSON + manifest under a bucket prefix), and matching CLI commands (firestore export-gcs, import-gcs, compare-gcs, firestore backup --mode gcs; see docs/firestore-gcs-export.md, docs/backup.md)
• Operator toolbox CLI — Diagnose app↔catalog access (Firestore catalogBindings + simulated listCatalogItemsWithOutcome), create or repair bindings, or disable a binding (unbind-catalog); see docs/cli-toolbox.md
• Operator items CLI — Validate, list, get, upsert/patch/delete, batch-upsert, export one catalog’s items, and manage relations; see docs/cli-items.md

Catalox does not own UI, workflow orchestration, remote execution, artifact blobs, or secret storage.

When to use Catalox (and when not to)

Use Catalox when you need governed catalogs: reusable item inventories that apps must discover, render, search, reference, and validate.

Examples that fit Catalox well:

• Product or plan definitions used by checkout/order flows
• Routing rules, policies, feature flags, lookup tables
• Graph templates, execution-matrix templates, prompt templates
• Mappings and “configuration-as-data” that many services read

Do not use Catalox as the transaction itself (OLTP / event/state engine):

• Orders, payments, live execution runs, audit/event streams
• Queues, locks, balances, high-write mutable operational state

A good rule: Catalox stores the definition or selectable item; your host system stores the transaction/run/event/state transition that uses that item.

Public format notes:

• See docs/catalog-format.md for the public shapes: catalog meta, catalog lists, and catalog items.
• See docs/migration-catalog-model.md for the one-time backfill to the newer catalog model.

Quick onboarding: docs/onboarding-happy-path.md (use catalox seed validate to check a manifest before seed apply). App↔catalog access troubleshooting (CLI): docs/cli-toolbox.md. Per-item operators and inventory-style export: docs/cli-items.md (includes catalox export vs items export).

Host runtime contracts

Catalox stays product-agnostic. Hosts document their own rules for:

• Canonical source — design catalogs vs runtime config stores (docs/host-canonical-source.md)
• Versioned item.data shapes — schema packs, registry ids, CI validation (docs/schema-packs.md, docs/cli-validate-payload.md)

Scoped registries: catalog types, domains, agents

Catalox supports scoped “enum registries” for three common dimensions:

• Catalog types: allowed catalogType values per scope
• Domains: allowed domain ids per scope
• Agents: allowed agent ids per scope

Scopes are resolved in priority order: store → app → global.

Item association:

• Catalog items should carry metadata.domainIds: string[] and metadata.agentIds: string[] to associate the item (for example: a graph/template) to one or more domains/agents in your host runtime.
• Catalox stores and returns these arrays; their semantics are host-defined.

Identity & outcomes (vNext docs)

If you embed Catalox behind a BFF/UI and need clarity around tenancy/identity axes (store/app vs account/group/channel/user/visitor vs domain/agent) and empty vs denied vs misconfigured semantics, start here:

• docs/identity-model.md
• docs/authorization.md
• docs/outcomes.md

Install

This repo is currently set up as a workspace package.

• Node: >=20
• TypeScript: builds to dist/

@x12i/helpers is a required dependency (installed as @x12i/helpers@^1.4.0; GCS backup uses @x12i/helpers/gcs).

Direct dependency: @google-cloud/storage is declared for GCS export/import/compare; it uses the same Application Default Credentials pattern as Firestore Admin (bucket IAM required).

Package exports (v4)

| Import path | Intended use | |-------------|----------------| | @x12i/catalox | Full surface: embedder API + operator tooling (markdown, diagrams, migrations, Firebase stores, backup/GCS helpers, etc.). | | @x12i/catalox/embedder | Catalog runtime only: createCatalox, Catalox, withContext / CataloxBound, contracts, errors, adapters, validateMappingSpec / executeMapping. | | @x12i/catalox/operator | Markdown/diagrams/JSX, validation, bindings, ContextResolver, backup/GCS transfer helpers, identity/json-io, etc. | | @x12i/catalox/firebase | Firestore-backed store classes for advanced wiring. | | @x12i/catalox/mapping | Full mapping module (including helper-gap-report). |

Configuration (real connections)

Catalox is a library: you provide initialized clients + runtime env. For a full list of environment variables, CLI vs library behavior, and integration-test requirements, see docs/environment.md.

Firestore (Firebase Admin SDK)

This implementation of Catalox expects a Firebase Admin SDK Firestore instance (firebase-admin).

Important clarification:

• “Admin” here means privileged access to your Firebase/GCP project’s Firestore, using a service account.
• It is not “admin of the host machine / server OS”.
• Admin SDK access typically bypasses Firestore Security Rules (rules are for client SDKs).

Scoping:

• You can scope the service account to Firestore-related IAM roles and to a specific project.
• You generally cannot scope an Admin SDK credential to only certain collections/documents via IAM the way client rules work.

Firebase / Firestore credentials (createCataloxFromEnv, CLI, GCS helpers)

Bootstrap helpers in @x12i/catalox/firebase resolve credentials in this order (same order as JSDoc on resolveFirebaseAdminCredentialFromEnv):

1. serviceAccountBase64 option, else GOOGLE_SERVICE_ACCOUNT_BASE64 in the environment — base64-encoded standard Google service account JSON (the same JSON shape as a downloaded key file), used with cert(...). This is the recommended way to supply credentials in CI and scripts so behavior does not depend on a developer machine’s Application Default Credentials.
2. serviceAccountPath option only — your code passes a filesystem path string; Catalox reads that file and uses cert(...). No environment variable in this package supplies that path.
3. Application Default Credentials (applicationDefault()) when neither (1) nor (2) is set (for example GOOGLE_APPLICATION_CREDENTIALS, workload identity, or GCE metadata).

Optional: FIREBASE_PROJECT_ID and FIRESTORE_DATABASE_ID are read from the environment (or options) when using the bootstrap helpers.

v4: credential bootstrap is base64-first as above; use GOOGLE_SERVICE_ACCOUNT_BASE64 in .env or your secret store, or pass serviceAccountPath from your own code if you load a key file yourself.

This repo’s CLI + live tests typically load .env via @x12i/env. GCS-related code paths honor GOOGLE_SERVICE_ACCOUNT_BASE64 when set, then fall back to ADC.

Minimal example (ADC):

import { initializeApp, applicationDefault } from "firebase-admin/app";
import { getFirestore } from "firebase-admin/firestore";

initializeApp({ credential: applicationDefault() });
const firestore = getFirestore();

Official helper (library bootstrap; no dotenv):

import { createCataloxFromEnv } from "@x12i/catalox/firebase";

// Precedence: GOOGLE_SERVICE_ACCOUNT_BASE64 (or serviceAccountBase64 option), then serviceAccountPath option, else ADC.
// Optional: FIREBASE_PROJECT_ID, FIRESTORE_DATABASE_ID.
const { catalox } = createCataloxFromEnv();

Optional connectivity check (disposable Admin app, same resolution rules as createCataloxFromEnv). The read uses a Firestore-valid probe collection (not a reserved __…__ id). Exported from @x12i/catalox/firebase and re-exported from @x12i/catalox for a single import surface.

import { testFirestoreConnectionFromEnv } from "@x12i/catalox/firebase";
// or: import { testFirestoreConnectionFromEnv } from "@x12i/catalox";

const probe = await testFirestoreConnectionFromEnv();
if (!probe.ok) throw probe.error;

From the packaged CLI (after npm i / npx, with the same Firebase env vars as the library):

catalox firestore probe

Lifecycle-aware helper (recommended for long-lived hosts that may swap env/credentials at runtime):

import { createCataloxHostFromEnv } from "@x12i/catalox/firebase";

// Tip: set `appName` so you can safely replace the same Admin app instance.
const host = await createCataloxHostFromEnv({
  appName: "catalox-host",
  replaceExistingApp: true,
});

// Use the current runtime:
await host.catalox.listAppCatalogs({ appId: "myAppId" }, { appId: "myAppId" });

// Later, if credentials/project/database identity changes:
await host.rebuild({
  appName: "catalox-host",
  replaceExistingApp: true,
  env: {
    GOOGLE_SERVICE_ACCOUNT_BASE64: "<base64-encoded service account JSON>",
    FIREBASE_PROJECT_ID: "other-project",
    FIRESTORE_DATABASE_ID: "(default)",
  },
});

// Optional: on shutdown.
await host.dispose();

Construction helper (avoid implicit “default app” binding; use your own Admin app):

import { initializeApp, cert, type App } from "firebase-admin/app";
import { createCataloxFromFirebaseApp } from "@x12i/catalox/firebase";

const app: App = initializeApp({ credential: cert(serviceAccountJson) }, "my-admin-app");
const { catalox } = createCataloxFromFirebaseApp({ app, databaseId: "(default)" });

Google Cloud Storage (optional export / import / compare)

When you use catalox firestore export-gcs, import-gcs, compare-gcs, firestore backup --mode gcs, or the matching Catalox methods, the runtime needs a GCS bucket and a credential that can read/write objects there (typically the same service account as Firestore, with Storage roles on that bucket). GCS backup uses @x12i/helpers/gcs (ADC); export/import/compare still use @google-cloud/storage directly. See docs/firestore-gcs-export.md, docs/backup.md, and docs/environment.md.

Mongo (mapped catalogs)

For Mongo-mapped catalogs, provide:

• MONGO_URI (or whatever you store in adapter config as mongoUriEnvVar)

Example .env (do not commit secrets):

MONGO_URI=mongodb://127.0.0.1:27017
# Optional: ADC via GOOGLE_APPLICATION_CREDENTIALS, workload identity, etc.
# GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
# Recommended for CLI + live tests: base64-encoded service account JSON (same shape as a key file)
# GOOGLE_SERVICE_ACCOUNT_BASE64=...
FIREBASE_PROJECT_ID=your-project-id
FIRESTORE_LIVE_TESTS=1
# Optional: record-history live test + CLI (`createCatalox`); use a dedicated test bucket:
# CATALOX_RECORD_HISTORY_BUCKET=your-test-bucket
# CATALOX_RECORD_HISTORY_PREFIX=catalox-record-history/
# Optional defaults for the packaged `catalox` CLI (Catalox app/store context, not GCP):
# CATALOX_APP_ID=myAppId
# CATALOX_STORE_ID=myStoreId
# CATALOX_USER_ID=user-123
# CATALOX_MONGO_URI=mongodb://127.0.0.1:27017

catalox CLI environment

The catalox binary loads .env via dotenv. Use CATALOX_* for Catalox runtime context (catalog appId, optional storeId, actor, Mongo URI for the bundled adapter). Use FIREBASE_* / FIRESTORE_* plus GOOGLE_SERVICE_ACCOUNT_BASE64 (when you want explicit service account JSON via env) for Firebase Admin / Firestore connection (GCP projectId, database id, live-test flags).

• CATALOX_APP_ID — Default appId in CLI context when a command does not pass --app (when present, --app overrides). Most items / toolbox / seed apply flows pass --app explicitly.
• CATALOX_STORE_ID — Default storeId for report, export, and other store-aware commands when --store is omitted (--store overrides).
• CATALOX_USER_ID — Optional user id / actor for authz-sensitive CLI paths.
• CATALOX_MONGO_URI — If set, enables the Mongo catalog adapter in the CLI (wired via createCatalox in src/catalox/create-catalox.ts, called from src/cli/index.ts).
• CATALOX_RECORD_HISTORY_BUCKET — If set, createCatalox wires recordHistory (GCS payloads + catalogItemHistory on native writes; also required for catalox history … to persist new events).
• CATALOX_BUCKET — Default GCS bucket for Catalox storage features (used by the CLI and live tests; also used as a fallback for CATALOX_RECORD_HISTORY_BUCKET when wiring recordHistory via env).
• CATALOX_RECORD_HISTORY_PREFIX — Optional GCS prefix for record history (default catalox-record-history/).
• CATALOX_RECORD_HISTORY_FAIL_CLOSED — When 1, failed history writes fail the parent Firestore mutation.
• CATALOX_LOGS_LEVEL — Catalox package log threshold via @x12i/logxer (off, error, warn, info, debug, verbose; default warn when unset). Example: CATALOX_LOGS_LEVEL=debug. Embedders can also pass logging on createCatalox for host-controlled levels in deep stacks.

Running commands from this repository: the shell command catalox is only on your PATH if the package is installed globally (npm i -g @x12i/catalox) or linked (npm link from this repo). Otherwise, after npm run build, use:

npm run cli -- firestore report-native-layout --app "narrix"
# same as: node dist/src/cli/index.js firestore report-native-layout --app "narrix"

firestore CLI quick reference

| Command | Purpose | |---------|---------| | firestore backup | Mirror metadata + native rows to Mongo, Firestore backup-*, or GCS NDJSON (--mode gcs --bucket …; docs/backup.md). | | firestore restore-backup / undo-restore-backup | Restore from backup-* with undo sidecars (docs/restore-firestore-backup.md). | | firestore restore-backup-from-gcs | Restore live Firestore from one backupData GCS run (same undo as above). | | firestore prune-gcs-backups / delete-gcs-backup-run | Retention and one-off run folder deletion in GCS. | | firestore gcs-backup-to-export-manifest | Emit import-gcs–compatible manifest JSON from a GCS backup run. | | firestore report-native-layout | Legacy vs flat native layout diagnostics. | | firestore migrate-native-catalog-data | Legacy catalogData/{id}/items → flat catalogData-{id}-items (docs/migration-native-catalog-data.md). | | firestore export-gcs / import-gcs | NDJSON per collection + optional manifest (docs/firestore-gcs-export.md). | | firestore compare-gcs | Diff live Firestore vs bucket NDJSON (single collection or manifest). |

history / catalog CLI (3.1+)

| Command | Purpose | |---------|---------| | history list / history show / history restore / history replay | Per-record native history from catalogItemHistory + GCS (docs/record-history.md). | | catalog delete / catalog restore / catalog rename | Hard delete with manifest, restore from manifest, or hard rename (docs/catalog-crud.md). |

toolbox CLI (binding / access diagnostics)

| Command | Purpose | |---------|---------| | toolbox check-access | Read catalogBindings/{appId}:{catalogId}, catalogs/{catalogId}, and run listCatalogItemsWithOutcome (real auth rules). Optional --show-all-bindings. | | toolbox ensure-binding | Create binding doc if missing (ensureBinding; no-op if row already exists). | | toolbox repair-binding | Merge-write binding to active + access flags (--god required). | | toolbox unbind-catalog | Disable app↔catalog binding (status: "disabled" on the same catalogBindings doc). |

Full workflow, field meanings, exit codes, and how this differs from report / export --god: docs/cli-toolbox.md.

seed / items CLI (catalog + item operators)

| Command | Purpose | |---------|---------| | seed apply | Idempotent manifest: catalogs, optional descriptors/bindings/native items (docs/onboarding-happy-path.md). | | seed validate | Parse + AJV-validate --file or --preset manifest only (no Firestore writes). | | items validate / get / list | validateCatalog / validateCatalogItem, getCatalogItem, listCatalogItemsWithOutcome. | | items upsert / patch / delete / batch-upsert | Native item writes (delete requires --confirm). Body: --file or stdin JSON. | | items export | Stable { catalogId, items } JSON for one catalog (paginates nextCursor or offset). | | items relation upsert / delete | upsertCatalogItemRelation / deleteCatalogItemRelation (JSON body). |

Full JSON shapes, auth, and when to use seed apply vs items batch-upsert: docs/cli-items.md.

Firestore data model (logical collections)

For a full layout (subcollections, document id conventions, snapshot runs, and query notes), see docs/firestore-data-model.md. For native catalogs specifically (per-catalog catalogData-*-items, layout resolution, listCatalogItems, filters, troubleshooting), see docs/native-catalog-storage-and-api.md.

Metadata:

• apps/{appId}
• catalogs/{catalogId}
• catalogBindings/{bindingId} (many-to-many app↔catalog)
• storeAppBindings/{bindingId} (store↔app, multi-app export/report)
• catalogDefinitions/{catalogId} (native vs mapped specifics)
• catalogAdapters/{adapterId} (mongo/api adapter definitions)
• catalogMappings/{mappingId} (field mapping specs)
• catalogDescriptors/{catalogId} (descriptor metadata for generic consumption)
• catalogRendererSnippets/{catalogId}:{role}[:{mode}] (stored renderer snippets for list/grid/item/report/dashboard rendering)
• catalogReferences/{referenceId} (standardized reference records)
• catalogItemHistory/{eventId} (optional native write history index; payloads in GCS — docs/record-history.md)

Data:

• catalogData-{catalogId}-items/{itemId} (native item rows; top-level collection per catalog)
• catalogData/{catalogId} (index and metadata for that catalog’s native storage; not item payloads)
• catalogSnapshots/{catalogId}/items/{itemId} (mapped snapshot mode)

Backups (optional operator feature):

• backup-* and {timestamp}__backup-* (Firebase mode, same database)
• Mongo database catalox-backups (Mongo mode)
• gs:// bucket prefixes — backupData mode: "gcs" (NDJSON + catalox-backup-manifest.json per run; default prefix catalox-firestore-backups/)

See docs/backup.md for backupData / CLI backup (including mode: "gcs"), docs/restore-firestore-backup.md for restoring Firebase mirrors to live data with undo, docs/migration-native-catalog-data.md for moving legacy native rows into catalogData-{catalogId}-items, and docs/firestore-native-layout-vs-backup.md for how live paths differ from backup-* mirrors and what backupData reports in nativeItemSourceLayoutByCatalogId. Gap analysis for GCS backup vs restore: .reports/gcs-backup-gap-analysis.md. For how native storage, layout resolution, listCatalogItems, and filters work together, read docs/native-catalog-storage-and-api.md. For NDJSON export/import of Firestore collections to a GCS bucket (single collection, all roots, optional recursive subcollections, manifest restore) and compare live data vs bucket NDJSON (firestore compare-gcs), see docs/firestore-gcs-export.md and CLI firestore export-gcs / firestore import-gcs / firestore compare-gcs. If the console still shows catalogData/{catalogId}/items, run catalox firestore report-native-layout then catalox firestore migrate-native-catalog-data (backup + copy to flat; optional --delete-legacy after you trust backups).

Core usage (app-first, generic from appId)

Create Catalox (recommended)

Use createCatalox so you do not wire every Firestore store by hand. Optional mongoUri enables Mongo mapped catalogs; API mapped catalogs use an internal adapter unless you set enableApiAdapter: false.

import { createCatalox } from "@x12i/catalox";
import { getFirestore } from "firebase-admin/firestore";

const firestore = getFirestore();
const catalox = createCatalox({
  firestore,
  // Optional: per-record native history to GCS + `catalogItemHistory` (or set CATALOX_RECORD_HISTORY_BUCKET for CLI)
  // recordHistory: { gcsBucket: "my-bucket", gcsPrefix: "catalox-record-history/" },
});

// Optional: bind context once (no globals; same semantics as passing context each call).
// App-first (classic):
const scoped = catalox.withContext({ appId: "myApp" });
const list = await scoped.listCatalogItems("myCatalog", { limit: 50 });
// list.listOutcome is "ok" | "mapping_blocked"; empty items with "ok" means zero rows.

CatalogId-first usage (no appId required)

If you already have a catalogId (deep link/bookmark/operator tool), you can omit appId for catalog-specific operations. Catalox will infer an appId from active catalogBindings when there is exactly one viable binding (or allow omission when superAdmin: true).

const ctx = {}; // tenant/agent coords optional
const items = await catalox.listCatalogItems(ctx, "signals", { limit: 50 });
const descriptor = await catalox.getCatalogDescriptor(ctx, "signals");

App-first discovery/bootstrap APIs (e.g. listAppCatalogs, getAppCatalogBootstrap) still require an explicit appId.

Advanced: manual CataloxDependencies

If you need a non-default dependency graph, import stores from @x12i/catalox/firebase, construct AuthorizationService, and pass new Catalox(deps) as before (see source under src/catalox/catalox.ts).

Descriptor contract (planning-critical)

Catalox is designed so upstream packages can be “generic” (no hardcoded catalog registrations). The stable contract is the persisted descriptor:

• Stored at catalogDescriptors/{catalogId}
• Retrieved via getCatalogDescriptor(...) or getAppCatalogBootstrap(...)
• Used for identity, query, and client rendering metadata

Below is the actual current descriptor shape (TypeScript), with the most planning-relevant subtypes.

export type CatalogCapabilitiesDescriptor = {
  canList: boolean;
  canGet: boolean;
  canCreate: boolean;
  canEdit: boolean;
  canDelete: boolean;
  canImport?: boolean;
  canExport?: boolean;
  canSync?: boolean;
  canValidate?: boolean;
  canViewReferences?: boolean;
};

export type CatalogFieldDescriptor = {
  key: string;
  label: string;
  type:
    | "string"
    | "number"
    | "boolean"
    | "date"
    | "datetime"
    | "enum"
    | "object"
    | "array"
    | "reference";
  // Optional path into the stored item payload. If omitted, `key` is assumed.
  path?: string;

  // Query + indexing metadata.
  filterable?: boolean;
  sortable?: boolean;
  indexed?: boolean;
  multiValue?: boolean;

  // Presentation + contract metadata.
  listVisible?: boolean;
  detailVisible?: boolean;
  required?: boolean;
  enumValues?: Array<string | number>;
  reference?: { targetCatalogId?: string; targetField?: string };
  metadata?: Record<string, unknown>;
};

export type CatalogIdentityDescriptor = {
  itemIdStrategy: "natural" | "composite" | "generated";
  itemIdField?: string;
  compositeFields?: string[];

  // Optional display decoration fields.
  titleField?: string;
  subtitleField?: string;
  statusField?: string;
  updatedAtField?: string;
};

export type CatalogDescriptor = {
  catalogId: string;
  label: string;
  description?: string;
  itemLabel?: string;
  sourceMode: "native" | "mapped";
  mappedSourceType?: "mongo" | "api";
  status: "active" | "disabled" | "draft";
  visibility?: "visible" | "hidden";
  defaultSort?: { field: string; direction: "asc" | "desc" };
  defaultFilters?: Record<string, unknown>;
  capabilities: CatalogCapabilitiesDescriptor;
  queryableFields: CatalogFieldDescriptor[];
  queryCapabilities?: Record<string, unknown>;
  filterSpec?: Record<string, unknown>;
  presentationSpec?: Record<string, unknown>;
  /**
   * Optional UI escape hatch: registry components and/or stored snippet refs per role.
   * See `CatalogCustomRenderer` in the published types (`renderers[]` with `role` + optional `mode`).
   */
  customRenderer?: Record<string, unknown>;
  identity: CatalogIdentityDescriptor;
  /** Descriptor-driven FK fields (see Smart properties below). */
  smartProperties?: CatalogSmartPropertyDescriptor[];
  smartPropertyRules?: CatalogSmartPropertyRules;
  // Optional governance for AI-assisted create/modify:
  creationRules?: Record<string, unknown>;
  modificationRules?: Record<string, unknown>;
  metadata?: Record<string, unknown>;
};

export type CatalogSmartPropertyDescriptor = {
  key: string;
  targetCatalogId: string;
  targetField?: string;
  targetType?: string;
  autoCreate?: boolean;
  multiValue?: boolean;
  indexed?: boolean;
};

export type CatalogSmartPropertyRules = {
  /** At least N smart properties must have one or more codes after normalization. */
  minPropertiesFilled?: number;
};

Smart properties

Smart properties are descriptor-driven foreign keys: named fields on item.data (usually string[] of codes) that point at rows in target catalogs. They complement static queryableFields[].reference metadata and catalogReferences / relationRules.

On upsertNativeCatalogItem / updateNativeCatalogItem:

1. Validate smartPropertyRules (e.g. minPropertiesFilled: 1).
2. For each code in each configured property: resolve the target catalog, optionally auto-create the target row, bump metadata.usageCount, set metadata.lastUsedAt, and union the source item scope into the target item.
3. Optionally mirror values into indexed when the smart property sets indexed: true.

Target row metadata (convention): usageCount, lastUsedAt, firstUsedAt on metadata — useful for ranking vocabulary by relevance.

Not the same as:

• relationRules — typed edges in catalogReferences
• Free-form JSON blobs (e.g. legacy classification[] arrays) without catalog-backed resolution

Example descriptor excerpt:

{
  "catalogId": "articles",
  "smartProperties": [
    { "key": "role", "targetCatalogId": "roles", "indexed": true },
    { "key": "useCase", "targetCatalogId": "use_cases", "indexed": true }
  ],
  "smartPropertyRules": { "minPropertiesFilled": 1 }
}

Example item payload:

{
  "code": "external-access-review",
  "title": "External Access Review",
  "role": ["ciso"],
  "useCase": ["vendorAccessReview", "accessReview"]
}

Deep dive: docs/smart-properties.md. Pagenti reference app: docs/pagenti-knowledge-usage.md.

UI metadata & custom renderers (optional)

Overview (all custom UI surfaces in one place): docs/catalog-custom-ui.md

Catalog descriptors can include optional UI metadata for generic presentation layers:

• filterSpec: declarative filter configuration (built on FieldSource)
• presentationSpec: declarative layout + view/edit semantics (grid/list/cards/form)
• catalogType (optional): a host-defined tag that allows applying shared presentation and design defaults across many catalogs
• presentationSpec.templates (optional): host-defined template assignments per role/mode (no executable code)
• presentationSpec.designRefs (optional): references to scoped design objects (opaque JSON) for tokens/branding/content/assets
• customRenderer: escape hatch with a renderers[] list. Each entry has a role (core|list|grid|item|report|dashboard) and optional mode (readonly|editable, only for list|grid|item). An entry may set either:
  • a host-resolved component (component: registry key), or
  • a separately-stored renderer snippet (snippetRef), plus optional syntax (html|jsx|tsx), props, and receiveMap (full|minimal).

Use role: "core" for defaults shared across roles; merge with resolveCustomRendererEntry(...) (exported from @x12i/catalox).

Custom renderer contracts:

• docs/catalog-list-render-map.md
• docs/catalog-item-render-map.md
• docs/custom-renderer-snippets-io.md — snippet default export, map I/O, actions, roles, and receiveMap
• docs/design-objects.md — scoped opaque JSON design payloads

Presentation bindings + design objects (recommended integration)

Catalox does not render UI. Instead it can return a resolved binding that tells your presentation layer how to render a catalog surface, plus optional merged design JSON.

Two sides (important):

• Orchestration / BFF side (server): builds map and resolves binding.\n Uses buildCatalogListPresentationSurface(...) / buildCatalogItemPresentationSurface(...) to return { map, binding }.
• Presentation layer side (UI): reads binding.strategy and chooses:\n - custom-renderer: mount a registry component or compile a stored snippet\n - template: render a host-owned template by templateId\n - native-auto: use map.presentation / descriptor hints for generic UI\n In all cases it may also apply binding.design.merged (opaque tokens/content) without Catalox interpreting keys.

Minimal server-side example:

import { createCatalox } from "@x12i/catalox";
import { getFirestore } from "firebase-admin/firestore";

const catalox = createCatalox({
  firestore: getFirestore(),
  includeRendererSnippets: true,       // optional (only needed for snippet-backed custom renderers)
  includePresentationProfiles: true,   // optional (type/catalog-scoped defaults)
  includeDesignObjects: true,          // optional (design objects API + merge)
});

const ctx = { appId: "myAppId" };

// List surface: returns render-map + resolved binding (no rendering).
const { map, binding } = await catalox.buildCatalogListPresentationSurface(ctx, "orders", {
  displayContext: "grid",
  includeSnippet: true,
  includeDesignObjects: true,
  designScope: { appId: "myAppId", storeId: "myStoreId" },
});

// Send `{ map, binding }` to your UI (or render on the server with your own engine).

Minimal UI-side decision:

switch (binding.strategy) {
  case "custom-renderer":
    // mount binding.customRenderer.entry.component OR compile binding.customRenderer.snippet
    break;
  case "template":
    // render host template registry[binding.template.templateId](...)
    break;
  case "native-auto":
    // generic UI using map.presentation + map.items + map.actions
    break;
}

// Apply design tokens/content if present (opaque JSON).
const design = binding.design?.merged;

Design objects can also be fetched directly when includeDesignObjects is enabled:

const design = await catalox.getDesignObject(ctx, "brand:default");
const designs = await catalox.listDesignObjects(ctx, { appId: "myAppId", storeId: "myStoreId" });

Important: host presentation layers are responsible for resolving FieldSource lookups (cross-catalog enums/refs) and populating resolvedSources for renderers.

Search items inside a catalog (text / AI)

Text search is local filtering on top of a normal listCatalogItems call; AI matching uses aifunctions-js match() over the candidate pool.

const ctx = { appId: "myAppId" };

const text = await catalox.searchCatalogItems(ctx, "signals", { text: "error", textFields: ["title", "details.message"] });
const ai = await catalox.findCatalogItemsByAi(ctx, "signals", { query: "the onboarding flow item", maxResults: 3 });

AI-assisted item create / modify (native catalogs)

These methods call aifunctions-js and return a structured envelope (issues, reason, etc.). When autoPersist: true, they persist via upsertNativeCatalogItem / updateNativeCatalogItem only for native catalogs.

const ctx = { appId: "myAppId" };

const created = await catalox.createCatalogItemByAi(ctx, "signals", {
  provided: { title: "New signal", categoryId: "core" },
  autoPersist: false,
});

const updated = await catalox.modifyCatalogItemByAi(ctx, "signals", "core:S1", {
  patch: { title: "Renamed" },
  autoPersist: false,
});

Stored renderer snippets (optional)

Descriptors reference snippets via customRenderer.renderers[].snippetRef.

Snippet documents (Firestore, default layout) live at:

• catalogRendererSnippets/{catalogId}:{role}[:{mode}] where role is core|list|grid|item|report|dashboard and mode is optional (readonly|editable)

Snippet record fields include rendererSource + optional syntax (html|jsx|tsx). Fetch via catalox.getCatalogRendererSnippet(ctx, catalogId, role, mode?) when rendererSnippets is wired on Catalox.

This package includes a small set of opt-in helpers for snippet compilation/validation:

• transpileRendererSourceToModuleSource(...): TSX/JSX → ESM module source (string)
• typecheckRendererSnippetIo(...) (optional): best-effort TypeScript validation that a snippet’s default export matches the role’s render-map I/O contract
• unsafeCreateRendererFunction(...): UNSAFE runtime evaluation of transpiled code
• renderRendererToHtml(...): best-effort HTML rendering if react + react-dom are installed

Persisted renderer metadata must already match the role/renderers[] model. There is no built-in migration CLI; see docs/custom-renderer-canonical-contract.md. The package still exports buildRendererSnippetDocId for consistent snippet document ids.

Worked example (catalog viewer + Catalox map): docs/new-feature/USAGE.md

What you need persisted for “generic consumption”

To let consumers operate purely from appId with no hardcoded catalog registrations, Catalox relies on these being present in Firestore:

• Bindings: catalogBindings determine which catalogs an app can see/use.
• Descriptors: catalogDescriptors/{catalogId} provide capabilities, query metadata, identity metadata, and field metadata.

Minimum viable setup for generic consumption is catalog + descriptor + binding.

For mapped catalogs, the minimum viable setup is also definition + mapping + adapter (created automatically if you use createCatalog with sourceMode: "mapped").

Discover catalogs for an app

const context = { appId: "myAppId" };
const catalogs = await catalox.listAppCatalogs(context, { appId: "myAppId" });

Discover catalogs across all apps (app-agnostic)

These APIs return the catalog lists themselves (metadata), not items. They merge catalogs/{catalogId} with catalogDescriptors/{catalogId} and apply visibility filtering (hidden catalogs are excluded unless context.superAdmin or includeHidden: true).

const ctx = { appId: "myAppId", superAdmin: true };

const all = await catalox.listAllCatalogs(ctx, { includeDisabled: false });
const hits = await catalox.findCatalogs(ctx, { text: "users", fields: ["label", "description"] });
const ai = await catalox.findCatalogsByAi(ctx, { query: "customer lists", maxResults: 5 });

Get a catalog descriptor

const descriptor = await catalox.getCatalogDescriptor(context, "signals");

Bootstrap (all descriptors accessible to an app)

const bootstrap = await catalox.getAppCatalogBootstrap(context, "myAppId");

Common “generic client” flow (no hardcoded catalog logic)

1. listAppCatalogs(appId) → get catalog list + access
2. getAppCatalogBootstrap(appId) → get descriptors (capabilities/query/identity/fields)
3. listCatalogItems(catalogId, filter/sort) → { listOutcome, items, issues? } (listOutcome distinguishes OK empty lists vs mapping validation blocking)
4. getCatalogItem(catalogId, itemId) → { outcome: "found" | "not_found" | "mapping_blocked", ... }
5. Optional: getCatalogItemReferences(...), validateCatalogItem(...)

Provisioning (create catalog + bind + descriptor)

Catalox includes provisioning helpers to reduce “manual Firestore wiring”.

Create a native catalog (also seeds a minimal descriptor)

const ctx = { appId: "myAppId", superAdmin: true }; // set only after host auth; often from AppRecord.superAdminApp

await catalox.createCatalog(ctx, {
  catalogId: "signals",
  name: "Signals",
  sourceMode: "native",
  native: { type: "native" },
});

Bind a catalog to an app (enables discovery + access)

await catalox.bindCatalogToApp(ctx, {
  appId: "myAppId",
  catalogId: "signals",
  access: { canRead: true, canWrite: true, canAdmin: true },
});

Patch/upsert a descriptor (recommended for identity + query metadata)

There is currently no Catalox.setCatalogDescriptor(...) method; consumers can upsert the persisted record via DescriptorStore.

await descriptors.upsert({
  catalogId: "signals",
  descriptorVersion: "2",
  descriptor: {
    catalogId: "signals",
    label: "Signals",
    sourceMode: "native",
    status: "active",
    capabilities: { canList: true, canGet: true, canCreate: true, canEdit: true, canDelete: true },
    queryableFields: [
      { key: "categoryId", label: "Category", type: "string", indexed: true, filterable: true },
      { key: "code", label: "Code", type: "string", indexed: true, filterable: true },
      { key: "title", label: "Title", type: "string" },
    ],
    identity: {
      itemIdStrategy: "composite",
      compositeFields: ["categoryId", "code"],
      titleField: "title",
    },
  },
  createdAt: new Date().toISOString(),
  updatedAt: new Date().toISOString(),
});

Ensure helpers (idempotent)

• ensureCatalog(...): creates a minimal catalog record if missing (requires admin binding access).
• ensureBinding(...): creates a binding if missing (enforces god-mode for cross-app provisioning).

Native catalogs (seed/import/export + batch upsert)

The packaged catalox CLI mirrors these APIs for operators: seed apply / seed validate, items upsert, items batch-upsert, items export, and related commands (docs/cli-items.md).

Import/Export JSON (helpers)

const items = catalox.importCatalogItemsFromJson<Array<Record<string, unknown>>>(jsonString);
const jsonOut = catalox.exportCatalogItemsToJson(items);

Upsert one item (descriptor-driven identity)

await catalox.upsertNativeCatalogItem({ appId: "myAppId" }, "signals", {
  categoryId: "core",
  code: "S1",
  title: "Example",
  indexed: { categoryId: "core", code: "S1" }
});

Native write contract (exact)

• Input shape: the write API accepts a plain object payload. indexed is a reserved top-level field used only for query performance and is not part of the domain payload.
• Stored shape (native): items are stored as { itemId, catalogId, indexed, data }.
• Identity: itemId is resolved from descriptor.identity.
  • natural: uses identity.itemIdField from the input payload.
  • composite: joins identity.compositeFields with :.
  • generated: currently requires caller-supplied id (Catalox will throw if descriptor uses generated and no id is provided).

Batch upsert

await catalox.batchUpsertNativeCatalogItems({ appId: "myAppId" }, "signals", items);

List native items with equality filtering

const res = await catalox.listCatalogItems({ appId: "myAppId" }, "signals", {
  filter: { categoryId: "core" }
});
// `res.listOutcome`: "ok" (list ran) or "mapping_blocked" (see `res.issues`). Empty `items` with "ok" means zero matches.

Filtering is performed on indexed.<field> in stored native records (payload remains in data). Blank filter values ("", whitespace-only strings, null, undefined) are not sent as Firestore constraints so empty UI fields do not zero out lists; see docs/native-catalog-storage-and-api.md and compactCatalogFilter in the published API.

Canonical indexed rule (native catalogs)

• Caller may provide indexed: indexed: { ... } is accepted on writes.
• Catalox may derive indexed: if omitted, Catalox derives indexed from the descriptor’s queryableFields (fields marked indexed: true, using field.path ?? field.key).
• Data cleanliness: indexed is treated as reserved metadata and is not duplicated inside data.
• If indexed is missing/wrong: equality filtering and indexed sorting may return incomplete results or fail due to missing Firestore indexes; the canonical fix is to correct the catalog descriptor and/or the write input and re-upsert.

References

const refs = await catalox.getCatalogItemReferences({ appId: "myAppId" }, "signals", "core:S1");

Reference / relations write model

Catalox supports generic item-to-item relations (graph links) via the persisted catalogReferences/{referenceId} collection.

• Storage record: CatalogItemReference (fromCatalogId/fromItemId -> toCatalogId/toItemId + relationType)\n
• Descriptor rules: CatalogDescriptor.relationRules[] defines which relation types are allowed and which are required.
• Render maps: item render maps include relations[] (and keep references[] for backward compatibility).

Create/update/delete relations

// Upsert (idempotent; deterministic referenceId)
await catalox.upsertCatalogItemRelation({ appId: "myAppId" }, {
  fromCatalogId: "signals",
  fromItemId: "core:S1",
  toCatalogId: "categories",
  toItemId: "core",
  relationType: "belongs_to",
  label: "Category",
});

// Delete by endpoints (or by referenceId)
await catalox.deleteCatalogItemRelation({ appId: "myAppId" }, {
  fromCatalogId: "signals",
  fromItemId: "core:S1",
  toCatalogId: "categories",
  toItemId: "core",
  relationType: "belongs_to",
});

Provide relations at item creation/update time (native catalogs)

For native item writes, you may include relations: [...] (or legacy references: [...]) in the write payload to satisfy required relation rules.

await catalox.upsertNativeCatalogItem({ appId: "myAppId" }, "signals", {
  categoryId: "core",
  code: "S1",
  title: "Example",
  relations: [{ toCatalogId: "categories", toItemId: "core", relationType: "belongs_to" }],
});

Validation

Validation APIs exist with standardized contracts. When a catalog descriptor defines relationRules, validateCatalogItem(...) returns issues for:\n

• missing required relations\n
• disallowed relation types\n
• disallowed target catalogs / catalogTypes\n
• cardinality violations (multiple: false)\n \n Native item upserts/updates will throw CatalogValidationError when relationRules are present and the provided relations would violate the rules.

Publishing

From a clean tree, npm publish runs prepublishOnly, which executes npm run build then npm test (unit tests). Ensure dist/ is build output only; the published tarball includes dist, README.md, LICENSE, and firestore.indexes.json (composite indexes for catalogItemHistory queries) per package.json files.

Tests

Unit tests (default):

npm test

Integration tests (live Firestore, no mocks/emulators):

npm run test:integration

Integration tests are live-only (no mocks, no emulators). They require:

• FIRESTORE_LIVE_TESTS=1
• FIREBASE_PROJECT_ID=...
• GOOGLE_SERVICE_ACCOUNT_BASE64 set to base64-encoded service account JSON, or valid Application Default Credentials for the test project

AI live integration tests (real LLM calls) are additionally gated behind:

• AIFUNCTIONS_LIVE_TESTS=1
• OPENROUTER_API_KEY or OPEN_ROUTER_KEY

The Firestore live integration test (test/integration/firestore.emulator.test.ts) uses createCatalox and asserts v3 contracts: listCatalogItems returns listOutcome: "ok", and getCatalogItem returns { outcome: "found", item } for the seeded row. The descriptor patch in that test includes queryableFields with indexed: true for filtered fields so deriveIndexed persists indexed.* (see Canonical indexed rule above).

When CATALOX_RECORD_HISTORY_BUCKET is set, test/integration/record-history.live.test.ts runs as well: ephemeral app/catalog, upsert → update → listCatalogItemHistory → getCatalogItemHistoryEvent → restoreCatalogItemFromHistory, then best-effort Firestore + GCS cleanup for the created history objects.

Live test safety (read before running)

• Never run against production credentials/projects. Use a dedicated Firebase project for tests.
• Touched collections: apps, catalogs, catalogBindings, catalogDefinitions, catalogDescriptors, catalogData/{catalogId}, catalogData-{catalogId}-items/..., and (when record history is enabled) catalogItemHistory/.... The catalox firestore restore-backup / undo-restore-backup commands additionally write backup-restoreSessions and {restoreSessionId}__preRestore-* sidecar collections (see docs/restore-firestore-backup.md).
• Cleanup: tests do best-effort deletes of the docs they created, but do not guarantee full teardown.

Changelog

4.0.4

• CLI: catalox seed validate — validate preset manifests without applying.
• CLI: catalox items — validate, get, list, upsert, patch, delete, batch-upsert, export, plus items relation upsert / delete (docs/cli-items.md).
• CLI: catalox toolbox unbind-catalog — disable an app↔catalog binding (docs/cli-toolbox.md).

4.0.0

• Breaking — Firebase bootstrap: resolveFirebaseAdminCredentialFromEnv, createCataloxFromEnv, and related helpers now resolve credentials in order: GOOGLE_SERVICE_ACCOUNT_BASE64 (or serviceAccountBase64 option), then optional serviceAccountPath (caller-supplied path only), then Application Default Credentials. GCS credential helpers in backup-data / record-history use the same base64-then-ADC pattern.
• New: testFirestoreConnectionFromEnv — minimal Firestore read using the same rules as createCataloxFromEnv, on a disposable named Admin app (probe uses collection cataloxConnectivityProbe, not reserved __…__ ids).
• CLI: catalox firestore probe prints JSON from testFirestoreConnectionFromEnv.
• New: applyCataloxSeedPreset, parseCataloxSeedManifest, loadCataloxSeedManifestFromPath (@x12i/catalox) — versioned JSON manifests for idempotent catalog + binding + descriptor + native item provisioning.
• CLI: catalox seed apply --app … --file … or --preset … with optional --god for descriptor sections; resolveCataloxSeedPresetPath and npm catalox.seedPreset (docs/onboarding-happy-path.md).
• New: Catalox.upsertCatalogDescriptor / CataloxBound.upsertCatalogDescriptor (super-admin) for host-controlled descriptor writes.
• Docs: docs/onboarding-happy-path.md, docs/native-map-catalog-preset.md, example presets/native-map-v1.json.

3.1.1

• Live tests: Firestore integration test now sets queryableFields on the patched descriptor so equality filters match stored indexed.* rows (same rule as production descriptors).
• Live tests: Record-history integration test covers update, get event, restore, and GCS object cleanup when a bucket is configured.
• Package: publish firestore.indexes.json in the npm tarball for operators deploying catalogItemHistory queries.

3.1.0

• Per-record history: optional recordHistory on createCatalox (or CATALOX_RECORD_HISTORY_* env for CLI) writes GCS NDJSON + Firestore catalogItemHistory on native upsert/update/delete/batch/move. APIs: listCatalogItemHistory, getCatalogItemHistoryEvent, restoreCatalogItemFromHistory, replayCatalogToPointInTime. CLI: catalox history …. See docs/record-history.md, firestore.indexes.json.
• Catalog lifecycle: deleteCatalog, restoreDeletedCatalog, renameCatalog (hard rename) + CLI catalox catalog …. See docs/catalog-crud.md.

3.0.0

• createCatalox(config) — single factory for Firestore-backed stores, authz, optional Mongo/API adapters, optional renderer snippet store (src/catalox/create-catalox.ts).
• catalox.withContext(ctx) / bindCataloxContext(catalox, ctx) — CataloxBound: same APIs without repeating CataloxContext on every call (src/catalox/catalox-bound.ts).
• Breaking — lists: CatalogListResult includes listOutcome: "ok" | "mapping_blocked". Mapping validation failures use mapping_blocked (see issues). Empty items with listOutcome === "ok" means zero matching rows.
• Breaking — get item: getCatalogItem returns CatalogGetItemResult (found | not_found | mapping_blocked), not null.
• Package: main / types and exports["."] resolve to dist/src/.... Subpaths @x12i/catalox/embedder, /operator, /mapping, /firebase. Root @x12i/catalox re-exports embedder + operator (preserves most existing root imports).

2.7.0

• GCS restore: restoreFirestoreBackupFromGcs + CLI firestore restore-backup-from-gcs — same pre-restore / undoFirestoreRestore model as mirror restore.
• GCS retention: pruneGcsBackupRuns, deleteCataloxGcsBackupRunObjects, CLI prune-gcs-backups, delete-gcs-backup-run.
• Manifest bridge: cataloxGcsBackupManifestToFirestoreExportManifest, CLI gcs-backup-to-export-manifest, type CataloxGcsBackupManifestV1; exported restoreFirestoreNdjsonStreamToCollection.
• Reliability: failed backupData GCS runs trigger best-effort deletion of the partial run folder.
• Contracts: RestoreFirestoreMirrorSource vs RestoreFirestoreBackupSource (adds gcs for session manifests).

2.6.0

• GCS backup: backupData supports mode: "gcs" — writes metadata, native catalogs, optional snapshots as NDJSON under gs://{bucket}/{prefix}{timestamp}/ via @x12i/helpers/gcs, with catalox-backup-manifest.json per run. Default prefix when omitted: catalox-firestore-backups/. CLI: firestore backup --mode gcs --bucket … (--gcs-prefix optional).
• Export: gcsBackupRunFolder helper (for tests and path clarity).
• Docs / report: docs/backup.md updated; gap analysis .reports/gcs-backup-gap-analysis.md.

2.5.0

• GCS: NDJSON export/import for one or many Firestore collections (exportFirestoreCollectionToGcs, exportAllFirestoreCollectionsToGcs, restore helpers, CLI firestore export-gcs / import-gcs). Optional gcsPrefix and objectNamePostfix on object keys; manifest-driven restore-all.
• GCS: Compare live collections to bucket NDJSON — identical / changed / only-in-Firestore / only-in-bucket (compareFirestoreCollectionWithGcsNdjson, manifest mode, CLI firestore compare-gcs). Helpers normalizeForCompare, dataFingerprint exported for tooling.
• Native: listCatalogItems compacts inert filter entries before Firestore queries; NativeItemStore.list can re-resolve flat vs legacy layout after an unconstrained empty first page.
• Docs: docs/native-catalog-storage-and-api.md, docs/firestore-gcs-export.md; README and environment docs updated for GCS.
• Dependency: @google-cloud/storage (^7.19).

Boundaries (important)

• Secrets: do not store secret material (API keys, cloud creds) in Catalox. Store only non-secret refs like credentialsRef.
• Artifacts: store descriptor metadata and remote keys; artifact blobs live in external object storage.