@exellix/jobs-db

Mongo-backed data tier for the Exellix job queue. This is the only package that imports mongodb. It exposes a small JobRunStore interface plus a native-driver implementation whose claim is a single atomic findOneAndUpdate.

A JobRun is one graph run on one input — the durable queue work-unit. It is distinct from a graph task node and from an ai-task (@exellix/ai-tasks). See temp/jobs/ for the full design.

Install

npm install @exellix/jobs-db

Usage

import { createJobRunStore } from '@exellix/jobs-db';

const { store, close } = await createJobRunStore({
  mongoUri: process.env.MONGO_URI,   // or set MONGO_URI in the environment
  // runtimeDb: 'exellix-jobs',       // default
  // configDb:  'exellix',            // default
  // ensureIndexes: true,             // default
});

const run = await store.claim('worker-1');   // atomic claim, or null
await store.close();                          // via close()

For tests, use the in-memory implementation:

import { createMemoryJobRunStore } from '@exellix/jobs-db';
const store = createMemoryJobRunStore();

JobRunStore interface

interface JobRunStore {
  insertMany(runs: JobRun[]): Promise<void>;
  update(id: string, patch: Partial<JobRun>): Promise<void>;
  updateMany(filter: JobRunFilter, patch: Partial<JobRun>): Promise<{ modifiedCount: number }>;

  claim(workerId: string, opts?: ClaimOpts): Promise<JobRun | null>;   // atomic

  get(id: string): Promise<JobRun | null>;
  find(filter: JobRunFilter): Promise<JobRun[]>;
  count(filter: JobRunFilter): Promise<number>;
  hasItem(jobDefId: string, itemId: string): Promise<boolean>;          // dedup

  decrementPendingDeps(id: string): Promise<JobRun | null>;             // atomic
  close(): Promise<void>;
}

Collections

| Collection | DB | Purpose | |------------|-----|---------| | job_runs | runtime (exellix-jobs) | the work-unit; source of truth | | job_defs | config (exellix) | one doc per JobDef | | graphs | config (exellix) | graph authoring JSON / per-graph defaults | | jobs_app_settings | runtime | singleton app settings (maxConcurrentJobs, admission interval) | | job_run_events (optional) | runtime | thin append-only audit log |

App settings (jobs_app_settings)

One document (id: 'default') caps how many job runs are admitted app-wide:

| Field | Default | Role | |-------|---------|------| | maxConcurrentJobs | 10 | Max admitted runs (pending + running, claimable) | | queueAdmissionIntervalMs | 15000 | How often held runs are promoted when capacity frees |

Edited via jobs-ui → Settings (dashboard) or JobsAppSettingsStore.save(). See parallel execution for sizing guidance.

Job context sources (ContextSource)

Work definitions, on-demand runs, and sync graph execute can attach optional context sources — linked-object and same-object Memorix slices resolved before dispatch into jobMemory.context:

| Type | Shape | Role | |------|-------|------| | LinkedObjectContextSource | kind: 'linked', join fields, optional filter, mandatory | Resolve related records (e.g. subnets inferences joined on data.subnetIp) | | SameObjectContextSource | kind: 'same-object', contentTypes[] | Load other content types from the same record id |

Exported types: ContextSource, LinkedObjectContextSource, SameObjectContextSource, ContextFilter, ContextSourceQueryOptions (from context-sources.ts). Resolution logic lives in @exellix/jobs; the dispatcher receives pre-resolved jobMemory only.

The (jobDefId, itemId) index (by_job_item) is intentionally non-unique — one item produces one job run per graph. Dedup is enforced by hasItem at enqueue time, not by a unique constraint.

Why a separate package

@x12i/xronox-store has no atomic conditional update (findOneAndUpdate), which forced the old Execution Matrix into in-process withClaimLock. A durable cross-process queue needs structural atomicity, so the data tier uses the native mongodb driver and is kept out of the queue logic.

Scripts

npm run build      # tsc → dist/
npm test           # unit + live (live runs only when MONGO_URI is set)
npm run test:live  # live Mongo integration tests only

Live tests read MONGO_URI from the environment, falling back to ../../graph-packages/graph-engine/.env. Each run uses an isolated exellix-jobs-live-* database that is dropped on teardown.

License

exellix-license