@semiont/jobs

Filesystem-based job queue, worker infrastructure, and annotation workers for Semiont.

Architecture Context

In production, the job queue and workers are created by @semiont/make-meaning's startMakeMeaning() function. Workers emit commands on the EventBus — the Stower actor (in @semiont/make-meaning) handles all persistence to the Knowledge Base.

Workers are not actors. They use a polling loop, not RxJS subscriptions. But they emit the same EventBus commands as any other caller in the system.

Installation

npm install @semiont/jobs

Dependencies:

• @semiont/core — Core types, EventBus
• @semiont/api-client — OpenAPI types
• @semiont/inference — InferenceClient for AI operations

Quick Start

import { JobQueue, type PendingJob, type GenerationParams } from '@semiont/jobs';
import { EventBus, userId, resourceId, annotationId } from '@semiont/core';
import { jobId } from '@semiont/api-client';

// Initialize
const eventBus = new EventBus();
const jobQueue = new JobQueue({ dataDir: './data' }, logger, eventBus);
await jobQueue.initialize();

// Create a job
const job: PendingJob<GenerationParams> = {
  status: 'pending',
  metadata: {
    id: jobId('job-abc123'),
    type: 'generation',
    userId: userId('[email protected]'),
    userName: 'Jane Doe',
    userEmail: '[email protected]',
    userDomain: 'example.com',
    created: new Date().toISOString(),
    retryCount: 0,
    maxRetries: 3,
  },
  params: {
    referenceId: annotationId('ref-123'),
    sourceResourceId: resourceId('doc-456'),
    sourceResourceName: 'Source Document',
    annotation: { /* full W3C Annotation */ },
    title: 'Generated Article',
    prompt: 'Write about AI',
    language: 'en-US',
  },
};

await jobQueue.createJob(job);

Job Types

type JobType =
  | 'reference-annotation'     // Entity reference detection
  | 'generation'               // AI content generation
  | 'highlight-annotation'     // Key passage highlighting
  | 'assessment-annotation'    // Evaluative assessments
  | 'comment-annotation'       // Explanatory comments
  | 'tag-annotation'           // Structural role tagging

Job Metadata

All jobs share common metadata:

interface JobMetadata {
  id: JobId;
  type: JobType;
  userId: UserId;
  userName: string;       // For building W3C Agent creator
  userEmail: string;      // For building W3C Agent creator
  userDomain: string;     // For building W3C Agent creator
  created: string;
  retryCount: number;
  maxRetries: number;
}

The userName, userEmail, and userDomain fields are used by workers to build the W3C Agent for annotation creator attribution via userToAgent().

Annotation Workers

Six workers process different annotation types:

| Worker | Job Type | Constructor | |--------|----------|------------| | ReferenceAnnotationWorker | reference-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) | | GenerationWorker | generation | (jobQueue, config, inferenceClient, eventBus, logger) | | HighlightAnnotationWorker | highlight-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) | | AssessmentAnnotationWorker | assessment-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) | | CommentAnnotationWorker | comment-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) | | TagAnnotationWorker | tag-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) |

Workers emit EventBus commands (mark:create, job:start, job:complete, etc.) — the Stower actor in @semiont/make-meaning handles persistence.

Custom Workers

import { JobWorker, type AnyJob } from '@semiont/jobs';
import type { Logger } from '@semiont/core';

class MyWorker extends JobWorker {
  constructor(jobQueue: JobQueue, logger: Logger) {
    super(jobQueue, 1000, 5000, logger);
    //              ^^^^  ^^^^
    //              poll   error backoff
  }

  protected getWorkerName(): string {
    return 'MyWorker';
  }

  protected canProcessJob(job: AnyJob): boolean {
    return job.metadata.type === 'generation';
  }

  protected async executeJob(job: AnyJob): Promise<any> {
    // Your processing logic — return result object
  }
}

Discriminated Unions

Jobs use TypeScript discriminated unions for type safety:

function handleJob(job: AnyJob) {
  if (job.status === 'running') {
    console.log(job.progress);    // Available
    // console.log(job.result);   // Compile error
  }
  if (job.status === 'complete') {
    console.log(job.result);      // Available
    // console.log(job.progress); // Compile error
  }
}

Storage Format

Jobs are stored as individual JSON files organized by status:

data/jobs/
  pending/job-abc123.json
  running/job-def456.json
  complete/job-ghi789.json
  failed/job-jkl012.json
  cancelled/job-mno345.json

Documentation

• Job Queue Guide — JobQueue API and job management
• Workers Guide — Building custom workers
• Job Types Guide — All job type definitions
• Type System Guide — Discriminated unions and type safety
• Configuration Guide — Setup and options
• API Reference — Complete API reference

License

Apache-2.0

Related Packages

• @semiont/core — Domain types, EventBus
• @semiont/api-client — OpenAPI types
• @semiont/inference — AI inference client
• @semiont/make-meaning — Actor model, Knowledge Base, service orchestration