@semiont/jobs

Job queue, worker infrastructure, and annotation workers for Semiont.

Architecture Context

Workers run in a separate separate process and connect to the Knowledge System (KS) over HTTP/SSE using WorkerStateUnit from @semiont/api-client. Workers receive job assignments via SSE push, claim jobs atomically, and emit domain events back to the KS via HTTP. The KS ingests these events onto its EventBus for SSE delivery to the frontend.

Installation

npm install @semiont/jobs

Dependencies:

• @semiont/core — Core types, EventBus
• @semiont/api-client — OpenAPI types
• @semiont/inference — InferenceClient for AI operations

Quick Start

import { JobQueue, type PendingJob, type GenerationParams } from '@semiont/jobs';
import { EventBus, userId, resourceId, annotationId } from '@semiont/core';
import { jobId } from '@semiont/api-client';

// Initialize
const eventBus = new EventBus();
const jobQueue = new JobQueue({ dataDir: './data' }, logger, eventBus);
await jobQueue.initialize();

// Create a job
const job: PendingJob<GenerationParams> = {
  status: 'pending',
  metadata: {
    id: jobId('job-abc123'),
    type: 'generation',
    userId: userId('[email protected]'),
    userName: 'Jane Doe',
    userEmail: '[email protected]',
    userDomain: 'example.com',
    created: new Date().toISOString(),
    retryCount: 0,
    maxRetries: 3,
  },
  params: {
    referenceId: annotationId('ref-123'),
    sourceResourceId: resourceId('doc-456'),
    sourceResourceName: 'Source Document',
    annotation: { /* full W3C Annotation */ },
    title: 'Generated Article',
    prompt: 'Write about AI',
    language: 'en-US',
  },
};

await jobQueue.createJob(job);

Job Types

type JobType =
  | 'reference-annotation'     // Entity reference detection
  | 'generation'               // AI content generation
  | 'highlight-annotation'     // Key passage highlighting
  | 'assessment-annotation'    // Evaluative assessments
  | 'comment-annotation'       // Explanatory comments
  | 'tag-annotation'           // Structural role tagging

Job Metadata

All jobs share common metadata:

interface JobMetadata {
  id: JobId;
  type: JobType;
  userId: UserId;
  userName: string;       // For building W3C Agent creator
  userEmail: string;      // For building W3C Agent creator
  userDomain: string;     // For building W3C Agent creator
  created: string;
  retryCount: number;
  maxRetries: number;
}

The userName, userEmail, and userDomain fields are used by workers to build the W3C Agent for annotation creator attribution via userToAgent().

Annotation Workers

Six workers process different annotation types:

| Worker | Job Type | Constructor | |--------|----------|------------| | ReferenceAnnotationWorker | reference-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) | | GenerationWorker | generation | (jobQueue, config, inferenceClient, eventBus, logger) | | HighlightAnnotationWorker | highlight-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) | | AssessmentAnnotationWorker | assessment-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) | | CommentAnnotationWorker | comment-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) | | TagAnnotationWorker | tag-annotation | (jobQueue, config, inferenceClient, eventBus, contentFetcher, logger) |

Workers emit EventBus commands (mark:create, job:start, job:complete, etc.) — the Stower actor in @semiont/make-meaning handles persistence.

Custom Workers

import { JobWorker, type AnyJob } from '@semiont/jobs';
import type { Logger } from '@semiont/core';

class MyWorker extends JobWorker {
  constructor(jobQueue: JobQueue, logger: Logger) {
    super(jobQueue, 1000, 5000, logger);
    //              ^^^^  ^^^^
    //              poll   error backoff
  }

  protected getWorkerName(): string {
    return 'MyWorker';
  }

  protected canProcessJob(job: AnyJob): boolean {
    return job.metadata.type === 'generation';
  }

  protected async executeJob(job: AnyJob): Promise<any> {
    // Your processing logic — return result object
  }
}

Discriminated Unions

Jobs use TypeScript discriminated unions for type safety:

function handleJob(job: AnyJob) {
  if (job.status === 'running') {
    console.log(job:progress);    // Available
    // console.log(job.result);   // Compile error
  }
  if (job.status === 'complete') {
    console.log(job.result);      // Available
    // console.log(job:progress); // Compile error
  }
}

Storage Format

Jobs are stored as individual JSON files organized by status:

data/jobs/
  pending/job-abc123.json
  running/job-def456.json
  complete/job-ghi789.json
  failed/job-jkl012.json
  cancelled/job-mno345.json

Documentation

• Job Queue Guide — JobQueue API and job management
• Workers Guide — Building custom workers
• Job Types Guide — All job type definitions
• Type System Guide — Discriminated unions and type safety
• Configuration Guide — Setup and options
• API Reference — Complete API reference

License

Apache-2.0

Related Packages

• @semiont/core — Domain types, EventBus
• @semiont/api-client — OpenAPI types
• @semiont/inference — AI inference client
• @semiont/make-meaning — Actor model, Knowledge Base, service orchestration