npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2026 – Pkg Stats / Ryan Hefner

@ballistix.digital/task-scheduler

v0.11.0

Published

NestJS module that runs background work as durable task rows behind a pluggable queue engine.

Readme

@ballistix.digital/task-scheduler

A NestJS module that runs background work as durable task rows behind a pluggable queue engine. A task is a database row with a processing status, and the queue only transports the message that names that row. One engine ships, on pg-boss, behind its own entry, so an application installs pg-boss only when it uses that engine.

This file is the consumer guide. The pages that describe how the package is built live in the Ballistix wiki.

The model

The processingStatus column of the row is the lifecycle.

stateDiagram-v2
    classDef waiting fill:#FFD700,stroke:#333,color:#000
    classDef active fill:#87CEEB,stroke:#333,color:#00008B
    classDef done fill:#90EE90,stroke:#333,color:#006400
    classDef failed fill:#FFB6C1,stroke:#DC143C,color:#000

    [*] --> PENDING: trigger() saves the row
    PENDING --> PROCESSING: claim
    PROCESSING --> COMPLETED: handle() returns
    PROCESSING --> PENDING: handle() throws, status >= 500, attempts left
    PROCESSING --> FAILED: handle() throws, status < 500 or attempts exhausted
    COMPLETED --> [*]
    FAILED --> [*]

    class PENDING waiting
    class PROCESSING active
    class COMPLETED done
    class FAILED failed

Read the two exits of PROCESSING: a retry returns the row to PENDING, and every other failure settles it FAILED.

Three mechanisms carry the guarantees:

  • Transactional outbox. The task row and its task message commit in one transaction, or neither commits.
  • Claim. The processor sets PROCESSING under a row lock, only on a PENDING row or on a row a stopped run left (see A run that stops). A repeated delivery loses the claim and stops.
  • Retry guard. A thrown error with status >= 500 sends the row back to PENDING while attempts remain. Any other error settles the row FAILED. The queue owns the retry policy: the guard reads the retryLimit of the message and the processingMaxRetries of the row, and the lower one wins.

One run, step by step

sequenceDiagram
    box rgb(230,230,250) Application
        participant App as Application service
        participant P as ReportTaskProcessor
    end
    box rgb(144,238,144) Engine
        participant E as TaskEngine
    end
    box rgb(211,211,211) Postgres
        participant DB as Postgres
    end

    rect rgb(255,250,205)
        Note over App,DB: 1. trigger: row and message in one transaction
        App->>P: trigger({ reportId }, transaction)
        P->>DB: INSERT report_task (PENDING)
        P->>E: enqueue('report', { id }, transaction)
        E->>DB: INSERT task message (same transaction)
        P-->>App: the saved row
    end

    rect rgb(224,255,224)
        Note over App,DB: 2. delivery: claim, handle, land
        E->>P: deliver { id }, attempt 0
        P->>DB: UPDATE status = PROCESSING, retry count = 0 WHERE id AND status = PENDING
        P->>P: handle(task)
        P->>DB: UPDATE status = COMPLETED WHERE id AND status = PROCESSING AND retry count = 0
        P->>P: afterCompleted(task)
    end

Look at the first block: the row and the message travel on one transaction, so a rollback leaves neither.

Two processors

The difference between the two base classes is who creates the task row.

| | Task processor | Scheduled task processor | | --- | --- | --- | | Base class | AbstractTaskProcessor | AbstractScheduledTaskProcessor | | Who creates the row | Your code, with trigger() | The processor itself, on every tick, with the same trigger() | | When it runs | When a row is triggered | On the cron expression, and when a row is triggered | | Typical use | An upload to parse, an invitation to send | A nightly sync, an hourly cleanup | | Extra members | none | cronExpression, a queueOptions default that adds the singleton policy |

A tick is a message without a task id. The scheduled processor answers it with its own trigger(), and the { id } message that enqueues runs the work like a manual run. The queue of a scheduled processor runs one message at a time across every replica, so a tick never overlaps a manual run.

A third base class, AbstractWorkflowProcessor, moves one workflow row through more than one step. Each visit of a work step is one task row, and an internal task processor runs these rows. See Workflows.

Install

npm install @ballistix.digital/task-scheduler pg-boss

A browser build that reads the ./types entry alone installs the package by itself:

npm install @ballistix.digital/task-scheduler

The package needs Node 22.12 or later and has no dependencies of its own. Every peer dependency is optional, so npm installs none of them for you: install in the application what the entries it imports need.

| Package | Range | Needed by | | --- | --- | --- | | @nestjs/common | ^11 \|\| ^12 | the root entry | | @nestjs/core | ^11 \|\| ^12 | the root entry | | typeorm | >=0.3.0 <2.0.0 | the root entry | | reflect-metadata | ^0.2 | the root entry | | @ballistix.digital/exception-mapper | >=0.3.0 <1.0.0 | the root entry | | @ballistix.digital/exception-types | >=0.4.0 <1.0.0 | the root entry | | pg-boss | ^12 | the ./pg-boss entry |

| Entry | Peers it needs | | --- | --- | | @ballistix.digital/task-scheduler | all of them except pg-boss | | @ballistix.digital/task-scheduler/pg-boss | pg-boss, next to the peers of the root entry | | @ballistix.digital/task-scheduler/testing | the peers of the root entry | | @ballistix.digital/task-scheduler/types | none |

Only the ./pg-boss entry loads pg-boss, so an application with an engine of its own, or a package that only tests against TaskEngineMock, leaves pg-boss out. The ./types entry imports no framework at all: it carries ProcessingStatusEnum and the three workflow enums, so a browser application that shares DTO types with the backend re-exports an enum from that entry instead of copying it.

The two exception packages carry the error envelope and the mappers that build it. A processor stores that envelope in processingError and reads its status to decide the retry. A worker thread maps what its worker function threw through the same mappers and sends the envelope back.

A worker file that runs from its TypeScript source, which is what a Jest suite does, loads in its thread through ts-node/register. Install ts-node as a dev dependency for that, pass other loader flags through WorkerModuleOptions.execArgv, or test with WorkerRunnerMock, which needs no loader. See Run CPU-bound work in a worker thread.

Register the module

import { ExceptionModule } from '@ballistix.digital/exception-mapper';
import { Module } from '@nestjs/common';
import { TaskModule } from '@ballistix.digital/task-scheduler';
import { PgBossEngineService } from '@ballistix.digital/task-scheduler/pg-boss';

@Module({
	imports: [
		ExceptionModule.forRoot(),
		TaskModule.forRoot({
			engine: new PgBossEngineService({
				connection: { host: 'localhost', port: 5432, user: 'app', password: 'secret', database: 'app' },
				schema: 'pgboss',
				poolSize: 5,
			}),
		}),
	],
})
export class AppModule {}

If the options need injection, use TaskModule.forRootAsync({ imports, inject, useFactory }). The factory returns the same options object.

Both modules are global. A processor and a service alike inject TaskRuntime without a further import; nothing else of the package is injectable. TaskModule starts the engine in onModuleInit and stops it in onApplicationShutdown. Nest runs that hook after the HTTP server has drained its in-flight requests, so a request that is still running can enqueue until the end. Call app.enableShutdownHooks() in the bootstrap of the application: without it a SIGTERM ends the process at once and neither the drain nor the stop happens. A test of the application swaps the engine by overriding the TASK_ENGINE token the module provides it under; see Swap the engine in tests.

The engine keeps its own pool and its own schema, apart from the TypeORM connection of the application. It reads no environment variables. The pg-boss engine page gives the other options.

Give the task a row

Compose ProcessableTaskMixin onto the entity base class of the application:

import { ProcessableTaskMixin } from '@ballistix.digital/task-scheduler';
import { BaseEntity, Column, Entity, PrimaryGeneratedColumn } from 'typeorm';

@Entity({ name: 'report_task' })
export class ReportTask extends ProcessableTaskMixin(BaseEntity) {
	@PrimaryGeneratedColumn('uuid')
	id: string;

	@Column({ type: 'uuid', nullable: false })
	reportId: string;
}

The mixin contributes five columns:

| Column | Type | Default | | --- | --- | --- | | processingStatus | enum processing_status, indexed | PENDING | | processingError | jsonb, nullable | null | | processingRetryCount | int | 0 | | processingMaxRetries | int | 5 (DEFAULT_RETRY_LIMIT) | | processingDuration | int, nullable, milliseconds | null |

processingError holds an ExceptionDto. Import that type from @ballistix.digital/exception-types: this package does not re-export it.

Every entity on the mixin shares the one Postgres enum type processing_status. Generate a migration for the new table.

Write a task processor

import { GenericBadRequestException } from '@ballistix.digital/exception-types';
import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { AbstractTaskProcessor, TaskRuntime } from '@ballistix.digital/task-scheduler';
import { Repository } from 'typeorm';
import { ReportTask } from './report.task.entity';
import { RenderService } from './render.service';

@Injectable()
export class ReportTaskProcessor extends AbstractTaskProcessor<ReportTask> {
	protected readonly queue = 'report';

	constructor(
		taskRuntime: TaskRuntime,
		@InjectRepository(ReportTask) reportTaskRepository: Repository<ReportTask>,
		private readonly renderService: RenderService,
	) {
		super(taskRuntime, reportTaskRepository);
	}

	/** A 400 settles the task on the first attempt: the input cannot become valid on a retry. */
	protected async handle(task: ReportTask): Promise<void> {
		if (!task.reportId) {
			throw new GenericBadRequestException('Task carries no reportId');
		}

		await this.renderService.render(task.reportId);
	}

	/** Runs after COMPLETED has committed. A failure here never reverts COMPLETED. */
	protected async afterCompleted(task: ReportTask): Promise<void> {
		await this.renderService.publish(task.reportId);
	}
}

Register the class as a provider of the owning module. On bootstrap the base creates the queue with queueOptions and subscribes to it.

Queue options

The queue is the one place that holds the retry policy. A processor declares retryLimit, retryDelaySeconds and retryBackoff in queueOptions, the engine applies them on bootstrap, and every message on the queue inherits them: the message of trigger(), the message of a schedule tick, and the message a service enqueues for a row it saved itself. No enqueue of the package passes retry options of its own, so every run on one queue retries the same way.

A processor that declares nothing keeps the default of the base, DEFAULT_QUEUE_OPTIONS: retryLimit: 5 (DEFAULT_RETRY_LIMIT) with retryBackoff: true. A processor that must not repeat a run declares retryLimit: 0: the first throw settles the row FAILED. A processor whose work needs a pause before the next attempt adds retryDelaySeconds:

protected readonly queueOptions: QueueOptions = {
	...DEFAULT_QUEUE_OPTIONS,
	retryDelaySeconds: 30,
};

Every start reconciles the queue with queueOptions. A changed expireInSeconds, retryLimit, retryDelaySeconds or retryBackoff reaches a queue an earlier release created. A changed policy does not: a policy is fixed when the queue is created, so the engine keeps the queue and logs a warning that names both policies. Give a processor that needs another policy a new queue name.

The enqueue option startAfterSeconds holds one message back for at least that many seconds after the enqueue. trigger() takes it, and retryLimit for a row that must differ from its queue, as its third argument (see Trigger a run from a service).

super() takes two arguments. The first is the TaskRuntime the module provides. Pass it up and forget it: the type carries enqueue() and nothing else, the class behind it is not exported, and a new dependency of the base changes that class and no processor. The second is the repository. Name the parameter after the entity, such as reportTaskRepository. If the subclass needs the repository itself, keep it in a private field with that name.

You implement handle(). You can also override:

| Member | Default | Override when | | --- | --- | --- | | buildTask(input) | repository.create(input ?? {}) | the row needs values the entity defaults cannot give | | afterCompleted(task) | nothing | work must follow a committed COMPLETED, such as a notification | | afterFailed(task) | nothing | work must follow a committed terminal FAILED, such as an alert | | buildTelemetryTags(task) | {} | the telemetry line of the task must carry more tags (see Telemetry) |

handle() runs outside any held transaction, so a slow run holds no row lock. It opens its own transactions when it needs them. The queue is a plain string. Keep the queue names of the application in one enum.

Hook names

The prefix of a hook tells when it runs and in which transaction:

| Prefix | Runs | Transaction | A throw does | | --- | --- | --- | --- | | build* | Before a save, such as buildTask() in trigger(). It builds a value and writes nothing | The transaction of the caller | Rolls back the transaction of the caller | | beforeCommit* | Inside the transaction that lands a terminal FAILED, before its commit. The one hook is beforeCommitFailed(). A COMPLETED landing runs none | The landing transaction | Rolls back the landing | | after* | After the commit, best effort | None | Writes one log line. The row keeps its status |

Only database writes belong in beforeCommitFailed(): a throw rolls the landing back, and the row stays PROCESSING. The workflow processor uses it to set the workflow FAILED in the same commit. No hook name starts with on. buildTelemetryTags() also builds a value and writes nothing, but it runs after the commit, when the telemetry line is written.

Write a scheduled task processor

The constructor follows the task processor above.

@Injectable()
export class AfasSyncTaskProcessor extends AbstractScheduledTaskProcessor<AfasSyncTask> {
	protected readonly queue = 'afas-sync';
	protected readonly cronExpression = '5 */6 * * *';

	/** This sync must not repeat a partial run, so the queue allows no retry. */
	protected readonly queueOptions: QueueOptions = { policy: 'singleton', expireInSeconds: 1800, retryLimit: 0 };

	protected async handle(task: AfasSyncTask): Promise<void> {
		await this.afasService.syncCustomers();
		await this.afasService.syncProjects();
	}
}

A scheduled processor is a task processor plus cronExpression. Cron expressions are UTC. The base creates the queue with the singleton policy, so one run is active at a time across every replica.

queueOptions holds the time budget of one run (expireInSeconds, default 600) and the retry policy of the queue, which starts from DEFAULT_QUEUE_OPTIONS. A tick and a manual run inherit the same policy. The example gives the queue a budget of 30 minutes and no retry: retryLimit: 0 settles every run FAILED on its first throw, whatever processingMaxRetries the row carries.

A tick creates the row with trigger() and no input. If the entity has a column the tick cannot leave empty, override buildTask().

Schedules tick in every environment. Keep a processor safe to run from staging through its own configuration, not through the absence of a schedule.

Trigger a run from a service

@Injectable()
export class ReportService {
	constructor(
		private readonly dataSource: DataSource,
		private readonly processor: ReportTaskProcessor,
	) {}

	public async requestReport(reportId: string): Promise<ReportTask> {
		return this.dataSource.transaction(async (transaction) => {
			await transaction.update(Report, reportId, { requestedAt: new Date() });

			return this.processor.trigger({ reportId }, transaction);
		});
	}
}

trigger(input, transaction) saves the row and enqueues { id } on the same transaction. Without a transaction the processor opens one. A rolled-back transaction leaves no row and no task message. The message inherits the retry policy of the queue. A third argument, { startAfterSeconds?, retryLimit? }, delays the first delivery or sets another retry limit for this one message:

await this.processor.trigger({ reportId }, transaction, { startAfterSeconds: 60 });

trigger() returns the saved row, so the caller can poll processingStatus and processingError. A manual trigger() on a scheduled processor works the same way, because the schedule and the manual runs share the one subscriber.

A processor never reads an application authentication context. Put audit values in the trigger() input.

Enqueue a row you saved yourself

A service that owns the row, such as an upload flow that marks an attachment PENDING, enqueues the message itself through TaskRuntime:

@Injectable()
export class UploadService {
	constructor(private readonly taskRuntime: TaskRuntime) {}

	public async onUploadReady(upload: Upload, transaction: EntityManager): Promise<void> {
		await transaction.update(Upload, upload.id, { processingStatus: ProcessingStatusEnum.PENDING });
		await this.taskRuntime.enqueue('upload', { id: upload.id }, transaction, { dedupeKey: upload.id });
	}
}

enqueue() is the one call TaskRuntime exposes. The message rides the transaction you pass, so the row and the message commit together. Pass a dedupeKey when the policy of the queue needs one, and no retry options: the message inherits the retry policy of the queue, the same as a trigger(). The consumer surface diagram shows what sits behind the token and why you cannot reach it.

Errors and retries

When handle() throws, the ExceptionMapperRegistry normalizes the thrown value into an ExceptionDto. The processor stores that envelope in processingError, and the status of the envelope decides what happens next.

flowchart TD
    T["handle() throws"] --> N["error = exceptionMapperRegistry.toExceptionDto(thrown)"]
    N --> S{"error.status >= 500?"}
    S -- no --> F["FAILED, processingError stored, no rethrow, telemetry line"]
    S -- yes --> A{"attempt < min(message.retryLimit, processingMaxRetries)?"}
    A -- no --> F
    A -- yes --> R["PENDING, processingError stored, rethrow so the engine retries with backoff"]

    classDef decision fill:#FFD700,stroke:#333,stroke-width:2px,color:#000
    classDef retry fill:#87CEEB,stroke:#333,stroke-width:2px,color:#00008B
    classDef terminal fill:#FFB6C1,stroke:#DC143C,stroke-width:2px,color:#000
    classDef step fill:#F5F5F5,stroke:#333,stroke-width:1px,color:#000
    class S,A decision
    class R retry
    class F terminal
    class T,N step

Look at the first decision: the registry normalizes every thrown value, so the guard reads one field whatever the code threw.

If the code matters, throw a BaseException of the types package. Otherwise throw any value that carries a status:

| Thrown value | Stored ExceptionDto | Retried? | | --- | --- | --- | | new GenericBadRequestException('bad input') | title: 'bad input', status: 400, code: 'GENERIC_ERROR' | no, the input is wrong | | new QueryFailedError(...) with SQLSTATE 23505 | status: 400, code: 'DATABASE_ERROR' | no, the statement is wrong | | new QueryFailedError(...) with SQLSTATE 40P01 | status: 500, code: 'DATABASE_ERROR' | yes, while attempts remain | | { message, status: 400 } | title: message, status: 400, code: 'GENERIC_ERROR' | no, the input is wrong | | { message, status: 503 } | title: message, status: 503, code: 'GENERIC_ERROR' | yes, while attempts remain | | new Error('boom') | title: 'boom', status: 500, code: 'GENERIC_ERROR' | yes, while attempts remain | | 'a string' | title: 'a string', status: 500 | yes, while attempts remain |

The registry tries its mappers in order and the generic mapper is last. A thrown value that names no code therefore becomes GENERIC_ERROR, and one errors detail carries the message.

The queue decides the retry budget through its retryLimit, and the engine stamps that limit on every message it delivers. processingMaxRetries on the row is a row-level budget that can only lower the budget of the queue: the guard retries while attempt is below both numbers. A row that asks for more retries than its queue allows gets the retries of the queue, because the engine stops redelivering at its own limit and a row sent back to PENDING past that limit would wait for a message that never comes. Both numbers default to 5 (DEFAULT_RETRY_LIMIT).

processingRetryCount records how many attempts ran: the claim writes the attempt of the delivery, and a failed attempt writes the attempt plus one. A row that is triggered again gets a message at attempt 0, so it reads 0 while that run works. processingDuration records how long the last attempt of handle() ran, in milliseconds, whatever its outcome. The engine schedules the retries with the delay and the backoff of the queue. The row is the durable record you read.

A run that stops

A run can stop in the middle: the process crashes, or handle() runs past the expireInSeconds of the queue. The row then stays in PROCESSING, and the engine delivers the message again with the next attempt. The claim takes over a PROCESSING row when its processingRetryCount is lower than the attempt of the delivery, and logs one warning. The claim is one transaction that locks the row first, so a landing of the old run that commits while the claim waits is what the claim decides on: a retry the old run lands makes the row PENDING, and the new delivery claims it.

Every landing is conditional on PROCESSING and the attempt that claimed the row. When the row moved on, a landing changes nothing: the result of that run is dropped and one warning names the task, the queue and the attempt. A caller that resets a row to PENDING during a run therefore wins over the landing of that run.

These limits apply:

  • A message without retries left, or any message on a queue with retryLimit: 0, gets no new delivery. Its row stays in PROCESSING.
  • expireInSeconds must be longer than the longest run of handle(). Otherwise the engine delivers the message again while the first run still works, and the two runs overlap.
  • A row you reset to PENDING during a run is safe only when its new run claims after the old run landed. When the new run claims first at the same attempt, the old run lands its result on the row and the new run is dropped.

Workflows

AbstractWorkflowProcessor runs a flow of more than one durable step: create a document at a provider, send it, wait for the signatures, download the result. A workflow is a sequence of tasks. One workflow row holds one workflow instance, and each visit of a work step is one workflow step: a task row on the queue of the processor.

Use a workflow processor when the work has more than one step and one of these is true:

  • A step calls a provider, and the next step must not start before that call is recorded.
  • The flow waits for something outside: a webhook, a user action, a time.
  • A user or a provider can stop the flow while it runs, such as a cancel.

A workflow does not start other workflows and does not wait for them. For N items, use one step that loops over them.

The words

| Word | Meaning | | --- | --- | | workflow row | The row of one workflow instance, an entity with WorkflowMixin. Column workflowStepName says where it is | | work step | A step where your code runs: .work(step, { next, handle }) | | wait step | A step where nothing runs. A signal or its timer moves the row on: .wait(step, { on, after }) | | terminal step | The end. Nothing moves the row any more: .terminal(step) | | workflow step | The task row of one visit of a work step, an entity with WorkflowStepMixin. Wait and terminal steps get none | | current workflow step | The one workflow step that can move the row: column currentWorkflowStepId | | signal | A fact from outside that the application sends with signal() | | timer | The after of a wait step: a delayed workflow step | | patch | The columns of the application that a step or a signal writes with its move |

The status of the workflow row (WorkflowStatusEnum) says how the row goes:

| Status | Meaning | | --- | --- | | RUNNING | The row is in a work step. Its workflow step is pending or runs | | WAITING | The row is in a wait step and waits for a signal or its timer | | COMPLETED | The row is in a terminal step | | FAILED | The workflow step of the current work step failed terminally |

REJECTED in the example below is a terminal step: the business outcome is a refusal, and the status of the row is COMPLETED.

Give the workflow its two tables

import { WorkflowMixin, WorkflowStepMixin } from '@ballistix.digital/task-scheduler';
import { BaseEntity, Column, Entity, PrimaryGeneratedColumn } from 'typeorm';

@Entity({ name: 'esign_session' })
export class ESignSession extends WorkflowMixin(BaseEntity) {
	@PrimaryGeneratedColumn('uuid')
	id: string;

	@Column({ type: 'varchar', length: 255, nullable: true })
	providerDocumentId: string | null;

	@Column({ type: 'timestamptz', nullable: true })
	lastWebhookAt: Date | null;
}

@Entity({ name: 'esign_step' })
export class ESignStep extends WorkflowStepMixin(BaseEntity) {
	@PrimaryGeneratedColumn('uuid')
	id: string;
}

Each entity declares its own primary key. WorkflowMixin adds four columns:

| Column | Type | Default | | --- | --- | --- | | workflowStepName | text, indexed | none, trigger() sets it | | status | enum workflow_status, indexed | RUNNING | | currentWorkflowStepId | uuid, nullable | null | | workflowStepEnteredAt | timestamptz | now() |

WorkflowStepMixin adds the five columns of ProcessableTaskMixin and four more:

| Column | Type | Default | | --- | --- | --- | | workflowId | uuid, indexed, no foreign key | none | | workflowStepName | text | none | | enteredBy | text: START, NEXT, SIGNAL, TIMER | none | | signal | text, nullable. Set when enteredBy is SIGNAL | null |

The workflow steps of a row are its history: one row per visit, with the cause of the visit and the outcome of its attempts. Generate one migration. It creates the enum type workflow_status once, and the two tables. workflowStepName, enteredBy and signal are text, so a new step needs no ALTER TYPE.

Define the steps

WorkflowDefinitionBuilder takes three types: the workflow entity, the step enum and the signal enum. Its constructor takes the step enum object and the signal enum object. A workflow without signals leaves both out.

| Method | Meaning | | --- | --- | | initialWorkflowStep(step) | The step a new row starts in, of any kind. A wait step makes the row wait first | | work(step, config) | Declares a work step | | wait(step, config) | Declares a wait step | | terminal(step) | Declares a terminal step | | build() | Checks the definition and returns it |

| Config key | Of | Meaning | | --- | --- | --- | | next | work | Every step handle can return | | handle | work | (row) => Promise<{ next, patch? }>. It runs outside any transaction | | retryLimit | work | The retry limit of the workflow steps of this step. Left out, the limit of the queue applies. The processingMaxRetries of the workflow step still caps it | | on | work, wait | Signal to step. On a work step, only while its workflow step is PENDING | | onFailed | work | Signal to step, only while the row is FAILED in this step | | after | wait | { seconds, next }: the timer. After seconds, the work step next runs |

build() throws one error that lists every fault, one per line: no initialWorkflowStep or one that is not declared, a value of the step enum without a declaration, and a target of next, on, onFailed or after.next that is no declared step, and an after.next that is not a work step. A second declaration of a step replaces the first. The definition is a property of the processor, so a wrong definition fails when the processor is constructed.

The names of the steps and the signals are free: the package checks none. The examples use this naming, because it makes a definition easy to read:

  • A work step is a verb with an object: SEND_DOCUMENT, SYNC_SIGNATURES.
  • A wait step starts with AWAIT_ and names what it waits for: AWAIT_SIGNATURES.
  • A terminal step is the outcome, in the past tense: COMPLETED, CANCELLED.
  • A signal is a fact: WEBHOOK_RECEIVED, CANCEL_REQUESTED.

Write a workflow processor

The e-sign example: CREATE_DOCUMENT creates the document at the provider. When the provider needs a callback first, the row waits in AWAIT_PROVIDER, and SEND_DOCUMENT sends the document after the webhook. In AWAIT_SIGNATURES the row waits for the signers. Each webhook moves it to SYNC_SIGNATURES, which reads the state at the provider. The timer of one hour is a fallback poll for a lost webhook. When not all signers have signed, SYNC_SIGNATURES returns to AWAIT_SIGNATURES, and a new timer starts.

@Injectable()
export class ESignProcessor extends AbstractWorkflowProcessor<ESignSession, ESignStep, ESignStepEnum, ESignSignalEnum> {
	protected readonly queue = 'esign';
	protected readonly queueOptions: QueueOptions = { ...DEFAULT_QUEUE_OPTIONS, expireInSeconds: 300 };

	protected readonly definition = new WorkflowDefinitionBuilder<ESignSession, ESignStepEnum, ESignSignalEnum>(
		ESignStepEnum,
		ESignSignalEnum,
	)
		.initialWorkflowStep(ESignStepEnum.CREATE_DOCUMENT)
		.work(ESignStepEnum.CREATE_DOCUMENT, {
			next: [ESignStepEnum.AWAIT_PROVIDER, ESignStepEnum.AWAIT_SIGNATURES],
			handle: (session) => this.createDocument(session),
		})
		.wait(ESignStepEnum.AWAIT_PROVIDER, {
			on: {
				[ESignSignalEnum.WEBHOOK_RECEIVED]: ESignStepEnum.SEND_DOCUMENT,
				[ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT,
			},
		})
		.work(ESignStepEnum.SEND_DOCUMENT, {
			next: [ESignStepEnum.AWAIT_SIGNATURES],
			handle: (session) => this.sendDocument(session),
			on: { [ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT },
		})
		.wait(ESignStepEnum.AWAIT_SIGNATURES, {
			on: {
				[ESignSignalEnum.WEBHOOK_RECEIVED]: ESignStepEnum.SYNC_SIGNATURES,
				[ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT,
			},
			after: { seconds: 3600, next: ESignStepEnum.SYNC_SIGNATURES },
		})
		.work(ESignStepEnum.SYNC_SIGNATURES, {
			next: [ESignStepEnum.AWAIT_SIGNATURES, ESignStepEnum.DOWNLOAD_DOCUMENT, ESignStepEnum.REJECTED],
			handle: (session) => this.syncSignatures(session),
			onFailed: {
				[ESignSignalEnum.RETRY_REQUESTED]: ESignStepEnum.SYNC_SIGNATURES,
				[ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT,
			},
		})
		.work(ESignStepEnum.DOWNLOAD_DOCUMENT, {
			next: [ESignStepEnum.COMPLETED],
			handle: (session) => this.downloadDocument(session),
			on: { [ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT },
		})
		.work(ESignStepEnum.REVOKE_DOCUMENT, {
			next: [ESignStepEnum.CANCELLED],
			handle: (session) => this.revokeDocument(session),
		})
		.terminal(ESignStepEnum.COMPLETED)
		.terminal(ESignStepEnum.REJECTED)
		.terminal(ESignStepEnum.CANCELLED)
		.build();

	constructor(
		taskRuntime: TaskRuntime,
		@InjectRepository(ESignSession) eSignSessionRepository: Repository<ESignSession>,
		@InjectRepository(ESignStep) eSignStepRepository: Repository<ESignStep>,
		private readonly nitro: NitroClient,
		private readonly invitationService: InvitationService,
	) {
		super(taskRuntime, eSignSessionRepository, eSignStepRepository);
	}

	/** Runs after the commit that sets the session FAILED. A throw is logged, and the session stays FAILED. */
	protected async afterFailed(session: ESignSession): Promise<void> {
		await this.invitationService.markSigningError(session.id);
	}

	private async createDocument(session: ESignSession): Promise<Transition<ESignSession, ESignStepEnum>> {
		// The session id is the idempotency key: a second run gets the document of the first run.
		const document = await this.nitro.createDocument({ idempotencyKey: session.id });

		return {
			next: document.ready ? ESignStepEnum.AWAIT_SIGNATURES : ESignStepEnum.AWAIT_PROVIDER,
			patch: { providerDocumentId: document.id },
		};
	}

	private async syncSignatures(session: ESignSession): Promise<Transition<ESignSession, ESignStepEnum>> {
		const state = await this.nitro.getDocument(session.providerDocumentId);
		if (state.refused) {
			return { next: ESignStepEnum.REJECTED };
		}

		return { next: state.signed === state.signers ? ESignStepEnum.DOWNLOAD_DOCUMENT : ESignStepEnum.AWAIT_SIGNATURES };
	}

	// sendDocument, downloadDocument and revokeDocument follow the same shape.
}

super() takes three arguments: the TaskRuntime, the repository of the workflow entity and the repository of the workflow step entity. Register the two entities with TypeOrmModule.forFeature and the processor as a provider. On bootstrap the base creates the queue of the workflow steps with queueOptions and subscribes to it. Give the handlers names of their own: trigger and signal are public methods of the base.

| Member | Purpose | | --- | --- | | queue | Abstract. The queue of the workflow steps | | definition | Abstract. The steps, built with WorkflowDefinitionBuilder | | queueOptions | The time budget and the retry policy of every workflow step, from DEFAULT_QUEUE_OPTIONS | | trigger(input?, transaction?) | Saves the row in initialWorkflowStep and creates its first workflow step, in one commit. Without a transaction, it opens one. Returns the saved row | | signal(id, signal, patch?, transaction?) | Hands a signal to the row. Without a transaction, it opens one. Throws when the row is missing or its workflow step runs | | afterCompleted(row) | Protected hook. Runs after the commit that moves the row into a terminal step. Best effort | | afterFailed(row) | Protected hook. Runs after the commit that sets the row FAILED. Best effort |

The hooks follow the hook names of the task processor. A throw in afterCompleted() or afterFailed() writes one log line, and the row keeps its status. afterCompleted() runs after a workflow step moves the row into a terminal step. A signal that moves the row into a terminal step runs no hook: the caller acts after signal() returns.

The workflow processor is no task processor itself. It holds an internal task processor that runs the workflow steps, so cronExpression, buildTask() and the task hooks are not members of a workflow processor.

How a workflow step runs

A workflow step runs as a task (see The model): claim, run, land, with the retry guard and the takeover. Its run does this:

  1. If the workflow step is not the current workflow step, it does nothing and lands COMPLETED.
  2. If the row is WAITING, the workflow step is a timer. It moves the row into its step first.
  3. The handle of the step runs, outside any transaction.
  4. If next is not in the next list, the run throws a GenericErrorException with status 422 (code GENERIC_ERROR). The retry guard does not retry it.
  5. The move: one transaction locks the row, checks the guard again, and writes the next step, the patch and the next workflow step. A wait step gets WAITING, and its timer if it has after. A terminal step gets COMPLETED.
  6. After that commit, afterCompleted(row) runs when the row entered a terminal step.
  7. The landing: a separate transaction lands the workflow step COMPLETED. If the process stops between the two commits, the takeover runs the workflow step again. The guard sees that it is no longer current, so it does nothing and lands COMPLETED.

The guard. A workflow step moves its row only while currentWorkflowStepId is its id. A late, repeated or superseded workflow step therefore changes nothing. A signal or a timer supersedes the workflow step it replaces.

Each step must be idempotent. The process can stop after a provider call and before the move commits, and the step then runs again. Send an idempotency key, such as the row id, or look up your own reference before a create. handle does not write to the database itself: it returns a patch.

The timer. Entering a wait step with after creates the workflow step of after.next, delayed by after.seconds, as the current workflow step. A signal that moves the row first makes that timer stale. If the timer runs first, it moves the row into after.next. No periodic schedule exists.

A failure keeps the step. When a workflow step fails terminally, one transaction lands it FAILED and sets the row FAILED, while that workflow step is still current. After the commit, afterFailed(row) runs. When that transaction meets a database error, it rolls back, and the workflow step stays PROCESSING: the rules of A run that stops apply. The failed workflow step stays as history.

A retry is a signal. The onFailed table lets a signal move a FAILED row. To run a failed step again, add a signal that points back to the same step, as SYNC_SIGNATURES does in the example:

onFailed: {
	[ESignSignalEnum.RETRY_REQUESTED]: ESignStepEnum.SYNC_SIGNATURES,
	[ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT,
},

signal(id, RETRY_REQUESTED) then creates a new workflow step for SYNC_SIGNATURES with enteredBy SIGNAL and sets the row RUNNING. The same table moves the row on in other ways: the provider is down, SYNC_SIGNATURES fails, and the user clicks cancel. CANCEL_REQUESTED moves the row to REVOKE_DOCUMENT without a retry first.

Send a signal

@Post('nitro/webhook/:sessionId')
@HttpCode(200)
async onNitroWebhook(@Param('sessionId', ParseUUIDPipe) sessionId: string): Promise<void> {
	await this.eSignProcessor.signal(sessionId, ESignSignalEnum.WEBHOOK_RECEIVED, { lastWebhookAt: new Date() });
}

The controller checks nothing. signal() throws an exception of @ballistix.digital/exception-types, and the exception mapper turns it into the HTTP answer.

The controller does not know the step of the row: the definition decides. The same WEBHOOK_RECEIVED moves AWAIT_PROVIDER to SEND_DOCUMENT and AWAIT_SIGNATURES to SYNC_SIGNATURES. The webhook says "go and look", and SYNC_SIGNATURES reads the real state, so two webhooks in the wrong order do no harm.

signal() locks the row (SELECT ... FOR UPDATE). With the transaction of the caller, the lock holds until the caller commits, so a webhook can commit its own rows and the signal together. The signal routes by what it reads:

| Row | Result | | --- | --- | | No row with this id | Throws ResourceNotFoundException: 404, code RESOURCE_NOT_FOUND | | The current workflow step is PROCESSING or a claim holds its lock, also a timer | Throws PreconditionFailedException: 409, code PRECONDITION_FAILED. Nothing changes | | WAITING, the signal is in the on of the wait step | Moves the row | | RUNNING, the workflow step is PENDING, the signal is in the on of the work step | Moves the row | | FAILED, the signal is in the onFailed of the step | Moves the row | | Any other case, such as a terminal step | Returns, with one log line. Nothing changes |

A move writes the patch, the next step and the next workflow step with enteredBy SIGNAL. The workflow step it supersedes stays PENDING until it arrives, and then lands COMPLETED without a run. A signal that throws or that the step does not take writes no patch. If the data must stay in every case, save it before you call signal().

The 404 and the 409 tell the provider to send the webhook again later. A signal that the step does not take gives a 200. A signal that is not in the signal enum throws a GenericErrorException with status 422 (code GENERIC_ERROR) before the transaction opens.

WorkflowPatch<TEntity> is the type of a patch: the columns of the entity without the workflow columns and id.

Limits

  • expireInSeconds must be longer than the longest step. The limits of A run that stops apply to every workflow step.
  • Keep a removed step in the definition until no workflow row uses it.
  • One subscription runs one message at a time per instance. A slow step of one row delays the other rows on that instance.

Test a workflow

Boot the processor on a TaskEngineMock, as in Test a processor. deliverAll(queue) delivers until no workflow step is due, so the row waits or ends. A timer is a delayed message, so only includeDelayed: true delivers it. The mock does not return the row: read it back through the repository.

it('waits for the signatures, then polls through SYNC_SIGNATURES', async () => {
	const session = await processor.trigger();
	await engine.deliverAll('esign');
	expect(await sessionRepository.findOneByOrFail({ id: session.id })).toMatchObject({
		workflowStepName: ESignStepEnum.AWAIT_SIGNATURES,
		status: WorkflowStatusEnum.WAITING,
	});

	// The timer of AWAIT_SIGNATURES is the only delayed workflow step.
	await engine.deliverNext('esign', { includeDelayed: true });
	await engine.deliverAll('esign');

	// The stubbed provider reports both signatures, so SYNC_SIGNATURES and DOWNLOAD_DOCUMENT end the flow.
	expect(await sessionRepository.findOneByOrFail({ id: session.id })).toMatchObject({
		workflowStepName: ESignStepEnum.COMPLETED,
		status: WorkflowStatusEnum.COMPLETED,
	});
});

Fire a timer with deliverNext(), not with deliverAll(..., { includeDelayed: true }). The loop from SYNC_SIGNATURES back to AWAIT_SIGNATURES creates a new timer on every visit, so that call runs the loop forever.

Run CPU-bound work in a worker thread

Node runs JavaScript on one thread. A render of a document or an email blocks the event loop for its full duration, and while it runs the API answers nothing. WorkerRunnerService runs one function in a thread of its own, and the caller awaits the result the way it awaits any promise.

flowchart LR
    S["service or handle()<br/>workerRunner.run(__dirname, 'render.worker', input)"] --> R["WorkerRunnerService<br/>one thread per run"]
    R --> B["thread<br/>loads the worker file,<br/>calls its default export"]
    B -->|"result"| R
    B -->|"error envelope"| R
    R -->|"resolve"| OK["the result"]
    R -->|"reject"| E["WorkerException<br/>status, code, details kept"]

    classDef main fill:#87CEEB,stroke:#333,stroke-width:2px,color:#00008B
    classDef thread fill:#FFD700,stroke:#333,stroke-width:2px,color:#000
    classDef done fill:#90EE90,stroke:#333,color:#006400
    classDef failed fill:#FFB6C1,stroke:#DC143C,color:#000
    class S,R main
    class B thread
    class OK done
    class E failed

Register the module once, at the root. It is global, so any feature module injects the runner without importing anything.

import { WorkerModule } from '@ballistix.digital/task-scheduler';

WorkerModule.forRootAsync({
	inject: [ConfigService],
	useFactory: (config: ConfigService<EnvModel>) => ({
		timeoutSeconds: config.get('WORKER_TIMEOUT_SECONDS'),
	}),
});

timeoutSeconds is the timeout of every run that passes none of its own. The default is 300. execArgv is the list of flags a thread starts with when it loads a .ts worker file; the default is ['-r', 'ts-node/register'].

Write a worker file

A worker file is a plain module next to the service that calls it, named <name>.worker.ts. It exports the worker function as its default export, and it imports nothing from this package. The function takes one input and returns one result. Both cross the thread as structured clones, so a class instance arrives as a plain object and a Node Buffer arrives as a Uint8Array.

// render.worker.ts
export interface RenderInput {
	template: Uint8Array;
	data: Record<string, unknown>;
}

export interface RenderOutput {
	file: Uint8Array;
}

export default async function render(input: RenderInput): Promise<RenderOutput> {
	const file = renderWithSomeLibrary(input.template, input.data);

	return { file };
}

// Optional: the buffers of the result to move instead of copy.
export const transfer = (output: RenderOutput): ArrayBuffer[] => [output.file.buffer as ArrayBuffer];

The worker function is a pure function of its input: no Nest injection, no TypeORM, no request context. Everything else, the database reads, the blob storage and the task lifecycle, stays on the main thread. Export the input and the output types, so the caller imports the types and nothing else from the file.

Call the runner

import { WorkerRunnerService } from '@ballistix.digital/task-scheduler';
import type { RenderInput, RenderOutput } from './render.worker';

@Injectable()
export class ReportService {
	constructor(private readonly workerRunner: WorkerRunnerService) {}

	async render(template: Uint8Array, data: Record<string, unknown>): Promise<Uint8Array> {
		const output = await this.workerRunner.run<RenderInput, RenderOutput>(
			__dirname,
			'render.worker',
			{ template, data },
			{ timeoutSeconds: 120, transfer: [template.buffer as ArrayBuffer] },
		);

		return output.file;
	}
}

run(dir, workerName, input, options?) finds <workerName>.ts in dir when the source exists on disk and <workerName>.js otherwise, so the same call works under a test runner and from dist in production. Pass __dirname, and the worker file sits next to the caller in both trees.

A run that passes timeoutSeconds wins over the value of the module. A run that lists buffers in transfer moves them to the thread instead of copying them: the buffer has byte length 0 on the calling side afterwards. Only a buffer that owns its ArrayBuffer can move. A small Node Buffer shares a pooled slab with others, so copy it out first, with Buffer.allocUnsafeSlow or new Uint8Array(buffer).

A task processor calls the runner from handle() the same way. The row, the claim and the retry stay on the main thread; only the CPU-bound call moves.

What a failure becomes

The thread maps what the worker function threw through the exception mapper registry, the same mappers a processor uses, and posts the envelope. The run rejects with a WorkerException, a BaseException restored from that envelope, with the stack the error had in the thread. So the retry guard reads a worker failure like any other: a 400 lands FAILED, a 500 retries.

| What happened in the thread | The run rejects with | Status, code | | --- | --- | --- | | The worker function threw a BaseException | WorkerException | its own status and code, its details kept | | The worker function threw a class-validator ValidationException | WorkerException | 400, VALIDATION_FAILED | | The worker function threw anything else | WorkerException | 500, GENERIC_ERROR, the message as title | | An error escaped the promise chain, from a timer for example | WorkerException | as above, by what was thrown | | The thread could not start, for example on a buffer that cannot move | GenericErrorException | 500, GENERIC_ERROR, Worker <name> could not start: <cause> | | The run timed out | GenericErrorException | 500, GENERIC_ERROR, Worker <name> timed out after <n> s | | The thread exited before it posted a result | GenericErrorException | 500, GENERIC_ERROR, Worker <name> exited with code <code> before it posted a result | | The application shut down during the run | GenericErrorException | 500, GENERIC_ERROR, Worker <name> was terminated because the runner is shut down |

The run settles once, on the first of those events, and the runner terminates the thread after every settle, also after a result. A worker function that leaves a timer or a socket open cannot keep a thread alive.

On shutdown the runner terminates every thread still running and rejects its run, and it refuses every run that comes after. A task processor that was inside a run settles that attempt as a 500 and retries it after the restart. app.enableShutdownHooks() is what makes a signal reach the hook, the same as for the engine.

Telemetry

The base processor writes one structured line per attempt through the Nest Logger, with a flat context object as the second argument. Dashboards and monitors build on these fields.

| Field | Value | | --- | --- | | message | Task <id> on <queue> executed with status <status> in <n> ms | | level | info for completed, warn for pending, error for failed | | context | task_queue, task_id, task_status, task_duration_ms, task_scheduled, ddtags | | context, conditional | task_error_code when the attempt failed, and task_<key> with the tag task.<key>:<value> for each entry of buildTelemetryTags() |

buildTelemetryTags(task) returns extra tags as a Record<string, string>. Each entry becomes the field task_<key> and the tag task.<key>:<value>. In the tag, every character outside letters, digits, _, -, . and / becomes _. The field keeps the value as-is.

A workflow processor writes the line once per attempt of a workflow step, with the step it ran: task_step and the tag task.step:<step>. A timer is a workflow step, so it writes a line too. A signal move runs no workflow step and writes no line.

The status of the line is the ProcessingStatusEnum member the landing wrote on the row. The line lower-cases it, so the message, task_status and the task.status tag read completed, pending for a retry, or failed for a terminal failure. task_duration_ms is the processingDuration of the row, so the line and the row never disagree.

task_scheduled is true on every run of a processor that carries a cronExpression, whether a tick or a manual trigger() created the row, and false on every other run. The tag task.scheduled:true joins ddtags on the true runs only, because a tag exists to filter on. Query the field and the queue together: @task_scheduled:true @task_queue:<QUEUE>.

The architecture page gives the ddtags format.

Test a processor

The ./testing entry gives TaskEngineMock. It performs no I/O, records every call, and delivers a message on demand.

import { ExceptionModule } from '@ballistix.digital/exception-mapper';
import { Test } from '@nestjs/testing';
import { getRepositoryToken } from '@nestjs/typeorm';
import { ProcessingStatusEnum, TaskModule } from '@ballistix.digital/task-scheduler';
import { TaskEngineMock } from '@ballistix.digital/task-scheduler/testing';

it('completes the task the engine delivers', async () => {
	const engine = new TaskEngineMock();
	const repository = testDataSource.getRepository(ReportTask);

	const moduleRef = await Test.createTestingModule({
		imports: [ExceptionModule.forRoot(), TaskModule.forRoot({ engine })],
		providers: [ReportTaskProcessor, { provide: getRepositoryToken(ReportTask), useValue: repository }],
	}).compile();
	await moduleRef.init();

	const processor = moduleRef.get(ReportTaskProcessor);
	const task = await processor.trigger({ reportId: 'r-1' });
	expect(engine.enqueued[0].data).toEqual({ id: task.id });

	await engine.deliver('report', { id: task.id });

	const settled = await repository.findOneByOrFail({ id: task.id });
	expect(settled.processingStatus).toBe(ProcessingStatusEnum.COMPLETED);
});

moduleRef.init() runs the bootstrap hook, so the processor subscribes before the first deliver. Read these members of the mock:

| Member | Holds | | --- | --- | | enqueued | every enqueue call as an EnqueuedTaskMessage: queue, data, options, transaction, and its processingStatus and attempt | | queues | every createQueue call: queue, options | | subscriptions | the handler per queue | | schedules | every schedule call: queue, cron | | unscheduled | every queue that was unscheduled | | deliver(queue, data, attempt, retryLimit?) | hands a message to the subscribed handler | | deliverNext(queue, { includeDelayed? }) | delivers the oldest due message of the queue and returns it, or null | | deliverAll(queue, { includeDelayed? }) | delivers due messages until none is left and returns every delivery in order | | reset() | empties the recordings and keeps the subscriptions |

To test a scheduled processor, deliver an empty object. engine.deliver('afas-sync', {}) is a tick.

Each enqueued message has a processingStatus with the lifecycle of a task row: PENDING after the enqueue, PROCESSING while its handler runs, then COMPLETED, or PENDING again with the next attempt after a throw, or FAILED after a throw on the last attempt the retryLimit allows.

deliverNext() delivers the enqueued messages in the order of the enqueue. A message is due when it is PENDING. A message with startAfterSeconds above 0 that was never delivered is due only with includeDelayed. A handler throw reaches the caller.

deliverAll() calls the same delivery until no message is due. A handler throw does not stop it: the message runs again with the next attempt until it is COMPLETED or FAILED. A handler that enqueues a new message on every delivery keeps it running forever, for example a timer loop with includeDelayed. It throws when no handler is subscribed on the queue.

deliver() stamps a retryLimit on the message the way an engine does. Left out, it is the retryLimit the queue was created with, so a processor under test sees its own policy. A queue created without one gives infinity, so a spec delivers as many attempts as it asks for.

A spec that boots the module once and runs several cases on it empties the recordings between them, and keeps the handlers the processors subscribed on bootstrap:

beforeEach(() => engine.reset());

Run a worker function without a thread

The ./testing entry gives WorkerRunnerMock. It has the run() signature of the service and runs the worker function on the calling thread: the test runner transforms the .ts worker file the way it transforms any other module, so no loader is installed, and a thrown value becomes the same WorkerException a thread would send. It records every run in runs and empties them on reset(). There is no timeout, and a buffer listed in transfer stays readable.

import { WorkerRunnerService } from '@ballistix.digital/task-scheduler';
import { WorkerRunnerMock } from '@ballistix.digital/task-scheduler/testing';

const moduleRef = await Test.createTestingModule({ imports: [AppModule] })
	.overrideProvider(WorkerRunnerService)
	.useClass(WorkerRunnerMock)
	.compile();

const runner = moduleRef.get<WorkerRunnerMock>(WorkerRunnerService);
expect(runner.runs).toEqual([{ dir: expect.any(String), workerName: 'render.worker', input, options: undefined }]);

A service built by hand takes new WorkerRunnerMock() where it takes the runner. To run the real thread in a spec instead, install ts-node and use the service as it is.

Swap the engine in tests

TaskEngine is a type, not a provider token, so overrideProvider(TaskEngine) finds nothing to replace. TASK_ENGINE is the token the module holds the engine under, and it is exported for this: a test that boots the whole application overrides it and needs no token of its own.

import { TASK_ENGINE, TaskModule } from '@ballistix.digital/task-scheduler';
import { PgBossEngineService } from '@ballistix.digital/task-scheduler/pg-boss';
import { TaskEngineMock } from '@ballistix.digital/task-scheduler/testing';

// AppModule
TaskModule.forRootAsync({
	inject: [ConfigService],
	useFactory: (config: ConfigService) => ({ engine: new PgBossEngineService({ ... }) }),
});

// A test
const moduleRef = await Test.createTestingModule({ imports: [AppModule] })
	.overrideProvider(TASK_ENGINE)
	.useClass(TaskEngineMock)
	.compile();

The override replaces the provider the registration built, so the factory of forRootAsync never runs its engine and the application reaches the mock everywhere: TaskModule starts and stops it, every processor subscribes on it, and TaskRuntime.enqueue() records on it. Read it back with moduleRef.get<TaskEngineMock>(TASK_ENGINE), and drive the processors with engine.deliver() the same way as above. useValue(new TaskEngineMock()) works as well, when the spec wants the instance before it compiles.

API reference

Everything below comes from the package root, except PgBossEngineService and PgBossEngineOptions, which come from @ballistix.digital/task-scheduler/pg-boss, TaskEngineMock, CreatedQueue, EnqueuedTaskMessage, CreatedSchedule, WorkerRunnerMock and StartedRun, which come from @ballistix.digital/task-scheduler/testing, and ExceptionDto, which comes from @ballistix.digital/exception-types. ProcessingStatusEnum, WorkflowStatusEnum, WorkflowStepEnteredByEnum and WorkflowStepKindEnum also come from @ballistix.digital/task-scheduler/types, the entry that imports no framework.

| Entry | Holds | | --- | --- | | @ballistix.digital/task-scheduler | The modules, the processors, the workflow definition, TaskRuntime, the worker runner, the TASK_ENGINE token and the domain types | | @ballistix.digital/task-scheduler/pg-boss | PgBossEngineService and PgBossEngineOptions | | @ballistix.digital/task-scheduler/testing | TaskEngineMock, WorkerRunnerMock and their recorded-call types | | @ballistix.digital/task-scheduler/types | ProcessingStatusEnum and the three workflow enums, and nothing that imports a framework |

Every emitted module is also reachable by its path under dist/, because the Nest Swagger CLI plugin rebuilds an enum-typed DTO property as a require() of the declaration file the enum came from. Import through an entry in code you write.

Module

| Export | Purpose | | --- | --- | | TaskModule.forRoot(options) | Registers the module with an engine instance | | TaskModule.forRootAsync(options) | The same, with imports, inject and a useFactory | | TaskModuleOptions | { engine: TaskEngine } | | TaskModuleAsyncOptions | { imports?, inject?, useFactory } | | TASK_ENGINE | The token the module holds the engine under. A test overrides it to swap the engine | | PgBossEngineService | The pg-boss engine, constructed with PgBossEngineOptions; from the ./pg-boss entry | | PgBossEngineOptions | { connection, schema, poolSize?, superviseIntervalSeconds?, monitorIntervalSeconds?, queueCacheIntervalSeconds? } |

Processors

| Export | Purpose | | --- | --- | | AbstractTaskProcessor<T> | Base class of a task processor | | AbstractScheduledTaskProcessor<T> | Base class of a scheduled task processor | | TaskRuntime | The injectable of the task layer: the first argument of super() in a processor, and enqueue() for a service | | TaskEngine | The engine contract an application passes to TaskModule; a type, not something you inject |

AbstractTaskProcessor<T> declares two abstract members, queue and handle(task). trigger(input?, transaction?, options?) is its only public method; options takes startAfterSeconds and retryLimit. The protected members a subclass reads or overrides are queueOptions, buildTask(input), afterCompleted(task), afterFailed(task), buildTelemetryTags(task), repository and logger. The engine, the telemetry and the exception mapper registry sit behind TaskRuntime, and only the base reads them.

AbstractTaskProcessor<T> sets queueOptions to DEFAULT_QUEUE_OPTIONS. AbstractScheduledTaskProcessor<T> adds the abstract cronExpression and sets queueOptions to { ...DEFAULT_QUEUE_OPTIONS, policy: 'singleton', expireInSeconds: 600 }.

Workflows

| Export | Purpose | | --- | --- | | AbstractWorkflowProcessor<TWorkflow, TStep, TWorkflowStepEnum, TSignalEnum> | Base class of a workflow processor. An internal task processor runs its workflow steps. super(taskRuntime, workflowRepository, workflowStepRepository) | | WorkflowDefinitionBuilder<TWorkflow, TWorkflowStepEnum, TSignalEnum> | new WorkflowDefinitionBuilder(stepEnum, signalEnum?), then initialWorkflowStep(step), work(step, config), wait(step, config), terminal(step) and build(). build() throws one error that lists every fault of the definition | | WorkflowDefinition<TWorkflow, TWorkflowStepEnum, TSignalEnum> | { initialWorkflowStep, signals, workflowSteps }, what build() returns | | WorkflowWorkStep, WorkflowWorkStepConfig | A work step: { next, handle, retryLimit?, on?, onFailed? } | | WorkflowWaitStep, WorkflowWaitStepConfig, WorkflowWaitTimer | A wait step: { on?, after?: { seconds, next } } | | WorkflowTerminalStep, WorkflowStepDefinition | A terminal step, and the union of the three kinds | | Transition<TWorkflow, TWorkflowStepEnum> | What handle returns: { next, patch? } | | WorkflowPatch<TWorkflow> | The columns of the application a step or a signal writes: the entity without the workflow columns and id | | WorkflowMixin(Base), WorkflowRow | Adds workflowStepName, status, currentWorkflowStepId and workflowStepEnteredAt to an entity class, and their interface | | WorkflowStepMixin(Base), WorkflowStep | Adds the processing columns and workflowId, workflowStepName, enteredBy and signal to an entity class, and their interface | | WorkflowStatusEnum | RUNNING, WAITING, COMPLETED, FAILED. Also on the ./types entry | | WorkflowStepEnteredByEnum | START, NEXT, SIGNAL, TIMER. Also on the ./types entry | | WorkflowStepKindEnum | WORK, WAIT, TERMINAL. Also on the ./types entry |

AbstractWorkflowProcessor declares two abstract members, queue and definition. Its public methods are trigger(input?, transaction?) and signal(id, signal, patch?, transaction?). The protected members a subclass reads or overrides are queueOptions, afterCompleted(workflow), afterFailed(workflow), workflowRepository, workflowStepRepository and logger.

Domain

| Export | Purpose | | --- | --- | | ProcessableTaskMixin(Base) | Adds the five processing columns to an entity class | | ProcessableTask | The interface of a row with those columns | | ProcessingStatusEnum | PENDING, PROCESSING, COMPLETED, FAILED. Also on the ./types entry, for a browser build that shares the status | | ExceptionDto | { title, status, code, errors }, stored in processingError. Import it from @ballistix.digital/exception-types | | TaskMessage<T> | { data, attempt, retryLimit }, one delivery. retryLimit is how many redeliveries the engine allows | | EnqueueOptions | dedupeKey, startAfterSeconds, retryLimit, retryBackoff, retryDelaySeconds, expireInSeconds. The retry fields are for a message that must differ from its queue | | QueueOptions | policy, expireInSeconds, retryLimit, retryDelaySeconds, retryBackoff | | DEFAULT_QUEUE_OPTIONS | { retryLimit: DEFAULT_RETRY_LIMIT, retryBackoff: true }, the queueOptions a processor starts from | | DEFAULT_RETRY_LIMIT | 5, the default of processingMaxRetries and of the retryLimit of a queue | | QueuePolicy | standard, short, singleton, stately | | Constructor<T> | The constructor type the mixin accepts |

Worker

| Export | Purpose | | --- | --- | | WorkerModule.forRoot(options?) | Registers the worker runner with its options | | WorkerModule.forRootAsync(options) | The same, with imports, inject and a useFactory | | WorkerModuleOptions | { timeoutSeconds?, execArgv? }: the timeout of a run without one, default 300 s, and the flags of a thread that loads a .ts worker file, default -r ts-node/register | | WorkerModuleAsyncOptions | { imports?, inject?, useFactory } | | WorkerRunnerService | The injectable: run<TIn, TOut>(dir, workerName, input, options?) resol