@ballistix.digital/task-scheduler
v0.11.0
Published
NestJS module that runs background work as durable task rows behind a pluggable queue engine.
Maintainers
Keywords
Readme
@ballistix.digital/task-scheduler
A NestJS module that runs background work as durable task rows behind a pluggable queue engine. A task is a database row with a processing status, and the queue only transports the message that names that row. One engine ships, on pg-boss, behind its own entry, so an application installs pg-boss only when it uses that engine.
This file is the consumer guide. The pages that describe how the package is built live in the Ballistix wiki.
The model
The processingStatus column of the row is the lifecycle.
stateDiagram-v2
classDef waiting fill:#FFD700,stroke:#333,color:#000
classDef active fill:#87CEEB,stroke:#333,color:#00008B
classDef done fill:#90EE90,stroke:#333,color:#006400
classDef failed fill:#FFB6C1,stroke:#DC143C,color:#000
[*] --> PENDING: trigger() saves the row
PENDING --> PROCESSING: claim
PROCESSING --> COMPLETED: handle() returns
PROCESSING --> PENDING: handle() throws, status >= 500, attempts left
PROCESSING --> FAILED: handle() throws, status < 500 or attempts exhausted
COMPLETED --> [*]
FAILED --> [*]
class PENDING waiting
class PROCESSING active
class COMPLETED done
class FAILED failedRead the two exits of PROCESSING: a retry returns the row to PENDING, and
every other failure settles it FAILED.
Three mechanisms carry the guarantees:
- Transactional outbox. The task row and its task message commit in one transaction, or neither commits.
- Claim. The processor sets
PROCESSINGunder a row lock, only on aPENDINGrow or on a row a stopped run left (see A run that stops). A repeated delivery loses the claim and stops. - Retry guard. A thrown error with
status >= 500sends the row back toPENDINGwhile attempts remain. Any other error settles the rowFAILED. The queue owns the retry policy: the guard reads theretryLimitof the message and theprocessingMaxRetriesof the row, and the lower one wins.
One run, step by step
sequenceDiagram
box rgb(230,230,250) Application
participant App as Application service
participant P as ReportTaskProcessor
end
box rgb(144,238,144) Engine
participant E as TaskEngine
end
box rgb(211,211,211) Postgres
participant DB as Postgres
end
rect rgb(255,250,205)
Note over App,DB: 1. trigger: row and message in one transaction
App->>P: trigger({ reportId }, transaction)
P->>DB: INSERT report_task (PENDING)
P->>E: enqueue('report', { id }, transaction)
E->>DB: INSERT task message (same transaction)
P-->>App: the saved row
end
rect rgb(224,255,224)
Note over App,DB: 2. delivery: claim, handle, land
E->>P: deliver { id }, attempt 0
P->>DB: UPDATE status = PROCESSING, retry count = 0 WHERE id AND status = PENDING
P->>P: handle(task)
P->>DB: UPDATE status = COMPLETED WHERE id AND status = PROCESSING AND retry count = 0
P->>P: afterCompleted(task)
endLook at the first block: the row and the message travel on one transaction, so a rollback leaves neither.
Two processors
The difference between the two base classes is who creates the task row.
| | Task processor | Scheduled task processor |
| --- | --- | --- |
| Base class | AbstractTaskProcessor | AbstractScheduledTaskProcessor |
| Who creates the row | Your code, with trigger() | The processor itself, on every tick, with the same trigger() |
| When it runs | When a row is triggered | On the cron expression, and when a row is triggered |
| Typical use | An upload to parse, an invitation to send | A nightly sync, an hourly cleanup |
| Extra members | none | cronExpression, a queueOptions default that adds the singleton policy |
A tick is a message without a task id. The scheduled processor answers it with
its own trigger(), and the { id } message that enqueues runs the work like
a manual run. The queue of a scheduled processor runs one message at a time
across every replica, so a tick never overlaps a manual run.
A third base class, AbstractWorkflowProcessor, moves one workflow row through
more than one step. Each visit of a work step is one task row, and an internal
task processor runs these rows. See Workflows.
Install
npm install @ballistix.digital/task-scheduler pg-bossA browser build that reads the ./types entry alone installs the package by
itself:
npm install @ballistix.digital/task-schedulerThe package needs Node 22.12 or later and has no dependencies of its own. Every peer dependency is optional, so npm installs none of them for you: install in the application what the entries it imports need.
| Package | Range | Needed by |
| --- | --- | --- |
| @nestjs/common | ^11 \|\| ^12 | the root entry |
| @nestjs/core | ^11 \|\| ^12 | the root entry |
| typeorm | >=0.3.0 <2.0.0 | the root entry |
| reflect-metadata | ^0.2 | the root entry |
| @ballistix.digital/exception-mapper | >=0.3.0 <1.0.0 | the root entry |
| @ballistix.digital/exception-types | >=0.4.0 <1.0.0 | the root entry |
| pg-boss | ^12 | the ./pg-boss entry |
| Entry | Peers it needs |
| --- | --- |
| @ballistix.digital/task-scheduler | all of them except pg-boss |
| @ballistix.digital/task-scheduler/pg-boss | pg-boss, next to the peers of the root entry |
| @ballistix.digital/task-scheduler/testing | the peers of the root entry |
| @ballistix.digital/task-scheduler/types | none |
Only the ./pg-boss entry loads pg-boss, so an application with an engine of
its own, or a package that only tests against TaskEngineMock, leaves pg-boss
out. The ./types entry imports no framework at all: it carries
ProcessingStatusEnum and the three workflow enums, so a browser application
that shares DTO types with the backend re-exports an enum from that entry
instead of copying it.
The two exception packages carry the error envelope and the mappers that build
it. A processor stores that envelope in processingError and reads its
status to decide the retry. A worker thread maps what its worker function
threw through the same mappers and sends the envelope back.
A worker file that runs from its TypeScript source, which is what a Jest suite
does, loads in its thread through ts-node/register. Install ts-node as a
dev dependency for that, pass other loader flags through
WorkerModuleOptions.execArgv, or test with WorkerRunnerMock, which needs
no loader. See Run CPU-bound work in a worker thread.
Register the module
import { ExceptionModule } from '@ballistix.digital/exception-mapper';
import { Module } from '@nestjs/common';
import { TaskModule } from '@ballistix.digital/task-scheduler';
import { PgBossEngineService } from '@ballistix.digital/task-scheduler/pg-boss';
@Module({
imports: [
ExceptionModule.forRoot(),
TaskModule.forRoot({
engine: new PgBossEngineService({
connection: { host: 'localhost', port: 5432, user: 'app', password: 'secret', database: 'app' },
schema: 'pgboss',
poolSize: 5,
}),
}),
],
})
export class AppModule {}If the options need injection, use TaskModule.forRootAsync({ imports, inject,
useFactory }). The factory returns the same options object.
Both modules are global. A processor and a service alike inject TaskRuntime
without a further import; nothing else of the package is injectable.
TaskModule starts the engine in onModuleInit and stops it in
onApplicationShutdown. Nest runs that hook after the HTTP server has
drained its in-flight requests, so a request that is still running can
enqueue until the end. Call app.enableShutdownHooks() in the bootstrap of
the application: without it a SIGTERM ends the process at once and neither
the drain nor the stop happens. A test of the application swaps the engine by
overriding the TASK_ENGINE token the module provides it under; see
Swap the engine in tests.
The engine keeps its own pool and its own schema, apart from the TypeORM connection of the application. It reads no environment variables. The pg-boss engine page gives the other options.
Give the task a row
Compose ProcessableTaskMixin onto the entity base class of the application:
import { ProcessableTaskMixin } from '@ballistix.digital/task-scheduler';
import { BaseEntity, Column, Entity, PrimaryGeneratedColumn } from 'typeorm';
@Entity({ name: 'report_task' })
export class ReportTask extends ProcessableTaskMixin(BaseEntity) {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column({ type: 'uuid', nullable: false })
reportId: string;
}The mixin contributes five columns:
| Column | Type | Default |
| --- | --- | --- |
| processingStatus | enum processing_status, indexed | PENDING |
| processingError | jsonb, nullable | null |
| processingRetryCount | int | 0 |
| processingMaxRetries | int | 5 (DEFAULT_RETRY_LIMIT) |
| processingDuration | int, nullable, milliseconds | null |
processingError holds an ExceptionDto. Import that type from
@ballistix.digital/exception-types: this package does not re-export it.
Every entity on the mixin shares the one Postgres enum type
processing_status. Generate a migration for the new table.
Write a task processor
import { GenericBadRequestException } from '@ballistix.digital/exception-types';
import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { AbstractTaskProcessor, TaskRuntime } from '@ballistix.digital/task-scheduler';
import { Repository } from 'typeorm';
import { ReportTask } from './report.task.entity';
import { RenderService } from './render.service';
@Injectable()
export class ReportTaskProcessor extends AbstractTaskProcessor<ReportTask> {
protected readonly queue = 'report';
constructor(
taskRuntime: TaskRuntime,
@InjectRepository(ReportTask) reportTaskRepository: Repository<ReportTask>,
private readonly renderService: RenderService,
) {
super(taskRuntime, reportTaskRepository);
}
/** A 400 settles the task on the first attempt: the input cannot become valid on a retry. */
protected async handle(task: ReportTask): Promise<void> {
if (!task.reportId) {
throw new GenericBadRequestException('Task carries no reportId');
}
await this.renderService.render(task.reportId);
}
/** Runs after COMPLETED has committed. A failure here never reverts COMPLETED. */
protected async afterCompleted(task: ReportTask): Promise<void> {
await this.renderService.publish(task.reportId);
}
}Register the class as a provider of the owning module. On bootstrap the base
creates the queue with queueOptions and subscribes to it.
Queue options
The queue is the one place that holds the retry policy. A processor declares
retryLimit, retryDelaySeconds and retryBackoff in queueOptions, the
engine applies them on bootstrap, and every message on the queue inherits
them: the message of trigger(), the message of a schedule tick, and the
message a service enqueues for a row it saved itself. No enqueue of the
package passes retry options of its own, so every run on one queue retries the
same way.
A processor that declares nothing keeps the default of the base,
DEFAULT_QUEUE_OPTIONS: retryLimit: 5 (DEFAULT_RETRY_LIMIT) with
retryBackoff: true. A processor that must not repeat a run declares
retryLimit: 0: the first throw settles the row FAILED. A processor whose
work needs a pause before the next attempt adds retryDelaySeconds:
protected readonly queueOptions: QueueOptions = {
...DEFAULT_QUEUE_OPTIONS,
retryDelaySeconds: 30,
};Every start reconciles the queue with queueOptions. A changed
expireInSeconds, retryLimit, retryDelaySeconds or retryBackoff reaches
a queue an earlier release created. A changed policy does not: a policy is
fixed when the queue is created, so the engine keeps the queue and logs a
warning that names both policies. Give a processor that needs another policy a
new queue name.
The enqueue option startAfterSeconds holds one message back for at least
that many seconds after the enqueue. trigger() takes it, and retryLimit
for a row that must differ from its queue, as its third argument (see
Trigger a run from a service).
super() takes two arguments. The first is the TaskRuntime the module
provides. Pass it up and forget it: the type carries enqueue() and nothing
else, the class behind it is not exported, and a new dependency of the base
changes that class and no processor. The second is the repository. Name the
parameter after the entity, such as reportTaskRepository. If the subclass
needs the repository itself, keep it in a private field with that name.
You implement handle(). You can also override:
| Member | Default | Override when |
| --- | --- | --- |
| buildTask(input) | repository.create(input ?? {}) | the row needs values the entity defaults cannot give |
| afterCompleted(task) | nothing | work must follow a committed COMPLETED, such as a notification |
| afterFailed(task) | nothing | work must follow a committed terminal FAILED, such as an alert |
| buildTelemetryTags(task) | {} | the telemetry line of the task must carry more tags (see Telemetry) |
handle() runs outside any held transaction, so a slow run holds no row lock.
It opens its own transactions when it needs them. The queue is a plain string.
Keep the queue names of the application in one enum.
Hook names
The prefix of a hook tells when it runs and in which transaction:
| Prefix | Runs | Transaction | A throw does |
| --- | --- | --- | --- |
| build* | Before a save, such as buildTask() in trigger(). It builds a value and writes nothing | The transaction of the caller | Rolls back the transaction of the caller |
| beforeCommit* | Inside the transaction that lands a terminal FAILED, before its commit. The one hook is beforeCommitFailed(). A COMPLETED landing runs none | The landing transaction | Rolls back the landing |
| after* | After the commit, best effort | None | Writes one log line. The row keeps its status |
Only database writes belong in beforeCommitFailed(): a throw rolls the
landing back, and the row stays PROCESSING. The workflow processor uses it
to set the workflow FAILED in the same commit. No hook name starts with on. buildTelemetryTags() also
builds a value and writes nothing, but it runs after the commit, when the
telemetry line is written.
Write a scheduled task processor
The constructor follows the task processor above.
@Injectable()
export class AfasSyncTaskProcessor extends AbstractScheduledTaskProcessor<AfasSyncTask> {
protected readonly queue = 'afas-sync';
protected readonly cronExpression = '5 */6 * * *';
/** This sync must not repeat a partial run, so the queue allows no retry. */
protected readonly queueOptions: QueueOptions = { policy: 'singleton', expireInSeconds: 1800, retryLimit: 0 };
protected async handle(task: AfasSyncTask): Promise<void> {
await this.afasService.syncCustomers();
await this.afasService.syncProjects();
}
}A scheduled processor is a task processor plus cronExpression. Cron
expressions are UTC. The base creates the queue with the singleton policy, so
one run is active at a time across every replica.
queueOptions holds the time budget of one run (expireInSeconds, default
600) and the retry policy of the queue, which starts from
DEFAULT_QUEUE_OPTIONS. A tick and a manual run inherit the same policy. The
example gives the queue a budget of 30 minutes and no retry: retryLimit: 0
settles every run FAILED on its first throw, whatever processingMaxRetries
the row carries.
A tick creates the row with trigger() and no input. If the entity has a
column the tick cannot leave empty, override buildTask().
Schedules tick in every environment. Keep a processor safe to run from staging through its own configuration, not through the absence of a schedule.
Trigger a run from a service
@Injectable()
export class ReportService {
constructor(
private readonly dataSource: DataSource,
private readonly processor: ReportTaskProcessor,
) {}
public async requestReport(reportId: string): Promise<ReportTask> {
return this.dataSource.transaction(async (transaction) => {
await transaction.update(Report, reportId, { requestedAt: new Date() });
return this.processor.trigger({ reportId }, transaction);
});
}
}trigger(input, transaction) saves the row and enqueues { id } on the same
transaction. Without a transaction the processor opens one. A rolled-back
transaction leaves no row and no task message. The message inherits the retry
policy of the queue. A third argument, { startAfterSeconds?, retryLimit? },
delays the first delivery or sets another retry limit for this one message:
await this.processor.trigger({ reportId }, transaction, { startAfterSeconds: 60 });trigger() returns the saved row, so the caller can poll processingStatus
and processingError. A manual trigger() on a scheduled processor works the
same way, because the schedule and the manual runs share the one subscriber.
A processor never reads an application authentication context. Put audit values
in the trigger() input.
Enqueue a row you saved yourself
A service that owns the row, such as an upload flow that marks an attachment
PENDING, enqueues the message itself through TaskRuntime:
@Injectable()
export class UploadService {
constructor(private readonly taskRuntime: TaskRuntime) {}
public async onUploadReady(upload: Upload, transaction: EntityManager): Promise<void> {
await transaction.update(Upload, upload.id, { processingStatus: ProcessingStatusEnum.PENDING });
await this.taskRuntime.enqueue('upload', { id: upload.id }, transaction, { dedupeKey: upload.id });
}
}enqueue() is the one call TaskRuntime exposes. The message rides the
transaction you pass, so the row and the message commit together. Pass a
dedupeKey when the policy of the queue needs one, and no retry options: the
message inherits the retry policy of the queue, the same as a trigger(). The
consumer surface diagram shows
what sits behind the token and why you cannot reach it.
Errors and retries
When handle() throws, the ExceptionMapperRegistry normalizes the thrown
value into an ExceptionDto. The processor stores that envelope in
processingError, and the status of the envelope decides what happens next.
flowchart TD
T["handle() throws"] --> N["error = exceptionMapperRegistry.toExceptionDto(thrown)"]
N --> S{"error.status >= 500?"}
S -- no --> F["FAILED, processingError stored, no rethrow, telemetry line"]
S -- yes --> A{"attempt < min(message.retryLimit, processingMaxRetries)?"}
A -- no --> F
A -- yes --> R["PENDING, processingError stored, rethrow so the engine retries with backoff"]
classDef decision fill:#FFD700,stroke:#333,stroke-width:2px,color:#000
classDef retry fill:#87CEEB,stroke:#333,stroke-width:2px,color:#00008B
classDef terminal fill:#FFB6C1,stroke:#DC143C,stroke-width:2px,color:#000
classDef step fill:#F5F5F5,stroke:#333,stroke-width:1px,color:#000
class S,A decision
class R retry
class F terminal
class T,N stepLook at the first decision: the registry normalizes every thrown value, so the guard reads one field whatever the code threw.
If the code matters, throw a BaseException of the types package. Otherwise
throw any value that carries a status:
| Thrown value | Stored ExceptionDto | Retried? |
| --- | --- | --- |
| new GenericBadRequestException('bad input') | title: 'bad input', status: 400, code: 'GENERIC_ERROR' | no, the input is wrong |
| new QueryFailedError(...) with SQLSTATE 23505 | status: 400, code: 'DATABASE_ERROR' | no, the statement is wrong |
| new QueryFailedError(...) with SQLSTATE 40P01 | status: 500, code: 'DATABASE_ERROR' | yes, while attempts remain |
| { message, status: 400 } | title: message, status: 400, code: 'GENERIC_ERROR' | no, the input is wrong |
| { message, status: 503 } | title: message, status: 503, code: 'GENERIC_ERROR' | yes, while attempts remain |
| new Error('boom') | title: 'boom', status: 500, code: 'GENERIC_ERROR' | yes, while attempts remain |
| 'a string' | title: 'a string', status: 500 | yes, while attempts remain |
The registry tries its mappers in order and the generic mapper is last. A
thrown value that names no code therefore becomes GENERIC_ERROR, and one
errors detail carries the message.
The queue decides the retry budget through its retryLimit, and the engine
stamps that limit on every message it delivers. processingMaxRetries on the
row is a row-level budget that can only lower the budget of the queue: the
guard retries while attempt is below both numbers. A row that asks for more
retries than its queue allows gets the retries of the queue, because the
engine stops redelivering at its own limit and a row sent back to PENDING
past that limit would wait for a message that never comes. Both numbers
default to 5 (DEFAULT_RETRY_LIMIT).
processingRetryCount records how many attempts ran: the claim writes the
attempt of the delivery, and a failed attempt writes the attempt plus one. A
row that is triggered again gets a message at attempt 0, so it reads 0 while
that run works.
processingDuration records how long the last attempt of handle() ran, in
milliseconds, whatever its outcome. The engine schedules the retries with the
delay and the backoff of the queue. The row is the durable record you read.
A run that stops
A run can stop in the middle: the process crashes, or handle() runs past the
expireInSeconds of the queue. The row then stays in PROCESSING, and the
engine delivers the message again with the next attempt. The claim takes over a
PROCESSING row when its processingRetryCount is lower than the attempt of
the delivery, and logs one warning. The claim is one transaction that
locks the row first, so a landing of the old run that commits while the claim
waits is what the claim decides on: a retry the old run lands makes the row
PENDING, and the new delivery claims it.
Every landing is conditional on PROCESSING and the attempt that claimed the
row. When the row moved on, a landing changes nothing: the result of that run
is dropped and one warning names the task, the queue and the attempt. A caller
that resets a row to PENDING during a run therefore wins over the landing of
that run.
These limits apply:
- A message without retries left, or any message on a queue with
retryLimit: 0, gets no new delivery. Its row stays inPROCESSING. expireInSecondsmust be longer than the longest run ofhandle(). Otherwise the engine delivers the message again while the first run still works, and the two runs overlap.- A row you reset to
PENDINGduring a run is safe only when its new run claims after the old run landed. When the new run claims first at the same attempt, the old run lands its result on the row and the new run is dropped.
Workflows
AbstractWorkflowProcessor runs a flow of more than one durable step: create
a document at a provider, send it, wait for the signatures, download the
result. A workflow is a sequence of tasks. One workflow row holds one workflow
instance, and each visit of a work step is one workflow step: a task row on
the queue of the processor.
Use a workflow processor when the work has more than one step and one of these is true:
- A step calls a provider, and the next step must not start before that call is recorded.
- The flow waits for something outside: a webhook, a user action, a time.
- A user or a provider can stop the flow while it runs, such as a cancel.
A workflow does not start other workflows and does not wait for them. For N items, use one step that loops over them.
The words
| Word | Meaning |
| --- | --- |
| workflow row | The row of one workflow instance, an entity with WorkflowMixin. Column workflowStepName says where it is |
| work step | A step where your code runs: .work(step, { next, handle }) |
| wait step | A step where nothing runs. A signal or its timer moves the row on: .wait(step, { on, after }) |
| terminal step | The end. Nothing moves the row any more: .terminal(step) |
| workflow step | The task row of one visit of a work step, an entity with WorkflowStepMixin. Wait and terminal steps get none |
| current workflow step | The one workflow step that can move the row: column currentWorkflowStepId |
| signal | A fact from outside that the application sends with signal() |
| timer | The after of a wait step: a delayed workflow step |
| patch | The columns of the application that a step or a signal writes with its move |
The status of the workflow row (WorkflowStatusEnum) says how the row goes:
| Status | Meaning |
| --- | --- |
| RUNNING | The row is in a work step. Its workflow step is pending or runs |
| WAITING | The row is in a wait step and waits for a signal or its timer |
| COMPLETED | The row is in a terminal step |
| FAILED | The workflow step of the current work step failed terminally |
REJECTED in the example below is a terminal step: the business outcome is a
refusal, and the status of the row is COMPLETED.
Give the workflow its two tables
import { WorkflowMixin, WorkflowStepMixin } from '@ballistix.digital/task-scheduler';
import { BaseEntity, Column, Entity, PrimaryGeneratedColumn } from 'typeorm';
@Entity({ name: 'esign_session' })
export class ESignSession extends WorkflowMixin(BaseEntity) {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column({ type: 'varchar', length: 255, nullable: true })
providerDocumentId: string | null;
@Column({ type: 'timestamptz', nullable: true })
lastWebhookAt: Date | null;
}
@Entity({ name: 'esign_step' })
export class ESignStep extends WorkflowStepMixin(BaseEntity) {
@PrimaryGeneratedColumn('uuid')
id: string;
}Each entity declares its own primary key. WorkflowMixin adds four columns:
| Column | Type | Default |
| --- | --- | --- |
| workflowStepName | text, indexed | none, trigger() sets it |
| status | enum workflow_status, indexed | RUNNING |
| currentWorkflowStepId | uuid, nullable | null |
| workflowStepEnteredAt | timestamptz | now() |
WorkflowStepMixin adds the five columns of ProcessableTaskMixin and four
more:
| Column | Type | Default |
| --- | --- | --- |
| workflowId | uuid, indexed, no foreign key | none |
| workflowStepName | text | none |
| enteredBy | text: START, NEXT, SIGNAL, TIMER | none |
| signal | text, nullable. Set when enteredBy is SIGNAL | null |
The workflow steps of a row are its history: one row per visit, with the cause
of the visit and the outcome of its attempts. Generate one migration. It
creates the enum type workflow_status once, and the two tables.
workflowStepName, enteredBy and signal are text, so a new step needs no
ALTER TYPE.
Define the steps
WorkflowDefinitionBuilder takes three types: the workflow entity, the step
enum and the signal enum. Its constructor takes the step enum object and the
signal enum object. A workflow without signals leaves both out.
| Method | Meaning |
| --- | --- |
| initialWorkflowStep(step) | The step a new row starts in, of any kind. A wait step makes the row wait first |
| work(step, config) | Declares a work step |
| wait(step, config) | Declares a wait step |
| terminal(step) | Declares a terminal step |
| build() | Checks the definition and returns it |
| Config key | Of | Meaning |
| --- | --- | --- |
| next | work | Every step handle can return |
| handle | work | (row) => Promise<{ next, patch? }>. It runs outside any transaction |
| retryLimit | work | The retry limit of the workflow steps of this step. Left out, the limit of the queue applies. The processingMaxRetries of the workflow step still caps it |
| on | work, wait | Signal to step. On a work step, only while its workflow step is PENDING |
| onFailed | work | Signal to step, only while the row is FAILED in this step |
| after | wait | { seconds, next }: the timer. After seconds, the work step next runs |
build() throws one error that lists every fault, one per line: no
initialWorkflowStep or one that is not declared, a value of the step enum without a
declaration, and a target of next, on, onFailed or after.next that is
no declared step, and an after.next that is not a work step. A second declaration of a step replaces the first. The
definition is a property of the processor, so a wrong definition fails when
the processor is constructed.
The names of the steps and the signals are free: the package checks none. The examples use this naming, because it makes a definition easy to read:
- A work step is a verb with an object:
SEND_DOCUMENT,SYNC_SIGNATURES. - A wait step starts with
AWAIT_and names what it waits for:AWAIT_SIGNATURES. - A terminal step is the outcome, in the past tense:
COMPLETED,CANCELLED. - A signal is a fact:
WEBHOOK_RECEIVED,CANCEL_REQUESTED.
Write a workflow processor
The e-sign example: CREATE_DOCUMENT creates the document at the provider.
When the provider needs a callback first, the row waits in AWAIT_PROVIDER,
and SEND_DOCUMENT sends the document after the webhook. In
AWAIT_SIGNATURES the row waits for the signers. Each webhook moves it to
SYNC_SIGNATURES, which reads the state at the provider. The timer of one
hour is a fallback poll for a lost webhook. When not all signers have signed,
SYNC_SIGNATURES returns to AWAIT_SIGNATURES, and a new timer starts.
@Injectable()
export class ESignProcessor extends AbstractWorkflowProcessor<ESignSession, ESignStep, ESignStepEnum, ESignSignalEnum> {
protected readonly queue = 'esign';
protected readonly queueOptions: QueueOptions = { ...DEFAULT_QUEUE_OPTIONS, expireInSeconds: 300 };
protected readonly definition = new WorkflowDefinitionBuilder<ESignSession, ESignStepEnum, ESignSignalEnum>(
ESignStepEnum,
ESignSignalEnum,
)
.initialWorkflowStep(ESignStepEnum.CREATE_DOCUMENT)
.work(ESignStepEnum.CREATE_DOCUMENT, {
next: [ESignStepEnum.AWAIT_PROVIDER, ESignStepEnum.AWAIT_SIGNATURES],
handle: (session) => this.createDocument(session),
})
.wait(ESignStepEnum.AWAIT_PROVIDER, {
on: {
[ESignSignalEnum.WEBHOOK_RECEIVED]: ESignStepEnum.SEND_DOCUMENT,
[ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT,
},
})
.work(ESignStepEnum.SEND_DOCUMENT, {
next: [ESignStepEnum.AWAIT_SIGNATURES],
handle: (session) => this.sendDocument(session),
on: { [ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT },
})
.wait(ESignStepEnum.AWAIT_SIGNATURES, {
on: {
[ESignSignalEnum.WEBHOOK_RECEIVED]: ESignStepEnum.SYNC_SIGNATURES,
[ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT,
},
after: { seconds: 3600, next: ESignStepEnum.SYNC_SIGNATURES },
})
.work(ESignStepEnum.SYNC_SIGNATURES, {
next: [ESignStepEnum.AWAIT_SIGNATURES, ESignStepEnum.DOWNLOAD_DOCUMENT, ESignStepEnum.REJECTED],
handle: (session) => this.syncSignatures(session),
onFailed: {
[ESignSignalEnum.RETRY_REQUESTED]: ESignStepEnum.SYNC_SIGNATURES,
[ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT,
},
})
.work(ESignStepEnum.DOWNLOAD_DOCUMENT, {
next: [ESignStepEnum.COMPLETED],
handle: (session) => this.downloadDocument(session),
on: { [ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT },
})
.work(ESignStepEnum.REVOKE_DOCUMENT, {
next: [ESignStepEnum.CANCELLED],
handle: (session) => this.revokeDocument(session),
})
.terminal(ESignStepEnum.COMPLETED)
.terminal(ESignStepEnum.REJECTED)
.terminal(ESignStepEnum.CANCELLED)
.build();
constructor(
taskRuntime: TaskRuntime,
@InjectRepository(ESignSession) eSignSessionRepository: Repository<ESignSession>,
@InjectRepository(ESignStep) eSignStepRepository: Repository<ESignStep>,
private readonly nitro: NitroClient,
private readonly invitationService: InvitationService,
) {
super(taskRuntime, eSignSessionRepository, eSignStepRepository);
}
/** Runs after the commit that sets the session FAILED. A throw is logged, and the session stays FAILED. */
protected async afterFailed(session: ESignSession): Promise<void> {
await this.invitationService.markSigningError(session.id);
}
private async createDocument(session: ESignSession): Promise<Transition<ESignSession, ESignStepEnum>> {
// The session id is the idempotency key: a second run gets the document of the first run.
const document = await this.nitro.createDocument({ idempotencyKey: session.id });
return {
next: document.ready ? ESignStepEnum.AWAIT_SIGNATURES : ESignStepEnum.AWAIT_PROVIDER,
patch: { providerDocumentId: document.id },
};
}
private async syncSignatures(session: ESignSession): Promise<Transition<ESignSession, ESignStepEnum>> {
const state = await this.nitro.getDocument(session.providerDocumentId);
if (state.refused) {
return { next: ESignStepEnum.REJECTED };
}
return { next: state.signed === state.signers ? ESignStepEnum.DOWNLOAD_DOCUMENT : ESignStepEnum.AWAIT_SIGNATURES };
}
// sendDocument, downloadDocument and revokeDocument follow the same shape.
}super() takes three arguments: the TaskRuntime, the repository of the
workflow entity and the repository of the workflow step entity. Register the
two entities with TypeOrmModule.forFeature and the processor as a provider.
On bootstrap the base creates the queue of the workflow steps with
queueOptions and subscribes to it. Give the handlers names of their own:
trigger and signal are public methods of the base.
| Member | Purpose |
| --- | --- |
| queue | Abstract. The queue of the workflow steps |
| definition | Abstract. The steps, built with WorkflowDefinitionBuilder |
| queueOptions | The time budget and the retry policy of every workflow step, from DEFAULT_QUEUE_OPTIONS |
| trigger(input?, transaction?) | Saves the row in initialWorkflowStep and creates its first workflow step, in one commit. Without a transaction, it opens one. Returns the saved row |
| signal(id, signal, patch?, transaction?) | Hands a signal to the row. Without a transaction, it opens one. Throws when the row is missing or its workflow step runs |
| afterCompleted(row) | Protected hook. Runs after the commit that moves the row into a terminal step. Best effort |
| afterFailed(row) | Protected hook. Runs after the commit that sets the row FAILED. Best effort |
The hooks follow the hook names of the task processor. A throw
in afterCompleted() or afterFailed() writes one log line, and the row keeps
its status. afterCompleted() runs after a workflow step moves the row into a
terminal step. A signal that moves the row into a terminal step runs no hook:
the caller acts after signal() returns.
The workflow processor is no task processor itself. It holds an internal task
processor that runs the workflow steps, so cronExpression, buildTask() and
the task hooks are not members of a workflow processor.
How a workflow step runs
A workflow step runs as a task (see The model): claim, run, land, with the retry guard and the takeover. Its run does this:
- If the workflow step is not the current workflow step, it does nothing and
lands
COMPLETED. - If the row is
WAITING, the workflow step is a timer. It moves the row into its step first. - The
handleof the step runs, outside any transaction. - If
nextis not in thenextlist, the run throws aGenericErrorExceptionwith status 422 (codeGENERIC_ERROR). The retry guard does not retry it. - The move: one transaction locks the row, checks the guard again, and
writes the next step, the patch and the next workflow step. A wait step
gets
WAITING, and its timer if it hasafter. A terminal step getsCOMPLETED. - After that commit,
afterCompleted(row)runs when the row entered a terminal step. - The landing: a separate transaction lands the workflow step
COMPLETED. If the process stops between the two commits, the takeover runs the workflow step again. The guard sees that it is no longer current, so it does nothing and landsCOMPLETED.
The guard. A workflow step moves its row only while
currentWorkflowStepId is its id. A late, repeated or superseded workflow
step therefore changes nothing. A signal or a timer supersedes the workflow
step it replaces.
Each step must be idempotent. The process can stop after a provider call
and before the move commits, and the step then runs again. Send an
idempotency key, such as the row id, or look up your own reference before a
create. handle does not write to the database itself: it returns a patch.
The timer. Entering a wait step with after creates the workflow step of
after.next, delayed by after.seconds, as the current workflow step. A
signal that moves the row first makes that timer stale. If the timer runs
first, it moves the row into after.next. No periodic schedule exists.
A failure keeps the step. When a workflow step fails terminally, one
transaction lands it FAILED and sets the row FAILED, while that workflow
step is still current. After the commit, afterFailed(row) runs. When that
transaction meets a database error, it rolls back, and the workflow step stays
PROCESSING: the rules of A run that stops apply. The
failed workflow step stays as history.
A retry is a signal. The onFailed table lets a signal move a FAILED
row. To run a failed step again, add a signal that points back to the same
step, as SYNC_SIGNATURES does in the example:
onFailed: {
[ESignSignalEnum.RETRY_REQUESTED]: ESignStepEnum.SYNC_SIGNATURES,
[ESignSignalEnum.CANCEL_REQUESTED]: ESignStepEnum.REVOKE_DOCUMENT,
},signal(id, RETRY_REQUESTED) then creates a new workflow step for
SYNC_SIGNATURES with enteredBy SIGNAL and sets the row RUNNING. The
same table moves the row on in other ways: the provider is down,
SYNC_SIGNATURES fails, and the user clicks cancel. CANCEL_REQUESTED moves
the row to REVOKE_DOCUMENT without a retry first.
Send a signal
@Post('nitro/webhook/:sessionId')
@HttpCode(200)
async onNitroWebhook(@Param('sessionId', ParseUUIDPipe) sessionId: string): Promise<void> {
await this.eSignProcessor.signal(sessionId, ESignSignalEnum.WEBHOOK_RECEIVED, { lastWebhookAt: new Date() });
}The controller checks nothing. signal() throws an exception of
@ballistix.digital/exception-types, and the exception mapper turns it into
the HTTP answer.
The controller does not know the step of the row: the definition decides. The
same WEBHOOK_RECEIVED moves AWAIT_PROVIDER to SEND_DOCUMENT and
AWAIT_SIGNATURES to SYNC_SIGNATURES. The webhook says "go and look", and
SYNC_SIGNATURES reads the real state, so two webhooks in the wrong order do
no harm.
signal() locks the row (SELECT ... FOR UPDATE). With the transaction of
the caller, the lock holds until the caller commits, so a webhook can commit
its own rows and the signal together. The signal routes by what it reads:
| Row | Result |
| --- | --- |
| No row with this id | Throws ResourceNotFoundException: 404, code RESOURCE_NOT_FOUND |
| The current workflow step is PROCESSING or a claim holds its lock, also a timer | Throws PreconditionFailedException: 409, code PRECONDITION_FAILED. Nothing changes |
| WAITING, the signal is in the on of the wait step | Moves the row |
| RUNNING, the workflow step is PENDING, the signal is in the on of the work step | Moves the row |
| FAILED, the signal is in the onFailed of the step | Moves the row |
| Any other case, such as a terminal step | Returns, with one log line. Nothing changes |
A move writes the patch, the next step and the next workflow step with
enteredBy SIGNAL. The workflow step it supersedes stays
PENDING until it arrives, and then lands COMPLETED without a run. A signal
that throws or that the step does not take writes no patch. If the data must
stay in every case, save it before you call signal().
The 404 and the 409 tell the provider to send the webhook again later. A
signal that the step does not take gives a 200. A signal that is not in the
signal enum throws a GenericErrorException with status 422 (code
GENERIC_ERROR) before the transaction opens.
WorkflowPatch<TEntity> is the type of a patch: the columns of the entity
without the workflow columns and id.
Limits
expireInSecondsmust be longer than the longest step. The limits of A run that stops apply to every workflow step.- Keep a removed step in the definition until no workflow row uses it.
- One subscription runs one message at a time per instance. A slow step of one row delays the other rows on that instance.
Test a workflow
Boot the processor on a TaskEngineMock, as in
Test a processor. deliverAll(queue) delivers until no
workflow step is due, so the row waits or ends. A timer is a delayed
message, so only includeDelayed: true delivers it. The mock does not return
the row: read it back through the repository.
it('waits for the signatures, then polls through SYNC_SIGNATURES', async () => {
const session = await processor.trigger();
await engine.deliverAll('esign');
expect(await sessionRepository.findOneByOrFail({ id: session.id })).toMatchObject({
workflowStepName: ESignStepEnum.AWAIT_SIGNATURES,
status: WorkflowStatusEnum.WAITING,
});
// The timer of AWAIT_SIGNATURES is the only delayed workflow step.
await engine.deliverNext('esign', { includeDelayed: true });
await engine.deliverAll('esign');
// The stubbed provider reports both signatures, so SYNC_SIGNATURES and DOWNLOAD_DOCUMENT end the flow.
expect(await sessionRepository.findOneByOrFail({ id: session.id })).toMatchObject({
workflowStepName: ESignStepEnum.COMPLETED,
status: WorkflowStatusEnum.COMPLETED,
});
});Fire a timer with deliverNext(), not with deliverAll(..., { includeDelayed: true }).
The loop from SYNC_SIGNATURES back to AWAIT_SIGNATURES creates a new timer
on every visit, so that call runs the loop forever.
Run CPU-bound work in a worker thread
Node runs JavaScript on one thread. A render of a document or an email blocks
the event loop for its full duration, and while it runs the API answers
nothing. WorkerRunnerService runs one function in a thread of its own, and
the caller awaits the result the way it awaits any promise.
flowchart LR
S["service or handle()<br/>workerRunner.run(__dirname, 'render.worker', input)"] --> R["WorkerRunnerService<br/>one thread per run"]
R --> B["thread<br/>loads the worker file,<br/>calls its default export"]
B -->|"result"| R
B -->|"error envelope"| R
R -->|"resolve"| OK["the result"]
R -->|"reject"| E["WorkerException<br/>status, code, details kept"]
classDef main fill:#87CEEB,stroke:#333,stroke-width:2px,color:#00008B
classDef thread fill:#FFD700,stroke:#333,stroke-width:2px,color:#000
classDef done fill:#90EE90,stroke:#333,color:#006400
classDef failed fill:#FFB6C1,stroke:#DC143C,color:#000
class S,R main
class B thread
class OK done
class E failedRegister the module once, at the root. It is global, so any feature module injects the runner without importing anything.
import { WorkerModule } from '@ballistix.digital/task-scheduler';
WorkerModule.forRootAsync({
inject: [ConfigService],
useFactory: (config: ConfigService<EnvModel>) => ({
timeoutSeconds: config.get('WORKER_TIMEOUT_SECONDS'),
}),
});timeoutSeconds is the timeout of every run that passes none of its own.
The default is 300. execArgv is the list of flags a thread starts with when
it loads a .ts worker file; the default is ['-r', 'ts-node/register'].
Write a worker file
A worker file is a plain module next to the service that calls it, named
<name>.worker.ts. It exports the worker function as its default export, and
it imports nothing from this package. The function takes one input and returns
one result. Both cross the thread as structured clones, so a class instance
arrives as a plain object and a Node Buffer arrives as a Uint8Array.
// render.worker.ts
export interface RenderInput {
template: Uint8Array;
data: Record<string, unknown>;
}
export interface RenderOutput {
file: Uint8Array;
}
export default async function render(input: RenderInput): Promise<RenderOutput> {
const file = renderWithSomeLibrary(input.template, input.data);
return { file };
}
// Optional: the buffers of the result to move instead of copy.
export const transfer = (output: RenderOutput): ArrayBuffer[] => [output.file.buffer as ArrayBuffer];The worker function is a pure function of its input: no Nest injection, no TypeORM, no request context. Everything else, the database reads, the blob storage and the task lifecycle, stays on the main thread. Export the input and the output types, so the caller imports the types and nothing else from the file.
Call the runner
import { WorkerRunnerService } from '@ballistix.digital/task-scheduler';
import type { RenderInput, RenderOutput } from './render.worker';
@Injectable()
export class ReportService {
constructor(private readonly workerRunner: WorkerRunnerService) {}
async render(template: Uint8Array, data: Record<string, unknown>): Promise<Uint8Array> {
const output = await this.workerRunner.run<RenderInput, RenderOutput>(
__dirname,
'render.worker',
{ template, data },
{ timeoutSeconds: 120, transfer: [template.buffer as ArrayBuffer] },
);
return output.file;
}
}run(dir, workerName, input, options?) finds <workerName>.ts in dir when
the source exists on disk and <workerName>.js otherwise, so the same call
works under a test runner and from dist in production. Pass __dirname, and
the worker file sits next to the caller in both trees.
A run that passes timeoutSeconds wins over the value of the module. A run
that lists buffers in transfer moves them to the thread instead of copying
them: the buffer has byte length 0 on the calling side afterwards. Only a
buffer that owns its ArrayBuffer can move. A small Node Buffer shares a
pooled slab with others, so copy it out first, with Buffer.allocUnsafeSlow
or new Uint8Array(buffer).
A task processor calls the runner from handle() the same way. The row, the
claim and the retry stay on the main thread; only the CPU-bound call moves.
What a failure becomes
The thread maps what the worker function threw through the exception mapper
registry, the same mappers a processor uses, and posts the envelope. The run
rejects with a WorkerException, a BaseException restored from that
envelope, with the stack the error had in the thread. So the retry guard reads
a worker failure like any other: a 400 lands FAILED, a 500 retries.
| What happened in the thread | The run rejects with | Status, code |
| --- | --- | --- |
| The worker function threw a BaseException | WorkerException | its own status and code, its details kept |
| The worker function threw a class-validator ValidationException | WorkerException | 400, VALIDATION_FAILED |
| The worker function threw anything else | WorkerException | 500, GENERIC_ERROR, the message as title |
| An error escaped the promise chain, from a timer for example | WorkerException | as above, by what was thrown |
| The thread could not start, for example on a buffer that cannot move | GenericErrorException | 500, GENERIC_ERROR, Worker <name> could not start: <cause> |
| The run timed out | GenericErrorException | 500, GENERIC_ERROR, Worker <name> timed out after <n> s |
| The thread exited before it posted a result | GenericErrorException | 500, GENERIC_ERROR, Worker <name> exited with code <code> before it posted a result |
| The application shut down during the run | GenericErrorException | 500, GENERIC_ERROR, Worker <name> was terminated because the runner is shut down |
The run settles once, on the first of those events, and the runner terminates the thread after every settle, also after a result. A worker function that leaves a timer or a socket open cannot keep a thread alive.
On shutdown the runner terminates every thread still running and rejects its
run, and it refuses every run that comes after. A task processor that was
inside a run settles that attempt as a 500 and retries it after the restart.
app.enableShutdownHooks() is what makes a signal reach the hook, the same as
for the engine.
Telemetry
The base processor writes one structured line per attempt through the Nest
Logger, with a flat context object as the second argument. Dashboards and
monitors build on these fields.
| Field | Value |
| --- | --- |
| message | Task <id> on <queue> executed with status <status> in <n> ms |
| level | info for completed, warn for pending, error for failed |
| context | task_queue, task_id, task_status, task_duration_ms, task_scheduled, ddtags |
| context, conditional | task_error_code when the attempt failed, and task_<key> with the tag task.<key>:<value> for each entry of buildTelemetryTags() |
buildTelemetryTags(task) returns extra tags as a Record<string, string>.
Each entry becomes the field task_<key> and the tag task.<key>:<value>. In
the tag, every character outside letters, digits, _, -, . and / becomes
_. The field keeps the value as-is.
A workflow processor writes the line once per attempt of a workflow step, with
the step it ran: task_step and the tag task.step:<step>. A timer is a
workflow step, so it writes a line too. A signal move runs no workflow step
and writes no line.
The status of the line is the ProcessingStatusEnum member the landing wrote
on the row. The line lower-cases it, so the message, task_status and the
task.status tag read completed, pending for a retry, or failed for a
terminal failure. task_duration_ms is the processingDuration of the row, so
the line and the row never disagree.
task_scheduled is true on every run of a processor that carries a
cronExpression, whether a tick or a manual trigger() created the row, and
false on every other run. The tag task.scheduled:true joins ddtags on the
true runs only, because a tag exists to filter on. Query the field and the
queue together: @task_scheduled:true @task_queue:<QUEUE>.
The architecture page gives the ddtags format.
Test a processor
The ./testing entry gives TaskEngineMock. It performs no I/O, records every
call, and delivers a message on demand.
import { ExceptionModule } from '@ballistix.digital/exception-mapper';
import { Test } from '@nestjs/testing';
import { getRepositoryToken } from '@nestjs/typeorm';
import { ProcessingStatusEnum, TaskModule } from '@ballistix.digital/task-scheduler';
import { TaskEngineMock } from '@ballistix.digital/task-scheduler/testing';
it('completes the task the engine delivers', async () => {
const engine = new TaskEngineMock();
const repository = testDataSource.getRepository(ReportTask);
const moduleRef = await Test.createTestingModule({
imports: [ExceptionModule.forRoot(), TaskModule.forRoot({ engine })],
providers: [ReportTaskProcessor, { provide: getRepositoryToken(ReportTask), useValue: repository }],
}).compile();
await moduleRef.init();
const processor = moduleRef.get(ReportTaskProcessor);
const task = await processor.trigger({ reportId: 'r-1' });
expect(engine.enqueued[0].data).toEqual({ id: task.id });
await engine.deliver('report', { id: task.id });
const settled = await repository.findOneByOrFail({ id: task.id });
expect(settled.processingStatus).toBe(ProcessingStatusEnum.COMPLETED);
});moduleRef.init() runs the bootstrap hook, so the processor subscribes before
the first deliver. Read these members of the mock:
| Member | Holds |
| --- | --- |
| enqueued | every enqueue call as an EnqueuedTaskMessage: queue, data, options, transaction, and its processingStatus and attempt |
| queues | every createQueue call: queue, options |
| subscriptions | the handler per queue |
| schedules | every schedule call: queue, cron |
| unscheduled | every queue that was unscheduled |
| deliver(queue, data, attempt, retryLimit?) | hands a message to the subscribed handler |
| deliverNext(queue, { includeDelayed? }) | delivers the oldest due message of the queue and returns it, or null |
| deliverAll(queue, { includeDelayed? }) | delivers due messages until none is left and returns every delivery in order |
| reset() | empties the recordings and keeps the subscriptions |
To test a scheduled processor, deliver an empty object.
engine.deliver('afas-sync', {}) is a tick.
Each enqueued message has a processingStatus with the lifecycle of a task
row: PENDING after the enqueue, PROCESSING while its handler runs, then
COMPLETED, or PENDING again with the next attempt after a throw, or
FAILED after a throw on the last attempt the retryLimit allows.
deliverNext() delivers the enqueued messages in the order of the enqueue. A
message is due when it is PENDING. A message with startAfterSeconds above
0 that was never delivered is due only with includeDelayed. A handler throw
reaches the caller.
deliverAll() calls the same delivery until no message is due. A handler
throw does not stop it: the message runs again with the next attempt until it
is COMPLETED or FAILED. A handler that enqueues a new message on every
delivery keeps it running forever, for example a timer loop with
includeDelayed. It throws when no handler is subscribed on the queue.
deliver() stamps a retryLimit on the message the way an engine does. Left
out, it is the retryLimit the queue was created with, so a processor under
test sees its own policy. A queue created without one gives infinity, so a spec
delivers as many attempts as it asks for.
A spec that boots the module once and runs several cases on it empties the recordings between them, and keeps the handlers the processors subscribed on bootstrap:
beforeEach(() => engine.reset());Run a worker function without a thread
The ./testing entry gives WorkerRunnerMock. It has the run() signature of
the service and runs the worker function on the calling thread: the test
runner transforms the .ts worker file the way it transforms any other
module, so no loader is installed, and a thrown value becomes the same
WorkerException a thread would send. It records every run in runs and
empties them on reset(). There is no timeout, and a buffer listed in
transfer stays readable.
import { WorkerRunnerService } from '@ballistix.digital/task-scheduler';
import { WorkerRunnerMock } from '@ballistix.digital/task-scheduler/testing';
const moduleRef = await Test.createTestingModule({ imports: [AppModule] })
.overrideProvider(WorkerRunnerService)
.useClass(WorkerRunnerMock)
.compile();
const runner = moduleRef.get<WorkerRunnerMock>(WorkerRunnerService);
expect(runner.runs).toEqual([{ dir: expect.any(String), workerName: 'render.worker', input, options: undefined }]);A service built by hand takes new WorkerRunnerMock() where it takes the
runner. To run the real thread in a spec instead, install ts-node and use
the service as it is.
Swap the engine in tests
TaskEngine is a type, not a provider token, so
overrideProvider(TaskEngine) finds nothing to replace. TASK_ENGINE is the
token the module holds the engine under, and it is exported for this: a test
that boots the whole application overrides it and needs no token of its own.
import { TASK_ENGINE, TaskModule } from '@ballistix.digital/task-scheduler';
import { PgBossEngineService } from '@ballistix.digital/task-scheduler/pg-boss';
import { TaskEngineMock } from '@ballistix.digital/task-scheduler/testing';
// AppModule
TaskModule.forRootAsync({
inject: [ConfigService],
useFactory: (config: ConfigService) => ({ engine: new PgBossEngineService({ ... }) }),
});
// A test
const moduleRef = await Test.createTestingModule({ imports: [AppModule] })
.overrideProvider(TASK_ENGINE)
.useClass(TaskEngineMock)
.compile();The override replaces the provider the registration built, so the factory of
forRootAsync never runs its engine and the application reaches the mock
everywhere: TaskModule starts and stops it, every processor subscribes on
it, and TaskRuntime.enqueue() records on it. Read it back with
moduleRef.get<TaskEngineMock>(TASK_ENGINE), and drive the processors with
engine.deliver() the same way as above. useValue(new TaskEngineMock())
works as well, when the spec wants the instance before it compiles.
API reference
Everything below comes from the package root, except PgBossEngineService
and PgBossEngineOptions, which come from
@ballistix.digital/task-scheduler/pg-boss, TaskEngineMock,
CreatedQueue, EnqueuedTaskMessage, CreatedSchedule, WorkerRunnerMock and
StartedRun, which come from
@ballistix.digital/task-scheduler/testing, and ExceptionDto, which comes
from @ballistix.digital/exception-types. ProcessingStatusEnum,
WorkflowStatusEnum, WorkflowStepEnteredByEnum and WorkflowStepKindEnum
also come from @ballistix.digital/task-scheduler/types, the
entry that imports no framework.
| Entry | Holds |
| --- | --- |
| @ballistix.digital/task-scheduler | The modules, the processors, the workflow definition, TaskRuntime, the worker runner, the TASK_ENGINE token and the domain types |
| @ballistix.digital/task-scheduler/pg-boss | PgBossEngineService and PgBossEngineOptions |
| @ballistix.digital/task-scheduler/testing | TaskEngineMock, WorkerRunnerMock and their recorded-call types |
| @ballistix.digital/task-scheduler/types | ProcessingStatusEnum and the three workflow enums, and nothing that imports a framework |
Every emitted module is also reachable by its path under dist/, because the
Nest Swagger CLI plugin rebuilds an enum-typed DTO property as a require() of
the declaration file the enum came from. Import through an entry in code you
write.
Module
| Export | Purpose |
| --- | --- |
| TaskModule.forRoot(options) | Registers the module with an engine instance |
| TaskModule.forRootAsync(options) | The same, with imports, inject and a useFactory |
| TaskModuleOptions | { engine: TaskEngine } |
| TaskModuleAsyncOptions | { imports?, inject?, useFactory } |
| TASK_ENGINE | The token the module holds the engine under. A test overrides it to swap the engine |
| PgBossEngineService | The pg-boss engine, constructed with PgBossEngineOptions; from the ./pg-boss entry |
| PgBossEngineOptions | { connection, schema, poolSize?, superviseIntervalSeconds?, monitorIntervalSeconds?, queueCacheIntervalSeconds? } |
Processors
| Export | Purpose |
| --- | --- |
| AbstractTaskProcessor<T> | Base class of a task processor |
| AbstractScheduledTaskProcessor<T> | Base class of a scheduled task processor |
| TaskRuntime | The injectable of the task layer: the first argument of super() in a processor, and enqueue() for a service |
| TaskEngine | The engine contract an application passes to TaskModule; a type, not something you inject |
AbstractTaskProcessor<T> declares two abstract members, queue and
handle(task). trigger(input?, transaction?, options?) is its only public
method; options takes startAfterSeconds and retryLimit. The protected
members a subclass reads or overrides are queueOptions, buildTask(input),
afterCompleted(task), afterFailed(task), buildTelemetryTags(task),
repository and logger. The engine, the telemetry and the exception mapper
registry sit behind TaskRuntime, and only the base reads them.
AbstractTaskProcessor<T> sets queueOptions to DEFAULT_QUEUE_OPTIONS.
AbstractScheduledTaskProcessor<T> adds the abstract cronExpression and sets
queueOptions to { ...DEFAULT_QUEUE_OPTIONS, policy: 'singleton', expireInSeconds: 600 }.
Workflows
| Export | Purpose |
| --- | --- |
| AbstractWorkflowProcessor<TWorkflow, TStep, TWorkflowStepEnum, TSignalEnum> | Base class of a workflow processor. An internal task processor runs its workflow steps. super(taskRuntime, workflowRepository, workflowStepRepository) |
| WorkflowDefinitionBuilder<TWorkflow, TWorkflowStepEnum, TSignalEnum> | new WorkflowDefinitionBuilder(stepEnum, signalEnum?), then initialWorkflowStep(step), work(step, config), wait(step, config), terminal(step) and build(). build() throws one error that lists every fault of the definition |
| WorkflowDefinition<TWorkflow, TWorkflowStepEnum, TSignalEnum> | { initialWorkflowStep, signals, workflowSteps }, what build() returns |
| WorkflowWorkStep, WorkflowWorkStepConfig | A work step: { next, handle, retryLimit?, on?, onFailed? } |
| WorkflowWaitStep, WorkflowWaitStepConfig, WorkflowWaitTimer | A wait step: { on?, after?: { seconds, next } } |
| WorkflowTerminalStep, WorkflowStepDefinition | A terminal step, and the union of the three kinds |
| Transition<TWorkflow, TWorkflowStepEnum> | What handle returns: { next, patch? } |
| WorkflowPatch<TWorkflow> | The columns of the application a step or a signal writes: the entity without the workflow columns and id |
| WorkflowMixin(Base), WorkflowRow | Adds workflowStepName, status, currentWorkflowStepId and workflowStepEnteredAt to an entity class, and their interface |
| WorkflowStepMixin(Base), WorkflowStep | Adds the processing columns and workflowId, workflowStepName, enteredBy and signal to an entity class, and their interface |
| WorkflowStatusEnum | RUNNING, WAITING, COMPLETED, FAILED. Also on the ./types entry |
| WorkflowStepEnteredByEnum | START, NEXT, SIGNAL, TIMER. Also on the ./types entry |
| WorkflowStepKindEnum | WORK, WAIT, TERMINAL. Also on the ./types entry |
AbstractWorkflowProcessor declares two abstract members, queue and
definition. Its public methods are trigger(input?, transaction?) and
signal(id, signal, patch?, transaction?). The protected members a subclass
reads or overrides are queueOptions,
afterCompleted(workflow), afterFailed(workflow), workflowRepository,
workflowStepRepository and logger.
Domain
| Export | Purpose |
| --- | --- |
| ProcessableTaskMixin(Base) | Adds the five processing columns to an entity class |
| ProcessableTask | The interface of a row with those columns |
| ProcessingStatusEnum | PENDING, PROCESSING, COMPLETED, FAILED. Also on the ./types entry, for a browser build that shares the status |
| ExceptionDto | { title, status, code, errors }, stored in processingError. Import it from @ballistix.digital/exception-types |
| TaskMessage<T> | { data, attempt, retryLimit }, one delivery. retryLimit is how many redeliveries the engine allows |
| EnqueueOptions | dedupeKey, startAfterSeconds, retryLimit, retryBackoff, retryDelaySeconds, expireInSeconds. The retry fields are for a message that must differ from its queue |
| QueueOptions | policy, expireInSeconds, retryLimit, retryDelaySeconds, retryBackoff |
| DEFAULT_QUEUE_OPTIONS | { retryLimit: DEFAULT_RETRY_LIMIT, retryBackoff: true }, the queueOptions a processor starts from |
| DEFAULT_RETRY_LIMIT | 5, the default of processingMaxRetries and of the retryLimit of a queue |
| QueuePolicy | standard, short, singleton, stately |
| Constructor<T> | The constructor type the mixin accepts |
Worker
| Export | Purpose |
| --- | --- |
| WorkerModule.forRoot(options?) | Registers the worker runner with its options |
| WorkerModule.forRootAsync(options) | The same, with imports, inject and a useFactory |
| WorkerModuleOptions | { timeoutSeconds?, execArgv? }: the timeout of a run without one, default 300 s, and the flags of a thread that loads a .ts worker file, default -r ts-node/register |
| WorkerModuleAsyncOptions | { imports?, inject?, useFactory } |
| WorkerRunnerService | The injectable: run<TIn, TOut>(dir, workerName, input, options?) resol
