@cascadeflow/worker

Distributed step-level workflow execution with queue-based processing.

Installation

The worker requires a backend implementation to persist workflow state. Install the worker along with your chosen backend:

With Filesystem Backend (Recommended for Development)

npm install @cascadeflow/worker @cascadeflow/backend-filesystem

With PostgreSQL Backend (Production)

npm install @cascadeflow/worker @cascadeflow/backend-postgres

Usage

CLI

cf worker start [--mode unified|executor|scheduler] [--concurrency n]

Programmatic

import { StepWorker } from "@cascade-flow/worker";
import { FileSystemBackend } from "@cascade-flow/backend-filesystem";

const worker = new StepWorker(new FileSystemBackend("./.runs"), {
  mode: "unified",
  concurrency: 5,
  pollInterval: 5000,
});

await worker.start();

Note: Worker automatically discovers workflows with support for nested step groups. Steps can be organized in arbitrary directory hierarchies (e.g., steps/data-processing/extract/fetch-data/step.ts). Step IDs in the registry will reflect the full path.

Architecture

Events-as-queue pattern with 4 concurrent loops:

1. Executor - Claims & executes scheduled steps
2. Scheduler - Schedules ready steps, detects completion
3. Heartbeat - Proves worker liveness (5s interval)
4. Reclamation - Reclaims steps from crashed workers (30s threshold)

Modes

• Unified (default) - All 4 loops in one process
• Executor - Execute + heartbeat only (needs separate scheduler)
• Scheduler - Schedule + reclamation only (needs separate executors)

Timeouts

Step execution (3-tier fallback):

1. defineStep({ timeoutMs }) - Highest priority
2. submit({ timeout }) - Workflow-wide default
3. 300000ms (5 min) - System default

Worker health monitoring:

• Stale threshold (default 30s) - Detects crashed workers via missing heartbeats

Failure Recovery

• Worker crash → Stale heartbeat → Step reclaimed → Rescheduled
• Step timeout → StepFailed event → Retry with delay (if configured)
• Terminal failure → Scheduler stops scheduling → WorkflowFailed

Checkpoints

Workers track checkpoint progress within steps via StepCheckpoint and StepCheckpointFailed events. This enables:

• Fine-grained progress tracking for long-running steps
• Replay of completed checkpoints on retry (skip already-done work)
• Pinpoint error locations when failures occur mid-step

API

interface StepWorkerOptions {
  workerId?: string;
  mode?: "unified" | "executor" | "scheduler";
  concurrency?: number;
  pollInterval?: number;
  heartbeatInterval?: number;
  staleThreshold?: number;
  schedulerInterval?: number;
  shutdownTimeout?: number;
  workflowsDir?: string;
  baseDir?: string;
}

class StepWorker {
  constructor(backend?: Backend, options?: StepWorkerOptions);
  async start(): Promise<void>;
  async stop(): Promise<void>;
  getStats(): StepWorkerStats;
}