@gomani/jobs
v0.14.0
Published
A durable job queue, workers, and cron for splitting workloads by failure/resource profile — a crashing worker backs jobs up while the web process keeps serving.
Maintainers
Readme
@gomani/jobs
A durable job queue, workers, and cron. Split your workloads by their failure and resource profile: the latency-sensitive web process only enqueues (cheap); a separate worker process drains. A durable queue sits between them — so a slow or crashing worker just backs jobs up in the store while requests keep being served, and the worker picks them up on restart. That is the real "one thing dies, the rest survive" win — from workload isolation, not one-service-per-domain.
import { createJobRunner, memoryStore, fileStore } from '@gomani/jobs';
const runner = createJobRunner({
store: fileStore('.goma/queue.json'), // local-default; survives a restart
jobs: [
{
name: 'transcode',
concurrency: 2, // at most 2 in flight
maxAttempts: 5, // then dead-letter
backoff: { baseMs: 1000 }, // exponential + jitter
handler: async (payload) => {
await doHeavyWork(payload);
},
},
],
});
// From a request handler / loader (latency-safe — just a write):
await runner.enqueue('transcode', { fileId }, { idempotencyKey: `transcode:${fileId}` });
// In the worker process (`gomani worker` runs this):
await runner.work({ signal }); // drains forever, honoring per-job concurrencyWhat it guarantees
- At-least-once + idempotency. A job runs until it succeeds;
enqueuewith anidempotencyKeydedupes while a job with that key is pending. A worker that crashes mid-run leaves the job in the store; on restart, jobs leftactiveare reset toqueuedand run again. - Retry with backoff → dead-letter. A failing handler is retried with exponential backoff (reusing
@gomani/resilience) up tomaxAttempts, then moved todeadfor inspection — a poison job never spins forever. - Per-job concurrency. The worker loop runs at most
concurrencyof each job at once, so one heavy job can't starve the others.
Cron
Scheduled work flows through the same durable pipeline (so a missed cron run is retried, and nothing runs "outside" the system):
import { defineCron, cronsToJobs, createJobRunner, createCronScheduler } from '@gomani/jobs';
const nightly = defineCron('royalties', '0 2 * * *', async () => {
await runRoyalties();
});
const runner = createJobRunner({ store, jobs: cronsToJobs([nightly]) });
const scheduler = createCronScheduler({ crons: [nightly], runner });
await scheduler.run({ signal }); // ticks each minute; enqueues a job when a schedule is duecronMatches(schedule, date) implements the standard 5-field syntax (minute hour day-of-month month
day-of-week) with *, */n, a-b, a-b/n, and comma lists (and the Vixie day-of-month/day-of-week OR).
Stores (local-default, pluggable)
memoryStore()— dev + tests.fileStore(path)— single-box; persists to JSON, survives a restart. Single-writer.- Your own — a Postgres/Redis
QueueStoreis a drop-in for the multi-process / split-service topologies where several workers claim concurrently. The interface is the seam — the same local-default-then-real-backend pattern as the payment providers.
Every clock/id source is injectable (now, newId), so the whole engine is deterministically testable.
Topology — the same code, single / multi / split
An app declares its worker in app/worker.ts (default export: a WorkerApp = { store, jobs, crons }).
The web side imports it to enqueue; the worker side drains it. Where the worker runs is a config
choice in gomani.topology.json, not a code change:
{ "mode": "single" } // single | multi | splitsingle(default) — web + workers + cron in one process.gomani devruns the worker in-process; a small deploy needs nothing more.multi— a web process + a separategomani workerprocess on one box, coordinated by the durable queue (afileStoreworks — both re-read the same file).split— web and workers as separate deployables against a shared queue/DB (a realQueueStoreadapter) — "if you outgrow the box".
gomani dev # single: serves + drains + cron, one process
gomani worker # a standalone worker process (multi/split)runWorker(app, { signal }) is the worker entry (drain loop + cron scheduler); webRunsWorker(topology)
and resolveTopology(config) are the deploy-time helpers. Constraint-first: single is the default;
distribution is opt-in, escalated deliberately — the same shape as @gomani/native's PWA-first.
