TaskQueuer

A tiny TypeScript task runner with retries, backoff, persistence (SQLite), and optional encryption for queued task payloads.

Install

npm install taskqueuer
# or
pnpm add taskqueuer

Requires Node 18+ (ESM) and bundles SQLite for persistence.

Quick start

import TaskQueue from "taskqueuer";

const queue = new TaskQueue(); // in-memory, no persistence

queue.enqueue(
  "hello",
  async () => {
    console.log("Hello from the queue!");
  },
  undefined, // validator
  3, // maxAttempts
  false // requiresNetwork
);

queue.on("events", (evt) => {
  console.log("event:", evt.name, evt.data);
});

queue.processQueue();

Axios + network retry example

requiresNetwork = true makes transient network errors (5xx, timeouts, DNS, axios network errors) back off and retry with exponential jitter. The sample below intentionally fails twice, then succeeds, so you can observe retries:

import axios from "axios";
import TaskQueue from "taskqueuer";

const queue = new TaskQueue(
  true,
  false,
  undefined,
  "./.database.sqlite3",
  "./encrypt.key"
);
const http = axios.create({ timeout: 2000 });

let attempts = 0;
queue.enqueue(
  "axios-flaky",
  async () => {
    attempts++;
    const url =
      attempts < 3
        ? "https://postman-echo.com/status/503"
        : "https://postman-echo.com/delay/1"; // returns 200 after ~1s

    console.log(`[axios] attempt ${attempts} -> ${url}`);
    const res = await http.get(url);
    console.log(`[axios] status ${res.status}`);
  },
  undefined,
  5,
  true, // requiresNetwork -> retry on transient errors
  750 // backoff base ms to keep the demo quick
);

queue.enqueue(
  "axios-no-network-flag",
  async () => {
    await http.get("https://postman-echo.com/status/503"); // fails, no retry
  },
  undefined,
  2,
  false // requiresNetwork = false -> no retry on transient network errors
);

queue.on("events", (evt) => console.log("event:", evt.name, evt.data));
queue.processQueue();

Run the compiled dist demo

npm run example:dist

Persistence + encryption

// Persist tasks to SQLite and encrypt serialized task payloads with AES-GCM.
const queue = new TaskQueue(
  true, // persist to SQLite
  true, // encrypt serialized task bodies
  undefined, // optional config object (see below)
  "./.database.sqlite3", // optional custom DB path (default: ./.database.sqlite3)
  "./encrypt.key" // optional custom key path (default: ./queue-descriptor.key)
);
queue.resumePersistedTasks(); // hydrate and start processing persisted tasks

When persist is true, tasks are stored in the task_queue table; completed/failed/skipped tasks are removed automatically.

API surface

• new TaskQueue(persist?, encrypt?, config?, databasePath?, keyFilePath?, sortFunc?)

  • persist (default false): store tasks in SQLite so they survive restarts.
  • encrypt (default false): AES-GCM encrypt serialized task functions on disk.
  • config (default: internal defaults): tune polling and backoff; see Configuration below.
  • databasePath (default ./.database.sqlite3): custom SQLite file path.
  • keyFilePath (default ./queue-descriptor.key): where to store/read the encryption key used when encrypt is true.
  • sortFunc (optional (a, b) => number): comparator to control processing order.

• enqueue(name, method, validator?, maxAttempts = 3, requiresNetwork = true, backoffBaseMs = 60_000)

  • name string identifier.
  • method async function to run.
  • validator optional sync/async function; returning false skips the task.
  • maxAttempts max times to try the task (incremented on any thrown error).
  • requiresNetwork toggles network-aware retry/backoff. If true, transient network errors trigger exponential backoff with jitter (cap 30s). If false, the task fails immediately on any error.
  • backoffBaseMs starting backoff for transient network errors.

• processQueue() – start the processing loop.

• pause(reason?) / resume(reason?) – pause/resume; reason is offline or manual.

• stop() – stop processing loop altogether.

• isEmpty() / size() – queue state helpers.

• isTaskProcessing(name) / isTaskProcessingById(id) – whether a task is in the queue.

• resumePersistedTasks() – hydrate persisted tasks without immediately starting processing.

Configuration

Pass a QueueConfig as the third argument to the constructor to tweak timings and sizing. All keys are optional; defaults shown:

const queue = new TaskQueue(
  true,
  false,
  {
    waiting_timeout: 2, // seconds to wait before marking the queue ready to process
    default_backoff_base_ms: 60_000, // base for network backoff when enqueue doesn't override
    max_queue_size: 50_000, // queue size before trimming
    overflow_drop_ratio: 0.5, // drop 50% of oldest tasks when overflowing
    loop_sleep_ms: 500, // delay between process loop iterations
  },
  "./.database.sqlite3",
  "./encrypt.key"
);

Events

The queue extends EventEmitter and emits a single events stream with { name, data }:

• enqueue, ready, start, retry, skipped, completed, failed, paused, resumed

Example:

queue.on("events", (evt) => {
  if (evt.name === "failed") {
    console.error("Task failed", evt.data);
  }
});

Sorting custom priority

Pass a comparator to the constructor to control ordering:

const sortNewestFirst = (a, b) => b.id - a.id;
const queue = new TaskQueue(
  false,
  false,
  undefined,
  undefined,
  sortNewestFirst
);

Notes

• Tasks are plain async functions; if you need validation, pass a validator that returns true/false to skip.
• When requiresNetwork is true, transient network errors trigger exponential backoff (max 30s between retries).
• Failed tasks are removed after exceeding maxAttempts. Increase maxAttempts to keep retrying longer.