power-queues

A lightweight, scalable, and high-performance queue engine for Node.js built on Redis Streams + Lua scripts.

The library is designed for real-world distributed systems that require high throughput, idempotent task execution, automatic recovery, and predictable performance under heavy load.

Unlike traditional Redis-based queues that rely on lists or complex abstractions, power-queues focuses on low-level control, atomic operations, and minimal overhead, making it ideal for high-load backends, microservices, schedulers, telemetry pipelines, and data-processing clusters.

Extends power-redis.

📚 Documentation

Full documentation is available here:
👉 https://power-queues.docs.ihor.bielchenko.com

📦 Installation

npm install power-queues

OR

yarn add power-queues

🧪 Basic usage

const queue = new PowerQueues({
	stream: 'email',
	group: 'workers',
});

await queue.loadScripts(true);

await queue.addTasks('email', [
	{ payload: { type: 'welcome', userId: 42 } },
	{ payload: { type: 'hello', userId: 51 } }
]);

Worker:

class EmailWorker extends PowerQueues {
	async onExecute(id, payload) {
		await sendEmail(payload);
	}
}

⚖️ power-queues vs Existing Solutions

|Feature |power-queues |BullMQ |Bee-Queue |Custom Streams| |----------------------|----------------|----------- |------------|--------------| |Bulk XADD (Lua) |✅ Yes |❌ No |❌ No |Rare | |Idempotent workers |✅ Built-in |Partial |❌ No |❌ No | |Stuck-task recovery |✅ Advanced |Basic |❌ No |Manual | |Heartbeats |✅ Yes |Limited |❌ No |Manual | |Retry logic |✅ Flexible |Good |Basic |Manual | |DLQ |✅ Native |Basic |❌ No |Manual | |Pure Streams |✅ Yes |Partial |❌ No |Yes | |Lua optimization |✅ Strong |Minimal |❌ No |Manual | |Throughput |🔥 Very high |High |Medium |Depends | |Overhead |Low |Medium |Low |Very high |

🚀 Key Features & Advantages

✔ Ultra‑Fast Bulk XADD (Lua‑Powered)

• Adds thousands of messages per second using optimized Lua scripts.
• Minimizes round‑trips to Redis.
• Supports batching based on:
  • number of tasks
  • number of Redis arguments (safe upper bound)
• Outperforms typical list‑based queues and generic abstractions.

✔ Built‑in Idempotent Workers

Every task can carry an idemKey, guaranteeing exactly‑once execution even under:

• worker crashes
• network interruptions
• duplicate task submissions
• process restarts

Idempotency includes:

• Lock key
• Start key
• Done key
• TTL‑managed execution lock
• Automatic release on failure
• Heartbeat mechanism
• Waiting on TTL for contended executions

This makes the engine ideal for:

• payment processing
• external API calls
• high‑value jobs
• distributed pipelines

✔ Stuck Task Recovery (Advanced Stream Scanning)

If a worker crashes mid‑execution, power-queues automatically detects:

• abandoned tasks
• stalled locks
• unfinished start keys

The engine then recovers these tasks back to active processing safely and efficiently.

✔ High‑Throughput Workers

• Batch execution support
• Parallel or sequential processing mode
• Configurable worker loop interval
• Individual and batch‑level error hooks
• Safe retry flow with per‑task attempt counters

✔ Native DLQ (Dead‑Letter Queue)

When retries reach the configured limit:

• the task is moved into ${stream}:dlq
• includes: payload, attempt count, job, timestamp, error text
• fully JSON‑safe

Perfect for monitoring or later re‑processing.

✔ Zero‑Overhead Serialization

power-queues uses:

• safe JSON encoding
• optional "flat" key/value task format
• predictable and optimized payload transformation

This keeps Redis memory layout clean and eliminates overhead.

✔ Complete Set of Lifecycle Hooks

You can extend any part of the execution flow:

• onSelected
• onExecute
• onSuccess
• onError
• onRetry
• onBatchError
• onReady

This allows full integration with:

• monitoring systems
• logging pipelines
• external APM tools
• domain logic

✔ Atomic Script Loading + NOSCRIPT Recovery

Scripts are:

• loaded once
• cached
• auto‑reloaded if Redis restarts
• executed safely via SHA‑based calls

Ensures resilience in failover scenarios.

✔ Job Progress Tracking

Optional per‑job counters:

• job:ok
• job:err
• job:ready

Useful for UI dashboards and real‑time job progress visualization.

🧩 Extensibility

power-queues is ideal for building:

• task schedulers
• distributed cron engines
• ETL pipelines
• telemetry processors
• notification workers
• device monitoring systems
• AI job pipelines
• high-frequency background jobs

🧱 Reliability First

Every part of the engine is designed to prevent:

• double execution
• stuck tasks
• orphan locks
• lost messages
• zombie workers
• script desynchronization

The heartbeat + TTL strategy guarantees that no task is "lost" even in chaotic cluster environments.

📜 License

MIT - free for commercial and private use.