QueueX 🚀

A powerful, feature-rich job queue system for Node.js with advanced retry strategies, job chaining, and intelligent job processing.

✨ Features

• 🔄 Advanced Retry Strategies

  • Exponential, Linear, and Fixed backoff
  • Configurable delays and attempts
  • Maximum retry limit

• ⏱️ Smart Job Processing

  • Job timeout handling
  • Time-to-live (TTL) support
  • Multiple queue processing strategies
  • Concurrent job execution

• 🔗 Job Dependencies & Chaining

  • Sequential job execution
  • Result passing between jobs
  • Complex workflow support
  • Dependency graph management

• 📊 Queue Management

  • FIFO/LIFO processing
  • Priority queues
  • Round-robin distribution
  • Rate limiting

🚀 Quick Start

npm install queuex-sdk

import { QueueX } from 'queuex-sdk';

// Initialize QueueX
const queuex = new QueueX({ 
  redisConnection: 'redis://localhost:6379' 
});

// Create a queue
await queuex.createQueue('emailQueue', { 
  maxConcurrency: 5 
});

// Add a job with retry strategy
await queuex.enqueue('emailQueue', 
  { to: '[email protected]', subject: 'Welcome!' },
  {
    retries: 3,
    backoff: {
      type: 'exponential',
      delay: 1000,
      maxDelay: 30000
    }
  }
);

// Process jobs
queuex.startWorker('emailQueue', async (job) => {
  await sendEmail(job.data);
  return { sent: true };
});

🔥 Advanced Features

Retry Strategies

// Exponential Backoff
await queuex.enqueue('queue', data, {
  retries: 5,
  backoff: { 
    type: 'exponential', 
    delay: 1000,
    maxDelay: 60000 
  }
});

// Linear Backoff
await queuex.enqueue('queue', data, {
  retries: 3,
  backoff: { 
    type: 'linear', 
    delay: 5000 
  }
});

// Fixed Delay
await queuex.enqueue('queue', data, {
  retries: 2,
  backoff: { 
    type: 'fixed', 
    delay: 10000 
  }
});

Job Timeouts & TTL

// Job with timeout
await queuex.enqueue('queue', data, {
  timeout: 5000  // 5 seconds timeout
});

// Job with TTL
await queuex.enqueue('queue', data, {
  ttl: 3600000  // 1 hour TTL
});

// Combined options
await queuex.enqueue('queue', data, {
  timeout: 5000,
  ttl: 3600000,
  retries: 3,
  backoff: { type: 'exponential', delay: 1000 }
});

Job Chaining

// Create a chain of jobs
await queuex.enqueue('videoQueue', { videoId: '123' }, {
  chain: [
    {
      data: { step: 'compress' },
      options: { 
        priority: 'high',
        timeout: 300000 
      }
    },
    {
      data: { step: 'thumbnail' },
      options: { 
        retries: 2,
        backoff: { type: 'linear', delay: 5000 }
      }
    }
  ]
});

// Access previous job's result
queuex.startWorker('videoQueue', async (job) => {
  if (job.context) {
    console.log('Previous job result:', job.context);
  }
  // Process current job...
});

Queue Processing Strategies

// FIFO Queue (default)
await queuex.createQueue('fifoQueue', { 
  strategy: QueueStrategy.FIFO 
});

// LIFO Queue
await queuex.createQueue('lifoQueue', { 
  strategy: QueueStrategy.LIFO 
});

// Priority Queue
await queuex.createQueue('priorityQueue', { 
  strategy: QueueStrategy.PRIORITY 
});

// Round Robin Queue
await queuex.createQueue('roundRobinQueue', { 
  strategy: QueueStrategy.ROUND_ROBIN 
});

📊 Event Handling

queuex.on('jobStarted', (job) => {
  console.log(`Job ${job.id} started`);
});

queuex.on('jobCompleted', (job) => {
  console.log(`Job ${job.id} completed`);
});

queuex.on('jobFailed', (job) => {
  console.error(`Job ${job.id} failed:`, job.logs);
});

queuex.on('jobDelayed', (job) => {
  console.log(`Job ${job.id} delayed until:`, 
    new Date(job.scheduledAt!).toISOString()
  );
});

🔧 Configuration Options

Job Options

interface JobOptions {
  priority?: 'high' | 'medium' | 'low';
  retries?: number;
  backoff?: {
    type: 'exponential' | 'linear' | 'fixed';
    delay: number;
    maxDelay?: number;
  };
  timeout?: number;
  ttl?: number;
  delay?: number;
  concurrency?: number;
  dependsOn?: string[];
  cron?: string;
  chain?: Array<{
    data: any;
    options?: JobOptions;
  }>;
}

Queue Options

interface QueueOptions {
  maxConcurrency?: number;
  strategy?: QueueStrategy;
  rateLimit?: {
    max: number;
    interval: number;
  };
}

📚 Best Practices

1. Retry Strategies

   • Use exponential backoff for network operations
   • Use linear backoff for resource-intensive tasks
   • Use fixed delay for scheduled retries

2. Timeouts & TTL

   • Set reasonable timeouts based on operation type
   • Use TTL for time-sensitive tasks
   • Consider queue processing time in TTL calculations

3. Job Chaining

   • Keep chains focused and minimal
   • Handle errors appropriately in each step
   • Use context passing judiciously

4. Queue Strategies

   • Use FIFO for standard operations
   • Use LIFO for real-time updates
   • Use Priority for important tasks
   • Use Round Robin for fair resource distribution