@dot-ready/redis-service-template

Production-grade framework for building fault-tolerant Redis-based background processing services with zero message loss and complete accountability.

Features

• Zero Message Loss - Triple persistence (Redis, Memory, Disk)
• Automatic Recovery - Handles Redis failures, crashes, and network issues
• Complete Accountability - Track every message from receipt to completion
• Worker Pool Management - Concurrent processing with configurable workers
• Smart Retry Logic - Exponential backoff with dead letter queue
• Health Monitoring - Built-in health checks and auto-recovery
• Comprehensive Logging - Multi-output logging with audit trails
• HTTP API - Built-in health, stats, metrics, and control endpoints

Installation

npm install @dot-ready/redis-service-template

Add to your .npmrc:

@dot-ready:registry=https://npm.pkg.github.com

Quick Start

Create a new service by extending BaseRedisService:

require('dotenv').config();
const { BaseRedisService } = require('@dot-ready/redis-service-template');

class MyService extends BaseRedisService {
  constructor() {
    super({
      serviceName: 'my-service',
      queueName: 'my-service:queue',
      port: 3001,
      redis: {
        host: process.env.REDIS_HOST || 'localhost',
        port: parseInt(process.env.REDIS_PORT) || 6379,
      },
      workers: { maxWorkers: 10 },
      fallback: { directory: './fallback-models' },
      logging: { level: 'info', outputs: ['console', 'file'] },
    });
  }

  validateServiceModel(model) {
    if (!model.payload) throw new Error('payload is required');
  }

  async processModel(model) {
    // Your business logic here
    return { success: true };
  }
}

const service = new MyService();
service.start();

Exports

const {
  // Core - extend these to build services
  BaseRedisService,          // Queue-driven service base class
  BaseEventListenerService,  // Redis keyspace event listener base class

  // Core components (auto-initialized by BaseRedisService)
  ModelTracker,              // In-memory model tracking with audit trail
  FallbackManager,           // Disk persistence fallback
  WorkerPoolManager,         // Concurrent processing pool
  HealthMonitor,             // Component health monitoring
  ProgressTracker,           // Cross-instance coordination

  // Services (usable standalone or via BaseRedisService)
  RedisManager,              // Enhanced ioredis wrapper
  Logger,                    // Winston-based structured logging
  ConfigManager,             // Hierarchical configuration
  ExpressServer,             // HTTP API for monitoring/control
  MySQLService,              // MySQL connection pool (optional)

  // Utilities
  UidGenerator,              // Traceable UID generation
} = require('@dot-ready/redis-service-template');

Architecture

┌─────────────────────────────────────────────┐
│           Express HTTP API                  │
│     /health /stats /metrics /pause /resume  │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────┴──────────────────────────┐
│           BaseRedisService                  │
│         (Main BRPOP Processing Loop)        │
└──────────────────┬──────────────────────────┘
                   │
        ┌──────────┼──────────┬───────────────┐
        │          │          │               │
┌───────▼──────┐ ┌─▼──────┐ ┌▼──────────┐ ┌─▼────────┐
│ModelTracker  │ │Worker  │ │Fallback   │ │Health    │
│(In-Memory)   │ │Pool    │ │Manager    │ │Monitor   │
└──────────────┘ └────────┘ │(Disk)     │ └──────────┘
                             └───────────┘

What You Implement

When extending BaseRedisService, you only need to implement two methods:

| Method | Purpose | |---|---| | validateServiceModel(model) | Validate the model's payload before processing | | processModel(model) | Your actual business logic |

Everything else (queue consumption, retry logic, disk persistence, health monitoring, logging, HTTP API) is handled by the framework.

Built-in HTTP Endpoints

Every service automatically gets:

| Endpoint | Method | Description | |---|---|---| | /health | GET | Health status (200 if healthy, 503 if not) | | /stats | GET | Processing statistics | | /metrics | GET | Prometheus-format metrics | | /models | GET | Model tracker stats | | /models/:uid | GET | Detailed model audit report | | /workers | GET | Worker pool status | | /fallback | GET | Disk fallback stats | | /config | GET | Sanitized configuration | | /dead-letter | GET | Dead letter queue count | | /completed | GET | Paginated completed models | | /pause | POST | Pause processing | | /resume | POST | Resume processing |

Configuration

Configuration is loaded in this priority order:

1. Constructor config (highest)
2. Environment variables
3. config/default.json in your project root
4. Package defaults (lowest)

Key Environment Variables

SERVICE_NAME=my-service
QUEUE_NAME=my-service:queue
PORT=3001

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0

MAX_CONCURRENT_JOBS=10
MAX_RETRIES=3
RETRY_DELAY=1000
MAX_RETRY_DELAY=60000

FALLBACK_DIR=./fallback-models
LOG_LEVEL=info
HEALTH_CHECK_INTERVAL=30000

Model Structure

Every message follows this structure:

{
  uid: "unique-identifier",
  service_type: "service-name",
  queue_name: "queue:name",
  created_at: "2024-01-01T00:00:00Z",
  retry_count: 0,
  max_retries: 3,
  status: "pending",
  priority: "normal",
  payload: {
    // Service-specific data
  },
  wildcard: {},
  errors: [],
  checkpoints: []
}

Testing

npm test

License

MIT