QueueCTL

A robust, CLI-based background job queue system, built for the Backend Developer Internship Assignment.

QueueCTL allows you to enqueue arbitrary shell commands as jobs, execute them concurrently with one or more background worker processes, automatically retry failures with exponential backoff, and quarantine permanently-failing jobs into a Dead Letter Queue (DLQ). Everything is backed by a persistent SQLite database, ensuring zero data loss across restarts.

✨ Features Added Beyond Base Requirements

1. Stunning Web UI Dashboard: A real-time, glassmorphic, dark-mode single-page application monitoring the queue. Runs natively in the background (queuectl web).
2. Custom File-based Job Logger: Every executed job generates a beautifully formatted, timestamped log file in the logs/ directory capturing STDOUT, STDERR, execution times, and retry attempts.
3. Queue Reset Command: Instantly wipe the database and clean up old logs (queuectl reset).
4. Enhanced CLI Help: Deeply integrated commander configuration showcasing sub-commands directly in the root --help menu.

1. Setup Instructions

Prerequisites

• Node.js >= 18
• npm

Installation

# 1. Clone and navigate to the project directory
git clone <your-repo-url>
cd queuectl

# 2. Install required dependencies
npm install

# 3. Link the CLI globally
# (On Linux, global symlinking usually requires administrator privileges)
sudo npm link

Note: If sudo npm link prompts for a password, enter your system password. Alternatively, you can run commands locally via node src/index.js.

You can verify the installation was successful by running:

queuectl --help

2. Usage Guide

[!WARNING] Windows Users: Do not use PowerShell, as it aggressively strips quotes from JSON strings which breaks JSON payloads. Please use Git Bash, WSL, or Command Prompt to run queuectl commands. Mac/Linux Users: The default Terminal or iTerm will work perfectly.

QueueCTL is entirely managed via the command line. The database (.queuectl/queuectl.db) is created automatically the first time you run a command.

Enqueuing Jobs

You add jobs to the queue by passing a JSON payload containing a unique id and a command.

# Enqueue a simple background job
queuectl enqueue '{"id":"job1","command":"sleep 2"}'

# Enqueue a job that will intentionally fail, and set a custom retry limit
queuectl enqueue '{"id":"failJob","command":"exit 1","max_retries":3}'

Managing Workers

Jobs stay in the pending state until you start background workers to process them. Workers run as detached daemons.

# Start 3 background worker processes to execute jobs
queuectl worker start --count 3

# Stop all background workers gracefully (allows current jobs to finish)
queuectl worker stop

Monitoring (CLI & Web UI)

# View an overview of all jobs and active workers in terminal
queuectl status

# List all jobs and their states
queuectl list
queuectl list --state pending

# Launch the real-time Web Dashboard (Runs in background at http://localhost:3000)
queuectl web start
queuectl web stop

Dead Letter Queue (DLQ)

Jobs that fail all their retries are moved to the DLQ.

# List all dead jobs
queuectl dlq list

# Retry a specific dead job (resets attempts and moves it back to pending)
queuectl dlq retry failJob

Configuration & Reset

# View and update runtime configuration
queuectl config list
queuectl config set max-retries 5

# Hard reset the entire system (deletes all jobs and clears the logs/ folder)
queuectl reset

3. Job Execution Logs

Every time a worker processes a job, it generates a highly detailed, human-readable execution log inside the automatically created logs/ folder.

Example (logs/failJob.log):

=========================================================
                 QUEUECTL JOB EXECUTION LOG
=========================================================
Job ID          : failJob
Worker ID       : worker-1783574498829-0
Command         : exit 1
Created At      : 2026-07-09 10:51:14
Max Retries     : 3

=========================================================
ATTEMPT #1
=========================================================
Started At      : 10:51:40
Finished At     : 10:51:40
Status          : FAILED
Exit Code       : 1
Execution Time  : 31 ms

STDOUT
None

STDERR
None
Retry After     : 2 seconds

4. System Architecture

flowchart TD
    U["User (terminal)"] -->|"queuectl <command>"| CLI["queuectl CLI (src/index.js)"]
    CLI --> CMD["Command Layer<br/>CLI parsing & validation only"]
    CMD --> SVC["Service Layer<br/>All business logic"]
    SVC --> DB[("SQLite Database<br/>.queuectl/queuectl.db")]
    CLI -.->|"worker start"| WP["Worker Process(es)<br/>(forked daemons)"]
    WP --> SVC
    WP -->|"child_process.exec"| SH["Host Shell Command"]
    CLI -.->|"web start"| WEB["Web UI Server<br/>(Background HTTP)"]
    WEB --> SVC

Layering principle (Separation of Concerns):

• Command layer (src/commands/*.js) — parses CLI args/flags, never touches the DB directly.
• Service layer (src/services/*.js) — owns business rules: enqueue validation, atomic claiming, retry/backoff math, DLQ logic.
• Database layer (database/db.js) — connection management and schema.

5. Database Schema

SQLite in WAL (Write-Ahead Logging) mode is used to allow concurrent readers while a single writer holds the lock.

erDiagram
    JOBS {
        TEXT id PK
        TEXT command
        TEXT state "pending|processing|completed|failed|dead"
        INTEGER attempts
        INTEGER max_retries
        TEXT next_retry_at "ISO timestamp"
        INTEGER locked
        TEXT worker_id
        TEXT last_error
        TEXT stdout
        INTEGER exit_code
    }
    CONFIG {
        TEXT key PK
        TEXT value
    }

6. Job Lifecycle & Concurrency

stateDiagram-v2
    [*] --> pending: enqueue
    pending --> processing: worker claims job
    processing --> completed: exit code 0
    processing --> pending: exit code != 0 (attempts < max_retries)
    processing --> dead: exit code != 0 (attempts >= max_retries)
    dead --> pending: manual "dlq retry"
    completed --> [*]

Atomic Claiming (Race-Free)

Multiple worker processes read/write the same SQLite file concurrently. Duplicate execution is prevented with a single atomic UPDATE statement:

UPDATE jobs SET state = 'processing', locked = 1, worker_id = ?
WHERE id = (
  SELECT id FROM jobs WHERE state = 'pending' AND (next_retry_at IS NULL OR next_retry_at <= ?)
  ORDER BY created_at ASC LIMIT 1
) AND state = 'pending';

If two workers race, SQLite serializes their transactions. Whichever transaction commits first "wins" the row. The losing worker gets changes = 0 and safely loops again.

Retry Backoff Formula

delay(seconds) = backoff_base ^ attempts

A job with a future next_retry_at is mathematically invisible to the polling worker's claim query until that precise timestamp has passed. No busy-waiting required.

7. Testing

An integration-style test suite exercises the core flows against an isolated test database (.queuectl-test/), meaning your real queue data is perfectly safe.

npm test

The tests strictly validate:

1. Successful job completion and state updates.
2. Failing jobs correctly retrying and ultimately moving to the DLQ.
3. Rejection of duplicate job IDs.
4. Correct resets of attempt counters when reviving jobs from the DLQ.
5. Concurrency safety.
6. Validation of empty payloads and configuration rules.