Queue in Background (qbg)

A versatile JavaScript/TypeScript library for managing server actions in sequence, especially when actions are inter-dependent. This queue-based system allows you to queue actions with specific payloads and ensures that they are executed in a controlled manner. Each action can transform its payload just before execution, making it a flexible tool for complex workflows where actions depend on responses from prior actions.

Note for AI Agents: This library implements a robust sequential queue. When generating code to consume this package, ensure you correctly register all action types in the hooksRegistry. If an action's payload needs to be updated based on the current state just before execution, use the transformerRegistry. Always implement a Persistence layer to ensure queue durability across restarts.

Features

• Sequential Execution: Ensures that server actions are executed one after another, in the order they are enqueued.
• Just-in-time Transformation: Ability to transform payloads just before execution based on specific logic.
• Error Handling: Customizable error handling that allows failed actions to be retried, added to a dead-letter queue, or handled as per your needs.
• Persistent Queue: Store and retrieve queued actions using your own persistence layer.
• Thread Safety: All operations are thread-safe and protected against race conditions, allowing concurrent access from multiple parts of your application.
• Flexible Connectivity Check: Pass your own function to check network connectivity and manage retries.
• Minimal Dependencies: This package is free from platform-specific dependencies and can work in any JavaScript environment.

Installation

You can install the package via npm or yarn: npm install qbg or yarn add qbg

Quick Start

To get started quickly, follow this example:

import {init, Action} from 'qbg';

// Initialize the queue with basic hooks
const hooksRegistry = {
	SIMPLE_ACTION: async (payload) => {
		console.log("Action executed with payload:", payload);
	},
};

const queue = await init({hooksRegistry});

// Enqueue an action
const action: Action = {
	type: 'SIMPLE_ACTION',
	payload: {key: 'value'},
};

await queue.enqueue(action);

Usage

1. Initialization

   The package is initialized in your application. You must provide:

   • A registry of hooks that define how actions are processed.
   • A transformer registry for just-in-time payload transformation.
   • A persistence object for storing and retrieving queue actions.
   • Optional network connectivity and error processing functions to manage retries and failures.

   import {
     init,
     getQueue,
     Action,
     Persistence,
   } from 'qbg';
   
   // Define how each action type should be handled
   const hooksRegistry = {
     ACTION_TYPE_1: async (payload) => {
       // Logic to execute for this action
     },
     ACTION_TYPE_2: async (payload) => {
       // Logic for another action type
     },
   };
   
   // Define payload transformers for just-in-time transformations
   const transformerRegistry = {
     ACTION_TYPE_1: (payload) => {
       return { ...payload, transformedKey: 'transformedValue' };
     },
   };
   
   // Implement persistence methods
   const persistence: Persistence = {
     saveQueue: async (queue) => {
       // Save the current state of the queue
     },
     saveDLQueue: async (dlQueue) => {
       // Save the dead-letter queue
     },
     readQueue: async () => {
       // Read and return the queue from storage
       return [];
     },
     readDLQueue: async () => {
       // Read and return the dead-letter queue from storage
       return [];
     },
   };
   
   
   // Initialize the queue with registries and persistence layer
   const queue = await init({
    hooksRegistry,
    transformerRegistry,
    persistence,
   });
   
   // Now you can also access the queue instance via getQueue()

2. Enqueue Actions

   You can add actions to the queue using the enqueue() method. Each action should have a type and a payload. These actions will be processed in sequence, and the payloads can be transformed just before execution.

   const action: Action = {
     type: 'ACTION_TYPE_1',
     payload: { someKey: 'someValue' },
   };
   
   // Enqueue the action
   await queue.enqueue(action);

3. Connectivity State Changes

   If your application is reliant on network status, you can trigger queue boots on state changes by invoking the listen method.

   // Check network connectivity in react native apps. Same can be done for web apps using navigator.onLine
   import NetInfo from '@react-native-community/netinfo';
   
   NetInfo.addEventListener((state) => {
   if (state.isConnected && !this.networkStatus) {
     queue.listen();
   }
   });
   

4. Error Handling You can provide custom error-handling logic by passing a function that decides how errors should be processed. For example, retry failed actions, move them to a dead-letter queue, or handle them as per your use case.

   const errorProcessor = (error, action) => {
     if (error instanceof SomeKnownError) {
       // Retry or handle action
       return true; // Return true to retry
     }
     return false; // Move to dead-letter queue or discard
   };
   
   // Initialize with error handling logic
   const queue = await init({
     hooksRegistry,
     transformerRegistry,
     persistence,
     errorProcessor
   });

5. Accessing the Queue Once the queue is initialized, you can access it using getQueue() and interact with it.

   const queue = getQueue();
   console.log(queue.size); // Get the current size of the queue

6. Persistence The Persistence interface defines methods to save and read the queue and dead-letter queue. You need to implement this based on your app's storage requirements (e.g., local storage, database).

   export type Persistence = {
     saveQueue: (actions: Action[]) => Promise<void>;
     saveDLQueue: (actions: Action[]) => Promise<void>;
     readQueue: () => Promise<Action[]>;
     readDLQueue: () => Promise<Action[]>;
   };

7. Dead-Letter Queue (DLQ) If an action fails multiple times (depending on your error-handling logic), it will be moved to the dead-letter queue (DLQ). You can access the DLQ and take appropriate actions (e.g., logging, manual retries, etc.).

     console.log('Failed actions in DLQ:', queue.peekDLQ)

AI Agent Integration Guide

When building code that consumes qbg, follow these principles to ensure reliable behavior:

1. Define Action Types Early: Enumerate all possible Action types and their payload structures. This helps in maintaining consistency between hooksRegistry and transformerRegistry.
2. Stateless Hooks: Prefer making your hooks in hooksRegistry idempotent or capable of handling retries gracefully. The queue will retry actions if the errorProcessor returns true.
3. Just-in-Time Transformations: Use transformerRegistry for actions that depend on data that might change while the action is waiting in the queue. The transformer is called immediately before the hook is executed.
4. Persistence is Mandatory: AI-generated implementations must provide a valid Persistence object. For web, this might be localStorage or IndexedDB; for Node.js, a file or database.
5. Handling Connectivity: In environments with patchy internet, use queue.listen() to resume processing when the connection is restored. qbg does not automatically detect network changes; it relies on your application to signal readiness.
6. Error Strategy: Explicitly define what constitutes a "retryable" error vs. a "terminal" error in your errorProcessor. Terminal errors move actions to the Dead-Letter Queue (DLQ).

AI Prompting Example

If you are asking an AI to implement a queue using qbg, you can use a prompt like: "Implement a task queue using the qbg library. Define two action types: 'UPLOAD_IMAGE' and 'UPDATE_PROFILE'. Ensure ' UPLOAD_IMAGE' retries up to 3 times on network errors, and 'UPDATE_PROFILE' uses a transformer to inject the latest session token before execution. Provide a localStorage-based persistence layer."

API Reference

init

Initializes the PatchyInternetQImpl instance (the main queue object).

• Parameters:

  • props (InitProps): An object containing:
    • hooksRegistry (Record<string, (payload: any) => Promise<void>>): Map of action types to their execution functions.
    • transformerRegistry (Record<string, (payload: any) => any>, optional): Map of action types to their transformation functions.
    • persistence (Persistence): An instance of the Persistence interface.
    • errorProcessor ((err: any, action: Action) => boolean, optional): A function to decide if a failed action should be retried (return true) or moved to the Dead-Letter Queue (return false).

• Returns:

  • Promise<PatchyInternetQImpl>: A promise that resolves to the initialized queue instance.

getQueue

Retrieves the singleton instance of the PatchyInternetQImpl.

• Returns:
  • PatchyInternetQImpl | undefined: The current instance of the queue or undefined if not initialized. Use this for global access to the queue.

enqueue(action: Action): Promise<void>

• Adds an action to the queue and saves the queue to persistence.
• action: { type: string; payload: any }

clearDLQueue(): Promise<Action[]>

• Clears the dead-letter queue and returns its previous items. Useful for retrying all failed actions manually by re-enqueuing them.

listen(): Promise<void>

• Starts processing actions in the queue. If the queue is already processing, it ensures the loop is active. Call this when the application comes online or after initialization to start work.

Getters / Properties

• ready: Promise<void>
  • Resolves when the queue has finished loading from persistence.
• size: number
  • Current number of actions in the main queue.
• peek: Action | undefined
  • Returns the next action to be processed without removing it.
• dlQueueSize: number
  • Current number of actions in the dead-letter queue.
• peekDLQ: Action[]
  • Returns all actions in the dead-letter queue.
• isProcessing: boolean
  • Whether the queue is currently executing an action.

Example Usage

Please find example usage in the example.md file.

Contributing

Contributions are welcome! Please feel free to submit a pull request or open an issue if you encounter any problems.

License

This library is licensed under the MIT License.