Batch Executor

A lightweight async batch executor with concurrency control and exponential backoff.

Installation

npm install @chandrafsd/batch-executor

Usage

Here is a real-life example: processing thousands of API calls with retries and exponential backoff.

import { batchExecutor } from "@chandrafsd/batch-executor";

// Example: sending notifications to users
async function sendNotification(userId: string) {
  // Fake API call
  if (Math.random() < 0.3) {
    throw new Error(`Failed to notify user ${userId}`);
  }
  return `Notification sent to ${userId}`;
}

async function main() {
  const users = Array.from({ length: 20 }, (_, i) => `user-${i + 1}`);

  const result = await batchExecutor({
    items: users,
    batchSize: 5, // run 5 at a time
    retryCount: 3, // retry each failed item 3 times
    backoffMs: 500, // wait 500ms between retries
    operation: sendNotification,
  });

  console.log(JSON.stringify(result, null, 2));
}

main();

Example Output

[
  { "status": "fulfilled", "value": "Notification sent to user-1" },
  { "status": "rejected", "reason": "Failed to notify user-4" },
  { "status": "fulfilled", "value": "Notification sent to user-7" }
]

The results always contain one entry per item, showing either success or failure.

📘 Why Use @chandrafsd/batch-executor?

Real-world systems often need to process thousands or millions of async tasks, such as:

• Sending notifications or emails
• Updating records in the database
• Making API calls with rate-limits
• Syncing data from external services
• Processing large datasets in batches

Doing everything in a single Promise.all() causes: ❌ High memory usage
❌ API throttling / rate-limit errors
❌ Crashes due to unhandled rejections

This package solves all of that with:

✅ Batching

✅ Retries

✅ Exponential backoff

✅ Full result tracking using Promise.allSettled

✅ Clean and simple API

🔄 How It Works (Visual Diagram)

items[] → split into batches → execute concurrently → retry on failure → return final result per item

Example (batchSize = 3):

Items: [1,2,3,4,5,6,7]

Batch 1 → [1,2,3]
Batch 2 → [4,5,6]
Batch 3 → [7]

Each batch runs in parallel, results are collected, failures retry with backoff.

🧠 Real Life Example 1: Sending Email Notifications

import { batchExecutor } from "@chandrafsd/batch-executor";

async function sendEmail(email: string) {
  if (Math.random() < 0.2) throw new Error(`Unable to email: ${email}`);
  return `Email sent to ${email}`;
}

async function run() {
  const emails = [
    "[email protected]", "[email protected]", "[email protected]",
    "[email protected]", "[email protected]", "[email protected]" 
  ];

  const results = await batchExecutor({
    items: emails,
    batchSize: 2,
    retryCount: 3,
    backoffMs: 300,
    operation: sendEmail,
  });

  console.log(results);
}

run();

Output looks like:

[
  { "status": "fulfilled", "value": "Email sent to [email protected]" },
  { "status": "rejected", "reason": "Unable to email: [email protected]" },
  { "status": "fulfilled", "value": "Email sent to [email protected]" }
]

🧠 Real Life Example 2: Fetching Data From an API With Rate Limits

async function fetchUser(userId: number) {
  const response = await fetch(`https://jsonplaceholder.typicode.com/users/${userId}`);
  if (!response.ok) throw new Error(`Failed to fetch user ${userId}`);
  return response.json();
}

const result = await batchExecutor({
  items: [1,2,3,4,5,6,7,8,9,10],
  batchSize: 3,
  retryCount: 2,
  backoffMs: 500,
  operation: fetchUser,
});

🧠 Real Life Example 3: Database Writes in Batches (MongoDB / PostgreSQL)

async function saveRecord(record) {
  try {
    return await db.records.insertOne(record);
  } catch (err) {
    throw new Error(`DB write failed for id ${record.id}`);
  }
}

await batchExecutor({
  items: recordsArray,
  batchSize: 50,
  retryCount: 5,
  backoffMs: 1000,
  operation: saveRecord,
});

🔧 API Reference

batchExecutor(options)

| Option | Type | Required | Description | |--------|------|----------|-------------| | items | T[] | ✔️ | List of items to process | | operation | (item: T) => Promise<any> | ✔️ | Async function executed per item | | batchSize | number | ❌ | Default 10 | | retryCount | number | ❌ | Default 3 | | backoffMs | number | ❌ | Default 300 |

Returns

Always returns an array of Promise.allSettled results:

Array<{
  status: "fulfilled" | "rejected",
  value?: any,
  reason?: any,
}>

🛠 Installation

npm install @chandrafsd/batch-executor

🚀 Quick Start

import { batchExecutor } from "@chandrafsd/batch-executor";

await batchExecutor({
  items: [1,2,3],
  batchSize: 2,
  operation: async (x) => x * 2,
});

❤️ Contribution

Feel free to open issues or PRs. This package is designed to help developers avoid complex async loops and retry logic.

📄 License

MIT