@vitkuz/local-pipline

A utility for creating resumable task pipelines with full support for both CommonJS and ES Modules.

Installation

npm install @vitkuz/local-pipline

Features

• Create task pipelines that can be paused and resumed
• Automatically save progress to a file
• Skip already completed tasks when restarting
• Access results from previous tasks
• Dual Format Support: Works in both CommonJS and ES Modules projects

Usage

ES Modules

import { createPipeline } from '@vitkuz/local-pipline';

const tasks = [
  {
    id: 'fetchData',
    execute: async () => {
      // Fetch data from an API
      return ['item1', 'item2'];
    }
  },
  {
    id: 'processData',
    execute: async ({ getResult }) => {
      // Access results from previous task
      const data = getResult('fetchData');
      return data.map(item => `processed ${item}`);
    }
  }
];

async function run() {
  const pipeline = await createPipeline('./logs/progress.json');
  const results = await pipeline.runTaskPipeline(tasks);
  console.log('Pipeline results:', results);
}

run().catch(console.error);

CommonJS

const { createPipeline } = require('@vitkuz/local-pipline');

// Same usage as above

API

createPipeline(progressFilePath, options)

Creates a new pipeline instance.

Parameters:

• progressFilePath: Path where progress will be saved (JSON format)
• options: Optional configuration
  • verbose: Whether to log progress (default: true)
  • restart: Whether to restart the pipeline from scratch (default: false)

Returns an object with:

• runTaskPipeline(tasks): Runs the given array of tasks
• getProgressPath(): Returns the path to the progress file
• clearProgress(): Clears all saved progress data

Task Definition

type Task = {
  id: string;
  execute: (ctx: {
    getResult: (id: string) => any;
    task: Task;
  }) => Promise<any>;
};

License

ISC