Deburst

Description

Take a burst call and transform it into a call with an array from this burst

When developing a streaming ETL on the cloud sometimes you hit the quota limit because of recurring calls that could be grouped. Wit deburst you can do exactly this.

sequenceDiagram
  Origin ->> Processor: File1.zip
  activate Processor
  Origin ->> Processor: File2.zip
  Origin ->> Processor: File3.zip
  Note right of Processor: Timedout waiting for new calls
  Processor ->> Destination : [File1.csv, File2.csv,File3.csv]
  deactivate Processor

Instead of this

sequenceDiagram
  Origin ->> Processor: File1.zip
  activate Processor
  Processor ->> Destination: File1.csv
  deactivate Processor
  Origin ->> Processor: File2.zip
  activate Processor
  Processor ->> Destination: File2.csv
  deactivate Processor
  Origin ->> Processor: File3.zip
  activate Processor
  Processor ->> Destination: File3.csv
  deactivate Processor

Install

$ yarn add deburst

Usage

The following code will produce

const {deburst} = require('deburst');

const DESIRED_TIMEOUT = 1000;
function myBurstedFunction(changingParam, repeatingParam){
  console.log(changingParam, '-->', repeatingParam);
}

const myDeburstedFunction = deburst(myBurstedFunction, DESIRED_TIMEOUT);

(async ()=>{
  myDeburstedFunction('file1.png', 'folder-A');
  myDeburstedFunction('file2.png', 'folder-B');
  myDeburstedFunction('file3.png', 'folder-A');
  myDeburstedFunction('file4.png', 'folder-B');
  myDeburstedFunction('file5.png', 'folder-A');
  await new Promise(res => {setTiemout(res, DESIRED_TIMEOUT * 2)});
  myDeburstedFunction('file6.png', 'folder-A');
  myDeburstedFunction('file7.png', 'folder-A');
  myDeburstedFunction('file8.png', 'folder-A');
})();

the followgin output:

> ['file1.png', 'file3.png', 'file5.png'] --> folder-A
> ['file2.png', 'file4.png'] --> folder-B
> ['file1.png', 'file2.png', 'file3.png'] --> folder-A