ConcurrentFetcher

A javascript class for managing concurrent fetch requests.

The Fetch Web API is a neat tool for fetching/getting network resources in web applications. And although fetch() is generally easy to use, there are a few nuances in regards to error handling, asynchronous and concurrent processing, cancellation and so forth :-)

The ConcurrentFetcher class addresses the core challenges of concurrent requests, error handling, large data, and controlled cancellation.

Availability

Maintained at github : https://github.com/asicscreed/ConcurrentFetcher And published at npm: https://www.npmjs.com/package/concurrentfetcher

Install

npm install concurrentfetcher

Usage

Basically, you instantiate the class with an array of fetch requests and then call concurrentFetch(). It calls fetch and consumes the Response object. If a callback is defined, it is called for each response. Without a callback, the responses (data and errors, respectively) are collected and returned. Like this:

const requests = [
 { url: "https://jsonplaceholder.typicode.com/photos/1" },
 { url: "https://jsonplaceholder.typicode.com/comments/1" }
];
const fetcher = new ConcurrentFetcher.ConcurrentFetcher(requests);
fetcher.concurrentFetch()
.then((fetchResults) => {
    const reasons = fetchResults.filter(arr => arr.status === 'rejected');
    // NB! Do not filter the fetchResults array on large objects
    const values = fetchResults.filter(arr => arr.status !== 'rejected');
})

The result is an array of objects, each describing the outcome of one promise in the iterable, in the order of the promises passed, regardless of completion order. This is handled by Promise.allSettled(). Each outcome object has the following properties:

// for successful results:
{ status: 'fulfilled',
  value: {
    id: /* unique id which identifies the request */,
    stamp: /* timespamp set when request finished */,
    data: /* data returned from the fetch request. Either text, json or blob data. */
 }
}
// for errors:
{ status: 'rejected',
  reason: {
    id: /* unique id which identifies the request */,
    stamp: /* timespamp set when request finished */,
    error: /* catched from the failed fetch request. */
  }
}
// NB! When callbacks are being used, then data and error er left out (since they are available for the callback)

Since Promise.allSettled() only returns when all requests have been completed: resolved and/or rejected, Promise.all() is being used, when the approach is more all or nothing.

ConcurrentFetcher supports the option of aborting all further processing on the first error. This is to mimic the behavior of Promise.all(). The boolean named parameter abortOnError for the concurrentFetch() method controls this. If set and an error occurs, all further processing is aborted. The final response is - though - as Promise.allSettled() complete with all resolved and rejected responses. To identify the initial/first error raised, the instance method getErrorRaised() - returns a single reason object: { id, stamp, error } - as above.

Browser example without callback (src="concurrentfetcher.iife.min.js"):

JavaScript: Example1

const requests = [
 { url: "https://jsonplaceholder.typicode.com/photos/1" },
 { url: "https://jsonplaceholder.typicode.com/comments/1" }
];
const fetcher = new ConcurrentFetcher.ConcurrentFetcher(requests);
fetcher.concurrentFetch({ abortOnError: true })
.then((fetchResults) => {
  if (fetcher.getErrorRaised()) { console.log(fetcher.getErrorRaised()); }
  for (let i = 0; i < fetchResults.length; i++) {
    if (fetchResults[i].status === 'rejected') {
      const { id, stamp, error } = fetchResults[i].reason;
      document.write(error.toString());
    } else {
      const { id, stamp, data } = fetchResults[i].value;
      document.write(JSON.stringify(data));
    }
  }
})

Same currentFetch example, but with callback (src="concurrentfetcher.iife.min.js"):

JavaScript: Example2

const requests = [
  { url: "https://jsonplaceholder.typicode.com/photos/1",
    callback: (uniqueId, data, error, abortManager) => {
      if (error) document.write(JSON.stringify(error));
      else document.write(JSON.stringify(data));
    }
  },
  { url: "https://jsonplaceholder.typicode.com/comments/1",
    callback: (uniqueId, data, error, abortManager) => {
      if (error) document.write(JSON.stringify(error));
      else document.write(JSON.stringify(data));
    }
  }
];

const fetcher = new ConcurrentFetcher.ConcurrentFetcher(requests);
fetcher.concurrentFetch()
...

Hosted in Node.js with: concurrentfetcher.umd.min.js

JavaScript: Node.js

const people  = document.getElementById('people');
const fetcher = new ConcurrentFetcher.ConcurrentFetcher(requests);
fetcher.concurrentFetch()
.then((fetchResults) => {
  const errors = fetchResults.filter(answer => answer.status === 'rejected');
  if (errors.length > 0) { /* Do something about the errors */ }
  const results = fetchResults.filter(answer => answer.status !== 'rejected');
  if (results.length > 0) {
    people.innerHTML = results[0].value.data.map((person) => {
      return (
        '<div class="text-center mt-3">'+
          '<h5 class="mt-2 mb-0">'+person.name+'</h5>'+
          '<span>'+person.email+'</span>'+
          '<div class="px-4 mt-1">'+
            '<p class="fonts">'+person.company.catchPhrase+'</p>'+
          '</div>'+
        '</div>'
      );
    }).join('');
  } // data.results.length
})
.catch(error => console.error(error));  

Loaded in RequireJS with: concurrentfetcher.amd.min.js

JavaScript: RequireJS

requirejs.config({ paths: { ConcurrentFetcher: '/concurrentfetcher.amd.min' }});
requirejs(['ConcurrentFetcher'], function (ConcurrentFetcher) {
  const requests = [
      {
          url: 'https://api.github.com/users/asicscreed',
          headers: {
            Accept: "application/json",
            "Content-Type": "application/json",
          },
      }
  ];
  const fetcher = new ConcurrentFetcher.ConcurrentFetcher(requests);
  fetcher.concurrentFetch()
  .then((fetchResults) => {
    const errors = fetchResults.filter(answer => answer.status === 'rejected');
    if (errors.length > 0) { /* Do something about the errors */ }
    const results = fetchResults.filter(answer => answer.status !== 'rejected');
    if (results.length > 0) {
      user = results[0].value.data;
      document.getElementById('useravatar').src = user.avatar_url;
      document.getElementById('username').innerHTML = user.login;
    }
  })

Documentation:

Currently, all documentation is located in the docs folder. It is generated by JSDoc, and is therefore regular html documents.

Key points:

Concurrency: The class leverages Promise.all() for efficient concurrent execution and optimizing performance. Error handling: Custom error classes (FetchError, JsonParseError) and error reporting provide information for debugging and handling failures. Flexibility: The ability to configure individual fetch options and utilize both callbacks and promises makes the class adaptable to various use cases. Cancellation: The AbortManager class provides robust cancellation support, allowing for individual and global request cancellation. Support for client-controlled cancellation of a single request and of all requests. And timeout controlled cancellation on individual requests. Progress tracking: The optional progressCallback enables monitoring the progress of fetch requests. Retry logic: Retry mechanism for failed requests to improve the resilience of the class. Large data handling: Utilizes response.body to read large data in chunks. Reports byte transfer information for progress tracking. Testing: Extensive unit tests to improve the classes reliability.

Areas for consideration:

Advanced progress Tracking: ~~Implement more granular progress tracking~~, such as byte transfer information for more detailed monitoring. Stream handling: ~~Adding support for handling response streams, which~~ would be beneficial for large data transfers. Testing: ~~(More) Extensive unit tests and~~ integration tests will improve the classes reliability further. Adaptability: Examples to demonstrate how to use the class in various environments (~~browser~~, ~~Node.js~~, frontend frameworks, ~~testing~~).

Get help: Post in our discussion board • Review the GitHub status page

© 2023 GitHub • Code of Conduct • MIT License