npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

esqlate-queue

v2.0.0

Published

Push to a AsyncIterableIterator

Downloads

12

Readme

EsqlateQueue - Push to a AsyncIterableIterator

Why

Sometimes you want to process something, but you're not interested in the result immediately because the task is not a priority to you, or maybe even the user.

For this you may well use a Queue of some variety, something like RabbitMQ, Amazon AWS's SQS or maybe even ZeroMQ.

These are all fantastic technologies but:

  • The first two require infrastructure (so perhaps not great for an OSS project you want people to use).
  • The latter, you just get a message out, which is untyped and is perhaps overly complicated / higher barrier to entry for some use cases.

However if your task is mainly just running a few PostgreSQL queries:

  • Your CPU requirements for the process are probably small (you're doing mostly IO) and PostgreSQL is taking the load.
  • You can't simply scale to many nodes without complications such as pgBouncer or similar because of how PostgreSQL handles connections (memory).
  • You want to keep it super simple as you know the demand for the service will be small.

If these are your requirements and you're using TypeScript, you may want a typed solution for a really simple queue this may be the answer.

What it does

This allows you to use one simple worker function (EsqlateQueueWorker<Q, R> = (item: Q) => Promise<R>) which has a parameter of a queue item, and transforms it into the item you want as the finished product and the end of the queue.

Passing this EsqlateQueueWorker to the getEsqlateQueue() function will return an object with two methods, these are:

  • push() which you use to add things for processing.
  • results(), which will when called, return an AsyncIterableIterator which you can use a for-of to get the results.

UPDATE: This now performs n jobs in parallel by wrapping the excellent the excellent async library!

Example

import { EsqlateQueueWorker as QueueWorker } from '../src/index';
import getQueue from '../src/index';


// Create a worker. This will be used to process the items in the Queue.
const worker: QueueWorker<number,string> = (n) => {
    return new Promise((resolve) => {
        setTimeout(() => {
            resolve("Number: A" + n);
        }, 5);
    });
};

// Create an instance of the Queue
const esqlateQueue = getQueue(worker, 2); // UPDATE: Can do two things at once!

// Push items onto the Queue... afterwards, otherwise we'd never get to the loop
setTimeout(
    async () => {
        results.push("ADD");
        esqlateQueue.push(1);
        esqlateQueue.push(2);
    },
    500
);

let n = 1;

// Process the Queue Results (which also start the queue processing)
for await (const s of esqlateQueue.results()) {
    assert(s == "Number: A" + (n++));
}

Installation

To install, use NPM:

npm install esqlate-queue

Versions

1.0.0 - Initial Release 2.0.0 - Supports parallelism

License

The code is licensed under MIT.