streamifi

streamifi your backends

It is what it sounds like it is. This package allows you to stream data from various backends.

Usage

// example to stream a file from s3 to stdout line by line
import { s3 } from "../index";

// we highly recommend the use of pipeline that propagates errors unlike pipe. you could still use pipe if you desire so
import { pipeline } from "stream/promises";
import { s3, util_transform } from "../index";

pipeline(
  s3.reader(
    {
      credentials: {
        accessKeyId: ...,
        secretAccessKey: ...,
      },
      Bucket: ...,
      Key: ...,
    },
    // codec is used to tell the reader how to break the stream into chunks/msgs
    // in the below line, we are asking the reader to break the stream into lines with each line being delimited by a new line char (\n)
    [
      {
        codec: "line", 
        codecOptions: { lineDelimiter: "\n" } 
      }
      // multiple codecs can be specified allowing you to perform things like JSONParse per line
    ]
  ),  
  // util_transform is a utility function provided that help create functions that transform a stream
  util_transform([
    (x) => {
      console.log(x);
    },
  ])
).catch((e) => {    // this is how error handling is performed
  console.error("ERROR", e);
});

Backends

Various backends are available (including bring your own backend*).