feeds-sdk

An SDK that simplifies feed development by handling all Faros API interactions. A feed is a tool that extracts data from an external data source, typically an IaaS or PaaS API, and loads it into Faros.

Uploading data

The SDK includes two functions for uploading data to the Faros platform: withEntryUploader and entryUploader. They have slightly different signatures, but they both wrap all the necessary API calls to the Faros platform, so you can focus on what's unique to your feed. Choosing one comes down to whether your feed requires state to be persisted between runs.

Stateless feeds

Feeds which do not need to persist state between runs can use entryUploader. This will give you a stream.Writable whose entries are uploaded to the API.

Example of using entryUploader:

import {entryUploader} from 'faros-feeds-sdk';

const writer = await entryUploader({
  name: 'EC2',
  // Other required parameters...
});

// Written entries will be streamed to the API
writer.write({
  ec2_Instance: {
    instanceId: 'i-0ef827cbfe244614f',
    instanceType: 't2.micro',
    state: {
      code: 80,
      name: 'stopped',
    },
    launchedAt: new Date(1607473249000),
  },
});

// Write more entries and finally:
writer.end();

Stateful feeds

When possible, it's good practice to support incremental data ingestion. In order to do so, use withEntryUploader, which comes with a callback you can use to read the existing feed state, and return the new state.

Example of using withEntryUploader:

import {withEntryUploader} from 'faros-feeds-sdk';

export interface FeedState {
  readonly lastLaunchedAt: number;
}

const uploaderCfg = {
  name: 'EC2',
  // Other required parameters...
};
await withEntryUploader<FeedState>(uploaderCfg, (writer, state) => {
  // Write entries to writer using the old state...
  return /* new state */;
});

Logging

The FarosEnv passes a logger to the feed. This logger should be used by the feed for all logging to ensure logs are correctly stored.

Links

1. Canonical models. This library can be used for the sdl property of EntryUploaderConfig. See the docs for details.