gulp-etl-savestate

Extract STATE record and generate STATE messages from records (optional) coming in at a save frequency from the Message Stream and (optionally) save it to a State File.

Usage

POSSIBLE USE CASES FOR THIS PLUGIN WITH NEW ADDED FUNCTIONALITY:

1. Incoming State Messages (DEPICTED BY USECASE1 IN GULPFILE)
   • It can save the incoming state messages (if it has a filename, no filename means no saving)
   • It can pass it back into the stream (if SaveInStream is true, if its false it will not pass it through)
2. Generated State Messages (DEPICTED BY USECASE2 IN GULPFILE)
   • It can do this (If it has bookmarkProp, if it does not have these, the functionality will be disabled)
   • It can pass it back into the stream (if SaveInStream is true, if its false it will not pass it through)

gulp-etl plugins accept a configObj as its first parameter. The configObj will contain any info the plugin needs.

This plugin will check for the following parameters in the configObj:

• fileName: string - optionally pass in a path to save the state state, default to creating a state file in teh active directory and dumping it there. If you don't want to save the incoming state hard code null in this parameter
• saveInStream: boolean - remove the State message from the stream or keep it, in case bookmarp prop is not passed it just takes care of the incoming state messages, if bookmark prop is present this also decides whether the generated state messages will be also passed into the stream, defaults to true
• bookmarkProp?: string - decides the property of the record which the STATE message will generated based upon, if this is null, the STATE message generation feature will be turned off, defaults to null
• saveFrequency?: number - this is the frequency at which the STATE messages would be generated, defaulted to 1000

Example gulpfile below:

import {saveState} from '../src/plugin';

function runSaveState(callback: any) {
  let result
  result =
    gulp.src('../testdata/*.ndjson' )//,{ buffer: false }
    //USE_CASE-1
      .pipe(saveState({fileName:'../testdata/output/state1.json', saveInStream: true, bookmarkProp: 'Module, Trailer, or Single Bale', saveFrequency:2}))
      .on('error', console.error.bind(console))
      .pipe(gulp.dest('../testdata/processed/Stream1'))

    //USE_CASE-2
      // .pipe(saveState({fileName:'../testdata/output/state2.json', saveInStream: false}))
      // .on('error', console.error.bind(console))
      // .pipe(gulp.dest('../testdata/processed/Stream2'))
      .on('end', function () {
        console.log('end')
        callback()
      })
      .on('error', function (err: any) {
        console.error(err)
        callback(err)
      })

  return result;
}

This is a gulp-etl plugin, and as such it is a gulp plugin. gulp-etl plugins processes ndjson data streams/files which we call Message Streams and which are compliant with the Singer specification. Message Streams look like this:

{"type": "SCHEMA", "stream": "users", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}}}}
{"type": "RECORD", "stream": "users", "record": {"id": 1, "name": "Chris"}}
{"type": "RECORD", "stream": "users", "record": {"id": 2, "name": "Mike"}}
{"type": "SCHEMA", "stream": "locations", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}}}}
{"type": "RECORD", "stream": "locations", "record": {"id": 1, "name": "Philadelphia"}}
{"type": "STATE", "value": {"users": 2, "locations": 1}}

Model Plugin

This plugin is intended to be a model gulp-etl plugin, usable as a template to be forked to create new plugins for other uses. It is compliant with best practices for gulp plugins, and it properly handles both buffers and streams.

Quick Start

• Dependencies:

  • git
  • nodejs - At least v6.3 (6.9 for Windows) required for TypeScript debugging
  • npm (installs with Node)
  • typescript - installed as a development dependency

• Clone this repo and run npm install to install npm packages

• Debug: with VScode use Open Folder to open the project folder, then hit F5 to debug. This runs without compiling to javascript using ts-node

• Test: npm test or npm t

• Compile to javascript: npm run build

• Run using included test data (be sure to build first): gulp --gulpfile debug/gulpfile.ts

Testing

We are using Jest for our testing. Each of our tests are in the test folder.

• Run npm test to run the test suites

Notes

Note: This document is written in Markdown. We like to use Typora and Markdown Preview Plus for our Markdown work.