bunyan-elasticsearch-bulk

• A Bunyan stream for saving logs into Elasticsearch.
• Saves logs in memory instead of sending them to the Elasticsearch server one by one.
• The logs are accumulated in memory until a certain number of logs reached or a certain amount of time passed without sending any logs.
• The goal is to save resources on both your application and Elastic Search server (ex: lowering costs in AWS).

Installation

npm i bunyan-elasticsearch-bulk

Usage with Node

const bunyan = require('bunyan')
const createESStream = require('bunyan-elasticsearch-bulk')

const config = {
  name: 'Application Name',
  streams: [{
    level: 'debug',
    stream: createESStream({
      indexPattern: '[my-app-]YYYY.MM.DD',
      node: 'http://localhost:9200'
    })
  }]
}

const log = bunyan.createLogger(config)

// From here, you can log things according to the best practices of Bunyan.
// Please familiarize yourself with it here: https://github.com/trentm/node-bunyan

log.info('Log this message!')
log.info({ otherInfo: 'What else do you need to log here?' }, 'Log this message!')

List of Configuration Parameters

Parameters Specific to Elasticsearch Official Client

These Parameters are passed to the client without any change.

You need to provide the following one. The rest are up to you.

| Field | Required? | Example | Description | | --- | --- | --- | --- | | node or nodes | Yes | http://localhost:9200 | The Elasticsearch endpoint to use. No trailing slash! |

Parameters Specific to this Module

| Field | Default | Description | | --- | --- | --- | | indexPattern | [my-app-]YYYY[-]MM[-]DD | Pattern for the daily segmentation of log indices. This will be passed to moment.utc().format(indexPattern) from moment.js | | type | logs | Type of the logs for Elasticsearch servers prior to v7.x | | limit | 100 | How many logs to collect before submitting them to ES | | interval | 5000 | Time in milliseconds before submitting logs even if their count has not reached the limit. | | client | Client(options) | If you don't want to use the included version of the client v7.x, you can configure the one you want and pass it here. |

Emitted Events

Chech server.js for example usages:

| Event | Fires when .. | | --- | --- | | log_received | we receive a log | | log_submitted | we bulk-submit logs to ES server. With a reason limit_exceeded or timeout | | error | an error happens in the whole bulk submission or when saving an individual message fails |

Changes since 1.0.x

For you as a client, you no longer need to use the keyword new with this module. There is a factory function now. Example above is updated.

Development

• A docker-compose.yaml file has been added to make it easier to develop and test locally.

Author(s)

• Milad Kawas Cale

License

MIT