npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

pino-elasticsearch

v8.0.0

Published

Load pino logs into ElasticSearch

Downloads

80,234

Readme

pino-elasticsearch  Build Status Coverage Status

Load pino logs into Elasticsearch.

Install

npm install pino-elasticsearch -g

Usage

  pino-elasticsearch

  To send pino logs to elasticsearch:

     cat log | pino-elasticsearch --node http://localhost:9200

  Flags
  -h  | --help              Display Help
  -v  | --version           display Version
  -n  | --node              the URL where Elasticsearch is running
  -i  | --index             the name of the index to use; default: pino
                            will replace %{DATE} with the YYYY-MM-DD date
  -t  | --type              the name of the type to use; default: log
  -f  | --flush-bytes       the number of bytes for each bulk insert; default: 1000
  -t  | --flush-interval    time that the helper will wait before flushing; default: 30000
  -l  | --trace-level       trace level for the elasticsearch client, default 'error' (info, debug, trace).
      | --es-version        specify the major version number of Elasticsearch (eg: 5, 6, 7)
                            (this is needed only if you are using Elasticsearch <= 7)
  -u  | --username          Username to specify with authentication method
                            (can only be used in tandem with the 'password' flag)
  -p  | --password          Password to specify with authentication method
                            (can only be used in tandem with the 'username' flag)
  -k  | --api-key           Api key for authentication instead of username/password combination
  -c  | --cloud             Id of the elastic cloud node to connect to
  -r  | --read-config       the name of config file
  --rejectUnauthorized      Reject any connection which is not authorized with the list of supplied CAs; default: true

Usage as module

const pino = require('pino')
const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  esVersion: 7,
  flushBytes: 1000
})

const logger = pino({ level: 'info' }, streamToElastic)

logger.info('hello world')
// ...

Using multiple streams (output to Console and Elasticsearch)

If you want to output to multiple streams (transports and console)), the simplest way is to use the pino-multi-stream library and register a stream per output.

const pinoElastic = require('pino-elasticsearch');
const pinoMultiStream = require('pino-multi-stream').multistream;

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  esVersion: 7,
  flushBytes: 1000
});

const pinoOptions = {};

return pino(pinoOptions, pinoMultiStream([
  { stream: process.stdout },
  { stream: streamToElastic },
]));

You can learn more about pino-multi-stream here: https://github.com/pinojs/pino-multi-stream.

Custom Connection

If you want to use a custom Connection class for the Elasticsearch client, you can pass it as an option when using as a module. See the Elasticsearch client docs for Connection.

const pino = require('pino')
const pinoElastic = require('pino-elasticsearch')

const Connection = <custom Connection>

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  esVersion: 7,
  flushBytes: 1000,
  Connection
})

const logger = pino({ level: 'info' }, streamToElastic)

logger.info('hello world')
// ...

Troubleshooting

Assuming your Elasticsearch instance is running and accessible, there are a couple of common problems that will cause indices or events (log entries) to not be created in Elasticsearch when using this library. Developers can get feedback on these problems by listening for events returned by the stream handler.

The stream handler returned from the default factory function is an EventEmitter. Developers can use this interface to debug issues encountered by the pino-elasticsearch library.

const pinoElastic = require('pino-elasticsearch');

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  esVersion: 7,
  flushBytes: 1000
})

streamToElastic.on('<event>', (error) => console.log(event));

The following table lists the events emitted by the stream handler:

| Event | Callback Signature | Description | | ----- | ------------------ | ----------- | | unknown | (line: string, error: string) => void | Event received by pino-elasticsearch is unparsable (via JSON.parse) | | insertError | (error: Error & { document: Record<string, any> }) => void | The bulk insert request to Elasticsearch failed (records dropped). | | insert | (stats: Record<string, any>) => void | Called when an insert was successfully performed | | error | (error: Error) => void | Called when the Elasticsearch client fails for some other reason |

There are a few common problems developers will encounter when initially using this library. The next section discusses these issues.

Pino output is not JSON

Any transform of the stream (like pino-pretty) that results in an non-JSON stream output will be ignored (the unknown event will be emitted).

const pino = require('pino');
const pinoElastic = require('pino-elasticsearch');

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  esVersion: 7,
  flushBytes: 1000
})

streamToElastic.on(
  'unknown',
  (line, error) =>
    console.log('Expect to see a lot of these with Pino Pretty turned on.')
);

const logger = pino({
  prettyPrint: true,
}, streamToElastic)

Events do not match index schema/mappings

Elasticsearch schema mappings are strict and if events do not match their format, the events will be dropped. A typical example is if you use the default pino format with the logs- index in Elasticsearch. The default installation of Elasticsearch includes a data pipeline mapped to the logs- index prefix. This is intended to be used by the Beats and Logstash aggregators and requires @timestamp and @message fields. The default pino setup uses time and msg. Attempting to write events to the logs- index without mapping/transforming the pino schema will result in events being dropped.

Developers can troubleshoot insert errors by watching for the insertError event.

streamToElastic.on(
  'insertError',
  (error) => {
    const documentThatFailed = error.document;
    console.log(`An error occurred insert document:`, documentThatFailed);
  }
);

ECS support

If you want to use Elastic Common Schema, you should install @elastic/ecs-pino-format, as the ecs option of this module has been removed.

const pino = require('pino')
const ecsFormat = require('@elastic/ecs-pino-format')()
const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  esVersion: 7,
  flushBytes: 1000
})

const logger = pino({ level: 'info',  ...ecsFormat  }, streamToElastic)

logger.info('hello world')
// ...

You can then use Kibana to browse and visualize your logs.
Note: This transport works only with Elasticsearch version ≥ 5.

Dynamic index

It is possible to customize the index name for every log line just providing a function to the index option:

const pino = require('pino')
const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: function (logTime) {
    // the logTime is a ISO 8601 formatted string of the log line
    return `awesome-app-${logTime.substring(5, 10)}`
  },
  node: 'http://localhost:9200'
})
// ...

The function must be sync, doesn't throw and return a string.

Datastreams

Indexing to datastreams requires the opType to be set to create:

const pino = require('pino')
const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: "type-dataset-namespace",
  node: 'http://localhost:9200',
  opType: 'create'
})
// ...

Error handling

const pino = require('pino')
const ecsFormat = require('@elastic/ecs-pino-format')
const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  esVersion: 7,
  flushBytes: 1000
})

// Capture errors like unable to connect Elasticsearch instance.
streamToElastic.on('error', (error) => {
  console.error('Elasticsearch client error:', error);
})
// Capture errors returned from Elasticsearch, "it will be called every time a document can't be indexed".
streamToElastic.on('insertError', (error) => {
  console.error('Elasticsearch server error:', error);
})

const logger = pino({ level: 'info',  ...ecsFormat()  }, streamToElastic)

logger.info('hello world')

Authentication

If you need to use basic authentication to connect with the Elasticsearch cluster, pass the credentials in the URL:

cat log | pino-elasticsearch --node https://user:pwd@localhost:9200

Alternatively you can supply a combination of username and password OR api-key:

cat log | pino-elasticsearch --node https://localhost:9200 -u user -p pwd
cat log | pino-elasticsearch --node https://localhost:9200 --api-key=base64EncodedKey

Elastic cloud option cloud is also supported:

cat log | pino-elasticsearch --cloud=name:bG9jYWxob3N0JGFiY2QkZWZnaA== --api-key=base64EncodedKey

Note: When using the cli, if you pass username/password AND an apiKey the apiKey will take precedence over the username/password combination.

You can also include the auth field in your configuration like so:

const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  auth: {
    username: 'user',
    password: 'pwd'
  },
  esVersion: 7,
  flushBytes: 1000
})

Alternatively you can pass an apiKey instead:

const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: 'an-index',
  node: 'http://localhost:9200',
  cloud: {
    id: 'name:bG9jYWxob3N0JGFiY2QkZWZnaA=='
  },
  auth: {
    apiKey: 'apikey123'
  },
  esVersion: 7,
  flushBytes: 1000
})

For a full list of authentication options when using elastic, check out the authentication configuration docs

Use as a module

use pino-elasticsearch as a module is simple, use pino-multi-stream to send log to multi transport, for example:

const pinoms = require('pino-multi-stream')
const pinoEs = require('pino-elasticsearch')({
    node: 'http://192.168.1.220:9200',
    index: 'zb'
})

const logger = pinoms({
    streams: [
      {level: 'error', stream: process.stderr}, // an "error" level destination stream
      {level: 'info', stream: process.stdout}, // an "info" level destination stream
      {stream: pinoEs}
    ]
  })


logger.info({'msg': {'info': 'info'}})
logger.debug('debug')
logger.warn('warn')
logger.error('error')

Setup and Testing

Setting up pino-elasticsearch is easy, and you can use the bundled docker-compose.yml file to bring up both Elasticsearch and Kibana.

You will need docker and docker-compose, then in this project folder, launch docker-compose -f docker-compose-v8.yml up.

You can test it by launching node example | pino-elasticsearch, in this project folder. You will need to have pino-elasticsearch installed globally.

Acknowledgements

This project was kindly sponsored by nearForm.

License

Licensed under MIT.