gelf-pro-plus

Graylog2 client library for Node.js. Sends logs to Graylog2 server in GELF (Graylog Extended Log Format) format.

Fork notice: This is a fork of gelf-pro by Kanstantsin Kamkou. It adds persistent TCP connection reuse (keepAlive) for high-throughput scenarios, with automatic reconnection, message queuing, and backoff. All credit for the original library goes to the original author and contributors.

Features:

• JS object marshalling
• UDP/TCP/TLS support
• Persistent TCP connection reuse with automatic reconnection and connection pooling
• Filtering, Transforming, Broadcasting.

Installation

npm install gelf-pro-plus

Library only depends on: lodash#~4.17

Initialization

var log = require('gelf-pro-plus');

Adapters

• UDP (with deflation and chunking)
  • Input: GELF UDP
• TCP
  • Input: GELF TCP (with Null frame delimiter)
• TCP via TLS(SSL)
  • Input: GELF TCP (with Null frame delimiter and Enable TLS)

[!NOTE] By default, the TCP and TCP-TLS adapters create a new connection for each message. For high-throughput use cases, you can enable persistent connection reuse with the keepAlive option. See TCP Connection Reuse below.

[!NOTE] Within a more or less stable network (which is most likely), I would recommend using the udp adapter. I would also recommend it for an average to high-loaded project. For sensitive information, the tcp-tls adapter is recommended.

Configuration

// simple
log.setConfig({adapterOptions: {host: 'my.glog-server.net'}});

// advanced
log.setConfig({
  fields: {facility: "example", owner: "Tom (a cat)"}, // optional; default fields for all messages
  filter: [], // optional; filters to discard a message
  transform: [], // optional; transformers for a message
  broadcast: [], // optional; listeners of a message
  levels: {}, // optional; default: see the levels section below
  aliases: {}, // optional; default: see the aliases section below
  adapterName: 'udp', // optional; currently supported "udp", "tcp" and "tcp-tls"; default: udp
  adapterOptions: { // this object is passed to the adapter.connect() method
    // common
    host: '127.0.0.1', // optional; default: 127.0.0.1
    port: 12201, // optional; default: 12201
    // ... and so on
    
    // tcp adapter example
    family: 4, // tcp only; optional; version of IP stack; default: 4
    timeout: 1000, // tcp only; optional; default: 10000 (10 sec)
    keepAlive: false, // tcp/tcp-tls only; optional; enable persistent connection reuse; default: false
    
    // udp adapter example
    protocol: 'udp4', // udp only; optional; udp adapter: udp4, udp6; default: udp4
    
    // tcp-tls adapter example
    key: fs.readFileSync('client-key.pem'), // tcp-tls only; optional; only if using the client certificate authentication
    cert: fs.readFileSync('client-cert.pem'), // tcp-tls only; optional; only if using the client certificate authentication
    ca: [fs.readFileSync('server-cert.pem')] // tcp-tls only; optional; only for the self-signed certificate
  }
});

log.setConfig merges the data. Therefore, you can call it multiple times.

Basic functionality

var extra = {tom: 'cat', jerry: 'mouse', others: {spike: 1, tyke: 1}};

log.info("Hello world", extra, function (err, bytesSent) {});
log.info("Hello world", function (err, bytesSent) {});
log.info("Hello world", extra);
log.info("Hello world");

log.error('Oooops.', new Error('An error message'));
// ^-- extra becomes: {short_message: 'Oooops.', _error_message: 'An error message', _error_stack: Error's stack}

log.error(new Error('An error message'));
// ^-- extra becomes: {short_message: 'An error message', full_message: Error's stack}

log.message(new Error('An error message'), 3); // same as previous

Extra

In case extra is a plain object, the library converts it to a readable format. Other values are converted to string. The acceptable format of a key is: ^[\w-]$

log.info(
  'a new msg goes here',
  {me: {fname: 'k', lname: 'k', bdate: new Date(2000, 01, 01)}}
);
// ^-- extra becomes: {_me_fname: 'k', _me_lname: 'k', _me_bdate: 'Tue Feb 01 2000 00:00:00 GMT+0100 (CET)'}

Filtering

Sometimes we have to discard a message which is not suitable for the current environment. It is NOT possible to modify the data.

log.setConfig({
  filter: [
    function (message) { // rejects a "debug" message
      return (message.level < 7);
    }
  ]
});

Transforming

transforming happens after filtering. It is possible to modify the data.

log.setConfig({
  transform: [
    function (message) { // unwind an error
      if (_.isError(message.error)) {
        message.error = {message: message.error.message, stack: message.error.stack};
      }
      return message;
    }
  ]
});

Broadcasting

broadcasting happens after transforming. It is NOT possible to modify the data.

log.setConfig({
  broadcast: [
    function (message) { // broadcasting to console
      console[message.level > 3 ? 'log' : 'error'](message.short_message, message);
    }
  ]
});

Levels (1, 2, 3)

Default:
{emergency: 0, alert: 1, critical: 2, error: 3, warning: 4, notice: 5, info: 6, debug: 7}
Example: log.emergency(...), log.critical(...), etc.
Custom example: {alert: 0, notice: 1, ...}

TCP Connection Reuse

By default, the tcp and tcp-tls adapters create a new connection for every message. For high-throughput scenarios (e.g. hundreds of messages per second), you can enable persistent connection reuse with the keepAlive option.

When enabled, the adapter maintains persistent TCP connections, queues messages during outages, and automatically reconnects with exponential backoff.

log.setConfig({
  adapterName: 'tcp', // also works with 'tcp-tls'
  adapterOptions: {
    host: '127.0.0.1',
    port: 12201,
    keepAlive: true,
    keepAliveOptions: {
      poolSize: 1,                // number of persistent connections (round-robin); default: 1
      maxQueueSize: 5000,         // max messages buffered per connection while disconnected; default: 5000
      reconnectBaseDelay: 100,    // initial reconnect delay in ms; default: 100
      reconnectMaxDelay: 5000,    // max reconnect delay in ms (backoff cap); default: 5000
      reconnectMaxAttempts: 0,    // 0 = unlimited retries; default: 0
      writeTimeout: 5000,         // per-message write timeout in ms; default: 5000
      queueFullBehavior: 'drop-oldest', // 'drop-oldest', 'drop-newest', or 'error'; default: 'drop-oldest'
      onBatchDisconnect: 'retry'  // 'retry' = re-queue in-flight batch; 'drop' = discard; default: 'retry'
    }
  }
});

When you are done, close the connection to clean up:

log.close(function () {
  console.log('GELF connection closed');
});

How it works:

• Messages are queued and sent over persistent connections using socket.write()
• If a connection drops, messages are buffered and the adapter reconnects automatically
• Reconnection uses exponential backoff: 100ms -> 200ms -> 400ms -> ... -> 5s (cap)
• In-flight messages that fail mid-write are re-queued and retried by default (configurable via onBatchDisconnect)
• When the queue is full, the oldest messages are dropped by default (configurable)
• Message timestamps are set when log.info() is called, not when delivered, so queued messages retain their original timestamps

Connection Pooling

Setting poolSize greater than 1 creates multiple persistent TCP connections. Messages are distributed across them in round-robin order, multiplying the effective send buffer and increasing throughput.

Note on ordering: With poolSize greater than 1, messages may arrive at Graylog in a slightly different order than they were sent. Each connection drains independently, so two messages sent nanoseconds apart may be delivered out of sequence. The GELF timestamp field is always set at log time and remains correct, but log viewers that sort by receive time may display events out of order. If strict ordering matters for your logging strategy, keep poolSize at 1 or use the UDP adapter, which sends each message independently.

Batch Disconnect Policy

The onBatchDisconnect option controls what happens to messages that are mid-write when a connection drops:

• 'retry' (default): Re-queues the in-flight batch and retries on reconnect. Guarantees at-least-once delivery, but may produce rare duplicates if the server received some messages before the connection was lost.
• 'drop': Discards the in-flight batch and calls their callbacks with an error. Guarantees at-most-once delivery with no duplicates, but accepts potential message loss for the failed batch.

Third party adapters

You can force using custom adapter by setting the adapter right after initialisation. The signature might be found here.

  var log = require('gelf-pro-plus');
  var myFancyAdapter = require('...');
  log.adapter = myFancyAdapter;
  // (!) adapterName and adapterOptions will be ignored

Aliases

Default: {log: 'debug', warn: 'warning'}
Example: log.log(...) -> log.debug(...), log.warn(...) -> log.warning(...), etc.
Custom example: {red: 'alert', yellow: 'notice', ...}

Tests

Cli

npm install
npm test

Docker

[sudo] docker build --no-cache -t node-gelf-pro-plus .
[sudo] docker run -ti --rm -v "${PWD}:/opt/app" -w "/opt/app" node-gelf-pro-plus

Contributors

• Kanstantsin Kamkou
• corbinu
• joelharkes
• mkalam-alami

License

MIT