Logstash-Relay

A simple NodeJS service to relay JSNLogs or Log4Javascript/Log4JS(client) event messages to Logstash.

Table of Contents

1. Documentation
   1. Modes of Operation
      1. Stand-alone Service
         1. Installation
         2. Configuration
            1. Port Usage
            2. SSL/TLS Certs
         3. Running
      2. Embedded Service
         1. Installation
         2. Usage
         3. API
      3. AWS Lambda Function
         1. Installation, Packaging, and Deployment
         2. Usage
         3. Local Testing
      4. Docker Container
         1. Installation
         2. Configuration
         3. Running
   2. Configuration Object
   3. Logging Object
   4. Logstash
      1. Configuration
      2. Event Data
   5. REST API
      1. Endpoints
      2. Calling
2. License

Goals

logstash-relay is designed to be a simple pass-through service that relays external logging events from JSNLogs or Log4JS/Log4Javascript to Logstash with the goal of easily facilitating the collection of external logging centrally into ElasticSearch. It can be embedded within another service, setup and configured as a stand-alone service, or setup and configured as an AWS Lambda function.

Documentation

Modes of Operation

The logstash-relay service has four possible modes of operation:

• Stand-alone Service
• Embedded Service
• AWS Lambda Function
• Docker Container

Stand-alone Service

The stand-alone service option is available when it is desirable to use new/existing server architecture and more control over the environment is required.

Installation

To install, clone the git repository:

$ git clone https://github.com/OutOfSyncStudios/logstash-relay.git
$ cd logstash-relay

Configuration

To configure, run:

npm run config

This will ask a series of questions which provides the base configuration. Alternatively, it is possible to manually edit the <logstash-relay-home>/config/config.js file to make adjustments. The configuration file is an exported version of the Configuration Object.

Port Usage

By default, Logstash-Relay listens for HTTP request over port 8080 and HTTPS requests over port 8443 instead of ports 80 and 443 respectively. For security, on most Linux-based platforms the ports 80 and 443 are not available for services that do not run as the root user. To avoid any potential security issues or configuration hangups, it is recommended that Logstash-Relay is configured to use the default ports and that ports 80 and 443 are rerouted using iptables (or the system equivalent) to Logstash-Relay's default ports.

Additionally, firewall rules should be opened to allow traffic to the appropriate ports.

For Example:

Add Rules

$ iptables -t nat -A PREROUTING -i eth0 -p tcp --sport 80 --dport 8080 -j REDIRECT
$ iptables -t nat -A PREROUTING -i eth0 -p tcp --sport 443 --dport 8443 -j REDIRECT
$ service iptables save

Delete Rules

$ iptables -t nat -D PREROUTING -i eth0 -p tcp --sport 80 --dport 8080 -j REDIRECT
$ iptables -t nat -D PREROUTING -i eth0 -p tcp --sport 443 --dport 8443 -j REDIRECT
$ service iptables save

SSL/TLS Certs

If handling HTTPS requests are desired in stand-alone mode, then SSL/TLS certs must be included within the ./config/ssl folder of the project. Additionally, the file locations for the certificate files must be provided in the service configuration located in ./config/config.js. This file may be manually edited or configured using npm run config on the command-line. Both the Key and Cert files must be included and the files must have read permissions for the user that will be used to run the service.

Running

Before running, Logstash should be configured as outlined below. You should perform any of the operations listed below from the logstash-relay folder.

Note: When using stand-alone mode, it is recommended that a process manager, such as PM2, be used. Regardless of how the service is run, proper startup scripts will be needed to ensure that Logstash-Relay restarts whenever the server is rebooted.

With NodeJS

$ node main.js

With NodeJS and an external configuration

It is possible to pass an external configuration file.

$ node main.js -c <fullpath to config file>

With PM2

$ pm2 start main.js -n "Logstash-Relay" -i 1

With PM2 and an external configuration

It is possible to pass an external configuration file.

$ pm2 start main.js -n "Logstash-Relay" -i 1 -- -c <fullpath to config file>

Embedded Service

The embedded option is available if including the service bundled as a part of another service is desired.

Installation

npm install logstash-relay

Usage

Within a library or application, add the following code:

const LogstashRelay = require('logstash-relay');

// Create the relay service
let relay = new LogstashRelay(config, logger);

// Start the relay service
relay.init();

// Close the relay service
relay.close();

Note: The same considerations for port usage and certificates apply to using the service in embedded mode.

API

constructor(config[, logger])

Creates a new Logstash-Relay agent.

const LogstashRelay = require('logstash-relay');

let relay =  new LogstashRelay(config, logger);

Where the config and logger parameters are as outlined below. The logger is optional, and if no logger is provided then all logging is sent to /dev/null.

.init()

Initializes and starts the Logstash-Relay agent.

relay.init();

.close()

Shuts down the Logstash-Relay agent. Because the agent maintains an active thread, this operation must be performed to allow the application to gracefully shut down.

relay.close();

AWS Lambda Function

The service is also available to be run in a completely serverless environment by utilizing it as an AWS Lambda Function connected through CloudFormation and API Gateway. To use this mode, the AWS credentials must have permissions to create and use CloudFormation, S3, API Gateway, CloudFront, and Lambda resources.

Installation, Packaging, and Deployment

These steps support MacOS and Linux. When using Windows environment locally, it is recommended that a staging AWS EC2 Linux instance with NVM(NodeJS) and Python is setup and used.

1. If it is not already, install the AWS CLI on the staging environment (MacOS/Linux only).
2. Clone the git repo:

$ git clone https://github.com/OutOfSyncStudios/logstash-relay.git
$ cd logstash-relay

3. Run the base configurator:

$ npm run config

4. Run the AWS configurator:

$ npm run aws-config

5. Run the Setup:

$ npm run setup

This last step will create an S3 Bucket and a CloudFormation Stack. The CloudFormation Stack will in turn setup an AWS Lambda function and connect it to CloudFront and API GateWay. Additionally, routing through API Gateway and Route53 are possible to create a "pretty" URL that can connect to the API Gateway endpoint for the Lambda function. Please consult the AWS API Gateway documentation for additional details.

Usage

The Lambda function can be called by making the appropriate REST Endpoint request to the API Gateway prod endpoint. The correct prod Invoke URL can be retrieved throug the API Gateway Dashboard under APIs > (Your LogStashRelayAPI Name) > Stages.

Local Testing

To test as an AWS Lambda function, it is highly recommended to use the AWS SAM CLI Tool with Docker from your local environment and ensure that the Invoke URL returns data as expected with your configuration before deployment to AWS.

Docker Container

Installation

To install, clone the git repository:

$ git clone https://github.com/OutofSyncStudios/logstash-relay.git
$ cd logstash-relay

Configuration

To configure, run:

npm run config

This will ask a series of questions which provides the base configuration. Alternatively, it is possible to manually edit the <logstash-relay-base>/config/config.js file to make adjustments. The configuration file is an exported version of the Configuration Object.

Running

Before running, ensure that a compatible statsd service is configured to listen for events on the configured port and the any firewall rules are open between the service and the statsd service.

Building the Docker Image

$ npm run docker-build

Running the Built Docker Image

$ npm run docker-run

Configuration Object

{
  server: {
    port: 8080,
    shutdownTime: 1000,
    timeout: 5,
    sslEnabled: false,
    sslPort: 8443,
    sslKey: `${__dirname}/ssl/localhost.key`,
    sslCert: `${__dirname}/ssl/localhost.pem`
  },
  logging: {
    // Logging Configuration
    logDir: './logs',
    options: { json: false, maxsize: '10000000', maxFiles: '10', level: 'silly' },
    logstashLogging: false
  },
  logstash: {
    logging: { host: 'logstash-server', port: 5000, appName: 'logstash-relay' },
    relay: { host: 'logstash-relay-server', port: 5050, appName: 'client-errors' }
  }
}

|parameter|type|description| |---------|----|-----------| |server.port|Integer|The port to listen for HTTP requests| |server.shutdownTime|Integer|Time in millisecond to allow for graceful shutdown| |server.timeout|Integer|Time in seconds to wait to receive data from client before timeing out| |server.sslEnabled|Boolean|Enable handling of HTTPS requests (stand-alone and embedded modes only)| |server.sslPort|Integer|The port to listen for HTTPS request when sslEnabled === true| |server.sslKey|String|The full path to the Certificate .key file for HTTPS request when sslEnabled === true| |server.sslCert|String|The full path to the Certificate .pem/.crt file for HTTPS request when sslEnabled === true| |logging|Object|Required but only used in stand-alone mode. These options only pertain to logging events generated by the service, not events which are being relayed through the endpoints.| |logging.logDir|Integer|The full or relative path (from the Logstash-Relay base folder to store logs| |logging.options.json|Boolean|Store Logstash-Relay service events in JSON format| |logging.options.maxsize|Integer String|Max logfile size in bytes before logrotation| |logging.options.maxFiles|Integer String|Max number of rotated logfiles to keep for logrotation| |logging.options.level|String|The lowest log level to store in files (silly,debug,info,warn,error)| |logging.logstashLogging|Boolean|Send service log events to logstash| |logstash.logging|Object|Logstash information for Logstash-Relay log events when logging.logstashLogging === true| |logstash.relay|Object|Logstash information for relayed log events| |logstash.*.host|String|IP/Domain of the logstash server for this configuration| |logstash.*.port|Integer|UDP/TCP Port that handled the events| |logstash.*.appName|String|Unique identifying name that Logstash uses to classify events sent to ElasticSearch| |logstash.*.mode|String|Mode to send logstash messages. Expect the value 'udp' or 'tcp'. If no value is provided, then UDP is used|

Logging Object

The Logging object is an instance of any logging library, such as Winston or Bunyan, which support the .error(...), .info(...), .debug(...), and .log(...) methods. When in stand-alone mode, the service will use the configuration values to create an instance of Winston.

Logstash

Configuration

This package contains four files -- ./config/logstash-relay-tcp.conf, ./config/logstash-relay-udp.conf, ./config/logstash-logging-tcp.conf and ./config/logstash-logging-udp.conf -- that are necessary for proper configuration of the Logstash to listen for incoming events on the correct TCP/UDP ports. Only the associated one UDP or TCP file is needed depending on the configuration options.

logstash-logging-*.conf is only used for the stand-alone mode and handles events created by Logstash-Relay itself when the option logging.logstashLogging is set to true in the service configuration.

logstash-relay-*.conf handles all events that are passed to the relay and is necessary for proper functioning.

To setup Logstash with these files:

1. Copy the appropriate files to the Logstash server instance configuration folder. This is usually /etc/logstash/conf.d for most configurations.
2. Edit the file(s) and update the ElasticSearch server host address as appropriate for the environment.
3. Restart Logstash -- usually this is done with the command $ initctl restart logstash.

Event Data

Event data is passed to Logstash as a JSON object formatted like below:

{
  type: 'client_error',
  name: 'The name of the log',
  requestID: 'af4b33d2ae870d',
  client_error: 'error message or JSON object',
  actualIP: 'client_IP_address',
  ip: 'forwarded_client_IP_address_from_proxy',
  callID: 'UID for the relay request',
  headers: 'An object containing all http headers sent to the service',
  clientTimestamp: timestamp
}

|parameter|type|description| |---------|----|-----------| |type|String|Always 'client_error'| |name|String|The log name from JSNLogs events, not provided in Log4JS events| |requestID|String|A unique id used by JSNLogs to group events together, not provided in Log4JS events| |client_error|String or Object|The error message or object that was provided in the event| |actual_IP|String|The IPv4 or IPv6 of the request origin. If the request was proxied, this will be the proxy server address| |ip|String|The IPv4 or IPv6 of the request origin. If the request contains an x-forward header, then this will be the forwarded address| |callID|String|A UUIDv4 for the Logstash-Relay request, used for request tracking| |headers|Object|An object containing all the request headers from the origin. This usually contains the user-agent and other important details| |clientTimestamp|Timestamp|A timestamp of when the error was generated on the client. For JSNLogs, this will be the time the event was generated. For Log4JS, this will be the time the event was recieved by the relay.|

REST API

Once the service is setup and running, then the REST API microservice will be available. The service provides two identical POST endpoints.

Endpoints

OPTIONS *

This exists for CORS requests.

POST /api/logger

POST jsnlog.logger

Takes standard JSNLogs or Log4JS events passed in the POST body, and sends them to the configured Logstash service.

JSNLogs Events

JSNLogs events are structured as follows:

{
  r: 'requestID',
  lg: [
    {
      n: 'logName',
      l: 'logLevel',
      t: timestamp
      m: 'logMessage (may be a JSON object or string)'
    },
    ...
  ]
}

Log4JS Events

{
  level: 'logLevel',
  message: 'logMessage (string only)'
}

Calling

Setup JSNLogs or Log4JS to create AJAX requests that point to http(s)://<yourserviceURL>/api/logger or http(s)://<yourserviceURL>/jsnlog.logger. Alternatively, call these endpoints directly within an application using one of the log formats outlined above.

License

Copyright (c) 2017, 2018, 2019 Jay Reardon Copyright (c) 2019 Out of Sync Studios LLC -- Licensed under the MIT license.