Uriel

A simple service that pushes system information (e.g. system usage, memory, cpu, network, swap, diskio, and disk usage) to any compatible statsd service (e.g. StatsD, Telegraf, DogStatsD, etc.)

Note: Version 2.0.0 contains breaking changes to how some statistics are delivered to the statsd server, and how the Uriel process is configured. Please consult the documentation for information on these breakages.

Table of Contents

1. Migrating from 1.x
2. Documentation
   1. Modes of Operation
      1. Stand-alone Service
         1. Installation
         2. Configuration
         3. Running
      2. Docker Container
         1. Installation
         2. Configuration
         3. Running
      3. Embedded Service
         1. Installation
         2. Usage
         3. API
   2. Configuration Object
   3. Logging Object
   4. Statsd Buckets
3. Changelog
4. License

Goals

Uriel is designed to be a lightweight NodeJS agent that gathers system information and delivers it to a compatible statsd service over UDP. It can be embedded within another service, or setup and configured as its own stand-alone service that runs on the system being monitored. At the time of creation, other NodeJS systeminfo agents were unsuitable due to their inflexibility or poor implementations. Uriel was created to (ideally) bridge those shortcomings. It uses UDP, because of the decreased network overhead that is required in comparison to TCP.

The name Uriel is associated with Abrahamic religions for the Archangel which represents the light of the divine and as one who had dominion over another type of angel, the Gregori (a.k.a. The Watchers). The name was chosen both because of its association with illuminating dark places and watching things which are two abstract concepts related to system monitoring.

Migrating from 1.x

So that grouped dashboards are possible, version 2.x moves CPU and Disk Usage reporting for individual cpu/disks and the aggregate totals into tagged metrics. Any dashboards which rely on this reporting will need to be updated as follows:

• If you are monitoring total CPU or Disk stats, then you will need to update your dashboards to look for the cpu:total or disk:total respectively.
• If you are monitoring individual CPU or Disks, then you will need to update your dashboards from cpu.cpu0_usage_idle / disk.disk0_free to cpu.usage_idle / disk.free with tags cpu:0 / disk:0.

It is also now possible to configure global tags which are sent for all metrics in the server configuration. These are setup with the statsd.tags array of the server configuration. This allows for systems being monitored in auto-scaling groups or under load-balancing to be grouped together by shared tags in dashboards. This is especially useful under architectures where the exact specification (size/number of systems/etc.) is mutable.

Documentation

Modes of Operation

The Uriel service has three modes of operation:

• Stand-alone Service
• Docker Container
• Embedded Service

Stand-alone Service

The stand-alone service option is available when it is desirable to use new/existing server architecture and more control over the environment is required.

Installation

To install, clone the git repository:

$ git clone https://github.com/OutOfSyncStudios/uriel.git
$ cd uriel

Configuration

To configure, run:

npm run config

This will ask a series of questions which provides the base configuration. Alternatively, it is possible to manually edit the <uriel-base>/config/config.js file to make adjustments. The configuration file is an exported version of the Configuration Object.

Running

Before running, ensure that a compatible statsd service is configured to listen for events on the configured port and the any firewall rules are open between the service and the statsd service. All operations below should be performed from the Uriel base folder.

Note: When using stand-alone mode, it is recommended that a process manager, such as PM2, be used. Regardless of how the service is run, proper startup scripts will be needed to ensure that Uriel restarts whenever the server is rebooted.

With NodeJS

$ node app.js

With NodeJS and an external configuration

It is possible to pass an external configuration file.

$ node app.js -c <fullpath to config file>

With PM2

$ pm2 start app.js -n "Uriel" -i 1

With PM2 and an external configuration

$ pm2 start app.js -n "Uriel" -i 1 -- -c <fullpath to config file>

Docker Container

Installation

To install, clone the git repository:

$ git clone https://github.com/OutOfSyncStudios/uriel.git
$ cd uriel

Configuration

To configure, run:

npm run config

This will ask a series of questions which provides the base configuration. Alternatively, it is possible to manually edit the <uriel-base>/config/config.js file to make adjustments. The configuration file is an exported version of the Configuration Object.

Running

Before running, ensure that a compatible statsd service is configured to listen for events on the configured port and the any firewall rules are open between the service and the statsd service.

Building the Docker Image

$ npm run docker-build

Running the Built Docker Image

$ npm run docker-run

Embedded Service

The embedded option is available if including the service bundled as a part of another service is desired.

Installation

$ npm install uriel

Usage

Within a library or application, add the following code:

  const Uriel = require('uriel');

  // Create a new agent
  let statsd = new Uriel(config, logger);

  // Initialize and start the uriel agent
  statsd.init();

  // close and shutdown the uriel agent
  statsd.close();

Note: The same considerations for firewall rules must be made when running the Uriel service in embedded mode.

API

constructor(config[, logger])

Create a new Uriel agent.

const Uriel = require('uriel');

let statsd = new Uriel(config, logger);

Where the config and logger parameters are as outlined below. The logger is optional, and if no logger is provided then all logging is sent to /dev/null.

.init()

Initializes and starts the Uriel statsD agent

statsd.init();

.close()

Shuts down the Uriel StatsD agent. Because the agent maintains an active thread, this operation must be performed to allow the application to gracefully shut down.

statsd.close();

Configuration Object

The configuration parameter expects and object that contains the following (with defaults provided below):

{
  server: {
    shutdownTime: 1000,
    pollingTimer: 5000
  },
  logging: {
    // Logging Configuration
    logDir: './logs',
    options: { json: false, maxsize: '10000000', maxFiles: '10', level: 'silly' }
  },  
  statsd: {
    host: '127.0.0.1',
    port: '8125',
    name: 'Uriel',
    attachHostName: false,
    telegraf: false,
    tags: []
  }
}

|parameter|type|description| |---------|----|-----------| |server.shutdownTime|Integer|Time in millisecond to allow for graceful shutdown| |server.pollingTime|Integer|Time in millisecond to wait between polling the system information and delivering to statsd| |logging.logDir|Integer|The full or relative path (from the Uriel base folder to store logs| |logging.options.json|Boolean|Store Uriel service events in JSON format| |logging.options.maxsize|Integer String|Max logfile size in bytes before logrotation| |logging.options.maxFiles|Integer String|Max number of rotated logfiles to keep for logrotation| |logging.options.level|String|The lowest log level to store in files (silly,debug,info,warn,error)| |statsd.host|String|IP/Domain of the statsd server for this configuration| |statsd.port|Integer|UDP Port that handles the system information| |statsd.name|String|the serverName tag that is provided for all stats that are pushed. This allows info from differing servers to be distinguished from one another| |statsd.attachHostName|Boolean|true or false value that specifies that the os hostname should be appended to the serverName| |statsd.telegraf|Boolean|true or false value that specifies that the listening server is running telegraf| |statsd.tags|Array|An array of tag strings formatted key:value that are passed onto the statsd server

Logging Object

The Logging object is an instance of any logging library, such as Winston or Bunyan, which support the .error(...), .info(...), .debug(...), and .log(...) methods. When in stand-alone mode, the service will use the configuration values to create an instance of Winston.

Statsd Buckets

The following buckets are used to capture statistics:

|name|domain|description| |----|------|-----------| |system.uptime|System|System uptime| |system.load1|System|System 1-minute average load| |system.load5|System|System 5-minute average load| |system.load15|System|System 15-minute average load| |cpu.usage_user|CPU|% of CPU usage from user processes| |cpu.usage_nice|CPU|% of CPU usage from low-priority/background processes| |cpu.usage_system|CPU|% of CPU usage from system/kernel processes| |cpu.usage_idle|CPU|% of CPU idle| |cpu.usage_irq|CPU|% of CPU usage from system IRQs| |cpu.usage_total|CPU|% of non-idle CPU usage| |cpu.num_cpus|CPU|Count of CPUs on the system| |mem.free|Memory|Free Memory (in bytes)| |mem.free_percent|Memory|% of Free Memory| |mem.total|Memory|Total Memory (in bytes)| |mem.used|Memory|Used Memory (in bytes)| |mem.used_percent|Memory|% of Used Memory| |network.close_wait|Network|Count of network packets in CLOSED_WAIT state| |network.foreign|Network|Count of network packets in FOREIGN state| |network.established|Network|Count of network packets in ESTABLISHED state| |network.last_ack|Network|Count of network packets in LAST_ACK state| |network.listen|Network|Count of network packets in LISTEN state| |network.syn_sent|Network|Count of network packets in SYN_SENT state| |network.time_wait|Network|Count of network packets in TIME_WAIT state| |diskio.io_time|Disk I/O|Current time spent doing I/O (in milliseconds) (1s = 100% load)| |diskio.iops_in_progress|Disk I/O|Count of current number of disk operations| |diskio.weighted_io_time|Disk I/O|Measure of both I/O completion and estimated backlog| |diskio.read_bytes|Disk I/O|Total number of bytes read from disk| |diskio.read_time|Disk I/O|Total I/O time (1/100th of seconds) for read requests to disk| |diskio.write_bytes|Disk I/O|Total number of bytes written to disk| |diskio.write_time|Disk I/O|Total I/O time (1/100th of seconds) for write requests to disk| |disk.free|Disk Usage|Free Disk (in bytes)| |disk.free_percent|Disk Usage|% of Free Disk| |disk.total|Disk Usage|Total Disk (in bytes)| |disk.used|Disk Usage|Used Disk (in bytes)| |disk.used_percent|Disk Usage|% of Used Disk| |disk.num_disks|Disk Usage|Count of Disks in the system| |swap.free|Swap Usage|Free Swap (in bytes)| |swap.free_percent|Swap Usage|% of Free Swap| |swap.total|Swap Usage|Total Swap (in bytes)| |swap.used|Swap Usage|Used Swap (in bytes)| |swap.used_percent|Swap Usage|% of Used Swap|

Note: CPU or Disk Usage are delivered as tagged sets, with the 'cpu' and 'disk' tags respectively to mark the individual cpu or disk numbers or the 'total' for the aggregation of all cpus or disks.

Changelog

2.0.2

• Minor bug fixes

2.0.1

• Minor bug fixes

2.0.0

Note: This is a major version change that contains breaking changes to the old function & reporting. Please review the documentation for updates to how data is now reported.

• Improvements on delivery and handling of tagged metrics, an array of tags can now be configured to be delivered with each metric
• CPU and Disk Usage reporting are now delivered as tagged sets of data. The cpu and disk tags provide enumerate each of the individual cpus and disks; or have the tag value total to indicate the aggregation of all items.
• Refactored all monitors to be execute as Promises to reduce memory and cpu overhead
• Refactored statistics collection to use a factory pattern in an effort to further reduce memory overhead
• Migrated to Winston v3 logging

1.9.6

• Updating outdated dependencies again
• Additional improved logging
• Removed unnecessary dependencies

1.9.5

• Finally fixed CPU / Memory leak causing CPU panic
• Improved Logging
• Improved Unit Testing to check for the fixed bug

1.9.4

• Reverting hot-shots and systeminformation version to previous version in 1.8 branch as fix for the CPU race condition that is still being reported

1.9.3

• Reverting systeminformation dependency to version 3.51 -- the Physical CPU gathering logic added added in 3.52 causes major issues with cpu usage overhead

1.9.2

• Additional optimization of the inner monitoring loop

1.9.1

• Fixed bug causing excessive CPU usage when Uriel had been running for a while

1.9.0

• Moved some logging messages to silly level logging.
• Added tags configuration array so that other metrics may be hard passed to statsd

1.8.1

• Fixing console logging bug

1.8.0

• Added Docker support
• Added individual CPU and Disk reporting

1.7.0

• Fixed all outdated dependencies and security vulnerabilities

License

Copyright © 2017-2019 Jay Reardon Copyright © 2019-2021 Out of Sync Studios LLC -- Licensed under the MIT license.