Traceable Nodejsagent

nodejsagent provides a set of instrumentation & blocking features for collecting relevant data to be processed by the Traceable.ai platform.

Prerequisites

The agent supports Node 8.17 and up.

The package comes with precompiled extensions that support the following operating systems & associated versions:

• Debian 10+
• Ubuntu 18.04+
• Centos 7+

If you are using the agent on an unsupported operating system, the observability features will work; however blocking functionality will not be loaded.

Installing

To install the package: npm install @traceable/nodejsagent

Configuration

The agent supports two methods of configuration.

• Environment variables
• Configuration file

For the agent to use a config file you must set a environment variable TA_CONFIG_FILE=./config.yaml, where the value is a path to the config.yaml file.

A minimal config.yaml is below:

service_name: node_app
reporting:
  endpoint: http://agent.traceableai:4317
opa:
  endpoint: http://agent.traceableai:8181/
blocking_config:
  enabled: true
  remote_config:
    endpoint: agent.traceableai:5441

A complete config.yaml with default values is shown below:

service_name: nodeagent
reporting:
  endpoint: http://localhost:4317
  secure: false
  trace_reporter_type: OTLP
data_capture:
  http_headers:
    request: true
    response: true
  http_body:
    request: true
    response: true
  rpc_metadata:
    request: true
    response: true
  rpc_body:
    request: true
    response: true
  body_max_size_bytes: 131072
  body_max_processing_size_bytes: 131072
propagation_formats: ["TRACECONTEXT"]
enabled: true
resource_attributes: { }
opa:
  enabled: true
  endpoint: http://opa.traceableai:8181/
  poll_period_seconds: 60
blocking_config:
  enabled: true
  debug_log: true
  evaluate_body: true
  skip_internal_request: false
  modsecurity:
    enabled: true
  regionBlocking:
    enabled: true
  remote_config:
    enabled: true
    endpoint: localhost:5441/
    poll_period_seconds: 30

The full list of supported environment variables is listed below:

| Name |Default| Description | |------|------------|-------------| | TA_OPA_ENABLED |true| When true Open Policy Agent evaluation is enabled to block request | | TA_OPA_ENDPOINT |http://opa.traceableai:8181/| Represents the endpoint for polling OPA config file e.g. http://opa.traceableai:8181/ | | TA_OPA_POLL_PERIOD_SECONDS |60| Poll period in seconds to query OPA service | | TA_BLOCKING_CONFIG_ENABLED | false | The top level switch that enables/disables blocking functionality | TA_BLOCKING_CONFIG_DEBUG_LOG | false | Enables verbose logging of the native extensions | TA_BLOCKING_CONFIG_MODSECURITY_ENABLED | true | | TA_BLOCKING_CONFIG_EVALUATE_BODY | true | Determines if the request body should be analyzed | TA_BLOCKING_CONFIG_REGION_BLOCKING_ENABLED | true | Enables IP based blocking | TA_BLOCKING_CONFIG_REMOTE_CONFIG_ENABLED | true | Denotes if config needs to be fetched from remote or not | | TA_BLOCKING_CONFIG_REMOTE_CONFIG_ENDPOINT |localhost:5441| Denotes the agentmanager endpoint to connect to for config. eg: localhost:5441 | | TA_BLOCKING_CONFIG_REMOTE_CONFIG_POLL_PERIOD_SECONDS |30| Poll period in seconds to query for config updates | | TA_BLOCKING_CONFIG_SKIP_INTERNAL_REQUEST |true| When true, blocking evaluation will be skipped for internal requests i.e. requests coming from private IPs | | TA_SERVICE_NAME |nodeagent| Identifies the service/process running e.g. "my service" | | TA_REPORTING_ENDPOINT |http://localhost:4317| Represents the endpoint for reporting the traces For ZIPKIN reporter type use http://api.traceable.ai:9411/api/v2/spans For OTLP reporter type use http://api.traceable.ai:4317 | | TA_REPORTING_SECURE |false| When true, connects to endpoints over TLS. | | TA_DATA_CAPTURE_HTTP_HEADERS_REQUEST |true| When false it disables the capture for the request in a client/request operation | | TA_DATA_CAPTURE_HTTP_HEADERS_RESPONSE |true| When false it disables the capture for the response in a client/request operation | | TA_DATA_CAPTURE_HTTP_BODY_REQUEST |true| When false it disables the capture for the request in a client/request operation | | TA_DATA_CAPTURE_HTTP_BODY_RESPONSE |true| When false it disables the capture for the response in a client/request operation | | TA_DATA_CAPTURE_RPC_METADATA_REQUEST |true| When false it disables the capture for the request in a client/request operation | | TA_DATA_CAPTURE_RPC_METADATA_RESPONSE |true| When false it disables the capture for the response in a client/request operation | | TA_DATA_CAPTURE_RPC_BODY_REQUEST |true| When false it disables the capture for the request in a client/request operation | | TA_DATA_CAPTURE_RPC_BODY_RESPONSE |true| When false it disables the capture for the response in a client/request operation | | TA_DATA_CAPTURE_BODY_MAX_SIZE_BYTES |131072| Is the maximum size of captured body in bytes. Default should be 131_072 (128 KiB). | | TA_DATA_CAPTURE_BODY_MAX_PROCESSING_SIZE_BYTES |131072| Is maximum size of body being processed by filters in bytes. For uncompressed bodies we capture all bytes up to body_max_processing_size_bytes in memory and pass that through the filter. For compressed and GRPC bodies, if the size of the body is larger than this, we ignore it entirely, otherwise we decompress/decode the body and then pass it to the filter. | | TA_PROPAGATION_FORMATS |['TRACECONTEXT']| List the supported propagation formats e.g. TA_PROPAGATION_FORMATS="B3,TRACECONTEXT". | | TA_ENABLED |true| When false, disables the agent | | TA_LOG_LEVEL | info | Controls granularity of produced logs, one of: debug, info, warning, error|

Instrumenting

The agent offers two methods for instrumenting an application: 1.) Auto instrumentation 2.) Manual instrumentation

Auto instrumentation

Add the -r @traceable/nodejsagent to your apps node startup command. Ex: TA_CONFIG_FILE=./config.yaml node -r @traceable/nodejsagent server-a.js

Note: If using ES6 import syntax in your application you must use auto instrumentation instead of manual instrumentation Ex: import express from 'express' instead of const express = require('express'). The former requires auto instrumentation.

Manual instrumentation

To manually instrument an application you will need to edit the apps main entrypoint file.

Ex:

    const traceable = require('@traceable/nodejsagent')
    const agent = new traceable.TraceableAgent()
    agent.instrument() // Enables collection of observability data
    agent.addTraceableFilter() // Enables blocking features

Development

Setup

• Initialize agent-config repo

  git submodule update --init --recursive

Updating Libtraceable

1. Replace the header file ./src/ext/lib/libtraceable.h
2. Update the LIBTRACEABLE_VERSION in `./prebuild-ci/docker-compose.yaml

Building and Packaging

1. Build:

   cd prebuild-ci;  ./generate.sh;  cd -

2. Package:

   npm run build

3. Artifact: traceable-nodejsagent-0.0.0.tgz

Testing

1. Copy the artifact to ./examples

   cp traceable-nodejsagent-0.0.0.tgz ./examples

2. Edit config.yaml to set up traceable agent. By default it looks at localhost for traceable agent
3. Build the test app docker image

   docker build -t nodejsagent-test-app-image .

   For Alpine use Dockerfile-alpine.

   docker build -t nodejsagent-test-app-image -f Dockerfile-alpine .

4. Run the test app. By default, the sample nodejs app listens on 8001.

   docker run --name nodejsagent-test-app --rm -p 8001:8001 nodejsagent-test-app-image

5. Send test requests

   curl localhost:8001/test
   curl -H "x-forwarded-for: 1.39.0.0" -XPOST "http://localhost:8001/test-post" -H 'Accept: application/json, */*;q=0.5' -H 'Content-type: application/json' -d @request500b.json

6. Example blocked requests

   curl -vvv -H "x-forwarded-for: 1.40.0.0" "localhost:8001/test"
   curl -vvv -H "x-forwarded-for: 8.8.8.8" "localhost:8001/test?a=/usr/bin/perl"
   curl -vvv -H "x-forwarded-for: 8.8.8.8" "localhost:8001/test?a=blockmetyrone"
   curl -vvv -H "x-forwarded-for: 8.8.8.8" -XPOST "http://localhost:8001/test-post" -H 'Accept: application/json, */*;q=0.5' -H 'Content-type: application/json' -d @requestBAD.json