Node-RED OpenTelemetry

Distributed tracing with OpenTelemetry SDK and Prometheus metrics exporter for Node-RED

Key features

Traces

• based on OpenTelemetry JavaScript framework and Node-RED messaging hooks:
  • create spans on onSend(source) and postDeliver(destination) events,
  • end spans on onComplete and postDeliver(source) events.
• one trace per flow execution: every node reached by a message is a child span of a trace, regardless of any changes made to the message's identifier (see What is a run span?),
• completes the caller trace when the entry node receives a W3C trace context (http in headers, mqtt in v5 user properties, amqp-in headers),
• trace includes:
  • run id,
  • type of the node that triggered the run (node_red.trigger.type, on the run span),
  • message id,
  • flow id,
  • node id,
  • node type,
  • node name (if filled),
  • hostname,
  • optional http status code (for request node type),
  • optional exception,
  • optional custom attributes based on message data.

What is a run span?

A trace corresponds to an execution of your flow. The first node to emit a message opens a run span, each node reached by the message becomes a child span, and the run span ends when the last of those node spans ends. The run span carries node_red.trigger.type, which is the type of the node that triggered the execution. This allows a collector to route or filter it without having to read the spans of the individual nodes.

Node-RED gives a fresh _msgid to every message object a node emits (a function node returning {payload: ...} instead of the message it received, a split node, a join node, ...). To keep those in the same trace, the run is carried on the message in the otelRootMsgId property, which you will therefore see on messages in the debug sidebar.

Two situations deliberately produce more than one trace, following the messaging semantic conventions:

• a node that acknowledges a message and emits later (delay, trigger, a rate limiter) hands over to a new run, linked to the one it came from with a span link (node_red.link.type: continuation),
• a message published to a broker and consumed by another flow is a run of its own, correlated through the propagated trace context.

Nodes that never report completion (tcp in, server-events, ...) have their spans closed as soon as the message they created has been dispatched. A run that remains incomplete beyond the configured timeout is closed by a cleanup, and the spans of nodes that are still open are ended and flagged with node_red.span.incomplete instead of being dropped.

Metrics

• export of request metrics from http in nodes (for Prometheus scraping):
  • method,
  • route,
  • status,
  • ip
  • duration.

curl http://localhost:1881/metrics
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_name="Node-RED",telemetry_sdk_language="nodejs",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.30.0"} 1
# HELP http_request_duration Response time for incoming http requests in milliseconds
# UNIT http_request_duration ms
# TYPE http_request_duration histogram
http_request_duration_count{method="POST",route="/api/test",status="201",ip="127.0.0.1"} 5
http_request_duration_sum{method="POST",route="/api/test",status="201",ip="127.0.0.1"} 620
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="0"} 0
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="25"} 0
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="50"} 4
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="75"} 4
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="100"} 4
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="250"} 4
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="500"} 4
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="1000"} 5
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="2000"} 5
http_request_duration_bucket{method="POST",route="/api/test",status="201",ip="127.0.0.1",le="+Inf"} 5

Installation

Search node-red-contrib-opentelemetry within the palette manager or install with npm from the command-line (within your user data directory):

npm install node-red-contrib-opentelemetry

As with every node installation, you will need to restart Node-RED for it to pick-up the new nodes.

Usage

Traces

• Add OTEL node once (to any flow),
• Setup the node:
  • set OTEL exporter url (example for Jaeger: http://localhost:4318/v1/traces),
  • choose an OTLP transport protocol (http/json or http/protobuf),
  • set the auth scheme your collector requires (see Exporter authentication),
  • define a service name (will be displayed as span service),
  • define an optional root span prefix (will be added in Node-RED root span name),
  • define nodes that should not send traces (using comma-separated list like debug,catch),
  • define nodes that should propagate W3C trace context (in http request headers, using comma-separated list like http request,my-custom-node),
  • define time in seconds after which an inactive run is considered abandoned and closed,
  • define custom attributes you want to send (optionally).

Exporter authentication

Most hosted collectors require credentials. Select an authentication scheme on the OTEL configuration node:

| Scheme | Header sent | |----------------- |-------------------------------------------------------- | | None | none | | Bearer token | Authorization: Bearer <token> | | Basic | Authorization: Basic <base64 of user:password> | | Custom header | <header name>: <value>, for example x-api-key: ... |

The secret is kept in the node credentials, so it is stored apart from the flow: it does not end up in flows.json, nor in a flow you export or commit.

It is also possible to send non-confidential headers (such as the name of a dataset, an organization, or a feed) via the additional exporter headers.

Headers set in the OTEL_EXPORTER_OTLP_HEADERS environment variable are still sent, which is convenient for container deployments:

OTEL_EXPORTER_OTLP_HEADERS="authorization=Basic cm9vdDpwYXNz,x-tenant=acme"

A header configured on the node takes precedence over the environment for the same name.

Metrics

• Add Prometheus node once (to any flow),
• Setup the node:
  • set Prometheus export port and endpoint (example: 1881 and /metrics),
  • define a service name (will be displayed in export),
  • define a instrument name (will be displayed in export),
• Add middleware to your settings.js file:

    // import the prometheus middleware
    const { prometheusMiddleware } = require('node-red-contrib-opentelemetry/lib/prometheus-exporter.js')
    // ...
    // then add it to the existing httpNodeMiddleware attribute
    httpNodeMiddleware: prometheusMiddleware,
    // ...

Versioning

node-red-contrib-opentelemetry is maintained under the semantic versioning guidelines.

See the releases on this repository for changelog.

Contributors

• Nioc - Initial work
• Wodka - AMQP headers and CompositePropagator (Jaeger, W3C, B3)
• Akrpic77 - MQTT v5 context fields
• Joshendriks - Protobuf trace-exporter support
• Czepiec - node_red.flow.name span attribute
• Syron - Improve flow tracing (per run), exporter authentication and tests

See also the full list of contributors to this project.

Direct dependencies

• @opentelemetry (Apache-2.0)
• jmespath (Apache-2.0)
• on-finished (MIT)

License

This project is licensed under the GNU Lesser General Public License v3.0 - see the LICENSE file for details