Node Observability

Shared Node.js observability for Kittl services and workers.

@kittl/observability-node provides a single entry point for OpenTelemetry tracing and metrics.

How to use

Create telemetry.ts file and make sure it is imported into your entrypoint:

import { KittlNodeTelemetry } from '@kittl/observability-node';

const telemetry = new KittlNodeTelemetry({
  serviceName: 'api',
  serviceVersion: process.env.RELEASE_SHA,
  owner: 'developers',
});
telemetry.start();

With default parameters this sets up Node auto-instrumentation and exports both metrics and traces. Metrics will be exposed on port 9464 under /metrics path by default. If you use @kittl/logger, logs will be correlated with traces automatically.

It is recommended to shut down the instance gracefully based on SIGTERM signal:

process.on('SIGTERM', () => {
  telemetry
    .shutdown()
    .then(
      () => logger.info('SDK shut down successfully'),
      (err) => logger.error('Error shutting down SDK', err)
    )
    .finally(() => process.exit(0));
});

Node auto-instrumentations can be fine-tuned using nodeAutoInstrumentations parameter:

new KittlNodeTelemetry({
  serviceName: 'api',
  nodeAutoInstrumentations: {
    '@opentelemetry/instrumentation-express': {
      ignoreLayersType: [ExpressLayerType.MIDDLEWARE, ExpressLayerType.ROUTER],
    },
  },
});

NOTE: For each auto instrumentation configuration, only specified parameters will be overridden. Remaining parameters will be set to default values.

Check supported-instrumentations for information on what is included.

Additional instrumentations can be added via additionalInstrumentations parameter:

import { PrismaInstrumentation } from '@prisma/instrumentation';

new KittlNodeTelemetry({
  serviceName: 'api',
  additionalInstrumentations: [new PrismaInstrumentation()],
});

Kubernetes configuration

To make sure that service metrics are collected and traces are exported, following minimal values.yaml configuration is required:

env:
  - name: NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://$(NODE_IP):4318
deployment:
  metrics:
    enabled: true
    port: 9464 # change if different port is used
    path: /metrics

In addition following environment variables are recommended to be set via configmap:

OTEL_LOG_LEVEL: warn
# use new stable HTTP conventions
OTEL_SEMCONV_STABILITY_OPT_IN: http

User journeys

This package provides abstraction for tracking user journeys in a standardized way via Journey class. It supports recording step transitions as a counter metric, as well as recording step durations into a histogram.

Steps are particular operations inside the journey. For example, AI generation journey might contain separate steps to charge the user, generate image and refund the user in case of error. Each step has a particular outcome - for example, generation might fail with an error, be successful or it might time out. For more information on this topic see Business Metrics Taxonomy

Here's an example of tracking generation step in ai_generation journey using recordStep method:

import { Journey, ATTR_OUTCOME, ATTR_ERROR_CATEGORY } from '@kittl/observability-node';

const aiGenerationJourney = new Journey('ai_generation');

function generateAiImage(...) {
  try {
    /** some business logic */
  } catch {
    aiGenerationJourney.recordStep('generation', {
      [ATTR_OUTCOME]: 'failure',
      [ATTR_ERROR_CATEGORY]: 'generation_failed',
    });
    return;
  }
  aiGenerationJourney.recordStep('generation', {
    [ATTR_OUTCOME]: 'success',
  });
}

NOTE: Error category is always tracked, but defaults to none when it is not specified.

By default Journey only allows passing built-in ATTR_OUTCOME and ATTR_ERROR_CATEGORY attributes. To include additional attributes, they should be fined as an interface and passed to Journey as a generic type argument:

import { ATTR_PROVIDER } from '@kittl/observability-node';

const ATTR_CONTENT_TYPE = 'content.type';

interface AiGenerationJourneyAttributes {
  [ATTR_PROVIDER]?: 'nano-banana' | 'gemini' | 'stripe';
  [ATTR_CONTENT_TYPE]?: 'image' | 'video';
}

const aiGenerationJourney = new Journey<AiGenerationJourneyAttributes>(
  'ai_generation'
);
aiGenerationJourney.recordStep('generation', {
  [ATTR_OUTCOME]: 'success',
  [ATTR_PROVIDER]: 'nano-banana',
  [ATTR_CONTENT_TYPE]: 'image',
});

In cases where we are also interested in duration of a step, it is possible to also track step duration via recordStepWithDuration method. It does same as what recordStep does, but also records duration measurement in a histogram. Here's an example of tracking a different step in ai_generation journey:

function chargeUserForGeneration(...) {
  const startTime = performance.now();
  /** some business logic */
  aiGenerationJourney.recordStepWithDuration(
    'charge',
    (performance.now() - startTime) / 1000, // convert to seconds
    {
      [ATTR_OUTCOME]: 'success',
      [ATTR_PROVIDER]: 'stripe'
    }
  ),
}

NOTE: This example omits error handling for simplicity, but it is necessary when tracking durations in production code.

As this pattern of tracking start time and subtracting it from current time is so common, Journey class provides a useful createTimer method to simplify it:

const timer = aiGenerationJourney.createTimer();
/** some businees logic */
timer.recordStepWithDuration('charge', {
  [ATTR_OUTCOME]: 'success',
  [ATTR_PROVIDER]: 'stripe',
});