llm-observability

Structured observability for LLM requests — OpenTelemetry metrics + Pino logging.

Install

bun add @meistrari/llm-observability

Usage

import { recordSuccess, recordError } from '@meistrari/llm-observability'

// Record a successful LLM call
recordSuccess({
    attributes: { provider: 'openai', model: 'gpt-4' },
    metrics: {
        duration: 1200,
        cost: 0.03,
        totalTokens: 500,
        promptTokens: 400,
        completionTokens: 100,
    },
}, 'Chat completion succeeded')

// Record a failed LLM call
recordError({
    attributes: { provider: 'anthropic', model: 'claude-3' },
    metrics: { duration: 5000 },
    error: {
        errorType: 'rate_limit',
        httpStatus: 429,
        retriable: true,
        reason: 'Too many requests',
    },
}, 'Chat completion failed')

API

recordSuccess(params, message)

Records a successful LLM request: increments request counter, logs via Pino, records metric histograms.

| Parameter | Type | Description | |-----------|------|-------------| | params.attributes | Record<string, any> | Custom attributes (e.g. provider, model) | | params.metrics | MetricValues | Metric values to record | | message | string | Log message |

recordError(params, message)

Records a failed LLM request: increments request + error counters, logs error, optionally records duration.

| Parameter | Type | Description | |-----------|------|-------------| | params.attributes | Record<string, any> | Custom attributes | | params.metrics | { duration?: number } | Optional duration | | params.error | ErrorMetadata | Error details | | message | string | Log message |

MetricValues

| Field | Type | |-------|------| | duration | number? | | cost | number? | | totalTokens | number? | | promptTokens | number? | | completionTokens | number? | | reasoningTokens | number? | | pages | number? |

ErrorMetadata

| Field | Type | |-------|------| | errorType | string? | | errorMessage | string? | | retriable | boolean? | | reason | string? | | httpStatus | number? |

Metrics Emitted

| Metric Name | Type | Description | |-------------|------|-------------| | llm.request.count | Counter | Total requests (success + error) | | llm.error.count | Counter | Failed requests | | llm.request.duration | Histogram | Request duration | | llm.cost.total | Histogram | Cost per request | | llm.tokens.total | Histogram | Total tokens | | llm.tokens.prompt | Histogram | Prompt tokens | | llm.tokens.completion | Histogram | Completion tokens | | llm.tokens.reasoning | Histogram | Reasoning tokens | | llm.pages.total | Histogram | Pages processed |