@verydia/telemetry

Observability and telemetry system for Verydia. Provides distributed tracing, monitoring, and observability across LLM providers, workflows, and safety systems.

Features

• Global Telemetry Collector - Centralized event dispatcher with pluggable exporters
• Comprehensive Event Model - LLM calls, workflows, steps, tool calls, and safety scores
• Multiple Exporters - Console, JSONL file, and S3/S3-compatible storage
• Backend-Agnostic - Works in Node.js, browser, and edge environments
• Zero Runtime Dependencies - Except optional @aws-sdk/client-s3 for S3 exporter
• Non-Blocking - Telemetry never disrupts application flow
• TypeScript-First - Full type safety and IntelliSense support

Installation

pnpm add @verydia/telemetry

For S3 exporter support:

pnpm add @verydia/telemetry @aws-sdk/client-s3

Quick Start

import {
  globalTelemetryCollector,
  ConsoleExporter,
  JsonlExporter,
  S3Exporter,
} from "@verydia/telemetry";

// Configure exporters at application startup
globalTelemetryCollector.addExporter(new ConsoleExporter());

// Optional: Add file exporter for CI/CD
globalTelemetryCollector.addExporter(
  new JsonlExporter({ filepath: "./telemetry.jsonl" })
);

// Optional: Add S3 exporter for production
globalTelemetryCollector.addExporter(
  new S3Exporter({
    bucket: "my-telemetry-bucket",
    prefix: "verydia/",
    region: "us-east-1",
  })
);

Telemetry Event Model

All telemetry events extend the base TelemetryEvent interface:

interface TelemetryEvent {
  id: string;                    // Unique event ID
  timestamp: string;             // ISO 8601 timestamp
  type: TelemetryEventType;      // Event type discriminator
  workflowId?: string;           // Optional workflow ID
  agentName?: string;            // Optional agent name
  stepId?: string;               // Optional step ID
  providerId?: string;           // Optional LLM provider ID
  modelId?: string;              // Optional model ID
  latencyMs?: number;            // Optional latency
  costUsd?: number;              // Optional cost
  inputTokens?: number;          // Optional input tokens
  outputTokens?: number;         // Optional output tokens
  metadata?: Record<string, unknown>;  // Additional metadata
}

Event Types

• llm.call.start - LLM call initiated
• llm.call.end - LLM call completed
• workflow.start - Workflow execution started
• workflow.end - Workflow execution completed
• step.start - Workflow step started
• step.end - Workflow step completed
• tool.call.start - Tool/MCP call initiated
• tool.call.end - Tool/MCP call completed
• safety.score.computed - Safety scorecard computed

Creating and Dispatching Events

Manual Event Creation

import {
  globalTelemetryCollector,
  createLlmCallStartEvent,
  createLlmCallEndEvent,
} from "@verydia/telemetry";

// LLM call start
const startEvent = createLlmCallStartEvent({
  providerId: "openai",
  modelId: "gpt-4o-mini",
  agentName: "my-agent",
  metadata: {
    prompt: "Hello, world!",
    temperature: 0.7,
  },
});

await globalTelemetryCollector.dispatch(startEvent);

// ... perform LLM call ...

// LLM call end
const endEvent = createLlmCallEndEvent({
  providerId: "openai",
  modelId: "gpt-4o-mini",
  latencyMs: 1250,
  inputTokens: 10,
  outputTokens: 50,
  costUsd: 0.0001,
  metadata: {
    response: "Hello! How can I help you?",
  },
});

await globalTelemetryCollector.dispatch(endEvent);

Workflow Events

import {
  createWorkflowStartEvent,
  createWorkflowEndEvent,
  createStepStartEvent,
  createStepEndEvent,
} from "@verydia/telemetry";

const workflowId = "wf_123";

// Workflow start
await globalTelemetryCollector.dispatch(
  createWorkflowStartEvent({
    workflowId,
    metadata: { workflowName: "customer-support", input: { query: "..." } },
  })
);

// Step start
await globalTelemetryCollector.dispatch(
  createStepStartEvent({
    workflowId,
    stepId: "step_1",
    metadata: { stepName: "classify-intent" },
  })
);

// Step end
await globalTelemetryCollector.dispatch(
  createStepEndEvent({
    workflowId,
    stepId: "step_1",
    latencyMs: 500,
    metadata: { output: { intent: "billing" } },
  })
);

// Workflow end
await globalTelemetryCollector.dispatch(
  createWorkflowEndEvent({
    workflowId,
    latencyMs: 2500,
    metadata: { stepCount: 3 },
  })
);

Safety Score Events

import { createSafetyScoreComputedEvent } from "@verydia/telemetry";

await globalTelemetryCollector.dispatch(
  createSafetyScoreComputedEvent({
    totalScore: 85.5,
    classification: "Safe",
    breakdown: [
      { categoryId: "toxicity", score: 95, weight: 0.2 },
      { categoryId: "bias", score: 80, weight: 0.15 },
    ],
    environment: "production",
    suiteName: "default",
  })
);

Exporters

Console Exporter

Pretty-prints events to the console with colorization.

import { ConsoleExporter } from "@verydia/telemetry";

const exporter = new ConsoleExporter({
  colorize: true,           // Enable ANSI colors (default: true)
  prettyPrint: true,        // Pretty-print format (default: true)
  includeMetadata: true,    // Include metadata (default: true)
});

globalTelemetryCollector.addExporter(exporter);

JSONL Exporter

Writes events to a file in JSONL format (one JSON object per line).

import { JsonlExporter } from "@verydia/telemetry";

const exporter = new JsonlExporter({
  filepath: "./telemetry.jsonl",
  append: true,  // Append to existing file (default: true)
});

globalTelemetryCollector.addExporter(exporter);

Note: Requires Node.js environment (uses fs/promises).

S3 Exporter

Batches and uploads events to S3 or S3-compatible storage.

import { S3Exporter } from "@verydia/telemetry";

const exporter = new S3Exporter({
  bucket: "my-telemetry-bucket",
  prefix: "verydia/prod/",
  region: "us-east-1",
  batchSize: 100,           // Auto-flush after N events (default: 100)
  maxBatchTimeMs: 60000,    // Auto-flush after N ms (default: 60000)
});

globalTelemetryCollector.addExporter(exporter);

// Flush remaining events before shutdown
await globalTelemetryCollector.flush();

S3-Compatible Services:

// MinIO
new S3Exporter({
  bucket: "telemetry",
  endpoint: "http://localhost:9000",
  region: "us-east-1",
  forcePathStyle: true,
  accessKeyId: "minioadmin",
  secretAccessKey: "minioadmin",
});

// NetApp StorageGrid
new S3Exporter({
  bucket: "verydia-telemetry",
  endpoint: "https://storagegrid.example.com",
  region: "us-east-1",
  forcePathStyle: true,
  accessKeyId: process.env.STORAGEGRID_ACCESS_KEY,
  secretAccessKey: process.env.STORAGEGRID_SECRET_KEY,
});

Instrumentation Examples

Instrumenting LLM Providers

import {
  globalTelemetryCollector,
  createLlmCallStartEvent,
  createLlmCallEndEvent,
} from "@verydia/telemetry";

async function callLlm(provider: string, model: string, prompt: string) {
  const startTime = Date.now();

  // Emit start event
  await globalTelemetryCollector.dispatch(
    createLlmCallStartEvent({
      providerId: provider,
      modelId: model,
      metadata: { prompt },
    })
  );

  try {
    // Perform LLM call
    const response = await actualLlmCall(provider, model, prompt);

    // Emit end event
    await globalTelemetryCollector.dispatch(
      createLlmCallEndEvent({
        providerId: provider,
        modelId: model,
        latencyMs: Date.now() - startTime,
        inputTokens: response.usage?.inputTokens,
        outputTokens: response.usage?.outputTokens,
        costUsd: response.usage?.costUsd,
        metadata: { response: response.text },
      })
    );

    return response;
  } catch (error) {
    // Emit error event
    await globalTelemetryCollector.dispatch(
      createLlmCallEndEvent({
        providerId: provider,
        modelId: model,
        latencyMs: Date.now() - startTime,
        metadata: { error: String(error) },
      })
    );

    throw error;
  }
}

Instrumenting Workflows

import {
  createWorkflowStartEvent,
  createWorkflowEndEvent,
  createStepStartEvent,
  createStepEndEvent,
} from "@verydia/telemetry";

async function runWorkflow(workflowId: string, steps: Step[]) {
  const startTime = Date.now();

  await globalTelemetryCollector.dispatch(
    createWorkflowStartEvent({ workflowId })
  );

  try {
    for (const step of steps) {
      const stepStartTime = Date.now();

      await globalTelemetryCollector.dispatch(
        createStepStartEvent({
          workflowId,
          stepId: step.id,
          metadata: { stepName: step.name },
        })
      );

      const result = await step.execute();

      await globalTelemetryCollector.dispatch(
        createStepEndEvent({
          workflowId,
          stepId: step.id,
          latencyMs: Date.now() - stepStartTime,
          metadata: { output: result },
        })
      );
    }

    await globalTelemetryCollector.dispatch(
      createWorkflowEndEvent({
        workflowId,
        latencyMs: Date.now() - startTime,
        metadata: { stepCount: steps.length },
      })
    );
  } catch (error) {
    await globalTelemetryCollector.dispatch(
      createWorkflowEndEvent({
        workflowId,
        latencyMs: Date.now() - startTime,
        metadata: { error: String(error) },
      })
    );

    throw error;
  }
}

Custom Exporters

Create custom exporters by implementing the TelemetryExporter interface:

import type { TelemetryExporter, TelemetryEvent } from "@verydia/telemetry";

class CustomExporter implements TelemetryExporter {
  readonly id = "custom";

  async handle(event: TelemetryEvent): Promise<void> {
    // Send to external service, database, etc.
    await sendToExternalService(event);
  }

  async flush(): Promise<void> {
    // Optional: flush any buffered events
  }
}

globalTelemetryCollector.addExporter(new CustomExporter());

CI/CD Integration

Use JSONL exporter to capture telemetry in CI pipelines:

// ci-telemetry.ts
import { globalTelemetryCollector, JsonlExporter } from "@verydia/telemetry";

globalTelemetryCollector.addExporter(
  new JsonlExporter({ filepath: "./ci-telemetry.jsonl" })
);

// Run your tests/workflows
await runTests();

// Flush before exit
await globalTelemetryCollector.flush();

Then analyze the JSONL file:

# Count events by type
cat ci-telemetry.jsonl | jq -r '.type' | sort | uniq -c

# Calculate average LLM latency
cat ci-telemetry.jsonl | jq -r 'select(.type == "llm.call.end") | .latencyMs' | awk '{sum+=$1; count++} END {print sum/count}'

# Total LLM cost
cat ci-telemetry.jsonl | jq -r 'select(.type == "llm.call.end") | .costUsd // 0' | awk '{sum+=$1} END {print sum}'

API Reference

TelemetryCollector

class TelemetryCollector {
  addExporter(exporter: TelemetryExporter): void;
  removeExporter(id: string): boolean;
  getExporters(): TelemetryExporter[];
  dispatch(event: TelemetryEvent): Promise<void>;
  flush(): Promise<void>;
  setEnabled(enabled: boolean): void;
  isEnabled(): boolean;
}

Global Collector

import { globalTelemetryCollector } from "@verydia/telemetry";

// The global singleton used throughout Verydia

Best Practices

1. Configure at Startup - Add exporters during application initialization
2. Use Metadata Wisely - Don't include sensitive data or large payloads
3. Flush on Shutdown - Call flush() before process exit
4. Monitor Exporter Errors - Enable logErrors option in production
5. Batch for Performance - Use S3Exporter for high-volume production workloads
6. Filter Events - Create custom exporters that filter by event type or metadata

TypeScript Support

Full TypeScript support with exported types:

import type {
  TelemetryEvent,
  TelemetryEventType,
  TelemetryExporter,
  LlmCallStartEvent,
  LlmCallEndEvent,
  WorkflowStartEvent,
  WorkflowEndEvent,
  StepStartEvent,
  StepEndEvent,
  ToolCallStartEvent,
  ToolCallEndEvent,
  SafetyScoreComputedEvent,
} from "@verydia/telemetry";

License

MIT