@principal-ai/otel-collector-server

Node.js wrapper around the official OpenTelemetry Collector with dual-mode support for development and Electron integration.

🎯 Contract-Based Observability (Coming Soon)

This package is being enhanced to support contract-based observability - validate traces against expected workflows, reduce storage costs by 90%+, and surface unknown-unknowns automatically.

Key Features:

• ✅ Validate traces against Principal-View canvas contracts
• ✅ Emit metrics-only for expected workflows (95% of traces)
• ✅ Store full traces only for anomalies (5% of traces)
• ✅ Real-time contract violations in Electron UI
• ✅ Production mode for EC2 deployment

See CONTRACT-BASED-PLATFORM.md for full design.

Features

• 🚀 Standalone Mode - Console or file output for development and testing
• ⚡ Electron Mode - MessagePort routing to renderer windows
• 🔧 Auto-managed Binaries - Handles OpenTelemetry Collector binaries per platform
• 📊 Pretty Printing - Human-readable trace output
• 🎯 Source Routing - Routes traces based on origin URL
• 🔄 Auto-restart - Monitors and restarts collector on crashes

Quick Start

Installation

npm install @principal-ai/otel-collector-server

Standalone Mode (CLI)

# Install globally
npm install -g @principal-ai/otel-collector-server

# Start with console output
otel-collector-server

# Start with file output
otel-collector-server --output file --file ./traces.jsonl

# Custom configuration
otel-collector-server --port 4318 --log-level debug --pretty

# Using environment variables
OTLP_PORT=4318 OUTPUT=file OUTPUT_FILE=./traces.jsonl otel-collector-server

Production Deployment

AWS App Runner (Recommended) - Serverless, managed deployment:

• ✅ Currently deployed: https://5hmsn3wzue.us-east-1.awsapprunner.com
• 📋 Architecture & Decisions: production-deployment.md
• 🔧 Operational Guide: PRODUCTION.md
• 🚀 Setup Guide: APPRUNNER-SETUP.md
• Automatic deployments via GitHub Actions (push to main branch)
• Auto-scaling and zero-downtime updates
• ~$51-60/month for 24/7 operation

AWS EC2 - Self-hosted for more control:

• See DEPLOYMENT.md for complete guide
• Docker or PM2 deployment options
• Manual deployment and management
• ~$15/month for t3.small instance

Programmatic Usage

import { OTELCollectorServer } from '@principal-ai/otel-collector-server';

const server = new OTELCollectorServer({
  mode: 'standalone',
  output: 'console',
  prettyPrint: true,
  otlpPort: 4318,
  logLevel: 'info',
});

await server.start();

// Graceful shutdown
process.on('SIGINT', async () => {
  await server.stop();
  process.exit(0);
});

Configuration

Standalone Mode Options

The server can be configured via:

1. Environment variables (recommended for production)
2. Command-line arguments (overrides environment variables)
3. Programmatic config (for library usage)

Environment Variables:

OTLP_PORT=4318              # OTLP endpoint port
WRAPPER_PORT=4319           # Wrapper HTTP port
OUTPUT=console              # Output type: console | file
OUTPUT_FILE=./traces.jsonl  # File path (when OUTPUT=file)
PRETTY_PRINT=true           # Pretty print console output
LOG_LEVEL=info              # Log level: debug | info | warn | error
COLLECTOR_VERSION=v0.144.0  # OTEL collector version
BINARY_PATH=/path/to/binary # Custom binary path

TypeScript Interface:

interface OTELCollectorServerConfig {
  mode: 'standalone' | 'electron';
  otlpPort?: number;           // Default: 4318
  wrapperPort?: number;        // Default: 4319
  collectorVersion?: string;   // Default: latest
  logLevel?: 'debug' | 'info' | 'warn' | 'error';
  autoStart?: boolean;
  restartOnCrash?: boolean;

  // Standalone mode
  output?: 'console' | 'file';
  outputFile?: string;
  prettyPrint?: boolean;

  // Optional custom binary path
  binaryPath?: string;
}

Architecture

External App (instrumented with OTEL SDK)
         ↓ HTTP POST /v1/traces (OTLP)
OpenTelemetry Collector (Go binary)
         ↓ forwards to
Node.js Wrapper (this package)
         ↓
Console Output | File Output | MessagePort (Electron)

CLI Options

Usage: otel-collector-server [options]

Options:
  -p, --port <port>            OTLP endpoint port (default: 4318)
  --wrapper-port <port>        Wrapper HTTP port (default: 4319)
  -o, --output <type>          Output type: console | file (default: console)
  -f, --file <path>            Output file path (sets output to file)
  --pretty                     Pretty print console output (default: true)
  --no-pretty                  Disable pretty printing
  -l, --log-level <level>      Log level: debug | info | warn | error
  --version <version>          Collector version (default: latest)
  -h, --help                   Show help message

Sending Test Traces

Production Endpoint (App Runner)

# Send a test trace to production
curl -X POST https://5hmsn3wzue.us-east-1.awsapprunner.com/v1/traces \
  -H "Content-Type: application/json" \
  -d '{
    "resourceSpans": [{
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "test-service" } },
          { "key": "dev.server.url", "value": { "stringValue": "http://localhost:3000" } }
        ]
      },
      "scopeSpans": [{
        "spans": [{
          "traceId": "abc123",
          "spanId": "def456",
          "name": "test-span",
          "kind": 2,
          "startTimeUnixNano": "1737835200000000000",
          "endTimeUnixNano": "1737835200150000000"
        }]
      }]
    }]
  }'

Local Development

# Send a test trace to local instance
curl -X POST http://localhost:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{
    "resourceSpans": [{
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "test-service" } },
          { "key": "dev.server.url", "value": { "stringValue": "http://localhost:3000" } }
        ]
      },
      "scopeSpans": [{
        "spans": [{
          "traceId": "abc123",
          "spanId": "def456",
          "name": "test-span",
          "kind": 2,
          "startTimeUnixNano": "1737835200000000000",
          "endTimeUnixNano": "1737835200150000000"
        }]
      }]
    }]
  }'

Example Output

Pretty Print (Default)

======================================================================
TRACE FROM http://localhost:3000
Received at: 2025-01-25T12:00:00.000Z
======================================================================

Service: web-ade

Scope: my-tracer
  - HTTP GET /api/users (SERVER) 45.23ms [OK]
     http.method=GET, http.status_code=200
    └─ DB Query: SELECT * FROM users (CLIENT) 30.15ms [OK]
       db.system=postgresql, db.statement=SELECT * FROM users
    └─ Cache lookup (INTERNAL) 2.08ms [OK]

======================================================================

File Output (JSONL)

{"timestamp":"2025-01-25T12:00:00.000Z","source":"http://localhost:3000","payload":{...}}

Requirements

• Node.js >= 18.19.0
• OpenTelemetry Collector binary (auto-downloaded or bundled)

Supported Platforms

• macOS (Intel & Apple Silicon)
• Linux (x64 & ARM64)
• Windows (x64)

Development

Setup

# Install dependencies
npm install

# Download OpenTelemetry Collector binaries (required for first run)
npm run download-binaries

# Build
npm run build

# Run in dev mode
npm run dev:standalone

# Run tests
npm test

Download Binaries

The package bundles OpenTelemetry Collector binaries for all platforms. To download them:

npm run download-binaries

This downloads ~250-350MB of binaries for:

• macOS (Intel & Apple Silicon)
• Linux (x64 & ARM64)
• Windows (x64)

Binaries are automatically downloaded when publishing via prepublishOnly hook.

Project Status

✅ Completed (Phase 1 & 2)

• [x] Standalone mode with console output
• [x] Standalone mode with file output
• [x] Auto-managed collector binaries (platform detection)
• [x] Binary bundling for all platforms (v0.144.0)
• [x] Pretty-print trace output
• [x] Source URL routing
• [x] CLI interface
• [x] Comprehensive test coverage (57+ test cases)
• [x] TypeScript types and API exports
• [x] Full ESLint configuration and compliance
• [x] Tested and verified with live OTEL Collector

✅ Completed (Phase 3)

• [x] Integration tests with actual Go collector (14 comprehensive tests)
• [x] FileOutput unit tests (16 comprehensive tests)

📋 Roadmap (Future Phases)

• [ ] Electron mode with MessagePort routing (Phase 4)
• [ ] Desktop app integration (Phase 5)
• [ ] Metrics support (/v1/metrics)
• [ ] Logs support (/v1/logs)
• [ ] Custom processors configuration
• [ ] Multiple exporters support

License

MIT

Contributing

Issues and PRs welcome!

See Also

Documentation

• docs/architecture/DESIGN.md - Detailed system architecture
• .principal-views/production-deployment.md - Production architecture decisions
• docs/deployment/PRODUCTION.md - Production operational guide

External Resources

• OpenTelemetry Collector
• OTLP Specification