📊 Logs Interceptor

Enterprise-grade, production-ready log interceptor for Node.js applications with Grafana Loki integration. Zero-impact performance, automatic log collection from all sources, and built-in security features.

✨ Key Features

• 🚀 Zero Performance Impact - Circuit breaker, memory management, and adaptive sampling
• 🔒 Security First - Automatic sanitization of sensitive data (passwords, tokens, credit cards)
• 📝 Universal Logging - Intercepts console, Winston, Pino, Morgan, and Bunyan
• 🌐 Distributed Tracing - OpenTelemetry integration with trace/span correlation
• 💾 Smart Buffering - Automatic batching with memory-aware flushing
• 🔄 Resilient - Retry logic, circuit breaker, and emergency file fallback
• 📊 Observability - Built-in metrics, health checks, and performance monitoring
• 🎯 Context Propagation - AsyncLocalStorage for request-scoped logging
• 🗜️ Compression - Automatic gzip compression for network efficiency

📦 Installation

npm install logs-interceptor
# or
yarn add logs-interceptor
# or
pnpm add logs-interceptor

🚀 Quick Start

Basic Usage

import { init } from 'logs-interceptor';

const logger = init({
  transport: {
    url: 'https://loki.example.com/loki/api/v1/push',
    tenantId: 'my-tenant',
    authToken: process.env.LOKI_AUTH_TOKEN,
  },
  appName: 'my-app',
  environment: 'production',
  interceptConsole: true, // Automatically capture console.log
});

// Now all console.log, console.error, etc. are captured
console.log('This goes to Loki!');
logger.info('So does this!');

Environment Variables Setup

Create a .env file:

LOGS_INTERCEPTOR_URL=https://loki.example.com/loki/api/v1/push
LOGS_INTERCEPTOR_TENANT_ID=my-tenant
LOGS_INTERCEPTOR_AUTH_TOKEN=your-auth-token
LOGS_INTERCEPTOR_APP_NAME=my-app
LOGS_INTERCEPTOR_ENVIRONMENT=production
LOGS_INTERCEPTOR_ENABLED=true

Then simply:

import { init } from 'logs-interceptor';
const logger = init(); // Auto-configures from environment

🔧 Advanced Configuration

const logger = init({
  transport: {
    url: 'https://loki.example.com/loki/api/v1/push',
    tenantId: 'my-tenant',
    authToken: process.env.LOKI_AUTH_TOKEN,
    timeout: 10000,
    maxRetries: 5,
    compression: true,
  },
  
  appName: 'production-api',
  version: '2.0.0',
  environment: 'production',
  
  labels: {
    region: 'us-east-1',
    cluster: 'prod-cluster',
    service: 'api-gateway',
  },
  
  dynamicLabels: {
    hostname: () => require('os').hostname(),
    pid: () => process.pid,
  },
  
  buffer: {
    maxSize: 500,           // Max logs before auto-flush
    flushInterval: 10000,   // Auto-flush every 10s
    maxMemoryMB: 100,      // Max memory usage
  },
  
  filter: {
    levels: ['info', 'warn', 'error', 'fatal'],
    samplingRate: 0.8,     // Sample 80% of logs
    sanitize: true,        // Auto-sanitize sensitive data
    maxMessageLength: 8192,
    sensitivePatterns: [
      /password/i,
      /token/i,
      /api[_-]?key/i,
      /credit[_-]?card/i,
      /ssn/i,
      /cpf/i,
    ],
  },
  
  circuitBreaker: {
    enabled: true,
    failureThreshold: 10,
    resetTimeout: 120000,
  },
  
  integrations: {
    winston: true,
    pino: true,
    morgan: true,
  },
  
  performance: {
    useWorkers: true,
    compressionLevel: 6,
  },
});

🔌 Framework Integrations

Express.js

import express from 'express';
import { init } from 'logs-interceptor';
import { expressMiddleware } from 'logs-interceptor/middleware';

const app = express();
const logger = init({ /* config */ });

// Add request/response logging
app.use(expressMiddleware(logger));

// Morgan integration
import morgan from 'morgan';
app.use(morgan('combined', {
  stream: logger.getMorganStream(),
}));

Koa.js

import Koa from 'koa';
import { init } from 'logs-interceptor';
import { koaMiddleware } from 'logs-interceptor/middleware';

const app = new Koa();
const logger = init({ /* config */ });

app.use(koaMiddleware(logger));

Fastify

import fastify from 'fastify';
import { init } from 'logs-interceptor';
import { fastifyPlugin } from 'logs-interceptor/middleware';

const app = fastify();
const logger = init({ /* config */ });

app.register(fastifyPlugin(logger));

🔗 Logger Integrations

Winston

import winston from 'winston';

const logger = init({
  integrations: { winston: true },
});

const winstonLogger = winston.createLogger({
  transports: [
    logger.getWinstonTransport(),
    new winston.transports.Console(),
  ],
});

winstonLogger.info('This goes to both console and Loki!');

Pino

import pino from 'pino';

const logger = init({
  integrations: { pino: true },
});

const pinoLogger = pino({
  destination: logger.getPinoStream(),
});

pinoLogger.info('High-performance logging to Loki!');

🎯 Context Propagation

// Add context to all logs within a request
app.use((req, res, next) => {
  logger.runWithContext({
    userId: req.user?.id,
    sessionId: req.session?.id,
    requestId: req.headers['x-request-id'],
  }, next);
});

// All logs within this context will include the metadata
app.get('/api/users/:id', async (req, res) => {
  logger.info('Fetching user'); // Includes userId, sessionId, requestId
  
  try {
    const user = await getUserById(req.params.id);
    logger.info('User fetched successfully');
    res.json(user);
  } catch (error) {
    logger.error('Failed to fetch user', { error: error.message });
    res.status(500).json({ error: 'Internal server error' });
  }
});

🔒 Security Features

Automatic Sensitive Data Sanitization

// These will be automatically sanitized
logger.info('User login', {
  username: '[email protected]',
  password: 'secret123',        // → [REDACTED]
  creditCard: '4111111111111111', // → [REDACTED]
  apiKey: 'sk_live_abc123',     // → [REDACTED]
});

Custom Sanitization Rules

const logger = init({
  filter: {
    sanitize: true,
    sensitivePatterns: [
      /password/i,
      /secret/i,
      /\b\d{3}-\d{2}-\d{4}\b/, // SSN
      /custom-pattern/,
    ],
  },
});

📊 Monitoring & Health Checks

// Health endpoint
app.get('/health', (req, res) => {
  const health = logger.getHealth();
  const metrics = logger.getMetrics();
  
  res.json({
    status: health.healthy ? 'healthy' : 'unhealthy',
    health: {
      ...health,
      circuitBreaker: health.circuitBreakerState,
      memoryUsageMB: health.memoryUsageMB,
      bufferUtilization: health.bufferUtilization,
    },
    metrics: {
      logsProcessed: metrics.logsProcessed,
      logsDropped: metrics.logsDropped,
      logsSanitized: metrics.logsSanitized,
      avgFlushTime: metrics.avgFlushTime,
      errorCount: metrics.errorCount,
    },
  });
});

🛡️ Error Handling & Graceful Shutdown

// Capture unhandled errors
process.on('unhandledRejection', (reason, promise) => {
  logger.fatal('Unhandled Promise Rejection', { reason });
});

process.on('uncaughtException', (error) => {
  logger.fatal('Uncaught Exception', {
    error: error.message,
    stack: error.stack,
  });
  
  logger.flush().then(() => process.exit(1));
});

// Graceful shutdown
async function gracefulShutdown(signal) {
  console.log(`Received ${signal}, shutting down gracefully...`);
  
  server.close();           // Stop accepting new connections
  await logger.flush();      // Flush remaining logs
  await logger.destroy();    // Cleanup resources
  
  process.exit(0);
}

process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));

🚀 Performance Optimization

Adaptive Sampling

const logger = init({
  filter: {
    samplingRate: 0.8, // Sample 80% of logs in production
    levels: ['info', 'warn', 'error', 'fatal'], // Skip debug in production
  },
});

Circuit Breaker

Automatically stops sending logs when Loki is down to prevent application impact:

const logger = init({
  circuitBreaker: {
    enabled: true,
    failureThreshold: 5,    // Open after 5 failures
    resetTimeout: 60000,     // Try again after 1 minute
    halfOpenRequests: 3,     // Test with 3 requests
  },
});

Memory Management

Automatic memory pressure handling:

const logger = init({
  buffer: {
    maxMemoryMB: 50, // Auto-flush when memory usage exceeds 50MB
    maxSize: 1000,   // Max 1000 logs in buffer
  },
});

📝 API Reference

Logger Methods

logger.debug(message: string, context?: object): void
logger.info(message: string, context?: object): void
logger.warn(message: string, context?: object): void
logger.error(message: string, context?: object): void
logger.fatal(message: string, context?: object): void
logger.trackEvent(eventName: string, properties?: object): void
logger.flush(): Promise<void>
logger.getMetrics(): LoggerMetrics
logger.getHealth(): HealthStatus
logger.runWithContext<T>(context: object, fn: () => T): T
logger.destroy(): Promise<void>

Environment Variables

| Variable | Description | Default | |----------|-------------|---------| | LOGS_INTERCEPTOR_URL | Loki push endpoint | Required | | LOGS_INTERCEPTOR_TENANT_ID | Loki tenant ID | Required | | LOGS_INTERCEPTOR_AUTH_TOKEN | Authentication token | Optional | | LOGS_INTERCEPTOR_APP_NAME | Application name | Required | | LOGS_INTERCEPTOR_ENVIRONMENT | Environment name | production | | LOGS_INTERCEPTOR_VERSION | App version | 1.0.0 | | LOGS_INTERCEPTOR_BUFFER_SIZE | Buffer size | 100 | | LOGS_INTERCEPTOR_FLUSH_INTERVAL | Flush interval (ms) | 5000 | | LOGS_INTERCEPTOR_LOG_LEVEL | Log levels (comma-separated) | debug,info,warn,error,fatal | | LOGS_INTERCEPTOR_SAMPLING_RATE | Sampling rate (0-1) | 1.0 | | LOGS_INTERCEPTOR_CIRCUIT_BREAKER | Enable circuit breaker | true | | LOGS_INTERCEPTOR_SANITIZE | Sanitize sensitive data | true | | LOGS_INTERCEPTOR_MAX_MEMORY_MB | Max memory usage | 50 | | LOGS_INTERCEPTOR_DEBUG | Debug mode | false | | LOGS_INTERCEPTOR_ENABLED | Enable/disable logging | true |

🧪 Testing

// Test mode - logs to memory instead of Loki
const logger = init({
  transport: {
    url: 'memory://test',
    tenantId: 'test',
  },
  debug: true,
});

// Access logs in tests
logger.on('log', (entry) => {
  console.log('Log captured:', entry);
});

📊 Grafana Dashboard

Import our pre-configured Grafana dashboard for monitoring:

1. Import dashboard JSON from dashboards/logs-interceptor.json
2. Configure Loki data source
3. Set variables for app_name and environment

🐛 Troubleshooting

Logs not appearing in Loki

1. Check circuit breaker status: logger.getHealth()
2. Verify network connectivity to Loki
3. Check authentication token
4. Enable debug mode: LOGS_INTERCEPTOR_DEBUG=true

High memory usage

1. Reduce buffer size
2. Increase flush frequency
3. Enable sampling
4. Check for memory leaks in context data

Performance impact

1. Enable circuit breaker
2. Reduce sampling rate
3. Use worker threads
4. Increase compression level

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide for details.

📄 License

MIT © Leonardo Zwirtes

🔗 Links

• GitHub Repository
• NPM Package
• Documentation
• Examples

💬 Support

• GitHub Issues
• Discussions