@ypalav/log-pipeline v2.0.0

CLI tool to pipe logs to Log Pipeline server.

⚠️ Breaking Changes in v2.0.0

v2.0.0 is a complete rewrite. This version is CLI-only and no longer includes the SDK functionality from v1.x.

| v1.x (Old) | v2.0.0 (New) | |------------|--------------| | SDK for programmatic use | CLI tool only | | Optional configuration | Required environment variables | | Import as module | Global binary installation | | Flexible initialization | Stdin piping only |

👉 See Migration Guide below

Features

• ✅ CLI-only - Pure command-line tool
• ✅ Required environment variables - Fails fast if not configured
• ✅ Stdin piping - Works with any process output
• ✅ Batch sending - Efficient log transmission
• ✅ Auto log level detection - ERROR, WARN, INFO
• ✅ Graceful shutdown - Flushes remaining logs on exit
• ✅ Global install - Use anywhere in your system

Installation

npm install -g @ypalav/log-pipeline

Or install locally:

npm install @ypalav/log-pipeline
# Then use: npx log-pipeline

Environment Variables

Required

| Variable | Description | Example | |----------|-------------|---------| | LOG_PIPELINE_URL | Server API endpoint | https://your-server.com/api/logs | | LOG_PIPELINE_APP | Application name | my-app |

Optional

| Variable | Description | Default | |----------|-------------|---------| | LOG_PIPELINE_ENV | Environment name | docker | | LOG_PIPELINE_FLUSH_MS | Flush interval in milliseconds | 5000 |

Usage Examples

Pipe Docker Container Logs

docker run myapp | LOG_PIPELINE_APP=myapp LOG_PIPELINE_URL=https://your-server.com/api/logs log-pipeline

Pipe Any Process Output

node app.js 2>&1 | LOG_PIPELINE_APP=myapp LOG_PIPELINE_URL=https://your-server.com/api/logs log-pipeline

Pipe from File

cat app.log | LOG_PIPELINE_APP=myapp LOG_PIPELINE_URL=https://your-server.com/api/logs log-pipeline

Set Environment Variables Globally

export LOG_PIPELINE_URL=https://your-server.com/api/logs
export LOG_PIPELINE_APP=myapp

# Now just run:
node app.js 2>&1 | log-pipeline

Docker Compose Integration

version: '3.8'

services:
  app:
    image: myapp:latest
    command: sh -c "node app.js 2>&1 | log-pipeline"
    environment:
      - LOG_PIPELINE_APP=myapp
      - LOG_PIPELINE_URL=https://your-server.com/api/logs
      - LOG_PIPELINE_ENV=production
      - LOG_PIPELINE_FLUSH_MS=3000

Kubernetes Sidecar Pattern

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:latest
  - name: log-pipeline
    image: node:18
    command: ["/bin/sh", "-c"]
    args:
      - "tail -f /var/log/app.log | log-pipeline"
    env:
    - name: LOG_PIPELINE_APP
      value: "myapp"
    - name: LOG_PIPELINE_URL
      value: "https://your-server.com/api/logs"
    volumeMounts:
    - name: logs
      mountPath: /var/log
  volumes:
  - name: logs
    emptyDir: {}

Local Development

# Start your app and pipe logs
npm run dev 2>&1 | LOG_PIPELINE_APP=myapp-dev LOG_PIPELINE_ENV=development log-pipeline

Pipe Multiple Sources

# Combine stdout and stderr
(node app.js; exit 0) 2>&1 | LOG_PIPELINE_APP=myapp log-pipeline

Error Handling

Missing Environment Variables

If required environment variables are not set, the tool exits with a clear error:

ERROR: LOG_PIPELINE_URL environment variable is required
Example: export LOG_PIPELINE_URL=https://your-server.com/api/logs

Network Errors

If the server is unreachable, errors are logged to stderr but the tool continues:

[log-pipeline] ✗ Network error: fetch failed

Server Errors

If the server returns an error, it's logged but processing continues:

[log-pipeline] ✗ Server error 400: Invalid payload

Log Level Detection

The tool automatically detects log levels:

| Pattern | Detected Level | |---------|----------------| | [error], error:, error, err , exception, traceback | ERROR | | [warn], warn:, warn, warning | WARN | | Everything else | INFO |

Troubleshooting

"command not found: log-pipeline"

Make sure you installed globally:

npm install -g @ypalav/log-pipeline

Or use npx:

npx log-pipeline

Logs not appearing in dashboard

1. Check environment variables are set correctly
2. Verify server URL is accessible
3. Check stderr for error messages
4. Test with a simple echo:

   echo "test log" | LOG_PIPELINE_APP=test LOG_PIPELINE_URL=https://your-server.com/api/logs log-pipeline

Flush interval too slow/fast

Adjust with LOG_PIPELINE_FLUSH_MS:

LOG_PIPELINE_FLUSH_MS=1000 log-pipeline  # Flush every 1 second

Migration Guide from v1.x

What Changed

v2.0.0 is a complete rewrite focused on simplicity and Docker/CLI workflows. The SDK functionality from v1.x has been removed.

Migration Steps

1. Uninstall Old Version

npm uninstall -g @ypalav/log-pipeline

2. Install New Version

npm install -g @ypalav/log-pipeline

3. Update Your Code

v1.x (Old - SDK):

const LogPipeline = require('@ypalav/log-pipeline');

const logger = new LogPipeline({
  serverUrl: 'https://your-server.com/api/logs',
  app: 'my-app',
  env: 'production'
});

logger.info('Application started');
logger.error('Something failed');

v2.0.0 (New - CLI):

# Set environment variables
export LOG_PIPELINE_URL=https://your-server.com/api/logs
export LOG_PIPELINE_APP=my-app
export LOG_PIPELINE_ENV=production

# Pipe your app output to log-pipeline
node app.js 2>&1 | log-pipeline

4. Update Docker Compose

v1.x (Old):

services:
  app:
    image: myapp
    environment:
      - LOG_PIPELINE_SERVER=https://your-server.com/api/logs
      - LOG_PIPELINE_APP=myapp

v2.0.0 (New):

services:
  app:
    image: myapp
    command: sh -c "node app.js 2>&1 | log-pipeline"
    environment:
      - LOG_PIPELINE_URL=https://your-server.com/api/logs
      - LOG_PIPELINE_APP=myapp
      - LOG_PIPELINE_ENV=production

5. Update Logging Code

Since v2.0.0 captures all stdout/stderr, you no longer need to call logger methods. Just use console.log() as usual:

v1.x (Old):

logger.info('User logged in');
logger.error('Database error');

v2.0.0 (New):

// Just use console.log - log-pipeline will capture it
console.log('[INFO] User logged in');
console.error('[ERROR] Database error');

The CLI automatically detects log levels based on patterns like [INFO], [ERROR], [WARN].

What's Removed

• ❌ Programmatic SDK API
• ❌ new LogPipeline() constructor
• ❌ .info(), .error(), .warn() methods
• ❌ Callback-based initialization

What's New

• ✅ Global CLI binary (log-pipeline)
• ✅ Stdin piping for any process
• ✅ Automatic log level detection
• ✅ Batch sending for efficiency
• ✅ Graceful shutdown on SIGINT/SIGTERM

Need the SDK?

If you need programmatic logging in your code, consider:

1. Using standard console.log() and piping to log-pipeline
2. Using a different logging library (winston, pino, etc.) and piping output
3. Running log-pipeline as a sidecar container in Kubernetes

Version History

v2.0.0 (Current)

• ✅ CLI-only implementation
• ✅ Required environment variables (no defaults)
• ✅ Global install support
• ✅ Stdin piping
• ❌ SDK functionality removed

v1.0.0

• Initial SDK release
• Programmatic API

License

MIT