Server Log Monitor

AI-powered server log monitoring with BlackBox AI analysis and Twilio alerts.

Features

• 🔍 Polling-Based Log Monitoring: Monitors multiple log files and directories with configurable polling intervals
• 🤖 AI-Powered Analysis: Uses BlackBox AI to analyze log content intelligently
• 📱 Smart Alerts: Twilio SMS and phone call notifications for critical issues
• 🗣️ Conversational Alerts: ElevenLabs AI voice agents for interactive critical incident support
• ⚙️ Configurable: Flexible configuration via JSON files or environment variables
• 🚀 Easy Deployment: NPM package with CLI interface
• 📊 Controlled Analysis: Analyzes last N lines from logs on each poll

Quick Start

1. Installation

Global installation (recommended for CLI usage):

npm install -g server-log-monitor

On Linux/Ubuntu, you may need sudo:

sudo npm install -g server-log-monitor

Alternative - Use without installation:

npx server-log-monitor config create
npx server-log-monitor start

Or for local project:

npm install server-log-monitor

2. Create Configuration

server-log-monitor config create

This creates a config.sample.json file with example values.

⚠️ Important: You must copy the sample to config.json and add your real API keys:

# Copy the sample configuration
cp config.sample.json config.json

# Edit config.json with your actual API keys
nano config.json  # or use your preferred editor

Your config.json should look like this with real values:

{
  "blackboxAI": {
    "apiKey": "your_blackbox_api_key_here"
  },
  "twilio": {
    "accountSid": "your_twilio_account_sid",
    "authToken": "your_twilio_auth_token",
    "phoneNumber": "+12315999850",
    "toPhoneNumbers": ["+0987654321"]
  },
  "elevenlabs": {
    "apiKey": "your_elevenlabs_api_key_here",
    "agentId": "agent_id_will_be_set_after_setup",
    "phoneNumberId": "phone_number_id_will_be_set_after_setup",
    "voiceId": "your_preferred_voice_id_optional"
  },
  "logMonitor": {
    "paths": ["/var/log", "./logs"],
    "maxLines": 100,
    "pollInterval": 5000
  }
}

3. Start Monitoring

server-log-monitor start

Usage

CLI Commands

# Start monitoring (default command)
server-log-monitor start

# Monitor a specific file only
server-log-monitor start --file /var/log/nginx/access.log

# Monitor with verbose output (show log content before AI analysis)
server-log-monitor start --verbose

# Monitor specific file with verbose output
server-log-monitor start --file ../log-simulator/logs/app.log --verbose

# Analyze a specific log file
server-log-monitor analyze /var/log/nginx/error.log

# Analyze with specific number of lines
server-log-monitor analyze /var/log/nginx/error.log --lines 50

# Aggregate all logs into one file and analyze (one-time)
server-log-monitor aggregate

# Aggregate with custom settings (one-time)
server-log-monitor aggregate --lines 50 --output my-logs.txt --verbose

# Continuously aggregate and analyze logs (uses config default interval)
server-log-monitor aggregate-watch

# Continuous aggregation with 30-second interval
server-log-monitor aggregate-watch --interval 30

# Continuous aggregation with custom output and verbose mode
server-log-monitor aggregate-watch --output my-logs.txt --verbose

# Check status
server-log-monitor status

# Validate configuration
server-log-monitor config validate

# Health check
server-log-monitor health

# Test SMS alerts
server-log-monitor test-sms

# Test voice calls
server-log-monitor test-call

# ElevenLabs Conversational Agent Setup
server-log-monitor elevenlabs:setup

# Test ElevenLabs integration
server-log-monitor elevenlabs-test

# Update ElevenLabs agent configuration
server-log-monitor elevenlabs-update

# ElevenLabs Tools Integration
server-log-monitor tool-server

# Start tool server with custom settings
server-log-monitor tool-server --port 3002 --codebase /path/to/your/project

# Update to latest version
server-log-monitor update

# Show version
server-log-monitor --version

# Help
server-log-monitor help

Command Options

• --config <path> - Path to configuration file
• --file <path> - Monitor only a specific file (for start command)
• --verbose, --show-logs - Show detailed output (for start/aggregate commands)
• --lines <number> - Number of lines to collect/analyze (for analyze/aggregate commands)
• --output <path> - Output file for aggregated logs (for aggregate command, default: aggregated-logs.txt)
• --interval <number> - Analysis interval in seconds (for aggregate-watch command, uses config default if not specified)
• --version, -v - Show version number

Environment Variables

You can also configure using environment variables:

export BLACKBOX_API_KEY="your_api_key"
export TWILIO_ACCOUNT_SID="your_sid"
export TWILIO_AUTH_TOKEN="your_token"
export ELEVENLABS_API_KEY="your_elevenlabs_api_key"
export ELEVENLABS_VOICE_ID="your_preferred_voice_id"
export LOG_PATHS="/var/log,./logs"

Configuration

BlackBox AI Settings

• apiKey: Your BlackBox AI API key (required)
• model: AI model to use (default: "blackboxai/anthropic/claude-sonnet-4")
• timeout: Request timeout in milliseconds (default: 30000)

ElevenLabs Settings

• apiKey: Your ElevenLabs API key (required for conversational alerts)
• agentId: Agent ID (auto-set after running setup command)
• phoneNumberId: Phone number ID (auto-set after setup)
• voiceId: Voice ID from ElevenLabs voice library (optional, improves voice quality)

Log Monitoring Settings (Polling Mode)

• paths: Array of log file paths or directories to monitor
• patterns: File patterns to match (default: [".log", ".txt"])
• maxLines: Maximum lines to read from end of each log file per poll (default: 100)
• pollInterval: Polling frequency in milliseconds - how often to check log files (default: 200000)
• encoding: File encoding (default: 'utf8')
• recursive: Recursively scan subdirectories (default: false)

How Polling Works:

• Every pollInterval ms, reads the last maxLines from each monitored log file
• Only analyzes files that have been modified since last poll
• Sends the entire tail content (up to maxLines) to AI for analysis

Alert Settings

• cooldownPeriod: Minimum time between alerts (default: 5 minutes)
• enableSMS: Enable SMS notifications
• enableCalls: Enable phone call notifications
• onlyCallForCritical: Only make calls for critical issues

Log Aggregation

The aggregate command collects logs from all monitored files into a single file and analyzes them together. This is useful for getting a comprehensive view across all your services and detecting patterns that span multiple log files.

The aggregate-watch command provides continuous monitoring with periodic aggregation and analysis, perfect for ongoing system monitoring.

How It Works

1. Discovery: Finds all log files based on your configuration (paths and patterns)
2. Collection: Reads the last N lines from each file (default: 100 lines per file)
3. Aggregation: Combines all content into a single file with clear separators
4. Analysis: Sends the entire aggregated content to AI for comprehensive analysis

Usage Examples

# Basic aggregation (uses default config paths like /var/log)
server-log-monitor aggregate

# Aggregate with custom line count and output file
server-log-monitor aggregate --lines 50 --output today-logs.txt

# Aggregate with verbose output to see what's being collected
server-log-monitor aggregate --verbose

# Use custom config for specific log directories
server-log-monitor aggregate --config /path/to/custom-config.json

# Continuous aggregation and monitoring (uses config default interval)
server-log-monitor aggregate-watch

# Continuous monitoring with custom 30-second interval
server-log-monitor aggregate-watch --interval 30

# Continuous monitoring with custom output file and verbose mode
server-log-monitor aggregate-watch --output my-logs.txt --verbose

Benefits

• Cross-service analysis: Detect issues that affect multiple services
• Pattern recognition: Find correlations between different log sources
• Comprehensive reports: Get a single analysis covering your entire system
• Efficient troubleshooting: All relevant logs in one place

The aggregated file includes headers showing which file each section came from, making it easy to trace issues back to specific services.

Conversational AI Alerts (ElevenLabs)

For critical incidents, the system can make intelligent voice calls using ElevenLabs conversational AI agents. These agents can:

• Answer your questions about the specific alert during the call
• Explain technical issues in plain language
• Provide troubleshooting guidance based on the log context
• Assess severity and recommend next steps
• Access real-time log data during the conversation

Setup ElevenLabs Integration

1. Get ElevenLabs API Key: Sign up at ElevenLabs and get your API key
2. Add to Configuration: Update your config.json with the ElevenLabs section
3. Run Setup Command:

   server-log-monitor elevenlabs:setup

4. Test the Integration:

   server-log-monitor elevenlabs:test
   server-log-monitor elevenlabs:test-call  # Makes actual test call

How Conversational Alerts Work

When a critical alert is detected:

• The system prepares context from recent logs and error details
• Makes an outbound call using your ElevenLabs agent
• The AI agent has access to:
  • Current alert severity and summary
  • Specific error messages and counts
  • Recent log entries for context
  • System status information

Alert Routing Logic:

• Critical Alerts: Conversational voice calls only (no SMS)
• Warning/Normal Alerts: SMS only (no voice calls)
• Fallback: If voice calls fail, SMS is sent as backup

Example Conversation

When you answer a critical alert call:

Agent: "Hello! I'm your server monitoring assistant. I've detected a critical alert that needs your attention. I found a database connection failure affecting your application. Would you like me to explain what's happening?"

You: "Yes, what exactly is wrong?"

Agent: "I'm seeing 3 critical errors in your database logs. The main issue is connection timeouts to your database server at 192.168.1.100. This started about 5 minutes ago and is preventing users from accessing your application. I recommend checking if the database service is running and verifying network connectivity first."

ElevenLabs Tools Integration

The server log monitor includes a powerful tool server that extends ElevenLabs conversational agents with the ability to search codebases, execute diagnostic commands, and investigate errors directly in your project files.

Features

• 🔍 Codebase Search: Find code patterns, functions, or error messages in your project
• ⚡ Terminal Execute: Run safe diagnostic commands to investigate issues
• 📄 File Content Viewer: Examine specific files and line ranges
• 🎯 Error Location Finder: Locate where specific errors occur using messages or stack traces
• 📁 Project Structure: Get an overview of your project organization

Quick Start

1. Start the Tool Server:

   # Start with default settings (port 3001, blackboxHackathon codebase)
   server-log-monitor tool-server
   
   # Or customize the configuration
   server-log-monitor tool-server --port 3002 --codebase /path/to/your/project

2. Configure ElevenLabs Agent: Add the server tools to your ElevenLabs agent configuration using the URLs provided when the server starts.

3. Test the Integration: Your ElevenLabs agent can now search your codebase and execute diagnostic commands during conversations.

Available Tool Endpoints

When the tool server starts, it provides these endpoints for ElevenLabs integration:

• POST /tools/codebase-search - Search for code patterns in the project
• POST /tools/terminal-execute - Execute safe diagnostic commands
• POST /tools/file-content - View specific file contents
• POST /tools/find-error-location - Locate errors using messages/stack traces
• POST /tools/project-structure - Get project organization overview
• GET /health - Health check endpoint

Security Features

• Command injection prevention
• File access restricted to codebase root only
• Whitelist of allowed commands (ls, grep, find, cat, git, npm, python, etc.)
• All operations logged for audit purposes

Example Agent Workflow

When your ElevenLabs agent receives a critical alert:

1. Agent: "I see a database connection error. Let me search your codebase for database-related files."
2. Uses: Codebase Search tool to find database connection code
3. Agent: "I found the issue in src/database.js line 25. Let me examine that file."
4. Uses: File Content tool to view the problematic code
5. Agent: "The connection timeout is set too low. Let me check if the database service is running."
6. Uses: Terminal Execute tool to run ps aux | grep postgres
7. Agent: "The database service is running. I recommend increasing the timeout from 5 seconds to 30 seconds in your database configuration."

For complete setup and configuration details, see the ElevenLabs Tools Integration Guide.

AI Analysis

The system uses BlackBox AI to analyze log content and returns:

• Status: critical, warning, or normal
• Analysis: Brief description of findings
• Errors: Top 3 most problematic errors (if critical)
• Recommendations: Actionable suggestions

Status Classification

• Critical: System failures, crashes, security breaches, database failures
• Warning: High error rates, performance issues, timeouts
• Normal: Standard operations, successful requests

Development

Project Structure

server-log-monitor/
├── src/
│   ├── components/          # Core components
│   │   ├── AIAnalyzer.js    # BlackBox AI integration
│   │   └── LogMonitor.js    # Log file monitoring
│   ├── config/              # Configuration management
│   │   └── ConfigManager.js
│   └── index.js             # Main entry point
├── bin/
│   └── cli.js               # CLI executable
├── config/
│   └── config.sample.json   # Sample configuration
├── prompts/
│   └── log-analysis.js      # AI prompts
└── package.json

Running Tests

npm test

Linting

npm run lint

Troubleshooting

Common Configuration Issues

"BlackBox AI API key is required" Error

Failed to load configuration: Invalid configuration: BlackBox AI API key is required

Solution:

1. Make sure you created config.json (not just config.sample.json):

   cp config.sample.json config.json

2. Edit config.json and add your real BlackBox AI API key:

   {
     "blackboxAI": {
       "apiKey": "your_actual_api_key_here"
     }
   }

3. Get your API key from BlackBox AI

"Config file not found" Error

The monitor looks for config.json in the current directory. Either:

• Run from the directory containing config.json
• Use --config to specify the path: server-log-monitor start --config /path/to/config.json

Permission Issues (Linux/Ubuntu)

If you get permission errors:

# Install with sudo
sudo npm install -g server-log-monitor

# Or use npx (no installation)
npx server-log-monitor start

Twilio Configuration Issues

For SMS/voice alerts, make sure you have:

• Valid Twilio Account SID
• Valid Auth Token
• A Twilio phone number
• Destination phone numbers in E.164 format (+1234567890)

License

MIT License - see LICENSE file for details.