⚡️ Lightning Logs

Lightning-fast log analyzer powered by Go. Analyze gigabytes of logs in seconds.

Features

• 🚀 Blazing Fast - Process 5 GB logs in 5-15 seconds
• 🔥 Parallel Processing - Multi-threaded analysis with Go
• 💪 Low Memory - Streaming architecture
• 🎯 Smart Filtering - Pattern matching and grouping
• 📊 Detailed Stats - Top patterns and counts
• 🔒 Type Safe - Full TypeScript support
• 🌍 Cross Platform - Works on macOS, Linux, and Windows

Installation

npm install lightning-logs

Quick Start

TypeScript

import { analyzeLogs, AnalyzeResult } from 'lightning-logs';

async function analyze() {
  const result: AnalyzeResult = await analyzeLogs('/var/log/app.log', {
    filters: ['ERROR', 'FATAL'],
    workers: 8
  });

  console.log(`Total lines: ${result.totalLines}`);
  console.log(`Matched lines: ${result.matchedLines}`);
  console.log(`Analysis took: ${result.duration}ms`);
  
  // Top error patterns
  result.topPatterns.forEach(pattern => {
    console.log(`${pattern.text}: ${pattern.count} occurrences`);
  });
}

analyze();

JavaScript

const { analyzeLogs } = require('lightning-logs');

async function analyze() {
  const result = await analyzeLogs('/var/log/app.log', {
    filters: ['ERROR', 'FATAL'],
    workers: 8
  });

  console.log(`Found ${result.matchedLines} errors`);
}

analyze();

API Documentation

analyzeLogs(filePath, options)

Analyzes a log file and returns statistics.

Parameters

• filePath string - Path to the log file
• options AnalyzeOptions (optional)
  • filters string[] - Array of patterns to search for (default: [])
  • workers number - Number of parallel workers (default: 4)
  • caseSensitive boolean - Case-sensitive matching (default: false) (coming soon)

Returns

Promise<AnalyzeResult> - Analysis result with the following properties:

interface AnalyzeResult {
  totalLines: number;        // Total number of lines in file
  matchedLines: number;      // Number of lines matching filters
  groupedBy: Record<string, number>;  // Counts grouped by filter
  topPatterns: Pattern[];    // Top N most common patterns
  duration?: number;         // Analysis duration in milliseconds
}

interface Pattern {
  text: string;   // Pattern text
  count: number;  // Number of occurrences
}

Examples

Find All Errors

const result = await analyzeLogs('./app.log', {
  filters: ['ERROR'],
  workers: 4
});

console.log(`Found ${result.groupedBy.ERROR} errors`);

Multiple Patterns

const result = await analyzeLogs('./app.log', {
  filters: ['ERROR', 'WARN', 'FATAL'],
  workers: 8
});

console.log('Results:', result.groupedBy);
// { ERROR: 150, WARN: 45, FATAL: 3 }

Count All Lines

const result = await analyzeLogs('./app.log', {
  workers: 8
});

console.log(`Total lines: ${result.totalLines}`);

Express.js Integration

import express from 'express';
import { analyzeLogs } from 'lightning-logs';

const app = express();

app.get('/analyze-logs', async (req, res) => {
  try {
    const result = await analyzeLogs('/var/log/app.log', {
      filters: ['ERROR', 'FATAL'],
      workers: 8
    });
    
    res.json({
      success: true,
      data: result
    });
  } catch (error) {
    res.status(500).json({
      success: false,
      error: error.message
    });
  }
});

app.listen(3000);

NestJS Integration

import { Injectable } from '@nestjs/common';
import { analyzeLogs, AnalyzeResult } from 'lightning-logs';

@Injectable()
export class LogService {
  async analyzeServerLogs(): Promise<AnalyzeResult> {
    return analyzeLogs('/var/log/server.log', {
      filters: ['ERROR', 'CRITICAL'],
      workers: 8
    });
  }
}

Performance Benchmarks

Tested on MacBook Pro M1 (2021), 16GB RAM

| File Size | Lines | Traditional* | Lightning Logs | Speedup | |-----------|------------|--------------|----------------|---------| | 100 MB | 500K | ~30s | ~0.5s ⚡ | 60x | | 1 GB | 5M | ~5min | ~3s ⚡ | 100x | | 5 GB | 25M | ~25min | ~15s ⚡ | 100x | | 10 GB | 50M | ~50min | ~30s ⚡ | 100x |

*Traditional = Node.js fs.readFileSync + String operations

Real-world Example

# Analyze 2 GB production log (10M lines)
⚡ Lightning Logs: Starting analysis...
⚡ File: /var/log/production.log
⚡ Filters: [ 'ERROR', 'FATAL' ]
⚡ Workers: 8
⚡ Analysis complete in 6,234ms

Total lines: 10,000,000
Matched lines: 45,678
Duration: 6234ms

Top errors:
  - "Database connection timeout": 12,345
  - "Failed to connect to Redis": 8,901
  - "Out of memory error": 2,456

How It Works

Lightning Logs uses a hybrid architecture:

1. Go Binary - High-performance core written in Go

   • Parallel file reading
   • Concurrent pattern matching
   • Efficient memory usage

2. Node.js Wrapper - Easy-to-use JavaScript/TypeScript interface

   • Spawns Go binary as child process
   • Handles data serialization
   • Provides type-safe API

┌─────────────────┐
│  Your Node App  │
└────────┬────────┘
         │ TypeScript/JavaScript
         ↓
┌─────────────────┐
│  Node Wrapper   │
└────────┬────────┘
         │ JSON via stdin/stdout
         ↓
┌─────────────────┐
│   Go Binary     │ ⚡ Fast!
│  (8 workers)    │
└─────────────────┘

Platform Support

• ✅ macOS (Intel & Apple Silicon)
• ✅ Linux (x64)
• ✅ Windows (x64)

The package includes pre-built binaries for all platforms.

Requirements

• Node.js >= 16.0.0
• No need to install Go (binaries included)

TypeScript Support

Lightning Logs is written in TypeScript and includes full type definitions.

import { 
  analyzeLogs, 
  AnalyzeOptions, 
  AnalyzeResult, 
  Pattern 
} from 'lightning-logs';

// Full IntelliSense support
const options: AnalyzeOptions = {
  filters: ['ERROR'],
  workers: 8
};

const result: AnalyzeResult = await analyzeLogs('./app.log', options);

Troubleshooting

Binary not found

Error: Binary not found: /path/to/lightning-logs/bin/lightning-macos

Solution: The binary might not have execute permissions.

chmod +x node_modules/lightning-logs/bin/lightning-*

Out of memory

If analyzing extremely large files (50GB+), increase Node.js memory:

node --max-old-space-size=4096 your-script.js

Permission denied

Ensure you have read permissions for the log file:

chmod +r /var/log/app.log

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Development Setup

# Clone repository
git clone https://github.com/thechotinun/lightning-logs.git
cd lightning-logs

# Check required tools (Go, Node.js, npm)
make check

# Install dependencies
make install

# Build everything (Go binaries + TypeScript)
make build-all

# Run tests
make test

# Run example (optional)
cd example
node test.js

Available Make Commands

# Building
make build           # Build everything
make build-go        # Build Go binary for current platform
make build-node      # Build TypeScript only
make build-cross     # Build binaries for all platforms (macOS, Linux, Windows)

# Development
make watch           # Watch TypeScript files for changes
make dev             # Build + watch mode

# Testing
make test            # Run tests
make test-watch      # Run tests in watch mode
make test-coverage   # Run tests with coverage report

# Cleaning
make clean           # Clean build artifacts
make clean-all       # Clean everything including node_modules

# Publishing
make pack            # Create npm package (.tgz)
make publish-dry     # Dry run publish (see what will be published)
make publish         # Publish to npm (builds + tests + publishes)

# Utilities
make check           # Check if required tools are installed
make info            # Show project information
make size            # Show binary sizes
make help            # Show all available commands

License

MIT © [4l2T]

Why "Lightning Logs"?

Because it's as fast as lightning ⚡ when analyzing logs!

Acknowledgments

• Built with Go
• Powered by goroutines and channels
• Inspired by the need for faster log analysis

Links

• GitHub Repository
• npm Package
• Report Issues

Made with ⚡️ and ❤️