log-sink-mcp

AI-powered log analysis MCP server with HTTP ingestion and SQLite storage.

Overview

log-sink-mcp provides a complete log analysis solution for AI assistants like Claude. It combines:

• HTTP Server: REST endpoint for log ingestion from any application
• MCP Server: 7 AI-optimized tools for querying and analyzing logs
• SQLite Database: Fast local storage with FTS5 full-text search plus cached local embeddings
• Real-time Analysis: Query logs, trace distributed requests, analyze error patterns

Perfect for debugging, monitoring, and understanding application behavior through AI-powered analysis.

Features

• ✅ HTTP Log Ingestion: POST logs from any application via REST API
• ✅ 7 MCP Tools: Query, search, trace, analyze, and manage logs
• ✅ Hybrid Search: FTS5-powered search plus optional local semantic retrieval across messages and errors
• ✅ Distributed Tracing: Timeline view for trace IDs and span relationships
• ✅ Error Analysis: Pattern detection and error categorization
• ✅ Structured Logging: JSON metadata, trace IDs, span IDs
• ✅ Auto-Migration: Drizzle ORM with automatic schema migrations
• ✅ Type-Safe: Full TypeScript coverage with Drizzle schema inference

Architecture

┌─────────────┐        HTTP POST         ┌─────────────────┐
│ Application │ ───────────────────────> │  HTTP Server    │
│  (Pino,     │     /logs endpoint       │  (Hono)         │
│  Winston)   │                          │  Port 3100      │
└─────────────┘                          └────────┬────────┘
                                                  │
                                                  ▼
                                         ┌─────────────────┐
                                         │  LogStorage     │
                                         │  Service        │
                                         └────────┬────────┘
                                                  │
                                                  ▼
┌──────────────┐       MCP Protocol      ┌─────────────────┐
│   Claude /   │ <─────────────────────> │   MCP Server    │
│   AI Agent   │    7 Analysis Tools     │   (stdio)       │
└──────────────┘                         └────────┬────────┘
                                                  │
                                                  ▼
                                         ┌─────────────────┐
                                         │ SQLite + FTS5 + │
                                         │  local vectors   │
                                         │  ./logs/        │
                                         │  session.db     │
                                         └─────────────────┘

Singleton HTTP Server Management

log-sink-mcp implements automatic singleton coordination to prevent port conflicts when multiple MCP servers run concurrently.

How It Works

1. Service Discovery: When an MCP server starts, it checks .agiflow/registry/log-sink-mcp-http_{env}.json for an existing HTTP server
2. Health Check: Pings /health endpoint to verify the registered server is alive
3. Reuse or Start: If healthy, reuses existing server; otherwise starts a new one
4. PID Tracking: Process IDs stored in .pids/ for lifecycle management
5. Auto-Cleanup: Registry and PID files removed on graceful shutdown

Service Registry Schema

{
  "name": "log-sink-mcp-http",
  "port": 3100,
  "host": "localhost",
  "environment": "development",
  "pid": 12345,
  "status": "running",
  "startedAt": "2026-01-15T10:00:00.000Z",
  "metadata": {
    "healthCheckUrl": "http://localhost:3100/health",
    "dbPath": "./logs/session.db"
  }
}

Management Commands

# Check status of HTTP server
bun run src/cli.ts status

# Stop HTTP server
bun run src/cli.ts stop

# Start with auto-coordination
bun run src/cli.ts start

Installation

# From monorepo root
pnpm install

# Optional: install local embedding support for semantic search
pnpm add ruvector-onnx-embeddings-wasm

# Or install globally
pnpm add -g log-sink-mcp

If ruvector-onnx-embeddings-wasm is not installed, log-sink-mcp still starts and falls back to FTS5 search only.

Quick Start

1. Start the Servers

# Start both HTTP and MCP servers (recommended)
bun run src/cli.ts start

# Or start only HTTP server
bun run src/cli.ts start --http-only

# Or start only MCP server (stdio transport)
bun run src/cli.ts start --mcp-only

# With custom configuration
bun run src/cli.ts start --port 3100 --db-path ./logs/session.db

The start command uses singleton pattern - multiple MCP servers will share a single HTTP server automatically.

2. Configure MCP Server

Add to your mcp-config.yaml:

mcpServers:
  log-sink:
    type: stdio
    command: bun
    args:
      - ./packages/log-sink-mcp/src/cli.ts
      - mcp-serve
      - --db-path
      - ./logs/session.db
    disabled: false

3. Send Logs

# Send a test log
curl -X POST http://localhost:3100/logs \
  -H "Content-Type: application/json" \
  -d '{
    "logs": [{
      "level": "info",
      "message": "Application started",
      "service": "my-app",
      "metadata": {"version": "1.0.0"}
    }]
  }'

4. Query with AI

Ask Claude:

"Query all error logs from the database"

"Show me the trace timeline for trace ID abc123"

"Analyze error patterns in the logs"

HTTP API Reference

POST /logs

Ingest log entries in batch.

Request Body:

{
  logs: Array<{
    // Required fields
    level: 'debug' | 'info' | 'warn' | 'error';
    message: string;
    service: string;

    // Optional fields
    timestamp?: string;        // ISO 8601, defaults to now
    traceId?: string;          // 32-char hex for distributed tracing
    spanId?: string;           // 16-char hex for span identification
    parentSpanId?: string;     // 16-char hex for parent span
    hostname?: string;
    pid?: number;
    metadata?: Record<string, any>;
    errorType?: string;
    errorMessage?: string;
    errorStack?: string;
  }>
}

Response (201 Created):

{
  "success": true,
  "count": 5,
  "message": "Successfully inserted 5 logs"
}

Response (400 Bad Request):

{
  "success": false,
  "error": "Validation failed",
  "details": [...]
}

GET /health

Health check endpoint.

Response (200 OK):

{
  "status": "healthy",
  "service": "log-sink-mcp",
  "timestamp": "2026-01-15T10:00:00.000Z"
}

MCP Tools Reference

1. query_logs

Filter logs by level, service, or trace ID.

Parameters:

• level?: string - Filter by log level (debug, info, warn, error)
• service?: string - Filter by service name
• traceId?: string - Filter by trace ID
• limit?: number - Maximum results (default: 100)

Example:

Query all error logs from user-service

Returns: JSON array of matching log entries with timestamps, messages, metadata.

2. search_logs

Hybrid search across log messages and error messages. By default it combines:

• FTS5 exact / keyword matching
• Local semantic similarity from cached embeddings

Parameters:

• searchQuery: string (required) - Text to search for
• mode?: string - hybrid (default), fts, or semantic
• service?: string or string[] - Filter by service name
• level?: string or string[] - Filter by log level
• startTime?: string - ISO 8601 start time filter
• endTime?: string - ISO 8601 end time filter
• limit?: number - Maximum results (default: 100)

Example:

Search logs for "database connection timeout"

Returns: Matching logs with search relevance, count of results.

3. get_trace_timeline

Get chronological timeline for a distributed trace.

Parameters:

• traceId: string (required) - Trace ID to retrieve

Example:

Show me the trace timeline for trace ID a1b2c3d4e5f6789012345678901234ab

Returns: Ordered sequence of log entries for the trace, showing service call flow.

4. analyze_errors

Analyze error patterns and categorize by error type.

Parameters:

• limit?: number - Maximum errors to analyze (default: 100)

Example:

Analyze error patterns in the logs

Returns: Error categories, frequencies, example stack traces, common patterns.

5. get_log_stats

Get statistics grouped by log level.

Parameters: None

Example:

Show me log statistics

Returns: Count per log level (debug, info, warn, error), total count.

6. get_services

List all unique services that have logged.

Parameters: None

Example:

What services are logging to the database?

Returns: Array of service names with log counts.

7. clear_logs

Delete all logs from the database.

Parameters: None

Example:

Clear all logs from the database

Returns: Confirmation with count of deleted entries.

⚠️ Warning: This operation cannot be undone.

Integration Examples

Pino Logger

import pino from 'pino';

const logger = pino({
  transport: {
    target: 'pino-http-send',
    options: {
      url: 'http://localhost:3100/logs',
      method: 'POST',
    },
  },
});

logger.info({ traceId: 'abc123', userId: '456' }, 'User logged in');

Winston Logger

import winston from 'winston';

const logger = winston.createLogger({
  transports: [
    new winston.transports.Http({
      host: 'localhost',
      port: 3100,
      path: '/logs',
    }),
  ],
});

logger.error('Database connection failed', {
  service: 'api',
  errorType: 'ConnectionError'
});

Custom Fetch

async function sendLog(level: string, message: string, metadata?: any) {
  await fetch('http://localhost:3100/logs', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      logs: [{
        level,
        message,
        service: 'my-service',
        metadata,
        timestamp: new Date().toISOString(),
      }],
    }),
  });
}

await sendLog('info', 'Request processed', { duration: 245, status: 200 });

Configuration

CLI Options

start command (Recommended)

Start HTTP and/or MCP servers with automatic singleton coordination.

bun run src/cli.ts start [options]

Options:

• --port <number>: HTTP server port (default: 3100)
• --db-path <path>: SQLite database path (default: ./logs/session.db)
• --in-memory: Use in-memory database for testing (default: false)
• --http-only: Start only the HTTP server
• --mcp-only: Start only the MCP server (stdio transport)

Examples:

# Start both servers with defaults
bun run src/cli.ts start

# Production mode with persistent database
bun run src/cli.ts start --port 3100 --db-path ./logs/production.db

# Testing with in-memory database
bun run src/cli.ts start --in-memory

status command

Show diagnostic information about running services.

bun run src/cli.ts status

Displays:

• HTTP server status (running/stopped)
• Process ID and port
• Database file path and size
• Health check URL

stop command

Gracefully stop HTTP server and cleanup registry/PID files.

bun run src/cli.ts stop

Behavior:

• Sends SIGTERM to HTTP server process
• Removes .agiflow/registry/ entries
• Cleans up .pids/ files

http-serve command

Manually start HTTP log ingestion server (for advanced use).

bun run src/cli.ts http-serve [options]

Options:

• --port <number>: HTTP server port (default: 3000)
• --db-path <path>: SQLite database path (default: ./logs.db)
• --in-memory: Use in-memory database (default: false)

Note: Prefer using start command for automatic singleton coordination.

mcp-serve command

Manually start MCP server (for advanced use or custom transports).

bun run src/cli.ts mcp-serve [options]

Options:

• --type <type>: Transport type: stdio, http, or sse (default: stdio)
• --port <number>: Port for http/sse transport (default: 3000)
• --host <host>: Host for http/sse transport (default: localhost)
• --db-path <path>: SQLite database path (default: ./logs/session.db)
• --in-memory: Use in-memory database (default: false)
• --cleanup: Stop HTTP server on MCP shutdown (stdio only, default: false)

Examples:

# stdio transport (for Claude Desktop)
bun run src/cli.ts mcp-serve --type stdio

# HTTP transport
bun run src/cli.ts mcp-serve --type http --port 3001

# SSE transport
bun run src/cli.ts mcp-serve --type sse --port 3002

Note: For stdio transport, prefer using start --mcp-only for automatic HTTP coordination.

logs command

Run the same analysis capabilities exposed by the package MCP tools directly from the shell.

bun run src/cli.ts logs <subcommand> [options]

Subcommands:

• query: Equivalent to query_logs
• search: Equivalent to search_logs
• search --mode hybrid: Default hybrid search across FTS5 and semantic embeddings
• trace: Equivalent to get_trace_timeline
• analyze-errors: Equivalent to analyze_errors
• stats: Equivalent to get_log_stats
• services: Equivalent to get_services
• clear: Equivalent to clear_logs

All subcommands support:

• --db-path <path>: SQLite database path (default: ./logs/session.db)
• --in-memory: Use an in-memory database

Examples:

# Query error logs
bun run src/cli.ts logs query --level error --db-path ./logs/session.db

# Search for timeout failures
bun run src/cli.ts logs search "timeout OR retry" --service api-gateway user-service

# Inspect one trace
bun run src/cli.ts logs trace a1b2c3d4e5f6789012345678901234ab

# Analyze grouped errors in a time window
bun run src/cli.ts logs analyze-errors \
  --start-time 2026-01-15T10:00:00Z \
  --end-time 2026-01-15T11:00:00Z

# View service names discovered in the log database
bun run src/cli.ts logs services

Environment Variables

None required. All configuration via CLI flags.

Development

Setup

# Install dependencies
pnpm install

# Run development server
pnpm dev

# Run HTTP server in dev mode
bun run src/cli.ts http-serve --port 3100

Project Structure

log-sink-mcp/
├── src/
│   ├── cli.ts                    # CLI entry point
│   ├── commands/                 # CLI commands
│   │   ├── http-serve.ts        # HTTP server command
│   │   └── mcp-serve.ts         # MCP server command
│   ├── container/               # InversifyJS DI container
│   ├── models/                  # Drizzle schemas
│   │   └── schema.ts            # Log table schema
│   ├── server/
│   │   ├── http.ts              # Hono HTTP server
│   │   └── index.ts             # MCP server
│   ├── services/                # Business logic
│   │   ├── LogStorageService.ts
│   │   ├── LogQueryService.ts
│   │   ├── LogSearchService.ts
│   │   └── LogRetentionService.ts
│   ├── tools/                   # MCP tools
│   │   ├── QueryLogsTool.ts
│   │   ├── SearchLogsTool.ts
│   │   ├── GetTraceTimelineTool.ts
│   │   ├── AnalyzeErrorsTool.ts
│   │   ├── GetLogStatsTool.ts
│   │   ├── GetServicesTool.ts
│   │   └── ClearLogsTool.ts
│   └── types/                   # TypeScript types
├── migrations/                  # Drizzle migrations
├── tests/
│   ├── integration/             # Integration tests
│   ├── services/                # Service unit tests
│   ├── tools/                   # Tool unit tests
│   └── commands/                # Command tests
└── README.md

Testing

# Run all tests
pnpm test

# Run integration tests only
pnpm test integration

# Run with coverage
pnpm test --coverage

Test Coverage:

• HTTP server integration tests (8 tests)
• MCP tools integration tests (17 tests)
• Command tests (5 tests)

Adding New Tools

1. Create tool class in src/tools/
2. Implement Tool interface with getDefinition() and execute()
3. Register in src/server/index.ts
4. Add integration tests in tests/integration/

See existing tools for patterns.

Troubleshooting

Check Service Status

Use the status command to diagnose issues:

bun run src/cli.ts status

This shows:

• Whether HTTP server is running
• Process ID and port
• Database path and size
• Health check URL

Logs not appearing

1. Check HTTP server is running:

   # Using status command
   bun run src/cli.ts status
   
   # Or direct health check
   curl http://localhost:3100/health

2. Verify log format matches API schema

3. Check database path is accessible

4. Review HTTP server logs for errors

MCP tools not working

1. Verify database file exists and has logs:

   bun run src/cli.ts status

2. Check --db-path matches between HTTP and MCP servers

3. Restart MCP server after configuration changes:

   bun run src/cli.ts stop
   bun run src/cli.ts start

Database locked errors

SQLite uses WAL mode for better concurrency, but:

• Don't run multiple HTTP servers on same database
• Ensure file permissions allow writes
• Use separate databases for testing

Finding and Killing HTTP Server

If you need to manually stop the HTTP server:

# Graceful stop (recommended)
bun run src/cli.ts stop

# Find process manually
cat .pids/log-sink-mcp-http-development.pid

# Kill process (if graceful stop fails)
kill $(cat .pids/log-sink-mcp-http-development.pid)

# Force kill (last resort)
kill -9 $(cat .pids/log-sink-mcp-http-development.pid)

Stale Registry Entries

If the registry file exists but server isn't running:

# Clean up stale registry
rm .agiflow/registry/log-sink-mcp-http_development.json
rm .pids/log-sink-mcp-http-development.pid

# Restart
bun run src/cli.ts start

The health check will automatically detect stale registries and start a new server.

Port Already in Use

If port 3100 is already taken:

# Use different port
bun run src/cli.ts start --port 3101

# Or find what's using the port
lsof -i :3100

# Kill the process using the port
kill $(lsof -t -i :3100)

License

MIT

Contributing

This package is part of the Agiflow monorepo. Contributions welcome!

1. Follow coding standards in /docs/CODING_STANDARDS.md
2. Use architect MCP before modifying code
3. Add tests for new features
4. Run pnpm test before committing