@audit-llm/server

A server for auditing and logging LLM interactions, with improved message handling and robust MCP support.

Features

• Log LLM interactions with detailed context
• Track decision paths and reasoning
• Store model inferences and confidence scores
• MCP-compliant API endpoints
• PostgreSQL persistence
• Rate limiting and security features
• Support for multiple deployment methods
• Comprehensive error handling

Prerequisites

• Node.js (v16 or higher)
• PostgreSQL (v14 or higher)
• npm or yarn

Quick Start Guide

1. Database Setup

# Create database
createdb llm_audit

# Verify connection
psql -h localhost -U postgres -d llm_audit

2. Installation & Usage

Option A: NPX (Recommended)

# Run directly without installation
npx @audit-llm/server@latest "postgresql://postgres:password@localhost:5432/llm_audit?application_name=local_test" 4000

Note: The application_name parameter in the connection string is important for tracking database connections.

Option B: Global Installation

# Install globally (requires sudo/admin privileges)
sudo npm install -g @audit-llm/server  # On Unix/MacOS
# or
npm install -g @audit-llm/server      # On Windows with admin privileges

# Run server
audit-llm-server "postgresql://postgres:password@localhost:5432/llm_audit?application_name=local_test" 4000

Option C: Development Usage

# Clone and install
git clone https://github.com/sakhilchawla/audit-llm-decision.git
cd audit-llm-decision
npm install

# Build
npm run build

# Run directly with Node
node dist/server.js "postgresql://postgres:password@localhost:5432/llm_audit?application_name=local_test" 4000

3. Integration with AI Tools

LLM Usage

For LLMs (like Claude) to start the server directly, use this command:

node dist/server.js "postgresql://postgres:password@localhost:5432/llm_audit?application_name=llm_audit" 4000

Note: Make sure to:

1. Replace postgres:password with your actual database credentials
2. Adjust the port (4000) if needed
3. The application_name parameter helps track which client is connecting
4. The server will respond with {"status":"healthy"} on the health endpoint when running correctly

Cursor Integration

1. Create or edit the config file:
   • On macOS: ~/Library/Application Support/Cursor/config.json
   • On Windows: %APPDATA%\Cursor\config.json
   • On Linux: ~/.config/Cursor/config.json

{
  "mcpServers": {
    "audit-llm": {
      "command": "npx",
      "args": [
        "--yes",
        "@audit-llm/server@latest",
        "postgresql://postgres:password@localhost:5432/llm_audit?application_name=cursor_mcp",
        "4000",
        "--mcp"
      ],
      "env": {
        "NODE_ENV": "production",
        "DB_APPLICATION_NAME": "cursor_mcp"
      }
    }
  }
}

2. Restart Cursor to apply changes

3. Troubleshooting Cursor Integration:

   • Make sure Cursor has the necessary permissions to access the config directory
   • Check Cursor's logs for any MCP-related errors:
     • On macOS: ~/Library/Logs/Cursor/main.log
     • On Windows: %APPDATA%\Cursor\logs\main.log
     • On Linux: ~/.config/Cursor/logs/main.log
   • Verify PostgreSQL is running and accessible
   • Ensure the port (4000) is available
   • Try running the server manually first to verify database connection:

     npx @audit-llm/server@latest "postgresql://postgres:password@localhost:5432/llm_audit?application_name=cursor_test" 4000 --mcp

   • If you see authentication errors in Cursor, try signing out and back in
   • If you see ResizeObserver errors, these are usually harmless UI warnings and don't affect functionality

Note: The --mcp flag is required for Cursor integration. This ensures proper JSON-RPC message formatting.

Claude Desktop Integration

1. Create or edit ~/Library/Application Support/Claude/claude_desktop_config.json (MacOS) or %APPDATA%/Claude/claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "audit-llm": {
      "command": "npx",
      "args": [
        "--yes",
        "@audit-llm/server@latest",
        "postgresql://postgres:password@localhost:5432/llm_audit?application_name=claude_desktop",
        "4000"
      ],
      "env": {
        "NODE_ENV": "production",
        "DB_APPLICATION_NAME": "claude_desktop"
      }
    }
  }
}

Note:

• Replace postgres:password with your actual database credentials
• The application_name parameter helps identify connections in your database
• The server uses JSON-RPC formatted logs for MCP communication
• Make sure the database exists and is accessible before starting
• The --yes flag automatically accepts the package installation
• The server will automatically handle port conflicts by incrementing the port number
• All logs will be properly formatted for Claude Desktop's MCP protocol

2. Restart Claude Desktop to apply changes

MCP Integration Details

The server implements the Model Context Protocol (MCP) with the following features:

1. JSON-RPC Logging

   • All logs are formatted as JSON-RPC 2.0 messages
   • Proper error handling and status reporting
   • Structured logging for better integration

2. Message Handling

   • Robust message buffering
   • Proper encoding of special characters
   • Multi-line message support
   • Comprehensive error handling

3. Automatic Port Management

   • Automatically finds available ports if default is in use
   • Reports port changes through MCP logging

4. Database Integration

   • Automatic schema creation
   • Index optimization
   • Connection pooling
   • Error resilience

API Documentation

Endpoints

GET /health

Check server health status

curl http://localhost:4000/health

GET /schema

Get JSON schema for log entries

curl http://localhost:4000/schema

POST /api/v1/log

Log an LLM interaction

curl -X POST http://localhost:4000/api/v1/log \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "User query",
    "response": "Model response",
    "model_type": "claude",
    "model_version": "3",
    "metadata": {}
  }'

GET /api/v1/logs

Retrieve logged interactions

curl http://localhost:4000/api/v1/logs

Environment Variables

# Server Configuration
PORT=4000                    # Default port
NODE_ENV=production         # Environment (production/development)

# Database Configuration (Optional - can be provided via connection string)
DB_USER=postgres           # Database user
DB_PASSWORD=your_password  # Database password
DB_HOST=localhost         # Database host
DB_PORT=5432             # Database port
DB_NAME=llm_audit       # Database name
DB_APPLICATION_NAME=audit_llm  # Application name for connection tracking

Note: Environment variables are optional if you provide the full connection string with the application_name parameter.

Development

# Clone repository
git clone https://github.com/sakhilchawla/audit-llm-decision.git
cd audit-llm-decision

# Install dependencies
npm install

# Run in development mode
npm run dev

# Build
npm run build

# Run tests
npm test

Troubleshooting

Common Issues

1. Connection Failed

# Check database exists
createdb llm_audit

# Verify credentials
psql -h localhost -U postgres -d llm_audit

2. Port in Use

# Use different port
npx @audit-llm/server "postgresql://user:pass@localhost:5432/llm_audit?application_name=local_test" 4001

3. Permission Issues

# Grant database permissions
psql -U postgres
postgres=# GRANT ALL PRIVILEGES ON DATABASE llm_audit TO your_user;

Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

License

MIT License - see LICENSE file for details

Version History

Version 1.1.12 (Latest)

• Improved message handling and encoding:
  • Proper handling of multi-line messages
  • Robust JSON-RPC message buffering
  • Better error handling for malformed messages
  • Improved message validation
  • Fixed encoding issues with special characters
• Enhanced MCP protocol support:
  • Proper message framing
  • Better error reporting
  • Improved connection stability
• Database improvements:
  • Better error handling
  • More informative logging
  • Improved connection management

Version 1.1.11

• Fixed JSON parsing issues
• Improved message validation
• Enhanced error handling
• Added comprehensive test suite

Version 1.0.19

• Enhanced connection string parsing
• Added detailed connection logs
• Improved error handling