AI Tracking Library

A comprehensive AI interaction tracking library with advanced analytics and failure monitoring capabilities. This library provides the same functionality as the Raindrop AI SDK but with enhanced customization and extensibility.

🚀 Features

Core Tracking Capabilities

• ✅ 14+ Event Types - Complete event coverage for AI interactions
• ✅ 25+ User Traits - Comprehensive user profiling and identification
• ✅ 50+ Event Properties - Rich metadata and context tracking
• ✅ Multiple Attachments - Full conversation context support
• ✅ Tool Tracking - Complete tool usage monitoring
• ✅ Signal Tracking - Feedback and sentiment analysis
• ✅ Behavior Analytics - User interaction and engagement tracking
• ✅ Performance Metrics - System and response time monitoring
• ✅ Error Tracking - Complete error monitoring and classification
• ✅ Business Metrics - Value and satisfaction tracking

Advanced Failure Tracking

• ✅ 15 Failure Categories - Comprehensive failure classification
• ✅ 4 Severity Levels - Critical, High, Medium, Low
• ✅ 10 Auto-Detection Patterns - Automatic failure identification
• ✅ Rich Metadata - Confidence scores, error details, context
• ✅ Alerting System - High failure rate notifications
• ✅ Statistics API - Real-time failure analytics
• ✅ Manual Tracking - Custom failure reporting
• ✅ Batch Processing - Efficient bulk failure tracking

📦 Installation

npm install ai-tracking

🎯 Quick Start

Basic Setup

import { AITracker, FailureTracker } from 'ai-tracking';

// Initialize the tracker
const tracker = new AITracker({
  apiKey: 'your-api-key-here',
  debugLogs: true,
  redactPii: true
});

// Initialize failure tracker
const failureTracker = new FailureTracker(tracker);

Track AI Interactions

// Track a chat message
await tracker.trackChatMessage({
  userId: 'user-123',
  input: 'Hello, how are you?',
  output: 'I am doing well, thank you for asking!',
  model: 'gpt-4o-mini',
  conversationId: 'conv-123'
});

// Track user feedback
await tracker.trackFeedback({
  userId: 'user-123',
  eventId: 'event-123',
  type: 'thumbs_up',
  comment: 'Great response!',
  sentiment: 'POSITIVE'
});

// Track tool usage
await tracker.trackToolUsage({
  userId: 'user-123',
  toolName: 'web_search',
  input: 'weather in New York',
  output: 'Current weather is 72°F and sunny',
  success: true,
  duration: 1500
});

Track Failures

// Auto-detect failure category
await failureTracker.trackFailure({
  userId: 'user-123',
  errorMessage: 'API error: Rate limit exceeded',
  details: {
    statusCode: 429,
    retryAfter: 60
  }
});

// Manual failure category
await failureTracker.trackFailure({
  userId: 'user-123',
  errorMessage: 'Custom business logic error',
  category: 'business_logic_error',
  severity: 'medium',
  details: {
    ruleViolated: 'user_quota_exceeded',
    currentUsage: 95
  }
});

🔧 API Reference

AITracker Class

Constructor

new AITracker(config: TrackingConfig)

Methods

trackAi(params)

Track AI interactions with comprehensive metadata.

await tracker.trackAi({
  eventId?: string;
  event: string;
  userId: string;
  model?: string;
  input?: string;
  output?: string;
  properties?: Partial<EventProperties>;
  attachments?: Attachment[];
});

trackChatMessage(params)

Track chat messages with full conversation context.

await tracker.trackChatMessage({
  userId: string;
  input: string;
  output: string;
  model?: string;
  conversationId?: string;
  properties?: Partial<EventProperties>;
});

trackFeedback(params)

Track user feedback and ratings.

await tracker.trackFeedback({
  userId: string;
  eventId: string;
  type: 'thumbs_up' | 'thumbs_down';
  comment?: string;
  sentiment?: 'POSITIVE' | 'NEGATIVE';
});

trackToolUsage(params)

Track AI tool usage and execution.

await tracker.trackToolUsage({
  userId: string;
  toolName: string;
  input: string;
  output: string;
  success: boolean;
  duration?: number;
  properties?: Partial<EventProperties>;
});

identify(userId, traits)

Identify and profile users with comprehensive traits.

await tracker.identify(userId: string, traits: Partial<UserTraits>);

FailureTracker Class

Methods

trackFailure(params)

Track individual failure events with auto-detection or manual categorization.

await failureTracker.trackFailure({
  userId: string;
  errorMessage: string;
  category?: string;
  severity?: 'critical' | 'high' | 'medium' | 'low';
  details?: Record<string, any>;
  context?: Record<string, any>;
  autoDetected?: boolean;
});

trackFailures(failures)

Track multiple failures in batch for efficiency.

await failureTracker.trackFailures(failures: Array<FailureParams>);

getFailureStats()

Get comprehensive failure statistics and analytics.

const stats = failureTracker.getFailureStats();

getCriticalFailures()

Get critical failures requiring immediate attention.

const critical = failureTracker.getCriticalFailures();

📊 Event Types

The library tracks the following event types:

• chat_message - AI conversation interactions
• chat_error - Chat processing errors
• user_feedback - User ratings and feedback
• feedback_analytics - Feedback analysis and insights
• user_intent - User behavior and intent analysis
• user_behavior - Comprehensive user behavior tracking
• page_loaded - Page lifecycle tracking
• page_visibility_changed - User engagement tracking
• page_before_unload - Session duration tracking
• user_typing - Input behavior tracking
• input_focused - Interaction engagement
• user_scrolled - Content consumption tracking
• feedback_submitted - Feedback submission analytics
• tool_usage - AI tool execution tracking
• failure_detected - Failure and error tracking

🎛️ Configuration Options

interface TrackingConfig {
  apiKey: string;                    // Required: Your API key
  baseUrl?: string;                  // API endpoint URL
  debugLogs?: boolean;               // Enable debug logging
  redactPii?: boolean;               // Redact personally identifiable information
  batchSize?: number;                // Events per batch (default: 10)
  flushInterval?: number;            // Auto-flush interval in ms (default: 5000)
  retryAttempts?: number;            // Retry attempts for failed requests (default: 3)
  timeout?: number;                  // Request timeout in ms (default: 10000)
}

🔍 Failure Categories

The library automatically detects and categorizes failures:

• API Error - External API call failures
• Model Error - AI model errors and invalid responses
• Validation Error - Input validation failures
• Rate Limit - API rate limit exceeded
• Authentication Error - Auth and authorization failures
• Timeout Error - Request and operation timeouts
• Memory Error - Memory allocation failures
• Parsing Error - Data parsing failures
• Database Error - Database operation failures
• Configuration Error - Invalid or missing configuration
• Dependency Error - External dependency failures
• Business Logic Error - Business rule violations
• Security Error - Security violations
• Performance Error - Performance degradation
• Unknown Error - Unclassified errors

🧪 Testing

Run the comprehensive test suite:

npm test

The test suite includes:

• Basic AI tracking functionality
• User identification and profiling
• Chat message tracking
• Feedback and rating tracking
• Tool usage monitoring
• Behavior analytics
• Failure tracking and auto-detection
• Batch processing
• Configuration management
• Statistics and analytics

📈 Analytics and Monitoring

Failure Statistics

// Get failure statistics
const stats = failureTracker.getFailureStats();

// Check failure rates
const rate = failureTracker.getFailureRate('api_error', 3600000); // 1 hour window

// Get critical failures
const critical = failureTracker.getCriticalFailures();

Queue Management

// Get current queue size
const queueSize = tracker.getQueueSize();

// Manually flush events
await tracker.flush();

// Clear the queue
tracker.clearQueue();

🔒 Privacy and Security

• PII Redaction: Automatically redacts personally identifiable information
• Data Sanitization: Cleans sensitive data before transmission
• Secure Transmission: Uses HTTPS for all API communications
• Configurable Privacy: Granular control over data collection

🚀 Performance

• Batch Processing: Efficient bulk event transmission
• Auto-Flushing: Automatic event batching and transmission
• Retry Logic: Robust error handling and retry mechanisms
• Memory Efficient: Optimized for low memory usage
• Non-Blocking: Asynchronous operations that don't block your application

📝 TypeScript Support

Full TypeScript support with comprehensive type definitions:

import { AITracker, FailureTracker, UserTraits, EventProperties } from 'ai-tracking';

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

MIT License - see LICENSE file for details.

🆘 Support

For support and questions:

• Create an issue on GitHub
• Check the documentation
• Review the test suite for usage examples

Made with ❤️ for the AI community