auto-eval-sdk

Evaluate voice AI agent calls with automated quality checks, compliance validation, and actionable insights.

Installation

npm install auto-eval-sdk

Quick Start

import { AutoEvalClient } from 'auto-eval-sdk';

const client = new AutoEvalClient({
  apiKey: process.env.AUTO_EVAL_API_KEY,
  preset: 'healthcare', // Optional: pre-configured rules
});

const result = await client.evaluateCall({
  transcript: 'AGENT: Hello, how can I help? CUSTOMER: I need to cancel...',
  speakerTurns: [
    { speaker: 'agent', text: 'Hello, how can I help?', startTime: 0, endTime: 2000, timestamp: '2024-01-01T10:00:00Z' },
    { speaker: 'customer', text: 'I need to cancel my subscription', startTime: 2500, endTime: 5000, timestamp: '2024-01-01T10:00:05Z' },
  ],
  metadata: {
    callId: 'call-123',
    agentVersion: 'v1.2.3',
    callDuration: 60000,
    startTime: '2024-01-01T10:00:00Z',
    endTime: '2024-01-01T10:01:00Z',
  },
});

console.log(result.status); // 'PASS' | 'WARN' | 'FAIL' | 'BLOCK_RELEASE'
console.log(result.overallScore); // 0-100

Features

• Automated Quality Scoring - Response latency, verbosity, interruptions, talk ratio
• LLM-Powered Evaluations - Task resolution, policy compliance, hallucination detection, tone analysis
• Industry Presets - Pre-configured rules for healthcare, finance, debt collection, and more
• Audio File Support - Upload recordings directly; transcription handled automatically
• Compliance Checks - HIPAA, PCI-DSS, TCPA/FDCPA validation built-in
• Trend Detection - Automatic regression alerts across agent versions

Usage

With Transcript

const result = await client.evaluateCall({
  transcript: 'Full call transcript...',
  speakerTurns: [/* ... */],
  metadata: { /* ... */ },
});

With Audio File

const audioFile = document.querySelector('input[type="file"]').files[0];

const result = await client.evaluateCall({
  audioFile: audioFile,
  metadata: {
    callId: 'call-123',
    agentVersion: 'v1.2.3',
    callDuration: 60000,
    startTime: '2024-01-01T10:00:00Z',
    endTime: '2024-01-01T10:01:00Z',
  },
});

Supported formats: MP3, WAV, MP4, WebM (max 100MB)

Industry Presets

Presets configure compliance thresholds and critical flags automatically:

// Set during initialization
const client = new AutoEvalClient({
  apiKey: 'your-key',
  preset: 'healthcare', // HIPAA-compliant rules
});

// Or switch dynamically
await client.usePreset('finance');

// List available presets
const presets = await client.getPresets();

Available presets:

• healthcare - HIPAA compliance, patient privacy
• finance - PCI-DSS, data security
• debt-collection - TCPA/FDCPA compliance
• customer-support - Standard best practices
• e-commerce - Order management focus
• telemarketing - TCPA consent verification
• appointment-booking - Accuracy requirements
• tech-support - Problem resolution focus

Response Format

{
  status: 'PASS' | 'WARN' | 'FAIL' | 'BLOCK_RELEASE',
  overallScore: 85.5,
  metrics: {
    deterministic: {
      responseLatency: 1200,      // ms
      verbosity: 150,              // words/min
      interruptions: 2,
      talkRatio: 0.65              // 0-1
    },
    llmEvaluations: {
      taskResolution: { score: 90, confidence: 0.95, reasoning: '...' },
      policyCompliance: { score: 95, confidence: 0.98, reasoning: '...' },
      hallucinationRisk: { score: 10, confidence: 0.92, reasoning: '...' },
      toneAndEmpathy: { score: 88, confidence: 0.94, reasoning: '...' }
    }
  },
  flags: [
    { type: 'high_verbosity', severity: 'low', message: '...' }
  ],
  recommendations: [
    'Reduce response latency to under 2 seconds',
    'Improve policy compliance verification'
  ],
  regressionIndicators: {
    detected: true,
    metric: 'taskResolution',
    trend: 'degrading',
    confidence: 0.75
  }
}

API

AutoEvalClient

Constructor

new AutoEvalClient({
  apiKey: string,              // Required
  baseUrl?: string,            // Default: 'https://api.auto-eval.com'
  timeout?: number,            // Default: 30000ms
  webhookUrl?: string,         // Optional webhook for async results
  preset?: IndustryPreset      // Optional preset
})

Methods

evaluateCall(data) - Evaluate a single call

• Returns: Promise<EvaluationResponse>
• Supports: transcript + speakerTurns OR audioFile

evaluateCalls(calls[]) - Batch evaluation

• Returns: Promise<EvaluationResponse[]>

getEvaluationStatus(callId) - Get status for a call

• Returns: Promise<EvaluationResponse>

usePreset(preset) - Apply industry preset

• Returns: Promise<void>

getPresets() - List available presets

• Returns: Promise<Array<{ id, name, description }>>

TypeScript

Full TypeScript support with exported types:

import type {
  PostCallData,
  PostCallDataWithAudio,
  CallEvaluationInput,
  EvaluationResponse,
  IndustryPreset
} from 'auto-eval-sdk';

Examples

Healthcare Agent

const client = new AutoEvalClient({
  apiKey: process.env.AUTO_EVAL_API_KEY,
  preset: 'healthcare',
});

const result = await client.evaluateCall({
  transcript: '...',
  speakerTurns: [/* ... */],
  metadata: { /* ... */ },
});

if (result.flags.some(f => f.type === 'hipaa_violation')) {
  // Block release, notify compliance team
}

Debt Collection Agent

const client = new AutoEvalClient({
  apiKey: process.env.AUTO_EVAL_API_KEY,
  preset: 'debt-collection',
});

const result = await client.evaluateCall({
  audioFile: recordingFile,
  metadata: { /* ... */ },
});

if (result.status === 'BLOCK_RELEASE') {
  // TCPA/FDCPA violation detected
}

Error Handling

try {
  const result = await client.evaluateCall(data);
} catch (error) {
  if (error.message.includes('Validation failed')) {
    // Invalid data format
  } else if (error.message.includes('API error')) {
    // Server error
  }
}

License

MIT