ThinkHive SDK v4.2.2

The official JavaScript/TypeScript SDK for ThinkHive - AI Agent Observability Platform.

Features

• OpenTelemetry-Based Tracing: Built on OTLP for seamless integration with existing observability tools
• Run-Centric Architecture: Atomic unit of work tracking with claims, calibration, and linking
• Facts vs Inferences: Claims API for separating verified facts from inferences
• Deterministic Ticket Linking: 7 methods for linking runs to support tickets (Zendesk, Intercom, Salesforce, Freshdesk)
• Calibrated Predictions: Brier scores and ECE for prediction accuracy
• Configurable ROI Engine: Business impact calculation with customizable cost, deflection, and attribution models
• Auto-Instrumentation: Works with LangChain, OpenAI, Anthropic, and more
• Multi-Format Support: Normalizes traces from 25+ observability platforms

Installation

npm install @thinkhive/sdk

Quick Start

Initialization

import { init, runs, shutdown } from '@thinkhive/sdk';

init({
  apiKey: 'thk_your_api_key',
  serviceName: 'my-ai-agent',
  autoInstrument: true,
  frameworks: ['langchain', 'openai'],
});

// Create a run (atomic unit of work)
const run = await runs.create({
  agentId: 'weather-agent',
  conversationMessages: [
    { role: 'user', content: 'What is the weather in San Francisco?' },
    { role: 'assistant', content: 'The weather in San Francisco is currently 65F and sunny.' }
  ],
  outcome: 'success',
});

console.log(`Run ID: ${run.id}`);
await shutdown();

API Reference

Core APIs (V3)

| API | Import | Description | |-----|--------|-------------| | runs | @thinkhive/sdk | Create and manage runs (atomic work units) | | claims | @thinkhive/sdk | Manage facts/inferences for runs | | calibration | @thinkhive/sdk | Track prediction accuracy (Brier scores) | | linking | @thinkhive/sdk | Connect runs to support tickets (7 methods) | | customerContext | @thinkhive/sdk | Time-series customer snapshots | | roiAnalytics | @thinkhive/sdk | Configurable ROI calculation engine | | agents | @thinkhive/sdk | Agent CRUD management | | apiKeys | @thinkhive/sdk | API key management with scoped permissions |

Assessment APIs

| API | Description | |-----|-------------| | humanReview | Human-in-the-loop review queues | | nondeterminism | Multi-sample reliability testing | | evalHealth | Metric health monitoring | | deterministicGraders | Rule-based grading | | conversationEval | Multi-turn conversation grading | | transcriptPatterns | Pattern detection in transcripts |

Operations & Monitoring APIs

| API | Description | |-----|-------------| | evalRuns | Create, manage, and query evaluation runs | | signals | Behavioral signal configuration, stats, and trends | | notifications | Alert rules and notification management | | documents | Agent RAG document upload and management | | shadowTests | Shadow test creation and execution | | sessions | Trace session grouping and querying | | drift | Model/behavior drift detection | | llmCosts | LLM usage cost tracking and optimization |

Legacy APIs (V2)

| API | Description | |-----|-------------| | issues | Clustered failure patterns | | analyzer | User-selected trace analysis with cost estimation | | businessMetrics | Industry-driven metrics with historical tracking | | qualityMetrics | RAG scoring and hallucination detection |

Runs API

import { runs, createRunWithContext } from '@thinkhive/sdk';

// Create a run
const run = await runs.create({
  agentId: 'support-agent',
  conversationMessages: [
    { role: 'user', content: 'Help me cancel my order' },
    { role: 'assistant', content: 'I can help you cancel your order...' }
  ],
  outcome: 'resolved',
  durationMs: 1500,
});

// List runs with filters
const result = await runs.list({
  agentId: 'support-agent',
  outcome: 'resolved',
  startedAfter: '2025-01-01',
  limit: 50,
});

// Get a specific run
const existing = await runs.get('run_abc123');

// Get run statistics
const stats = await runs.stats('support-agent');

Claims API (Facts vs Inferences)

import { claims, isFact, isInference, getHighConfidenceClaims } from '@thinkhive/sdk';

// List claims for a run
const claimList = await claims.list(runId);

// Filter by type
const facts = claimList.filter(isFact);
const inferences = claimList.filter(isInference);

// Get high confidence claims
const confident = getHighConfidenceClaims(claimList, 0.9);

// Group by category
const byCategory = groupClaimsByCategory(claimList);

Calibration API

import { calibration, calculateBrierScore, isWellCalibrated } from '@thinkhive/sdk';

// Get calibration status for a prediction type
const status = await calibration.status('agent-123', 'churn_risk');

// Get all calibration metrics
const metrics = await calibration.allMetrics('agent-123');

// Single prediction
const score = calculateBrierScore(0.8, true); // 0.04

// Batch calculation
const batchScore = calculateBrierScore([
  { predicted: 0.8, actual: 1 },
  { predicted: 0.3, actual: 0 },
]);

if (isWellCalibrated(score)) {
  console.log('Agent predictions are well calibrated');
}

Ticket Linking (V3)

Deterministic linking between runs and support tickets using 7 methods with explicit confidence scores.

import {
  linking,
  generateZendeskMarker,
  linkRunToTicket,
  linkRunToZendeskTicket,
  getBestLinkMethod,
  LINK_METHOD_CONFIDENCE,
} from '@thinkhive/sdk';

// SDK explicit linking (confidence: 1.0)
const link = await linking.create({
  runId: 'run_abc123',
  ticketId: 'ticket_xyz',
  method: 'sdk_explicit',
});

// Zendesk marker linking
const zdLink = await linking.create({
  runId: 'run_abc123',
  externalTicketId: '12345',
  platform: 'zendesk',
  method: 'zendesk_marker',
});

// Convenience functions
await linkRunToTicket('run_abc123', 'ticket_xyz');
await linkRunToZendeskTicket('run_abc123', '12345');

// Get links for a run
const links = await linking.getForRun('run_abc123');

// Get links for a ticket
const ticketLinks = await linking.getForTicket('ticket_xyz');

// Verify a link
await linking.verify('link_id', { verified: true, notes: 'Confirmed by agent' });

// Auto-link using all available evidence
const autoLinked = await linking.autoLink('run_abc123');

// Get link statistics
const stats = await linking.stats();
// { totalLinks, byMethod, avgConfidence, verifiedCount, unverifiedCount }

// Generate a Zendesk marker to embed in responses
const marker = generateZendeskMarker('run_abc123');
// Returns: '[THID:run_abc123]'

// Server-side marker generation with format options
const serverMarker = await linking.generateMarker({
  traceId: 'trace_123',
  format: 'html_comment',
});

// Delete a link
await linking.delete('link_id');

Link Methods and Confidence

| Method | Confidence | Description | |--------|-----------|-------------| | sdk_explicit | 1.0 | Direct SDK call with ticket ID | | zendesk_marker | 1.0 | Embedded THID marker in response | | custom_field | 1.0 | Zendesk custom field | | manual | 1.0 | Human-assigned | | middleware_stamp | 0.98 | Middleware-injected trace ID | | session_match | 0.95 | Session ID correlation | | email_time_window | 0.6 | Email + 15min window |

ROI Analytics (V3)

Configurable ROI calculation engine with customizable cost models, deflection definitions, and attribution rules.

Configuration

import { roiAnalytics } from '@thinkhive/sdk';

// Get active ROI config (auto-provisions default if none exists)
const config = await roiAnalytics.getConfig();

// Create custom ROI config
const newConfig = await roiAnalytics.createConfig({
  costConfig: {
    humanAgentCostPerHour: 35,
    aiCostPer1kTokens: 0.01,
    escalationCostMultiplier: 2.5,
    infrastructureMonthlyCost: 200,
  },
  deflectionConfig: {
    definition: 'resolved_without_human',
    outcomesThatCount: ['resolved'],
    minConfidence: 0.8,
    excludeEscalated: true,
  },
  resolutionConfig: {
    definition: 'customer_confirmed',
    requiresSatisfaction: false,
    minSatisfactionScore: null,
    noRecontactWindowHours: 24,
  },
  attributionConfig: {
    timeSavedValuePerHour: 50,
    churnPreventionAttribution: 0.2,
    upsellAttribution: 0.1,
    useActualARR: true,
    defaultARRIfUnknown: 10000,
  },
});

// Update config (creates new version, keeps history)
const updated = await roiAnalytics.updateConfig({
  costConfig: { humanAgentCostPerHour: 40 },
});

// List config version history
const versions = await roiAnalytics.configVersions({ limit: 10 });

Calculate ROI

// Calculate ROI for a date range
const result = await roiAnalytics.calculateV3({
  startDate: '2025-01-01',
  endDate: '2025-02-01',
  includeBreakdown: true,
  includeConfidenceIntervals: false,
});

console.log(`Net ROI: $${result.summary.netROI}`);
console.log(`ROI %: ${result.summary.roiPercentage}%`);
console.log(`Deflection Rate: ${result.metrics.deflectionRate}`);
console.log(`Total Runs: ${result.metrics.totalRuns}`);

// Filter by agent
const agentROI = await roiAnalytics.calculateV3({
  agentId: 'support-agent',
  startDate: '2025-01-01',
  endDate: '2025-02-01',
});

ROI Trends

// Get ROI trend over time
const trend = await roiAnalytics.trendV3({
  granularity: 'day', // 'day' | 'week' | 'month'
  startDate: '2025-01-01',
  endDate: '2025-02-01',
});

for (const bucket of trend.trend) {
  console.log(`${bucket.periodStart}: ${bucket.runCount} runs, $${bucket.netROI} ROI`);
}

if (trend.comparison) {
  console.log(`Improving: ${trend.comparison.isImproving}`);
  console.log(`Change: ${trend.comparison.changePercent}%`);
}

Legacy V1 ROI (still available)

// V1 summary endpoint
const summary = await roiAnalytics.summary({
  startDate: '2025-01-01',
  endDate: '2025-01-31',
});

// V1 trends
const trends = await roiAnalytics.trends({
  startDate: '2025-01-01',
  endDate: '2025-01-31',
});

// V1 per-trace ROI calculation
const impact = await roiAnalytics.calculate({
  traceId: 'trace_abc123',
});

// V1 industry configs
const industries = await roiAnalytics.industries();

// V1 correlation analysis
const correlations = await roiAnalytics.correlations();

Manual Tracing

import { traceLLM, traceRetrieval, traceTool, traceChain } from '@thinkhive/sdk';

// Trace an LLM call
const response = await traceLLM({
  name: 'generate-response',
  modelName: 'gpt-4',
  provider: 'openai',
  input: { prompt: 'Hello!' }
}, async () => {
  return await openai.chat.completions.create({...});
});

// Trace a retrieval operation
const docs = await traceRetrieval({
  name: 'search-knowledge-base',
  query: 'refund policy',
  topK: 5
}, async () => {
  return await vectorStore.similaritySearch(query, 5);
});

// Trace a tool call
const result = await traceTool({
  name: 'lookup-order',
  toolName: 'order_lookup',
  parameters: { orderId: '12345' }
}, async () => {
  return await lookupOrder('12345');
});

// Trace a chain/workflow
const output = await traceChain({
  name: 'support-workflow',
  input: { query: 'help with refund' }
}, async () => {
  const docs = await retrieveContext(query);
  return await generateResponse(docs);
});

Analyzer API

import { analyzer } from '@thinkhive/sdk';

// Estimate cost before running analysis
const estimate = await analyzer.estimateCost({
  traceIds: ['trace-1', 'trace-2'],
  tier: 'standard',
});
console.log(`Estimated cost: $${estimate.estimatedCost}`);

// Analyze specific traces
const analysis = await analyzer.analyze({
  traceIds: ['trace-1', 'trace-2'],
  tier: 'standard',
  includeRootCause: true,
  includeLayers: true,
});

// Analyze traces by time window with smart sampling
const windowAnalysis = await analyzer.analyzeWindow({
  agentId: 'support-agent',
  startDate: new Date('2025-01-01'),
  endDate: new Date('2025-01-31'),
  filters: { outcomes: ['failure'], minSeverity: 'medium' },
  sampling: { strategy: 'smart', samplePercent: 10 },
});

// Get aggregated insights
const summary = await analyzer.summarize({
  agentId: 'support-agent',
  startDate: new Date('2025-01-01'),
  endDate: new Date('2025-01-31'),
});

Issues API

import { issues } from '@thinkhive/sdk';

const issueList = await issues.list('support-agent', { status: 'open', limit: 10 });
const issue = await issues.get('issue-123');
const fixes = await issues.getFixes('issue-123');

API Keys

import { apiKeys, hasPermission, canAccessAgent } from '@thinkhive/sdk';

const result = await apiKeys.create({
  name: 'CI Pipeline Key',
  permissions: { read: true, write: true, delete: false },
  scopeType: 'agent',
  allowedAgentIds: ['agent-prod-001'],
  environment: 'production',
  expiresAt: new Date(Date.now() + 90 * 24 * 60 * 60 * 1000),
});

if (hasPermission(key, 'write')) { /* can write */ }
if (canAccessAgent(key, 'agent-123')) { /* can access */ }

Business Metrics

import { businessMetrics, isMetricReady, getStatusMessage } from '@thinkhive/sdk';

const metric = await businessMetrics.current('agent-123', 'Deflection Rate');
const history = await businessMetrics.history('agent-123', 'Deflection Rate', {
  startDate: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000),
  granularity: 'daily',
});

await businessMetrics.record('agent-123', {
  metricName: 'CSAT/NPS',
  value: 4.5,
  unit: 'score',
  periodStart: '2025-01-01T00:00:00Z',
  periodEnd: '2025-01-07T23:59:59Z',
  source: 'survey_system',
});

Auto-Instrumentation

import { init } from '@thinkhive/sdk';

init({
  apiKey: 'thk_your_api_key',
  serviceName: 'my-ai-agent',
  autoInstrument: true,
  frameworks: ['langchain', 'openai', 'anthropic'],
});
// All LangChain, OpenAI, and Anthropic calls are now automatically traced

Sub-package Imports

// OpenAI instrumentation
import { instrumentOpenAIClient } from '@thinkhive/sdk/instrumentation/openai';

// LangChain instrumentation
import { ThinkHiveCallbackHandler } from '@thinkhive/sdk/instrumentation/langchain';

// Ticket linking (standalone)
import { linking, generateZendeskMarker } from '@thinkhive/sdk/integrations/ticket-linking';

// Customer context (standalone)
import { customerContext, captureCustomerContext } from '@thinkhive/sdk/integrations/customer-context';

Environment Variables

| Variable | Description | |----------|-------------| | THINKHIVE_API_KEY | Your ThinkHive API key | | THINKHIVE_ENDPOINT | Custom API endpoint (default: https://app.thinkhive.ai) | | THINKHIVE_SERVICE_NAME | Service name for traces (optional) | | THINKHIVE_AGENT_ID | Default agent ID (optional) |

Eval Runs

import { evalRuns } from '@thinkhive/sdk';

// Create an evaluation run
const run = await evalRuns.create('agent-123', { confidenceLevel: 'medium' });

// Get results
const results = await evalRuns.getResults(run.id, { limit: 50 });

// Estimate cost before running
const cost = await evalRuns.estimateCost('agent-123', { confidenceLevel: 'high' });

// List recent runs
const runs = await evalRuns.list({ agentId: 'agent-123', limit: 10 });

Signals

import { signals } from '@thinkhive/sdk';

// List all signals
const allSignals = await signals.list();

// Create a custom behavioral signal
await signals.create('Escalation Request', 'negative', {
  type: 'keywords',
  keywords: ['speak to manager', 'escalate'],
});

// Get signal stats and trends
const stats = await signals.getStats({ agentId: 'agent-123' });
const trends = await signals.getTrends({ granularity: 'daily' });

Notifications

import { notifications } from '@thinkhive/sdk';

// Create an alert rule
await notifications.createRule({
  agentId: 'agent-123',
  name: 'High failure rate alert',
  eventType: 'failure_spike',
  condition: { threshold: 0.3 },
  channel: 'email',
  target: '[email protected]',
});

// List notifications
const alerts = await notifications.list('agent-123', true);
// or: await notifications.listNotifications('agent-123', true);

Documents (RAG)

import { documents } from '@thinkhive/sdk';

// Upload a document
await documents.upload('agent-123', 'faq.txt', 'text/plain', 1024);

// List agent documents
const docs = await documents.list('agent-123');

// Delete a document
await documents.remove('agent-123', 'doc-456');

Shadow Tests

import { shadowTests } from '@thinkhive/sdk';

// Create a shadow test
await shadowTests.create({
  fixId: 'fix-456',
  agentId: 'agent-123',
  testName: 'Refund policy test',
  inputData: { message: 'How do I get a refund?' },
  expectedOutput: 'You can request a refund within 30 days.',
});

// List tests for an agent
const tests = await shadowTests.list('agent-123');

Sessions

import { sessions } from '@thinkhive/sdk';

// List conversation sessions
const allSessions = await sessions.list('agent-123', { limit: 20 });

// Get all traces in a session
const traces = await sessions.getTraces('session-789', 'agent-123');

Drift Detection

import { drift, hasDrift, getDriftSeverity } from '@thinkhive/sdk';

// Detect drift for an agent
const report = await drift.detect('agent-123');
if (hasDrift(report)) {
  console.log(`Drift severity: ${getDriftSeverity(report)}`);
}

// Detect drift across all agents
const allDrift = await drift.detectAll();

LLM Costs

import { llmCosts, formatCost } from '@thinkhive/sdk';

// Get cost summary
const summary = await llmCosts.summary({ period: '30d' });
// or: await llmCosts.getSummary({ period: '30d' });
console.log(`Total cost: ${formatCost(summary.totalCost)}`);

// Get per-agent breakdown
const breakdown = await llmCosts.getBreakdown('agent-123');

// Get optimization savings
const savings = await llmCosts.getSavings();

Error Handling

import { ThinkHiveError, RateLimitError } from '@thinkhive/sdk';

try {
  await runs.create({ /* ... */ });
} catch (error) {
  if (error instanceof RateLimitError) {
    console.log(`Retry after ${error.retryAfter}ms`);
  }
}

// Both forms work:
throw new RateLimitError(60000);
throw new RateLimitError('Custom message', 60000);

Upgrading

v4.2.0 → v4.2.2

Improvements in v4.2.2:

• RateLimitError now accepts both new RateLimitError(retryAfterMs) and new RateLimitError('message', retryAfterMs) signatures
• Claims helpers (isFact, isInference, isComputed) now accept both {claimType: 'observed'} (API format) and {type: 'fact'} (shorthand)
• groupClaimsByType returns both API names (observed/inferred/computed) and shorthand names (fact/inference)
• calculateBrierScore now accepts both single prediction (probability, outcome) and batch ([{predicted, actual}]) forms
• calculateECE now accepts pre-computed bucket objects [{averageConfidence, accuracy, count}] in addition to raw predictions
• isWellCalibrated accepts both a raw number and {brierScore: number} object
• calculatePassAtK now supports both (passRate, k) and (n, c, k) calling conventions
• isReliableEvaluation accepts both a CriterionAnalysis object and a raw reliability score
• Rule helpers (createRegexRule, createContainsRule, createLengthRule, createJsonSchemaRule) now return named rule objects with {type, name, config} structure
• allRulesPassed and getFailedRules accept any {passed: boolean} objects, not just RuleResult[]
• aggregateWorst / aggregateAverage now accept simple number[] arrays in addition to TurnEvaluation[]
• Method aliases added for cross-SDK consistency:
  • signals.delete() (alias for remove())
  • notifications.list() (alias for listNotifications())
  • llmCosts.summary() (alias for getSummary())
  • qualityMetrics.evaluate() (alias for evaluateRag())
  • guardrails.evaluate() (alias for scan())
  • customerContext.capture() (alias for captureSnapshot())

v4.1.0 → v4.2.0

New in v4.2.0:

• evalRuns — create, list, and manage evaluation runs programmatically
• signals — behavioral signal CRUD with stats and trends
• notifications — alert rule configuration and notification management
• documents — agent RAG document upload/list/delete
• shadowTests — shadow test creation and management
• sessions — trace session grouping and querying
• drift — model/behavior drift detection with helpers
• llmCosts — LLM usage cost tracking and optimization savings

v4.0 → v4.1.0

• roiAnalytics now includes V3 methods: getConfig(), createConfig(), updateConfig(), configVersions(), calculateV3(), trendV3()
• linking now includes: autoLink(), stats(), generateMarker()
• GET /roi/config auto-provisions a default configuration if none exists
• ROI date range limit increased to 92 days

License

MIT License - see LICENSE for details.