@thinkhive/sdk

v4.1.0

Published

4 days ago

ThinkHive SDK v4.0 - AI agent observability with business metrics, ROI analytics, and 25+ trace format support

ThinkHive SDK v4.1.0

The official JavaScript/TypeScript SDK for ThinkHive - AI Agent Observability Platform.

Features

OpenTelemetry-Based Tracing: Built on OTLP for seamless integration with existing observability tools
Run-Centric Architecture: Atomic unit of work tracking with claims, calibration, and linking
Facts vs Inferences: Claims API for separating verified facts from inferences
Deterministic Ticket Linking: 7 methods for linking runs to support tickets (Zendesk, Intercom, Salesforce, Freshdesk)
Calibrated Predictions: Brier scores and ECE for prediction accuracy
Configurable ROI Engine: Business impact calculation with customizable cost, deflection, and attribution models
Auto-Instrumentation: Works with LangChain, OpenAI, Anthropic, and more
Multi-Format Support: Normalizes traces from 25+ observability platforms

Installation

npm install @thinkhive/sdk

Quick Start

Initialization

import { init, runs, shutdown } from '@thinkhive/sdk';

init({
  apiKey: 'thk_your_api_key',
  serviceName: 'my-ai-agent',
  autoInstrument: true,
  frameworks: ['langchain', 'openai'],
});

// Create a run (atomic unit of work)
const run = await runs.create({
  agentId: 'weather-agent',
  conversationMessages: [
    { role: 'user', content: 'What is the weather in San Francisco?' },
    { role: 'assistant', content: 'The weather in San Francisco is currently 65F and sunny.' }
  ],
  outcome: 'success',
});

console.log(`Run ID: ${run.id}`);
await shutdown();

API Reference

Core APIs (V3)

| API | Import | Description | |-----|--------|-------------| | runs | @thinkhive/sdk | Create and manage runs (atomic work units) | | claims | @thinkhive/sdk | Manage facts/inferences for runs | | calibration | @thinkhive/sdk | Track prediction accuracy (Brier scores) | | linking | @thinkhive/sdk | Connect runs to support tickets (7 methods) | | customerContext | @thinkhive/sdk | Time-series customer snapshots | | roiAnalytics | @thinkhive/sdk | Configurable ROI calculation engine | | agents | @thinkhive/sdk | Agent CRUD management | | apiKeys | @thinkhive/sdk | API key management with scoped permissions |

Assessment APIs

| API | Description | |-----|-------------| | humanReview | Human-in-the-loop review queues | | nondeterminism | Multi-sample reliability testing | | evalHealth | Metric health monitoring | | deterministicGraders | Rule-based grading | | conversationEval | Multi-turn conversation grading | | transcriptPatterns | Pattern detection in transcripts |

Legacy APIs (V2)

| API | Description | |-----|-------------| | issues | Clustered failure patterns | | analyzer | User-selected trace analysis with cost estimation | | businessMetrics | Industry-driven metrics with historical tracking | | qualityMetrics | RAG scoring and hallucination detection |

Runs API

import { runs, createRunWithContext } from '@thinkhive/sdk';

// Create a run
const run = await runs.create({
  agentId: 'support-agent',
  conversationMessages: [
    { role: 'user', content: 'Help me cancel my order' },
    { role: 'assistant', content: 'I can help you cancel your order...' }
  ],
  outcome: 'resolved',
  durationMs: 1500,
});

// List runs with filters
const result = await runs.list({
  agentId: 'support-agent',
  outcome: 'resolved',
  startedAfter: '2025-01-01',
  limit: 50,
});

// Get a specific run
const existing = await runs.get('run_abc123');

// Get run statistics
const stats = await runs.stats('support-agent');

Claims API (Facts vs Inferences)

import { claims, isFact, isInference, getHighConfidenceClaims } from '@thinkhive/sdk';

// List claims for a run
const claimList = await claims.list(runId);

// Filter by type
const facts = claimList.filter(isFact);
const inferences = claimList.filter(isInference);

// Get high confidence claims
const confident = getHighConfidenceClaims(claimList, 0.9);

// Group by category
const byCategory = groupClaimsByCategory(claimList);

Calibration API

import { calibration, calculateBrierScore, isWellCalibrated } from '@thinkhive/sdk';

// Get calibration status for a prediction type
const status = await calibration.status('agent-123', 'churn_risk');

// Get all calibration metrics
const metrics = await calibration.allMetrics('agent-123');

// Calculate Brier score locally
const score = calculateBrierScore([
  { predicted: 0.8, actual: 1 },
  { predicted: 0.3, actual: 0 },
]);

if (isWellCalibrated(score)) {
  console.log('Agent predictions are well calibrated');
}

Ticket Linking (V3)

Deterministic linking between runs and support tickets using 7 methods with explicit confidence scores.

import {
  linking,
  generateZendeskMarker,
  linkRunToTicket,
  linkRunToZendeskTicket,
  getBestLinkMethod,
  LINK_METHOD_CONFIDENCE,
} from '@thinkhive/sdk';

// SDK explicit linking (confidence: 1.0)
const link = await linking.create({
  runId: 'run_abc123',
  ticketId: 'ticket_xyz',
  method: 'sdk_explicit',
});

// Zendesk marker linking
const zdLink = await linking.create({
  runId: 'run_abc123',
  externalTicketId: '12345',
  platform: 'zendesk',
  method: 'zendesk_marker',
});

// Convenience functions
await linkRunToTicket('run_abc123', 'ticket_xyz');
await linkRunToZendeskTicket('run_abc123', '12345');

// Get links for a run
const links = await linking.getForRun('run_abc123');

// Get links for a ticket
const ticketLinks = await linking.getForTicket('ticket_xyz');

// Verify a link
await linking.verify('link_id', { verified: true, notes: 'Confirmed by agent' });

// Auto-link using all available evidence
const autoLinked = await linking.autoLink('run_abc123');

// Get link statistics
const stats = await linking.stats();
// { totalLinks, byMethod, avgConfidence, verifiedCount, unverifiedCount }

// Generate a Zendesk marker to embed in responses
const marker = generateZendeskMarker('run_abc123');
// Returns: '[THID:run_abc123]'

// Server-side marker generation with format options
const serverMarker = await linking.generateMarker({
  traceId: 'trace_123',
  format: 'html_comment',
});

// Delete a link
await linking.delete('link_id');

Link Methods and Confidence

| Method | Confidence | Description | |--------|-----------|-------------| | sdk_explicit | 1.0 | Direct SDK call with ticket ID | | zendesk_marker | 1.0 | Embedded THID marker in response | | custom_field | 1.0 | Zendesk custom field | | manual | 1.0 | Human-assigned | | middleware_stamp | 0.98 | Middleware-injected trace ID | | session_match | 0.95 | Session ID correlation | | email_time_window | 0.6 | Email + 15min window |

ROI Analytics (V3)

Configurable ROI calculation engine with customizable cost models, deflection definitions, and attribution rules.

Configuration

import { roiAnalytics } from '@thinkhive/sdk';

// Get active ROI config (auto-provisions default if none exists)
const config = await roiAnalytics.getConfig();

// Create custom ROI config
const newConfig = await roiAnalytics.createConfig({
  costConfig: {
    humanAgentCostPerHour: 35,
    aiCostPer1kTokens: 0.01,
    escalationCostMultiplier: 2.5,
    infrastructureMonthlyCost: 200,
  },
  deflectionConfig: {
    definition: 'resolved_without_human',
    outcomesThatCount: ['resolved'],
    minConfidence: 0.8,
    excludeEscalated: true,
  },
  resolutionConfig: {
    definition: 'customer_confirmed',
    requiresSatisfaction: false,
    minSatisfactionScore: null,
    noRecontactWindowHours: 24,
  },
  attributionConfig: {
    timeSavedValuePerHour: 50,
    churnPreventionAttribution: 0.2,
    upsellAttribution: 0.1,
    useActualARR: true,
    defaultARRIfUnknown: 10000,
  },
});

// Update config (creates new version, keeps history)
const updated = await roiAnalytics.updateConfig({
  costConfig: { humanAgentCostPerHour: 40 },
});

// List config version history
const versions = await roiAnalytics.configVersions({ limit: 10 });

Calculate ROI

// Calculate ROI for a date range
const result = await roiAnalytics.calculateV3({
  startDate: '2025-01-01',
  endDate: '2025-02-01',
  includeBreakdown: true,
  includeConfidenceIntervals: false,
});

console.log(`Net ROI: $${result.summary.netROI}`);
console.log(`ROI %: ${result.summary.roiPercentage}%`);
console.log(`Deflection Rate: ${result.metrics.deflectionRate}`);
console.log(`Total Runs: ${result.metrics.totalRuns}`);

// Filter by agent
const agentROI = await roiAnalytics.calculateV3({
  agentId: 'support-agent',
  startDate: '2025-01-01',
  endDate: '2025-02-01',
});

ROI Trends

// Get ROI trend over time
const trend = await roiAnalytics.trendV3({
  granularity: 'day', // 'day' | 'week' | 'month'
  startDate: '2025-01-01',
  endDate: '2025-02-01',
});

for (const bucket of trend.trend) {
  console.log(`${bucket.periodStart}: ${bucket.runCount} runs, $${bucket.netROI} ROI`);
}

if (trend.comparison) {
  console.log(`Improving: ${trend.comparison.isImproving}`);
  console.log(`Change: ${trend.comparison.changePercent}%`);
}

Legacy V1 ROI (still available)

// V1 summary endpoint
const summary = await roiAnalytics.summary({
  startDate: '2025-01-01',
  endDate: '2025-01-31',
});

// V1 trends
const trends = await roiAnalytics.trends({
  startDate: '2025-01-01',
  endDate: '2025-01-31',
});

// V1 per-trace ROI calculation
const impact = await roiAnalytics.calculate({
  traceId: 'trace_abc123',
});

// V1 industry configs
const industries = await roiAnalytics.industries();

// V1 correlation analysis
const correlations = await roiAnalytics.correlations();

Manual Tracing

import { traceLLM, traceRetrieval, traceTool, traceChain } from '@thinkhive/sdk';

// Trace an LLM call
const response = await traceLLM({
  name: 'generate-response',
  modelName: 'gpt-4',
  provider: 'openai',
  input: { prompt: 'Hello!' }
}, async () => {
  return await openai.chat.completions.create({...});
});

// Trace a retrieval operation
const docs = await traceRetrieval({
  name: 'search-knowledge-base',
  query: 'refund policy',
  topK: 5
}, async () => {
  return await vectorStore.similaritySearch(query, 5);
});

// Trace a tool call
const result = await traceTool({
  name: 'lookup-order',
  toolName: 'order_lookup',
  parameters: { orderId: '12345' }
}, async () => {
  return await lookupOrder('12345');
});

// Trace a chain/workflow
const output = await traceChain({
  name: 'support-workflow',
  input: { query: 'help with refund' }
}, async () => {
  const docs = await retrieveContext(query);
  return await generateResponse(docs);
});

Analyzer API

import { analyzer } from '@thinkhive/sdk';

// Estimate cost before running analysis
const estimate = await analyzer.estimateCost({
  traceIds: ['trace-1', 'trace-2'],
  tier: 'standard',
});
console.log(`Estimated cost: $${estimate.estimatedCost}`);

// Analyze specific traces
const analysis = await analyzer.analyze({
  traceIds: ['trace-1', 'trace-2'],
  tier: 'standard',
  includeRootCause: true,
  includeLayers: true,
});

// Analyze traces by time window with smart sampling
const windowAnalysis = await analyzer.analyzeWindow({
  agentId: 'support-agent',
  startDate: new Date('2025-01-01'),
  endDate: new Date('2025-01-31'),
  filters: { outcomes: ['failure'], minSeverity: 'medium' },
  sampling: { strategy: 'smart', samplePercent: 10 },
});

// Get aggregated insights
const summary = await analyzer.summarize({
  agentId: 'support-agent',
  startDate: new Date('2025-01-01'),
  endDate: new Date('2025-01-31'),
});

Issues API

import { issues } from '@thinkhive/sdk';

const issueList = await issues.list('support-agent', { status: 'open', limit: 10 });
const issue = await issues.get('issue-123');
const fixes = await issues.getFixes('issue-123');

API Keys

import { apiKeys, hasPermission, canAccessAgent } from '@thinkhive/sdk';

const result = await apiKeys.create({
  name: 'CI Pipeline Key',
  permissions: { read: true, write: true, delete: false },
  scopeType: 'agent',
  allowedAgentIds: ['agent-prod-001'],
  environment: 'production',
  expiresAt: new Date(Date.now() + 90 * 24 * 60 * 60 * 1000),
});

if (hasPermission(key, 'write')) { /* can write */ }
if (canAccessAgent(key, 'agent-123')) { /* can access */ }

Business Metrics

import { businessMetrics, isMetricReady, getStatusMessage } from '@thinkhive/sdk';

const metric = await businessMetrics.current('agent-123', 'Deflection Rate');
const history = await businessMetrics.history('agent-123', 'Deflection Rate', {
  startDate: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000),
  granularity: 'daily',
});

await businessMetrics.record('agent-123', {
  metricName: 'CSAT/NPS',
  value: 4.5,
  unit: 'score',
  periodStart: '2025-01-01T00:00:00Z',
  periodEnd: '2025-01-07T23:59:59Z',
  source: 'survey_system',
});

Auto-Instrumentation

import { init } from '@thinkhive/sdk';

init({
  apiKey: 'thk_your_api_key',
  serviceName: 'my-ai-agent',
  autoInstrument: true,
  frameworks: ['langchain', 'openai', 'anthropic'],
});
// All LangChain, OpenAI, and Anthropic calls are now automatically traced

Sub-package Imports

// OpenAI instrumentation
import { instrumentOpenAIClient } from '@thinkhive/sdk/instrumentation/openai';

// LangChain instrumentation
import { ThinkHiveCallbackHandler } from '@thinkhive/sdk/instrumentation/langchain';

// Ticket linking (standalone)
import { linking, generateZendeskMarker } from '@thinkhive/sdk/integrations/ticket-linking';

// Customer context (standalone)
import { customerContext, captureCustomerContext } from '@thinkhive/sdk/integrations/customer-context';

Environment Variables

| Variable | Description | |----------|-------------| | THINKHIVE_API_KEY | Your ThinkHive API key | | THINKHIVE_ENDPOINT | Custom API endpoint (default: https://app.thinkhive.ai) | | THINKHIVE_SERVICE_NAME | Service name for traces (optional) | | THINKHIVE_AGENT_ID | Default agent ID (optional) |

Upgrading from v4.0

roiAnalytics now includes V3 methods: getConfig(), createConfig(), updateConfig(), configVersions(), calculateV3(), trendV3()
linking now includes: autoLink(), stats(), generateMarker()
GET /roi/config auto-provisions a default configuration if none exists
ROI date range limit increased to 92 days

License

MIT License - see LICENSE for details.