@countly/ai-sdk-anthropic

Countly AI observability adapter for the Anthropic TypeScript SDK.

Part of the Countly AI SDK — provider-agnostic LLM observability for every AI stack.

Install

npm install @countly/ai-sdk-anthropic

@countly/ai-sdk-core is pulled in automatically.

Peer dependency

@anthropic-ai/sdk >= 0.30.0

Quick Start

import Anthropic from "@anthropic-ai/sdk";
import { observeAnthropic } from "@countly/ai-sdk-anthropic";
import { AsyncLocalStorage } from "node:async_hooks";

const userStore = new AsyncLocalStorage<{ userId: string }>();

app.use((req, res, next) => {
  userStore.run({ userId: req.user.id }, next);
});

const anthropic = observeAnthropic(new Anthropic(), {
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getDeviceId: () => userStore.getStore()?.userId,
  observabilityLevel: 1,
  tags: ["chatbot", "customer-support"],
  environment: "production",
});

const message = await anthropic.messages.create({
  model: "claude-sonnet-4-20250514",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Hello" }],
});

Streaming

const stream = anthropic.messages.stream({
  model: "claude-sonnet-4-20250514",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Hello" }],
});

const finalMessage = await stream.finalMessage();
// Events reported automatically after stream completes

What's captured

• Token usage (input, output, cache read, cache write)
• Cost (computed from model pricing)
• Latency (total + TTFT for streaming)
• Tool use blocks (name, arguments)
• Error tracking with categorization
• APM traces, per-user aggregation

Configuration

| Field | Default | Description | |-------|---------|-------------| | appKey | required | Countly app key | | url | required | Countly server URL | | getDeviceId | — | Per-request user ID resolver (called at event enqueue time) | | deviceId | — | Static device ID (fallback) | | observabilityLevel | 0 | 0 = metrics only, 1 = + tool calls, 2 = + text previews | | tags | [] | Labels for cost attribution | | environment | "production" | Environment tag | | costModel | — | Custom pricing overrides | | getPromptId | — | Caller-supplied prompt_id resolver (called per interaction; falls back to an auto-generated id when it returns undefined) |

Caller-supplied prompt_id

By default the adapter generates a unique prompt_id for every tracked interaction. If you already mint your own request/trace id (e.g. per HTTP request or per chat turn), supply it via getPromptId so the interaction is stamped with your id instead — on both the streaming and non-streaming paths. This lets you correlate the [CLY]_llm_interaction event with your own logs and, in turn, with feedback recorded under the same id.

import { AsyncLocalStorage } from "node:async_hooks";

const requestStore = new AsyncLocalStorage<{ promptId: string }>();

const anthropic = observeAnthropic(new Anthropic(), {
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getPromptId: () => requestStore.getStore()?.promptId, // undefined → auto-generated fallback
});

The resolved id is what you record feedback against (see below) — pass the same value as prompt_id to feedback.track().

Feedback

User feedback (thumbs up/down, ratings, comments) is not auto-collected — wire it from your UI. Capture the prompt_id of each tracked interaction via the onPrompt callback, then record feedback against it with createFeedbackTracker (re-exported from this package, so no extra install is needed):

import Anthropic from "@anthropic-ai/sdk";
import { observeAnthropic, createFeedbackTracker, type PromptInfo } from "@countly/ai-sdk-anthropic";

const countly = { appKey: "YOUR_APP_KEY", url: "https://your-countly-server.com" };

let lastPrompt: PromptInfo | undefined;
const anthropic = observeAnthropic(new Anthropic(), {
  ...countly,
  onPrompt: (info) => { lastPrompt = info; }, // fires after every tracked call
});

const feedback = createFeedbackTracker(countly, { sdk_adapter: "anthropic" });

const message = await anthropic.messages.create({
  model: "claude-sonnet-4-5",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Explain quantum computing" }],
});

// ...later, when the user rates the answer:
feedback.track({
  prompt_id: lastPrompt!.prompt_id,
  rating: "thumbs_up", // or "thumbs_down", or any custom string
  score: 0.9, // optional 0-1 numeric score
  category: "helpful", // optional: hallucination, irrelevant, harmful, ...
  comment: "Great answer", // optional free-form text
  deviceId: user.id, // attribute to the same user as the interaction
});

Each track() call emits a [CLY]_llm_interaction_feedback event whose prompt_id links back to the [CLY]_llm_interaction event — powering prompt → feedback funnels and per-model satisfaction breakdowns in Countly. In a real app, store prompt_id alongside the rendered message (or return it to your client) and read it back when the user rates the answer. Feedback is batched like interaction events; call feedback.flush() to send immediately, or feedback.shutdown() on process exit.

Full documentation

See the Countly AI SDK repository for the unified data model, observability levels, cost calculation, privacy controls, and Countly plugin integration (Drill, Funnels, Cohorts, APM, Crash Analytics).

License

MIT