@countly/ai-sdk-anthropic

Countly AI observability adapter for the Anthropic TypeScript SDK.

Part of the Countly AI SDK — provider-agnostic LLM observability for every AI stack.

Install

npm install @countly/ai-sdk-anthropic

@countly/ai-sdk-core is pulled in automatically.

Peer dependency

@anthropic-ai/sdk >= 0.30.0

Tested against @anthropic-ai/sdk 0.88.x. The declared floor is not yet exercised in CI, so treat versions below 0.30 as unsupported and older 0.x versions as untested rather than verified.

Quick Start

import Anthropic from "@anthropic-ai/sdk";
import { observeAnthropic } from "@countly/ai-sdk-anthropic";
import { AsyncLocalStorage } from "node:async_hooks";

const userStore = new AsyncLocalStorage<{ userId: string }>();

app.use((req, res, next) => {
  userStore.run({ userId: req.user.id }, next);
});

const anthropic = observeAnthropic(new Anthropic(), {
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getDeviceId: () => userStore.getStore()?.userId,
  observabilityLevel: 1,
  tags: ["chatbot", "customer-support"],
  environment: "production",
});

const message = await anthropic.messages.create({
  model: "claude-sonnet-4-20250514",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Hello" }],
});

Streaming

Both streaming surfaces are instrumented, and reporting no longer depends on how you consume the stream:

// messages.stream() — reported however you consume it: finalMessage(),
// finalText(), done(), plain `for await`, or an .on("finalMessage") handler.
const stream = anthropic.messages.stream({
  model: "claude-sonnet-4-20250514",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Hello" }],
});
const finalMessage = await stream.finalMessage();

// messages.create({stream: true}) — the row is accumulated from the raw event
// stream (message_start usage, deltas, message_delta stop_reason/output tokens).
// controller, tee() and toReadableStream() are passed through untouched.
const raw = await anthropic.messages.create({ ...params, stream: true });
for await (const event of raw) { /* … */ }

A stream you abandon early (break) is still recorded, as status: "incomplete" with no usage — output tokens are only reported in message_delta, and pricing the input alone would present a partial cost as a complete one.

What's captured

• Token usage. usage_input is the provider's total input count, inclusive of both cache_read_input_tokens and cache_creation_input_tokens, so it reconciles with the Anthropic console; the fresh residual is reported separately as usage_input_fresh.
• Cost, computed from model pricing. When a model is not in the pricing table the cost keys are omitted and cost_priced: "unpriced_model" says why — never $0.
• Latency (total + TTFT for streaming, measured from the first content event).
• Tool calls: client tool_use, mcp_tool_use, and the built-in server_tool_use tools (web search / fetch → retrieval, the code-execution family → code_interpreter), each with its call_id.
• api_host_type / api_host — which class of endpoint served the call (vendor_direct, azure_openai, gateway, local_infra). The hostname only: path, query and credentials are never emitted.
• Error tracking with categorization; APM traces; per-user aggregation.

Not captured

• Client tool outcomes. A tool_use block is the model requesting a tool; its result travels in your next request, so the tool row carries status: "unknown" rather than a fabricated success. Server-side and MCP tools do return their result in the same message, so those rows carry the real outcome.
• client.beta.messages, client.messages.batches and client.completions. Generations made through them produce no rows. Set debug: true and the SDK warns once when your code touches one of these.
• Thinking/reasoning text is not emitted as a preview field.

Configuration

| Field | Default | Description | |-------|---------|-------------| | appKey | required | Countly app key | | url | required | Countly server URL | | getDeviceId | — | Per-request user ID resolver (called at event enqueue time) | | deviceId | — | Static device ID (fallback) | | observabilityLevel | 0 | 0 = metrics only, 1 = + tool calls, 2 = + text previews | | tags | [] | Labels for cost attribution | | environment | "production" | Environment tag | | costModel | — | Custom pricing overrides | | getPromptId | — | Caller-supplied turn id resolver (called per interaction; falls back to an auto-generated id when it returns undefined). The value becomes the row's run_id |

Caller-supplied turn id

By default the adapter generates a unique id for every tracked interaction. If you already mint your own request/trace id (e.g. per HTTP request or per chat turn), supply it via getPromptId so the row is stamped with your id instead — on both the streaming and non-streaming paths. This lets you correlate the [CLY]_llm_interaction event with your own logs and, in turn, with feedback recorded under the same id.

Your id lands on the wire as run_id, the turn identity shared by every row the turn emits. Each row also carries its own event_id (the primary key), and a prompt_id foreign key pointing at the row it belongs to — for an interaction that is its own event_id, for a tool row it is the generation that requested the tool.

import { AsyncLocalStorage } from "node:async_hooks";

const requestStore = new AsyncLocalStorage<{ promptId: string }>();

const anthropic = observeAnthropic(new Anthropic(), {
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getPromptId: () => requestStore.getStore()?.promptId, // undefined → auto-generated fallback
});

The resolved id is what you record feedback against (see below) — pass the same value as prompt_id to feedback.track().

Feedback

User feedback (thumbs up/down, ratings, comments) is not auto-collected — wire it from your UI. Capture each tracked interaction's identity via the onPrompt callback, then record feedback against it with createFeedbackTracker (re-exported from this package, so no extra install is needed):

import Anthropic from "@anthropic-ai/sdk";
import { observeAnthropic, createFeedbackTracker, type PromptInfo } from "@countly/ai-sdk-anthropic";

const countly = { appKey: "YOUR_APP_KEY", url: "https://your-countly-server.com" };

let lastPrompt: PromptInfo | undefined;
const anthropic = observeAnthropic(new Anthropic(), {
  ...countly,
  onPrompt: (info) => { lastPrompt = info; }, // fires after every tracked call
});

const feedback = createFeedbackTracker(countly, { sdk_adapter: "anthropic" });

const message = await anthropic.messages.create({
  model: "claude-sonnet-4-5",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Explain quantum computing" }],
});

// ...later, when the user rates the answer:
feedback.track({
  prompt_id: lastPrompt!.prompt_id,
  rating: "thumbs_up", // or "thumbs_down", or any custom string
  score: 0.9, // optional 0-1 numeric score
  category: "helpful", // optional: hallucination, irrelevant, harmful, ...
  comment: "Great answer", // optional free-form text
  deviceId: user.id, // attribute to the same user as the interaction
});

info.prompt_id is the turn (identical to info.run_id), so the rating above applies to the whole turn. To rate one generation instead, pass that generation's row id:

feedback.track({ prompt_id: lastPrompt!.event_id, run_id: lastPrompt!.run_id, rating: "thumbs_down" });

Each track() call emits a [CLY]_llm_interaction_feedback event that joins back to the interaction via run_id, with parent_event_key recording whether the rating is attached to the run or to a single generation — powering prompt → feedback funnels and per-model satisfaction breakdowns in Countly. In a real app, store the id alongside the rendered message (or return it to your client) and read it back when the user rates the answer. Feedback is batched like interaction events; call feedback.flush() to send immediately, or feedback.shutdown() on process exit.

Full documentation

See the Countly AI SDK repository for the schema v2 wire contract (one row per generation, RULE A dimensions, RULE B measures with their usage_state / cost_priced markers, and the common envelope), the adapter capability matrix, observability levels (0/1/2), cost calculation, privacy controls, and Countly plugin integration (Drill, Funnels, Cohorts, APM, Crash Analytics).

License

MIT