@countly/ai-sdk-vercel

Countly AI observability adapter for the Vercel AI SDK.

Part of the Countly AI SDK — provider-agnostic LLM observability for every AI stack.

Install

npm install @countly/ai-sdk-vercel

@countly/ai-sdk-core is pulled in automatically.

Peer dependency

ai >= 4.0.0

Quick Start

import { countlyTelemetry } from "@countly/ai-sdk-vercel";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { AsyncLocalStorage } from "node:async_hooks";

const userStore = new AsyncLocalStorage<{ userId: string }>();

app.use((req, res, next) => {
  userStore.run({ userId: req.user.id }, next);
});

const telemetry = countlyTelemetry({
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getDeviceId: () => userStore.getStore()?.userId,
  apiBaseURL: "https://api.openai.com/v1", // optional, see "API endpoint" below
});

const { text } = await generateText({
  model: openai("gpt-4o"),
  prompt: "Explain quantum computing",
  experimental_telemetry: { integrations: [telemetry] },
});

One row per generation

A [CLY]_llm_interaction event is one LLM generation. A multi-step tool-calling generateText is N generations, so it emits N rows: each carries its own model, token usage, finish reason, tool calls and latency, plus the shared turn context — run_id (identical on every row of the call), generation_index (0-based) and run_latency_total (the call end to end). totalUsage is deliberately not used: summing N turns into one row is what made cost-per-prompt and latency percentiles wrong.

embed(), embedMany() and rerank() are model calls but not generations, so they emit nothing at all rather than a row with no model and no tokens.

What's captured

• Token usage: usage_input / usage_output are the provider's totals — input includes both cache buckets, output includes reasoning — with usage_cache_read, usage_cache_write and usage_reasoning as subsets, so the cache premium is billed exactly once
• Model, provider, finish reason, status
• Latency: per generation (latency_total) and per call (run_latency_total)
• Tool calls, with the real outcome (success/error) and duration from onToolCallFinish; a tool whose outcome was never observed is recorded as unknown, never as a success
• Request settings (temperature, top_p, max_tokens, penalties)
• Reasoning text and prompt/answer previews (observability level 2 only)
• Error tracking

Nothing is fabricated: a value the AI SDK does not report is omitted, and the row's usage_state / cost_priced markers say why.

API endpoint (api_host_type)

The AI SDK's telemetry events do not carry the provider client's baseURL, so pass it as apiBaseURL if you want the endpoint dimension populated. It is used only to classify api_host_type (vendor_direct, azure_openai, gateway, local_infra) and to emit the bare hostname as api_host — path, query and credentials are never emitted. Without it, api_host_type is unknown.

Latency and concurrency

Latency is measured from the SDK's own lifecycle callbacks. Per-generation latency is exact even with many calls in flight. The call duration (run_latency_total, and the first generation's own latency) needs the onStart → onFinish pair, and AI SDK v6 integration events carry no call id — so when two calls are in flight simultaneously the pairing is unknowable and the latency is omitted rather than guessed.

Flush before shutdown

await telemetry.flush();    // send buffered events
await telemetry.shutdown(); // flush + stop the transport

Caller-supplied turn id

By default every tracked call gets an auto-generated id. If you already mint your own request/message id (and want feedback to correlate against it without round-tripping through onPrompt), provide getPromptId. It is called once per tracked call — every generation of that call shares the returned value as its run_id — so return a string to use it, or undefined to fall back to the generated id. Pair it with a per-request store so each call reads its own id:

import { AsyncLocalStorage } from "node:async_hooks";

const promptStore = new AsyncLocalStorage<{ promptId: string }>();

const telemetry = countlyTelemetry({
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getPromptId: () => promptStore.getStore()?.promptId, // undefined → generated fallback
});

// wrap each request so the interaction stamps your id:
promptStore.run({ promptId: myMessageId }, async () => {
  await generateText({
    model: openai("gpt-4o"),
    prompt: "Explain quantum computing",
    experimental_telemetry: { integrations: [telemetry] },
  });
});

The returned id becomes the turn's run_id, carried on every row of every event family for that call, so you can record feedback against your own id directly — feedback.track({ prompt_id: myMessageId, ... }) — no onPrompt capture needed. That rates the whole turn; to rate one generation of a multi-step call, pass that generation's event_id (from onPrompt) plus run_id. When getPromptId is absent (or returns undefined), behavior is unchanged.

Feedback

User feedback (thumbs up/down, ratings, comments) is not auto-collected — wire it from your UI. Capture the prompt_id of each tracked interaction via the onPrompt callback, then record feedback against it with createFeedbackTracker (re-exported from this package, so no extra install is needed):

import { countlyTelemetry, createFeedbackTracker, type PromptInfo } from "@countly/ai-sdk-vercel";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const countly = { appKey: "YOUR_APP_KEY", url: "https://your-countly-server.com" };

let lastPrompt: PromptInfo | undefined;
const telemetry = countlyTelemetry({
  ...countly,
  onPrompt: (info) => { lastPrompt = info; }, // fires once per generation
});

const feedback = createFeedbackTracker(countly, { sdk_adapter: "ai-sdk" });

const { text } = await generateText({
  model: openai("gpt-4o"),
  prompt: "Explain quantum computing",
  experimental_telemetry: { integrations: [telemetry] },
});

// ...later, when the user rates the answer:
feedback.track({
  prompt_id: lastPrompt!.prompt_id,
  rating: "thumbs_up", // or "thumbs_down", or any custom string
  score: 0.9, // optional 0-1 numeric score
  category: "helpful", // optional: hallucination, irrelevant, harmful, ...
  comment: "Great answer", // optional free-form text
  deviceId: user.id, // attribute to the same user as the interaction
});

Each track() call emits a [CLY]_llm_interaction_feedback event whose prompt_id links back to the rated turn (or, when you pass a generation's event_id, to that one [CLY]_llm_interaction row — parent_event_key records which) — powering prompt → feedback funnels and per-model satisfaction breakdowns in Countly. In a real app, store prompt_id alongside the rendered message (or return it to your client) and read it back when the user rates the answer. Feedback is batched like interaction events; call feedback.flush() to send immediately, or feedback.shutdown() on process exit.

Full documentation

See the Countly AI SDK repository for the schema v2 wire contract (one row per generation, RULE A dimensions, RULE B measures with their usage_state / cost_priced markers, and the common envelope), the adapter capability matrix, observability levels (0/1/2), cost calculation, privacy controls, and Countly plugin integration (Drill, Funnels, Cohorts, APM, Crash Analytics).

License

MIT