@countly/ai-sdk-vercel

Countly AI observability adapter for the Vercel AI SDK.

Part of the Countly AI SDK — provider-agnostic LLM observability for every AI stack.

Install

npm install @countly/ai-sdk-vercel

@countly/ai-sdk-core is pulled in automatically.

Peer dependency

ai >= 4.0.0

Quick Start

import { countlyTelemetry } from "@countly/ai-sdk-vercel";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { AsyncLocalStorage } from "node:async_hooks";

const userStore = new AsyncLocalStorage<{ userId: string }>();

app.use((req, res, next) => {
  userStore.run({ userId: req.user.id }, next);
});

const telemetry = countlyTelemetry({
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getDeviceId: () => userStore.getStore()?.userId,
});

const { text } = await generateText({
  model: openai("gpt-4o"),
  prompt: "Explain quantum computing",
  experimental_telemetry: { integrations: [telemetry] },
});

What's captured

• Token usage (input, output, reasoning, cached) — uses totalUsage to aggregate multi-step calls when available
• Model, provider, finish reason
• Latency (total)
• Tool calls and results
• Reasoning text (when the model returns it)
• Error tracking

Flush before shutdown

await telemetry.flush();    // send buffered events
await telemetry.shutdown(); // flush + stop the transport

Caller-supplied prompt_id

By default every tracked call gets an auto-generated prompt_id. If you already mint your own request/message id (and want feedback to correlate against it without round-tripping through onPrompt), provide getPromptId. It is called once per tracked interaction; return a string to use it, or undefined to fall back to the generated id. Pair it with a per-request store so each call reads its own id:

import { AsyncLocalStorage } from "node:async_hooks";

const promptStore = new AsyncLocalStorage<{ promptId: string }>();

const telemetry = countlyTelemetry({
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getPromptId: () => promptStore.getStore()?.promptId, // undefined → generated fallback
});

// wrap each request so the interaction stamps your id:
promptStore.run({ promptId: myMessageId }, async () => {
  await generateText({
    model: openai("gpt-4o"),
    prompt: "Explain quantum computing",
    experimental_telemetry: { integrations: [telemetry] },
  });
});

The returned id lands on the [CLY]_llm_interaction event's prompt_id, so you can record feedback against your own id directly — feedback.track({ prompt_id: myMessageId, ... }) — no onPrompt capture needed. When getPromptId is absent (or returns undefined), behavior is unchanged.

Feedback

User feedback (thumbs up/down, ratings, comments) is not auto-collected — wire it from your UI. Capture the prompt_id of each tracked interaction via the onPrompt callback, then record feedback against it with createFeedbackTracker (re-exported from this package, so no extra install is needed):

import { countlyTelemetry, createFeedbackTracker, type PromptInfo } from "@countly/ai-sdk-vercel";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const countly = { appKey: "YOUR_APP_KEY", url: "https://your-countly-server.com" };

let lastPrompt: PromptInfo | undefined;
const telemetry = countlyTelemetry({
  ...countly,
  onPrompt: (info) => { lastPrompt = info; }, // fires after every tracked call
});

const feedback = createFeedbackTracker(countly, { sdk_adapter: "ai-sdk" });

const { text } = await generateText({
  model: openai("gpt-4o"),
  prompt: "Explain quantum computing",
  experimental_telemetry: { integrations: [telemetry] },
});

// ...later, when the user rates the answer:
feedback.track({
  prompt_id: lastPrompt!.prompt_id,
  rating: "thumbs_up", // or "thumbs_down", or any custom string
  score: 0.9, // optional 0-1 numeric score
  category: "helpful", // optional: hallucination, irrelevant, harmful, ...
  comment: "Great answer", // optional free-form text
  deviceId: user.id, // attribute to the same user as the interaction
});

Each track() call emits a [CLY]_llm_interaction_feedback event whose prompt_id links back to the [CLY]_llm_interaction event — powering prompt → feedback funnels and per-model satisfaction breakdowns in Countly. In a real app, store prompt_id alongside the rendered message (or return it to your client) and read it back when the user rates the answer. Feedback is batched like interaction events; call feedback.flush() to send immediately, or feedback.shutdown() on process exit.

Full documentation

See the Countly AI SDK repository for the unified data model, observability levels, cost calculation, privacy controls, and Countly plugin integration (Drill, Funnels, Cohorts, APM, Crash Analytics).

License

MIT