@countly/ai-sdk-openai

Countly AI observability adapter for the OpenAI Node.js SDK.

Part of the Countly AI SDK — provider-agnostic LLM observability for every AI stack.

Install

npm install @countly/ai-sdk-openai

@countly/ai-sdk-core is pulled in automatically.

Peer dependency

openai >= 4.68.0

Quick Start

import OpenAI from "openai";
import { observeOpenAI } from "@countly/ai-sdk-openai";

// Per-user attribution via AsyncLocalStorage (Node.js)
import { AsyncLocalStorage } from "node:async_hooks";
const userStore = new AsyncLocalStorage<{ userId: string }>();

// In your middleware — run once per request
app.use((req, res, next) => {
  userStore.run({ userId: req.user.id }, next);
});

// Wrap your client — getDeviceId is called per event at enqueue time
const openai = observeOpenAI(new OpenAI(), {
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getDeviceId: () => userStore.getStore()?.userId,
});

// Use exactly as before — observability is automatic
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Explain quantum computing" }],
});

Streaming

const stream = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello" }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
// Events reported automatically after stream completes

What's captured

• Token usage (input, output, reasoning, cached)
• Cost (computed from model pricing)
• Latency (total + time to first token for streaming)
• Model config (temperature, top_p, max_tokens, frequency/presence penalties)
• Tool calls and their parameters
• Error tracking with categorization (rate_limit, context_length, content_filter, timeout, auth_error)
• APM traces for performance monitoring
• Per-user usage aggregation

Browser

In the browser there is no multi-user request mixing — each user runs in their own tab. Use a static device ID:

const openai = observeOpenAI(new OpenAI(), {
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  deviceId: loggedInUser.id,
});

Feedback

User feedback (thumbs up/down, ratings, comments) is not auto-collected — wire it from your UI. Capture the prompt_id of each tracked interaction via the onPrompt callback, then record feedback against it with createFeedbackTracker (re-exported from this package, so no extra install is needed):

import OpenAI from "openai";
import { observeOpenAI, createFeedbackTracker, type PromptInfo } from "@countly/ai-sdk-openai";

const countly = { appKey: "YOUR_APP_KEY", url: "https://your-countly-server.com" };

let lastPrompt: PromptInfo | undefined;
const openai = observeOpenAI(new OpenAI(), {
  ...countly,
  onPrompt: (info) => { lastPrompt = info; }, // fires after every tracked call
});

const feedback = createFeedbackTracker(countly, { sdk_adapter: "openai" });

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Explain quantum computing" }],
});

// ...later, when the user rates the answer:
feedback.track({
  prompt_id: lastPrompt!.prompt_id,
  rating: "thumbs_up", // or "thumbs_down", or any custom string
  score: 0.9, // optional 0-1 numeric score
  category: "helpful", // optional: hallucination, irrelevant, harmful, ...
  comment: "Great answer", // optional free-form text
  deviceId: user.id, // attribute to the same user as the interaction
});

Each track() call emits a [CLY]_llm_interaction_feedback event whose prompt_id links back to the [CLY]_llm_interaction event — powering prompt → feedback funnels and per-model satisfaction breakdowns in Countly. In a real app, store prompt_id alongside the rendered message (or return it to your client) and read it back when the user rates the answer. Feedback is batched like interaction events; call feedback.flush() to send immediately, or feedback.shutdown() on process exit.

Caller-supplied prompt IDs

By default each tracked call gets an auto-generated prompt_id. If you already mint your own request/trace ID upstream, supply it with the getPromptId callback — it is read once per create() call and stamped as the prompt_id on that interaction. Return undefined to fall back to auto-generation for that call:

const openai = observeOpenAI(new OpenAI(), {
  ...countly,
  getPromptId: () => currentRequestId(), // your own id, or undefined to auto-generate
});

Because this is the same prompt_id that lands on the [CLY]_llm_interaction event, you can correlate user feedback without the onPrompt round-trip: pass your known id straight into createFeedbackTracker().track({ prompt_id }). This is handy when the id already flows through your app (e.g. a chat message id), so both the interaction and its feedback are keyed on a value you control.

Full documentation

See the Countly AI SDK repository for the unified data model, observability levels (0/1/2), cost calculation, privacy controls, and Countly plugin integration (Drill, Funnels, Cohorts, APM, Crash Analytics).

License

MIT