@countly/ai-sdk-openai

Countly AI observability adapter for the OpenAI Node.js SDK.

Part of the Countly AI SDK — provider-agnostic LLM observability for every AI stack.

Install

npm install @countly/ai-sdk-openai

@countly/ai-sdk-core is pulled in automatically.

Peer dependency

openai >= 4.68.0

Quick Start

import OpenAI from "openai";
import { observeOpenAI } from "@countly/ai-sdk-openai";

// Per-user attribution via AsyncLocalStorage (Node.js)
import { AsyncLocalStorage } from "node:async_hooks";
const userStore = new AsyncLocalStorage<{ userId: string }>();

// In your middleware — run once per request
app.use((req, res, next) => {
  userStore.run({ userId: req.user.id }, next);
});

// Wrap your client — getDeviceId is called per event at enqueue time
const openai = observeOpenAI(new OpenAI(), {
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getDeviceId: () => userStore.getStore()?.userId,
});

// Use exactly as before — observability is automatic
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Explain quantum computing" }],
});

Streaming

const stream = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello" }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
// Events reported automatically after stream completes

Token usage for streams requires opting in. OpenAI only puts usage in the stream when you pass stream_options: { include_usage: true }. Without it the tokens genuinely are not reported, so the row carries usage_state: "not_reported" and no token or cost fields — rather than a 0 that would read as a free call:

const stream = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello" }],
  stream: true,
  stream_options: { include_usage: true }, // ← required for streaming cost data
});

Breaking out of the loop early, or aborting via stream.controller.abort(), still records the row — the request was billed for everything streamed so far — with status: "incomplete" and error: "stream_cancelled".

Tracked surfaces

| Entrypoint | Tracked | |---|---| | client.chat.completions.create() (incl. stream: true) | ✅ | | client.chat.completions.parse() | ✅ | | client.responses.create() (incl. stream: true) | ✅ | | client.responses.parse() | ✅ | | client.chat.completions.stream() / client.responses.stream() helper runners | ❌ — they build their own request through the unwrapped client | | embeddings, moderations, files, batches, … | ❌ by design — a non-generation operation must never produce an interaction row |

Set debug: true to get a one-time warning when an untracked generation entrypoint is accessed.

What's captured

• Token usage — input and output totals (inclusive of cache and reasoning), plus the cached / reasoning / audio subsets, and usage_state stating whether the provider reported any of it
• Cost from model pricing, plus cost_priced stating whether the model could be priced at all (an unpriced model emits no cost fields rather than $0)
• Latency (total + time to first token for streaming)
• Model config (temperature, top_p, max_tokens, frequency/presence penalties) — on streaming rows too
• api_host_type (vendor_direct / azure_openai / gateway / local_infra) and api_host — the client's baseURL hostname only; path, query and credentials are never emitted
• Tool calls and their parameters, with call_id so repeat calls of one tool stay distinguishable. A tool's status is "unknown" unless the outcome was actually observed (the hosted Responses tools — MCP, file search, code interpreter — do report theirs)
• Error tracking with categorization (rate_limit, context_length, content_filter, timeout, auth_error)
• APM traces for performance monitoring
• Per-user usage aggregation

Browser

In the browser there is no multi-user request mixing — each user runs in their own tab. Use a static device ID:

const openai = observeOpenAI(new OpenAI(), {
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  deviceId: loggedInUser.id,
});

Feedback

User feedback (thumbs up/down, ratings, comments) is not auto-collected — wire it from your UI. Capture the prompt_id of each tracked interaction via the onPrompt callback, then record feedback against it with createFeedbackTracker (re-exported from this package, so no extra install is needed):

import OpenAI from "openai";
import { observeOpenAI, createFeedbackTracker, type PromptInfo } from "@countly/ai-sdk-openai";

const countly = { appKey: "YOUR_APP_KEY", url: "https://your-countly-server.com" };

let lastPrompt: PromptInfo | undefined;
const openai = observeOpenAI(new OpenAI(), {
  ...countly,
  onPrompt: (info) => { lastPrompt = info; }, // fires after every tracked call
});

const feedback = createFeedbackTracker(countly); // stamped as sdk_adapter "openai"

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Explain quantum computing" }],
});

// ...later, when the user rates the answer:
feedback.track({
  prompt_id: lastPrompt!.prompt_id,
  rating: "thumbs_up", // or "thumbs_down", or any custom string
  score: 0.9, // optional 0-1 numeric score
  category: "helpful", // optional: hallucination, irrelevant, harmful, ...
  comment: "Great answer", // optional free-form text
  deviceId: user.id, // attribute to the same user as the interaction
});

Each track() call emits a [CLY]_llm_interaction_feedback event that joins back to the interaction — powering prompt → feedback funnels and per-model satisfaction breakdowns in Countly. lastPrompt.prompt_id (identical to lastPrompt.run_id) rates the whole turn; to rate one generation of a multi-call turn instead, pass that generation's row id:

feedback.track({ prompt_id: lastPrompt!.event_id, run_id: lastPrompt!.run_id, rating: "thumbs_down" });

In a real app, store the id alongside the rendered message (or return it to your client) and read it back when the user rates the answer. Feedback is batched like interaction events; call feedback.flush() to send immediately, or feedback.shutdown() on process exit.

Caller-supplied prompt IDs

By default each tracked call gets an auto-generated id. If you already mint your own request/trace ID upstream, supply it with the getPromptId callback — it is read once per call and becomes that turn's run_id, the join column every emitted row carries. Return undefined to fall back to auto-generation for that call:

const openai = observeOpenAI(new OpenAI(), {
  ...countly,
  getPromptId: () => currentRequestId(), // your own id, or undefined to auto-generate
});

Because that id becomes the turn's run_id, you can correlate user feedback without the onPrompt round-trip: pass your known id straight into createFeedbackTracker().track({ prompt_id }). This is handy when the id already flows through your app (e.g. a chat message id), so both the interaction and its feedback are keyed on a value you control.

On the wire, each row's own primary key is event_id, and prompt_id is a foreign key: on an interaction it points at that row's own event_id, and on a tool row at the generation that requested the tool. Your id is run_id.

Full documentation

See the Countly AI SDK repository for the schema v2 wire contract (one row per generation, RULE A dimensions, RULE B measures with their usage_state / cost_priced markers, and the common envelope), the adapter capability matrix, observability levels (0/1/2), cost calculation, privacy controls, and Countly plugin integration (Drill, Funnels, Cohorts, APM, Crash Analytics).

License

MIT