@countly/ai-sdk-langchain

Countly AI observability adapter for LangChain.

Part of the Countly AI SDK — provider-agnostic LLM observability for every AI stack.

Install

npm install @countly/ai-sdk-langchain

@countly/ai-sdk-core is pulled in automatically.

Peer dependency

@langchain/core >= 0.3.0

Quick Start

import { CountlyCallbackHandler } from "@countly/ai-sdk-langchain";
import { AsyncLocalStorage } from "node:async_hooks";

const userStore = new AsyncLocalStorage<{ userId: string }>();

app.use((req, res, next) => {
  userStore.run({ userId: req.user.id }, next);
});

const handler = new CountlyCallbackHandler({
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getDeviceId: () => userStore.getStore()?.userId,
});

// Works with any LangChain chain, agent, or LLM
const result = await chain.invoke(input, { callbacks: [handler] });

With ChatOpenAI

import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({ model: "gpt-4o" });
const response = await llm.invoke("Hello", { callbacks: [handler] });

What's captured

• LLM start/end lifecycle (latency, tokens, model)
• Tool start/end lifecycle (name, duration)
• Error tracking (LLM errors and tool errors tracked separately)
• Concurrent run tracking (each runId tracked independently)
• Per-user aggregation

Flush before shutdown

await handler.flush();   // send buffered events
await handler.shutdown(); // flush + stop the transport

Caller-supplied prompt_id

By default every tracked LLM call gets a freshly generated prompt_id. If you already mint your own prompt identifier upstream (e.g. a request-scoped id shared across services), supply it via the getPromptId config callback and the adapter will stamp it on every [CLY]_llm_interaction event instead:

import { AsyncLocalStorage } from "node:async_hooks";

const promptStore = new AsyncLocalStorage<{ promptId: string }>();

const handler = new CountlyCallbackHandler({
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getPromptId: () => promptStore.getStore()?.promptId, // return undefined to fall back to a generated id
});

// Resolved per-call: returning a value pins prompt_id; returning undefined restores default behavior.
promptStore.run({ promptId: "chat-req-42" }, () =>
  chain.invoke(input, { callbacks: [handler] }),
);

getPromptId is called once per LLM interaction (both success and error paths). Returning undefined is identical to not supplying it at all. Because the same prompt_id flows through to feedback correlation below, a caller-supplied id lets you link an interaction to feedback without threading the generated id back through your app.

Feedback

User feedback (thumbs up/down, ratings, comments) is not auto-collected — wire it from your UI. Capture the prompt_id of each tracked interaction via the onPrompt callback, then record feedback against it with createFeedbackTracker (re-exported from this package, so no extra install is needed):

import { CountlyCallbackHandler, createFeedbackTracker, type PromptInfo } from "@countly/ai-sdk-langchain";

const countly = { appKey: "YOUR_APP_KEY", url: "https://your-countly-server.com" };

let lastPrompt: PromptInfo | undefined;
const handler = new CountlyCallbackHandler({
  ...countly,
  onPrompt: (info) => { lastPrompt = info; }, // fires after every tracked LLM call
});

const feedback = createFeedbackTracker(countly, { sdk_adapter: "langchain" });

const result = await chain.invoke(input, { callbacks: [handler] });

// ...later, when the user rates the answer:
feedback.track({
  prompt_id: lastPrompt!.prompt_id,
  rating: "thumbs_up", // or "thumbs_down", or any custom string
  score: 0.9, // optional 0-1 numeric score
  category: "helpful", // optional: hallucination, irrelevant, harmful, ...
  comment: "Great answer", // optional free-form text
  deviceId: user.id, // attribute to the same user as the interaction
});

Each track() call emits a [CLY]_llm_interaction_feedback event whose prompt_id links back to the [CLY]_llm_interaction event — powering prompt → feedback funnels and per-model satisfaction breakdowns in Countly. In a real app, store prompt_id alongside the rendered message (or return it to your client) and read it back when the user rates the answer. Feedback is batched like interaction events; call feedback.flush() to send immediately, or feedback.shutdown() on process exit.

Full documentation

See the Countly AI SDK repository for the unified data model, observability levels, cost calculation, privacy controls, and Countly plugin integration (Drill, Funnels, Cohorts, APM, Crash Analytics).

License

MIT