@countly/ai-sdk-llamaindex

Countly AI observability adapter for LlamaIndex.TS.

Part of the Countly AI SDK — provider-agnostic LLM observability for every AI stack.

Install

npm install @countly/ai-sdk-llamaindex

@countly/ai-sdk-core is pulled in automatically.

Peer dependency

llamaindex >= 0.8.0

Quick Start

import { Settings } from "llamaindex";
import { CountlyLlamaIndexTracer } from "@countly/ai-sdk-llamaindex";
import { AsyncLocalStorage } from "node:async_hooks";

const userStore = new AsyncLocalStorage<{ userId: string }>();

app.use((req, res, next) => {
  userStore.run({ userId: req.user.id }, next);
});

const tracer = new CountlyLlamaIndexTracer({
  appKey: "YOUR_APP_KEY",
  url: "https://your-countly-server.com",
  getDeviceId: () => userStore.getStore()?.userId,
});

// Attach to LlamaIndex's callback manager
tracer.register(Settings.callbackManager);

The tracer listens to llm-start, llm-end, llm-tool-call, and llm-tool-result events on the dispatcher.

What's captured

• LLM start/end lifecycle (latency, tokens, model) — supports both snake_case and camelCase token fields
• Tool call lifecycle (name, arguments, status, latency)
• Multiple event format support across LlamaIndex versions
• Concurrent call tracking
• Error tracking with categorization
• Per-user aggregation

Flush before shutdown

await tracer.flush();    // send buffered events
await tracer.shutdown(); // flush + stop the transport
tracer.unregister(Settings.callbackManager); // detach handlers

Feedback

User feedback (thumbs up/down, ratings, comments) is not auto-collected — wire it from your UI. Capture the prompt_id of each tracked interaction via the onPrompt callback, then record feedback against it with createFeedbackTracker (re-exported from this package, so no extra install is needed):

import { Settings } from "llamaindex";
import { CountlyLlamaIndexTracer, createFeedbackTracker, type PromptInfo } from "@countly/ai-sdk-llamaindex";

const countly = { appKey: "YOUR_APP_KEY", url: "https://your-countly-server.com" };

let lastPrompt: PromptInfo | undefined;
const tracer = new CountlyLlamaIndexTracer({
  ...countly,
  onPrompt: (info) => { lastPrompt = info; }, // fires after every tracked LLM call
});
tracer.register(Settings.callbackManager);

const feedback = createFeedbackTracker(countly, { sdk_adapter: "llamaindex" });

const response = await queryEngine.query({ query: "Explain quantum computing" });

// ...later, when the user rates the answer:
feedback.track({
  prompt_id: lastPrompt!.prompt_id,
  rating: "thumbs_up", // or "thumbs_down", or any custom string
  score: 0.9, // optional 0-1 numeric score
  category: "helpful", // optional: hallucination, irrelevant, harmful, ...
  comment: "Great answer", // optional free-form text
  deviceId: user.id, // attribute to the same user as the interaction
});

Each track() call emits a [CLY]_llm_interaction_feedback event whose prompt_id links back to the [CLY]_llm_interaction event — powering prompt → feedback funnels and per-model satisfaction breakdowns in Countly. In a real app, store prompt_id alongside the rendered message (or return it to your client) and read it back when the user rates the answer. Feedback is batched like interaction events; call feedback.flush() to send immediately, or feedback.shutdown() on process exit.

Caller-supplied prompt_id

Instead of reading a generated prompt_id back through onPrompt, you can stamp your own id on each interaction via getPromptId. Return the correlation key you already track for the request (e.g. a message id or trace id) and the tracer uses it verbatim; return undefined to fall back to the auto-generated id. This lets you correlate feedback without a read-back round-trip:

const tracer = new CountlyLlamaIndexTracer({
  ...countly,
  getPromptId: () => currentRequest.id, // your own id → becomes the event's prompt_id
});
tracer.register(Settings.callbackManager);

// ...later, correlate feedback directly against the same id you supplied:
feedback.track({ prompt_id: currentRequest.id, rating: "thumbs_up" });

getPromptId is called once per interaction. It is the wrapped-client equivalent of the request-context prompt id used by other Countly AI adapters, so a non-empty return value is stamped on both the [CLY]_llm_interaction event and its tool events.

Full documentation

See the Countly AI SDK repository for the unified data model, observability levels, cost calculation, privacy controls, and Countly plugin integration (Drill, Funnels, Cohorts, APM, Crash Analytics).

License

MIT