@evalguard/llamaindex

Drop-in LlamaIndex.TS instrumentation that adds EvalGuard guardrails (firewall input checks) and observability (trace logging, cost tracking) to every LLM call inside your RAG pipeline. Hooks the standard Settings.callbackManager so it works with any LlamaIndex query / chat / agent flow with no other code changes.

Install

npm install @evalguard/llamaindex llamaindex @llamaindex/openai

llamaindex is a peer dependency (npm installs it for you, but naming it pins the major you want). @llamaindex/openai is the provider used by the quickstart below — as of [email protected] the provider classes live in their own packages, so import { OpenAI } from "llamaindex" is undefined. Swap it for @llamaindex/anthropic, @llamaindex/google, or whichever provider you actually use.

⚠️ ESM-only — requires "type": "module"

@evalguard/llamaindex ships ES modules only ("type": "module", no CJS build). The quickstart below will not type-check or run in a default CommonJS TypeScript project — you get TS1479 ("the referenced file is an ECMAScript module and cannot be imported with require") and, because the peer SDK's types then resolve under a different module mode, a confusing TS2345 … Property '#private' … refers to a different member on the very first call.

To use this package, your consuming project must be ESM:

// package.json
{ "type": "module" }

// tsconfig.json
{ "compilerOptions": { "module": "node16", "moduleResolution": "node16" } }

Staying on CommonJS? Load it with a dynamic import(). Unlike the SDK-shaped wrappers (@evalguard/openai, @evalguard/anthropic, @evalguard/gemini), llamaindex is dual-published, so your Settings import can stay static. Verified against [email protected].

Both awaits must sit inside an async function: top-level await is an ESM-only feature, so a bare await import(...) at file scope in a CJS module is TS1309: The current file is a CommonJS module and cannot use 'await' at the top level.

// ✅ compiles under module/moduleResolution "node16", no "type": "module"
import { Settings } from "llamaindex";

async function main() {
  const { installEvalguard } = await import("@evalguard/llamaindex");
  await installEvalguard({
    apiKey: process.env.EVALGUARD_API_KEY!,
    projectId: "proj-123",
    settings: Settings,
  });
}

void main();

If you swap in a provider package that is itself ESM-only, that import has to become dynamic as well — the rule is that no ESM-only specifier may be imported statically from a CJS file, not just this one.

Node.js ≥ 22.12 can also require() an ESM module directly (require(esm)), but TypeScript still type-checks the import under CJS rules, so the dynamic-import form above is the supported path.

Peer requires llamaindex >= 0.5.0.

Use

import { Settings } from "llamaindex";
import { OpenAI } from "@llamaindex/openai";
import { installEvalguard } from "@evalguard/llamaindex";

Settings.llm = new OpenAI({ model: "gpt-4o-mini" });

await installEvalguard({
  apiKey: process.env.EVALGUARD_API_KEY!,
  projectId: "proj-123",
  settings: Settings, // REQUIRED for blockOnViolation to block anything
});

// From here, every LlamaIndex chat / query call is wrapped automatically.
const response = await Settings.llm.chat({ messages: [{ role: "user", content: "hi" }] });

Two things about this snippet that are easy to get wrong, both measured against the installed [email protected]:

• settings: Settings is not optional. Omitting it — which this quickstart used to do — is exactly the configuration in which blockOnViolation (on by default) degrades to a console warning and cannot abort a call. See Where blocking is enforced.
• OpenAI does not come from llamaindex. As of 0.12 the provider classes moved to their own packages; typeof (await import("llamaindex")).OpenAI is undefined, so an import { Settings, OpenAI } from "llamaindex" line does not run. @llamaindex/openai is on the Install command above — it was added there on 2026-08-02, having previously been named only here, thirteen lines after the snippet that needs it.

Options

await installEvalguard({
  apiKey: "...",
  projectId: "proj-...",
  baseUrl: "https://...",        // self-hosted EvalGuard
  blockOnViolation: true,         // default: true — throws EvalguardBlockedError
  disableGuardrails: false,
  disableLogging: false,
  metadata: { feature: "rag", env: "prod" },
  settings: Settings,             // the LlamaIndex Settings object. Without it
                                  // blockOnViolation cannot block — see below.
});

The function returns an uninstall handle for graceful shutdown:

const uninstall = await installEvalguard({ apiKey: "..." });
process.on("SIGTERM", () => uninstall());

What gets instrumented

For every LlamaIndex LLM call (chat, complete, query-engine LLM calls, agent reasoning steps), we hook:

• llm-start — pre-call: collapse every message role (tool results and retrieved RAG documents are the untrusted channel, so they are scanned too, not just user turns) and run the firewall check.
• llm-end — post-call: compute latency, extract token usage from the response, estimate cost, log a trace fire-and-forget.

Both handlers read the event's .detail. Real LlamaIndex dispatches a LlamaIndexCustomEvent extends CustomEvent, so .detail is where the body lives; there is no .payload. (Through v1.0.0 this package read .payload and therefore observed nothing at all against the real library — see CHANGELOG / __A278_repro.test.ts.)

Blocking is not performed from these callbacks — see below.

Embedding calls and retrieval steps are NOT instrumented in v1 — they don't carry the same guardrail-relevant payload. Will land in v1.1 if customers ask.

Outage semantics — fail-CLOSED by default

Corrected 2026-07-29. This section previously promised a fail-open "guarantee" ("network errors ... silently allow the call through"). The wrappers went fail-closed on 2026-05-28; the docs were never updated.

With blockOnViolation: true (the default), an unreachable EvalGuard API throws EvalguardBlockedError carrying a guardrail_unavailable violation — the call does not proceed. Set blockOnViolation: false for availability-first (monitor-only) behaviour, where violations and outages are recorded on the trace and the call continues.

Trace-log failures are always silent and never bubble into your app.

Where blocking is enforced

blockOnViolation is enforced by wrapping Settings.llm.chat / .complete, so you must hand the Settings object over:

import { Settings } from "llamaindex";
await installEvalguard({ apiKey: process.env.EVALGUARD_API_KEY!, settings: Settings });

Without settings, EvalGuard can only observe: violations are reported and logged, but the call cannot be aborted, and a warning is printed at install time. This is a property of LlamaIndex, not a choice — CallbackManager.dispatchEvent fires handlers inside a queueMicrotask and never awaits them, so a throw from a callback cannot stop anything.

Only the LLM instance sitting on Settings.llm at install time is wrapped. If you build a second LLM yourself and hand it straight to an index or query engine, its calls cannot be blocked — but they are still scanned and still show up on traces, because the llm-start callback runs its own firewall check for any call the wrapper did not already check. The firewall is charged once per call, never zero times.

Cost estimates: unpriced models are reported as unpriced

estimateCost() used to invent a price for any model outside a ~60-row table: estimateCost("gpt-5", 1000, 500) and estimateCost("totally-unknown-model", 1000, 500) both returned 0.0105 from a blended $0.003/$0.015-per-1k fallback — with no flag and no warning. A FinOps figure you cannot tell apart from a real vendor price is worse than no figure at all.

Two things changed:

• Coverage — 2,200+ model ids now resolve to a real, sourced rate (generated from EvalGuard's own pricing database, which is synced from the LiteLLM catalogue). gpt-5 is priced correctly.
• Honesty — a genuinely unknown model is now visibly unknown.

import { estimateCostDetailed, isModelPriced } from "@evalguard/llamaindex";

estimateCostDetailed("gpt-5", 1000, 500);
// { model: "gpt-5", costUsd: 0.00625, priced: true, pricingSource: "catalog" }

estimateCostDetailed("totally-unknown-model", 1000, 500);
// { model: "totally-unknown-model",
//   costUsd: null,              // <- never a fabricated number
//   priced: false,
//   pricingSource: "unpriced",
//   blendedFallbackUsd: 0.0105 } // <- opt-in rough figure, clearly labelled

isModelPriced("totally-unknown-model"); // false

estimateCost() still returns a number for backwards compatibility, but it now emits a one-time console.warn naming the unpriced model. Traces carry costPricingSource alongside cost, and cost is null for an unpriced model rather than a guess, so your EvalGuard dashboard shows "unpriced" instead of a fake dollar amount.

License

Apache-2.0