Qualifire SDK

The official TypeScript SDK for evaluating LLM outputs with Qualifire. Detect hallucinations, prompt injections, PII leakage, content policy violations, and more.

Installation

npm install qualifire

Quick Start

import { Qualifire } from 'qualifire';
import OpenAI from 'openai';

const qualifire = new Qualifire({ apiKey: 'your-api-key' });
const openai = new OpenAI();

// Make your LLM call
const request = {
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'What is the capital of France?' },
  ],
};

const response = await openai.chat.completions.create(request);

// Evaluate the response
const evaluation = await qualifire.evaluate({
  framework: 'openai',
  request,
  response,
  hallucinationsCheck: true,
  groundingCheck: true,
});

console.log(evaluation);
// {
//   status: 'passed',
//   score: 100,
//   evaluationResults: [...]
// }

Supported Frameworks

| Framework | Value | SDK | |-----------|-------|-----| | OpenAI | openai | openai (Chat Completions & Responses API) | | Anthropic Claude | claude | @anthropic-ai/sdk | | Google Gemini | gemini | @google/genai | | Vercel AI SDK | vercelai | ai |

All frameworks support both streaming and non-streaming responses.

Available Evaluation Checks

| Check | Parameter | Description | |-------|-----------|-------------| | Hallucinations | hallucinationsCheck | Detect fabricated information | | Grounding | groundingCheck | Verify responses are grounded in context | | Prompt Injections | promptInjections | Detect prompt injection attempts | | PII Detection | piiCheck | Identify personally identifiable information | | Content Moderation | contentModerationCheck | Flag harmful content | | Instructions Following | instructionsFollowingCheck | Verify adherence to system instructions | | Tool Selection Quality | toolSelectionQualityCheck | Evaluate tool/function call accuracy | | Custom Assertions | assertions | Array of custom assertion strings |

Framework Examples

OpenAI

// Chat Completions API
const request = {
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello!' }],
};
const response = await openai.chat.completions.create(request);

await qualifire.evaluate({
  framework: 'openai',
  request,
  response,
  hallucinationsCheck: true,
});

// Streaming
const streamRequest = { ...request, stream: true };
const stream = await openai.chat.completions.create(streamRequest);

const chunks = [];
for await (const chunk of stream) {
  chunks.push(chunk);
}

await qualifire.evaluate({
  framework: 'openai',
  request: streamRequest,
  response: chunks,
  hallucinationsCheck: true,
});

Anthropic Claude

import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic();

const request = {
  model: 'claude-sonnet-4-20250514',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'Hello!' }],
};
const response = await anthropic.messages.create(request);

await qualifire.evaluate({
  framework: 'claude',
  request,
  response,
  promptInjections: true,
});

Google Gemini

import { GoogleGenAI } from '@google/genai';

const genai = new GoogleGenAI({ apiKey: 'your-key' });

const request = {
  model: 'gemini-2.0-flash',
  contents: [{ role: 'user', parts: [{ text: 'Hello!' }] }],
};
const response = await genai.models.generateContent(request);

await qualifire.evaluate({
  framework: 'gemini',
  request,
  response,
  contentModerationCheck: true,
});

Vercel AI SDK

import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';

const request = {
  model: openai('gpt-4o'),
  prompt: 'Hello!',
};
const response = await generateText(request);

await qualifire.evaluate({
  framework: 'vercelai',
  request,
  response,
  piiCheck: true,
});

Direct Message Mode

For cases where you don't use a supported framework, pass messages directly:

await qualifire.evaluate({
  messages: [
    { role: 'user', content: 'What is 2+2?' },
    { role: 'assistant', content: 'The answer is 4.' },
  ],
  hallucinationsCheck: true,
  groundingCheck: true,
});

Invoke Pre-configured Evaluations

Run evaluations configured in the Qualifire dashboard:

const result = await qualifire.invokeEvaluation({
  input: 'What is the capital of France?',
  output: 'Paris is the capital of France.',
  evaluationId: 'eval-123',
});

Configuration

Constructor Options

const qualifire = new Qualifire({
  apiKey: 'your-api-key',      // Required (or set QUALIFIRE_API_KEY env var)
  baseUrl: 'https://...',      // Optional, defaults to https://proxy.qualifire.ai
});

Environment Variables

| Variable | Description | |----------|-------------| | QUALIFIRE_API_KEY | API key for authentication | | QUALIFIRE_BASE_URL | Override the API base URL |

Response Format

interface EvaluationResponse {
  status: 'passed' | 'failed';
  score: number;  // 0-100
  evaluationResults: Array<{
    type: string;
    results: Array<{
      name: string;
      score: number;
      label: string;
      confidence_score: number;
      reason: string;
    }>;
  }>;
}

License

MIT