DeepEval is an open-source LLM evaluation framework. It's similar to Vitest — in fact it runs as Vitest — but specialized for testing LLM apps, with research-backed metrics like G-Eval, task completion, answer relevancy, and hallucination that use LLM-as-a-judge and run locally on your machine.

await expect(testCase).toPass([correctnessMetric]);

Whether you're building agents, RAG pipelines, or chatbots with LangChain, Mastra, the AI SDK, or OpenAI, you can evaluate your app end-to-end as a black box, over complete agent trajectories, or at individual steps like LLM calls, tool use, and retrieval.

This package is the TypeScript SDK. Python remains DeepEval's most complete implementation — see Python vs TypeScript for the gaps.

Quickstart

npm install --save-dev deepeval

Log in to Confident AI to keep evaluation results on the cloud and compare runs over time. It's free and takes no extra code, but it's optional — evals print to your terminal either way.

npx deepeval login

Then write a test file. Metrics judge with OpenAI by default, so set OPENAI_API_KEY first (.env.local and .env are auto-loaded).

import { LLMTestCase, SingleTurnParams } from "deepeval/test-case";
import { GEval } from "deepeval/metrics";
import { it, expect } from "vitest";
import "deepeval/vitest";

it("gives a correct answer", async () => {
  const correctnessMetric = new GEval({
    name: "Correctness",
    criteria:
      "Determine if the 'actual output' is correct based on the 'expected output'.",
    evaluationParams: [
      SingleTurnParams.ACTUAL_OUTPUT,
      SingleTurnParams.EXPECTED_OUTPUT,
    ],
    threshold: 0.5,
  });
  const testCase = new LLMTestCase({
    input: "What if these shoes don't fit?",
    // Replace this with the actual output from your LLM application
    actualOutput: "You have 30 days to get a full refund at no extra cost.",
    expectedOutput: "We offer a 30-day full refund at no extra costs.",
  });

  await expect(testCase).toPass([correctnessMetric]);
});

npx deepeval test run example.test.ts

Scores range from 0 to 1, and threshold decides whether the test passes. Every metric also explains itself, so a failure tells you why it failed. Read the docs for the full walkthrough.

Importing deepeval/vitest registers the toPass() matcher. npx deepeval test run also injects the matcher and the test-run reporter for you; to get the same from your own vitest command, register them in your config:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    setupFiles: ["deepeval/vitest"],
    globalSetup: ["deepeval/vitest/global-setup"],
    testTimeout: 120_000,
    hookTimeout: 120_000,
  },
});

Metrics

Every metric is a class from deepeval/metrics, takes an options object, and returns a score with a reason. Full list and parameters in the metrics docs.

Custom, all-purpose: GEval evaluates against any criteria you write in plain English; DAGMetric builds a deterministic decision tree of LLM judgements when you need repeatable verdicts.

TaskCompletionMetric, ToolCorrectnessMetric, GoalAccuracyMetric, StepEfficiencyMetric, PlanAdherenceMetric, PlanQualityMetric, ToolUseMetric, ArgumentCorrectnessMetric

AnswerRelevancyMetric, FaithfulnessMetric, ContextualRecallMetric, ContextualPrecisionMetric, ContextualRelevancyMetric

KnowledgeRetentionMetric, ConversationCompletenessMetric, TurnRelevancyMetric, TurnFaithfulnessMetric, RoleAdherenceMetric, TopicAdherenceMetric, TurnContextualPrecisionMetric, TurnContextualRecallMetric, TurnContextualRelevancyMetric, ConversationalGEval, ConversationalDAGMetric

MCPTaskCompletionMetric, MCPUseMetric, MultiTurnMCPUseMetric

TextToImageMetric, ImageEditingMetric, ImageCoherenceMetric, ImageHelpfulnessMetric, ImageReferenceMetric

HallucinationMetric, SummarizationMetric, BiasMetric, ToxicityMetric, JsonCorrectnessMetric, PromptAlignmentMetric, PIILeakageMetric, NonAdviceMetric, MisuseMetric, RoleViolationMetric

ExactMatchMetric and PatternMatchMetric need no LLM at all.

Metrics work standalone too, outside any test:

import { AnswerRelevancyMetric } from "deepeval/metrics";
import { LLMTestCase } from "deepeval/test-case";

const metric = new AnswerRelevancyMetric({ threshold: 0.7 });
await metric.measure(
  new LLMTestCase({
    input: "What if these shoes don't fit?",
    actualOutput: "We offer a 30-day full refund at no extra costs.",
  }),
);

console.log(metric.score, metric.reason);

Or score a whole batch at once with evaluate(testCases, metrics) from deepeval, which is better suited to scripts than to a test suite.

Tracing and Integrations

Wrap any function in observe() and DeepEval captures the ordered sequence of model decisions, tool calls, and intermediate steps. That trace is what lets you evaluate a complete agent trajectory rather than just its final answer, and score individual components along the way.

import { observe, updateCurrentSpan } from "deepeval/tracing";
import { AnswerRelevancyMetric } from "deepeval/metrics";
import { LLMTestCase } from "deepeval/test-case";

const retrieve = observe({
  type: "retriever",
  metrics: [new AnswerRelevancyMetric()],
  fn: async (query: string) => {
    const output = await search(query);
    updateCurrentSpan({
      testCase: new LLMTestCase({ input: query, actualOutput: output }),
    });
    return output;
  },
});

Run a dataset through the traced app with evalsIterator(). Trace-level metrics judge the whole trajectory; nextLlmSpan (and nextAgentSpan, nextToolSpan, nextRetrieverSpan) stage metrics onto the next matching component.

import { EvaluationDataset, Golden } from "deepeval/dataset";
import { TaskCompletionMetric } from "deepeval/metrics";

const dataset = new EvaluationDataset({
  goldens: [new Golden({ input: "What's the weather in Tokyo?" })],
});

for await (const golden of dataset.evalsIterator({
  metrics: [new TaskCompletionMetric()],
})) {
  await myAgent(golden.input);
}

Then replay the trace tree in your terminal with npx deepeval inspect.

If you use a framework, you don't need observe() at all — register the integration and its spans become the trace. Only the setup line differs; everything above stays the same.

| Framework | Setup | | ------------------------------------------------------------------------------- | ------------------------------------------------------------ | | OpenAI | instrumentOpenAI(client) from deepeval/openai | | LangChain / LangGraph | new DeepEvalCallbackHandler({}) passed as a callback | | OpenAI Agents | setTraceProcessors([new DeepEvalTracingProcessor()]) | | Mastra | new DeepEvalExporter() as an observability exporter | | AI SDK | configureAiSdkTracing() as experimental_telemetry.tracer | | OpenInference | instrumentOpenInference() |

Each lives at deepeval/integrations/<name>. AI SDK and OpenInference need isTestMode: true to reach evalsIterator.

Choosing a Judge Model

Metrics judge with OpenAI unless told otherwise, so OPENAI_API_KEY is all most people need. To switch, pass model to any metric or set a default from the CLI:

npx deepeval set-anthropic --model claude-opus-5

Anthropic, Azure OpenAI, Gemini, Bedrock, Ollama, DeepSeek, Grok, Kimi, OpenRouter, Portkey, the Vercel AI SDK, and any OpenAI-compatible endpoint are supported — see the model docs.

Python vs TypeScript

Nearly every metric has been ported, along with tracing, datasets, prompts, the CLI, and the Confident AI integration, so the day-to-day workflow is the same in both languages. TypeScript additionally has the Vitest toPass() matcher in place of pytest's assert_test, the npx deepeval inspect trace viewer, and Mastra and AI SDK integrations that Python doesn't have.

Still Python-only:

• Synthesizer — synthetic dataset generation. Write goldens by hand or pull them from Confident AI instead.
• Benchmarks — MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, GSM8K.
• Red teaming and prompt optimization.
• RAGAS metrics, plus AgentLoopDetectionMetric and ToolPermissionMetric.
• Integrations for CrewAI, LlamaIndex, Pydantic AI, Google ADK, AWS AgentCore, Strands, and the Anthropic client.

Both SDKs live in confident-ai/deepeval, where you'll find the Python package, contributing guide, license, and community links.