DeepEval.ts

TypeScript client for Confident AI's DeepEval API - a framework for evaluating and testing Large Language Models (LLMs).

Installation

npm install deepeval-ts

Authentication

DeepEval.ts requires a Confident AI API key to authenticate with the service. You can set up your API key in one of the following ways:

Option 1: Environment Variables

Set the CONFIDENT_API_KEY environment variable:

# In your terminal
export CONFIDENT_API_KEY="your-api-key-here"

# Or for Windows
set CONFIDENT_API_KEY=your-api-key-here

Option 2: .env File

Create a .env file in your project root:

# .env file
CONFIDENT_API_KEY="your-api-key-here"

Then use a package like dotenv to load it:

npm install dotenv

// At the top of your entry file
import 'dotenv/config';

Option 3: Pass API Key Directly

You can also pass your API key directly when creating an API instance:

import { Api } from 'deepeval-ts';

const api = new Api("your-api-key-here");

Usage Examples

Working with Datasets

import { EvaluationDataset, LLMTestCase } from 'deepeval-ts';
import * as path from 'path';

// Load dataset from CSV
const dataset = new EvaluationDataset();
await dataset.addTestCasesFromCsvFile(
  'path/to/dataset.csv',
  'input_column',
  'actual_output_column',
  'expected_output_column'
);

// Create dataset programmatically
const customDataset = new EvaluationDataset();
customDataset.addTestCase(
  new LLMTestCase({
    input: "What is the capital of France?",
    actualOutput: "Paris is the capital of France.",
    expectedOutput: "Paris"
  })
);

// Iterate through test cases
for (const testCase of dataset.testCases) {
  console.log(`Input: ${testCase.input}`);
  console.log(`Output: ${testCase.actualOutput}`);
}

API Reference

EvaluationDataset

The EvaluationDataset class manages collections of test cases for LLM evaluation.

// Create a new dataset
const dataset = new EvaluationDataset();

// Add test cases from CSV
await dataset.addTestCasesFromCsvFile(
  filePath,           // Path to CSV file
  inputColumn,        // Name of input column
  actualOutputColumn, // Name of actual output column
  expectedOutputColumn, // Name of expected output column (optional)
  contextColumn,      // Name of context column (optional)
  contextDelimiter,   // Delimiter for context values (optional)
  retrievalContextColumn, // Name of retrieval context column (optional)
  retrievalContextDelimiter // Delimiter for retrieval context values (optional)
);

// Add a test case programmatically
dataset.addTestCase(
  new LLMTestCase({
    input: "What is the capital of France?",
    actualOutput: "Paris is the capital of France.",
    expectedOutput: "Paris",
    context: ["France is a country in Europe.", "Paris is a city."],
    retrievalContext: ["Paris is the capital and most populous city of France."]
  })
);

LLMTestCase

The LLMTestCase class represents individual test cases for LLM evaluation.

const testCase = new LLMTestCase({
  input: "What is the capital of France?",
  actualOutput: "Paris is the capital of France.",
  expectedOutput: "Paris",
  context: ["France is a country in Europe.", "Paris is a city."],
  retrievalContext: ["Paris is the capital and most populous city of France."],
  toolCalls: [
    {
      name: "search",
      input: { query: "capital of France" },
      output: { result: "Paris is the capital of France" }
    }
  ]
});

Development

To build the package locally:

npm run build

To run tests:

npm test

License

MIT