Note: This package is a fork of @evalkit/core with added support for Ollama and other OpenAI-compatible providers.

🚀 Features, Metrics and Docs

Click here to navigate to the Official EvalKit Documentation

In the documentation, you can find information on how to use EvalKit, its architecture, including tutorials and recipes for various use cases and LLM providers.

Looking for a metric/feature that's not listed here? Open an issue and let us know!

Getting Started - Quickstart

Prerequisites

• Node.js 18+
• OpenAI API Key

Installation

EvalKit currently exports a core package that includes all evaluation related functionalities. Install the package by running the following command:

npm install --save-dev evalkit-ollama

Configuration

EvalKit supports multiple LLM providers through the configure() function. You can configure your preferred provider before running evaluations.

OpenAI

import { configure, evaluate, RelevancyMetric } from 'evalkit-ollama';

configure({
  apiKey: 'your-openai-api-key',
  model: 'gpt-4o-mini',  // optional, defaults to 'gpt-4o-mini'
});

const result = await evaluate(
  { input: 'What is the capital of France?', output: 'Paris is the capital of France.' },
  [RelevancyMetric]
);

Ollama (Local LLM)

EvalKit supports Ollama and other OpenAI-compatible providers:

import { configure, evaluate, RelevancyMetric } from 'evalkit-ollama';

configure({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama',  // Ollama doesn't require a real key
  model: 'llama3.2',
  embeddingModel: 'nomic-embed-text',  // for SemanticSimilarityMetric
});

const result = await evaluate(
  { input: 'What is the capital of France?', output: 'Paris is the capital of France.' },
  [RelevancyMetric]
);

Configuration Options

| Option | Description | Default | |--------|-------------|---------| | apiKey | API key for the LLM provider | - | | baseURL | Custom API endpoint URL | - | | model | Model name for chat completions | gpt-4o-mini | | embeddingModel | Model name for embeddings | text-embedding-ada-002 |

License

This repository's source code is available under the Apache 2.0 License.