@ai-test-harness/core

Core framework for AI quality testing with AWS Bedrock.

Installation

npm install @ai-test-harness/core

Usage

Programmatic API

import { runTests, LLMClientFactory, MockAdapter } from '@ai-test-harness/core';

// Run tests for a project
const result = await runTests({
  projectPath: './my-tests',
  configFile: 'config.yaml',
});

// Use LLM client directly
const client = LLMClientFactory.create({
  provider: 'bedrock',
  region: 'us-east-2',
});

const response = await client.chat({
  model: 'us.anthropic.claude-3-5-sonnet-20240620-v1:0',
  messages: [{ role: 'user', content: 'Hello!' }],
  temperature: 0.7,
  maxTokens: 1000,
});

// Unit testing with mock adapter
const mockClient = new MockAdapter();
mockClient.setDefaultResponse('Test response');

Exports

Test Runners

• TestRunner - For text-generation tests
• AgentTestRunner - For agent scenario tests
• AgentSimulationRunner - For dynamic agent simulation

LLM Components

• LLMJudge - Evaluates outputs against quality criteria
• LLMGenerator - Generates text from prompts
• UserSimulator - Simulates user responses

LLM Client Abstraction

• ILLMClient - Interface for LLM clients
• LLMClientFactory - Factory for creating clients
• BedrockAdapter - AWS Bedrock implementation
• LiteLLMAdapter - LiteLLM HTTP implementation
• MockAdapter - Testing mock

Quality Library

• registerAttribute() - Register custom quality attributes
• generateAssessmentPrompt() - Generate evaluation prompts
• buildQualitySchema() - Build Zod schemas for validation

Utilities

• Logger - Structured logging
• ConfigLoader - Load YAML configurations
• retryWithBackoff() - Resilient API calls

Environment Variables

# LLM Provider (default: bedrock)
LLM_PROVIDER=bedrock

# For LiteLLM integration
LITELLM_URL=https://litellm.company.com
LITELLM_API_KEY=sk-xxx

# AWS configuration
AWS_REGION=us-east-2

Documentation

For complete documentation, see the main repository.

Contributing

See CONTRIBUTING.md in the main repository.

License

MIT © AI Test Harness Contributors