@altairalabs/promptarena

PromptKit Arena - Multi-turn conversation simulation and testing tool for LLM applications

Installation

npx (No Installation Required)

npx @altairalabs/promptarena run -c ./examples/customer-support

Global Installation

npm install -g @altairalabs/promptarena

# Use directly
promptarena --version
promptarena run -c ./config

Project Dev Dependency

npm install --save-dev @altairalabs/promptarena

# Use via npm scripts
# Add to package.json:
{
  "scripts": {
    "test:prompts": "promptarena run -c ./tests/arena-config"
  }
}

Zero dependencies. The package ships a single, statically-linked, pure-Go binary for your platform. run, view, chat (text), reports — all work straight from npm install with no native libraries to install.

Voice mode (optional)

Hands-free voice (promptarena chat --voice) is the only feature with an extra requirement: it needs PortAudio installed on your machine for microphone capture. The binary loads PortAudio at runtime, on demand — so:

• You do not need PortAudio to install or to use any non-voice feature.
• If you run chat --voice without PortAudio, only voice is unavailable — you get a clear message with install instructions, and everything else keeps working.

Install PortAudio only if you want voice:

| OS | Install | |----|---------| | macOS | brew install portaudio | | Debian/Ubuntu | sudo apt install libportaudio2 | | Windows | place portaudio.dll on your PATH |

# Text chat — no PortAudio needed
promptarena chat

# Voice chat — needs PortAudio installed (see table above)
promptarena chat --voice

See the Voice Console guide for setup details.

What is PromptKit Arena?

PromptKit Arena is a comprehensive testing framework for LLM-based applications. It allows you to:

• 🎯 Test conversations across multiple LLM providers (OpenAI, Anthropic, Google, Azure)
• 🔄 Run multi-turn simulations with automated agent interactions
• ✅ Validate outputs using assertions and quality metrics
• 📊 Generate reports with detailed analysis and comparisons
• 🛡️ Test guardrails and safety measures
• 🔧 Validate tool usage and function calling

Quick Start

Get started in under 2 minutes:

# Create a new project from a template
npx @altairalabs/promptarena init my-test --quick

# Navigate to your project
cd my-test

# Set your API key (or use mock provider for testing)
export OPENAI_API_KEY=your-key-here

# Run your first test
npx @altairalabs/promptarena run

# View the Markdown report
open out/results.md

That's it! The template includes pre-configured scenarios, assertions, and examples to get you started.

Browse Available Templates

# List all available templates
npx @altairalabs/promptarena templates list

# Create from a specific template
npx @altairalabs/promptarena init my-project --template community/iot-maintenance-demo

# Interactive mode (choose template, provider, etc.)
npx @altairalabs/promptarena init

Key Features

• 🎯 Multi-Provider Testing - Compare OpenAI, Anthropic, Google, and Azure side-by-side
• 🔄 Self-Play Mode - AI agents simulate realistic user conversations with personas
• ✅ Turn-Level Assertions - Validate individual responses (content, tone, length, JSON)
• 📊 Conversation Assertions - Check patterns across entire conversations
• 🎭 Template & Persona System - Dynamic prompts with variables and reusable personas
• 🛡️ Guardrail Testing - Ensure tools and responses follow safety constraints
• 📈 Reports & Web UI - Markdown reports for archiving, promptarena serve for interactive review with cost tracking and metrics

Learn More

Assertion Types

• Turn-Level: content_includes, content_matches, json_schema, jsonpath, llm_judge, tone, length
• Conversation-Level: llm_judge_conversation, tools_not_called_with_args, max_tool_calls

See the Assertions Reference for examples and best practices.

Documentation

• Full Documentation - Comprehensive guides and tutorials
• Configuration Reference - Complete schema documentation
• Examples - Working examples:
  • Assertions Test - Turn and conversation-level assertions
  • Customer Support - Self-play with personas
  • Variables Demo - Template rendering
  • LLM Judge - AI-powered evaluation
• Multi-Turn Tutorial - Self-play patterns

License

Apache-2.0 - see LICENSE

Contributing

Contributions welcome! See CONTRIBUTING.md

Support

• GitHub Issues
• Discussions