llmtester

An interactive CLI tool for benchmarking LLMs across multiple benchmarks. Run via npx without installing.

Features

• Interactive CLI - Keyboard-driven benchmark selection and configuration
• Multi-Provider Support - OpenAI, Anthropic, Together.ai, Groq, Fireworks AI, Perplexity, OpenRouter, and any OpenAI-compatible API
• LLM-as-Judge - Optional secondary model evaluation for code, math, SQL, bash, and truthfulness benchmarks
• Progress Tracking - Resume interrupted evaluations from where you left off
• Result Explorer - Built-in TUI to browse past results, filter by pass/fail, and inspect individual responses
• Config Persistence - Saves provider, endpoint, and model settings between runs
• Shuffle & Sampling - Run a percentage of each benchmark with optional shuffling for diverse distribution

Screenshots

Select benchmarks to run

Progress during benchmark run

Browse past benchmark runs

Run a specific benchmark

Inspect individual test results

Supported Benchmarks

| Benchmark | Tests | Description | Judge | |-----------|-------|-------------|-------| | GSM8K | 1,319 | Grade School Math | | | MATH | 12,500 | Competition Math | Yes | | BIG-Bench Hard | 6,511 | 23 Challenging Tasks | | | ARC-Challenge | 1,172 | Advanced Reasoning Challenge | | | HellaSwag | 10,042 | Commonsense Reasoning | | | MMLU | 14,042 | Multitask Language Understanding | | | HumanEval | 164 | Python Code Generation | Yes | | MBPP | 500 | Mostly Basic Programming Problems | Yes | | APPS | 5,000 | Programming Progress System | Yes | | TypeScript (MultiPL-E) | 161 | TypeScript Code Generation | | | NL2Bash | 24 | Natural Language to Bash | Yes | | Spider | 1,034 | Text-to-SQL Generation | Yes | | TruthfulQA | 817 | Truthfulness Evaluation | Yes |

Installation

# Run directly with npx
npx llmtester

# Or clone and run locally
git clone https://github.com/officiallymarky/llmtester
cd llmtester
npm install
npm run build
npm start

Configuration

The app prompts for all configuration on first run and saves it to a config file (~/.config/llmtester/config.json on Linux). You can edit this file manually to change settings:

{
  "provider": "openai",
  "apiKey": "your_api_key",
  "baseUrl": "https://api.openai.com/v1",
  "modelName": "gpt-4o"
}

LLM-as-Judge

Some benchmarks use a secondary LLM ("judge") to evaluate responses instead of exact matching. This is necessary for:

• Code generation (HumanEval, MBPP, APPS) - since many code solutions can be functionally correct
• Math (MATH) - since multiple answer formats can represent the same solution
• SQL (Spider) - since multiple SQL queries can return the same result
• Truthfulness (TruthfulQA) - requires semantic evaluation

Recommended judge models:

• gpt-4o-mini - Fast and cost-effective, good quality
• gpt-4o - Higher quality for more accurate evaluation
• claude-sonnet-4-20250514 - Good alternative from Anthropic
• deepseek-ai/DeepSeek-V3 - Cost-effective option

The judge is configured separately from the main model when you run benchmarks that require one.

Usage

npx llmtester

1. Enter your API key
2. Select or enter the endpoint URL
3. Enter the model name
4. Select benchmarks to run (multi-select with Space, confirm with Enter)
5. Set percentage to run per benchmark (1-100)
6. Choose whether to shuffle samples
7. If any selected benchmark uses a judge, configure the judge model
8. Review configuration summary and confirm to start

Preset Endpoints

| Provider | URL | |----------|-----| | OpenAI | https://api.openai.com/v1 | | Together.ai | https://api.together.xyz/v1 | | Groq | https://api.groq.com/openai/v1 | | Fireworks AI | https://api.fireworks.ai/inference/v1 | | Perplexity | https://api.perplexity.ai | | OpenRouter | https://openrouter.ai/api/v1 |

Output

Results are stored in platform-specific application data directories:

| Platform | Data Directory | |----------|---------------| | Linux | ~/.local/share/llmtester/ | | macOS | ~/Library/Application Support/llmtester/ | | Windows | %APPDATA%/llmtester/ |

Within the data directory:

• results/eval_results_{timestamp}.json - Aggregate results per benchmark run
• detailed_logs/*.jsonl - Per-question logs (question, response, correctness, judge feedback)
• eval_progress/ - Resume files for interrupted runs

Config Directory

Config is stored in the platform config directory:

| Platform | Config Directory | |----------|-----------------| | Linux | ~/.config/llmtester/ | | macOS | ~/Library/Application Support/llmtester/ | | Windows | %APPDATA%/llmtester/ |

Development

npm install
npm run build
npm start

License

MIT