axo-eval

Measure how accurately AI models answer real questions about your product's docs — cold (no context) vs. grounded in the site's llms.txt — and see exactly what they get wrong.

npx axo-eval@latest run https://hono.dev --mock

--mock uses a deterministic mock model — no LLM calls, no API keys — the fastest way to see the output shape. It still fetches the docs URL over the network, so point it at a real site. Drop --mock and set the env vars below to run against a real model.

What it does

1. Ingests the site (sitemap + pages, plus llms.txt if present).
2. Derives real questions from the docs.
3. Answers each question twice — once cold, once grounded in the docs.
4. Judges both with an LLM-as-judge and scores accuracy.
5. Writes report.html, report.json, badge.svg, summary.md, and questions-audit.json (the derivation record, for debugging a run — versioned by auditSchemaVersion).

Usage

axo-eval run <url> [options]

Common options:

| Option | Description | |---|---| | --mock | Deterministic mock model — no LLM calls, no keys (still fetches the URL). | | --questions <n> | Max questions to derive (default 12). | | --out <dir> | Output directory (default ./axo-eval-report). | | --min-score <0-100> | Exit non-zero if grounded accuracy is below this (for CI). | | --json | Print report JSON to stdout instead of a summary. | | --preset local-small | Flag bundle tuned for a ~4k-context local model (Ollama gemma etc.); explicit flags still win. |

Run axo-eval --help for the full list (there is no --version flag).

Configuration (real runs)

All LLM calls speak the OpenAI-compatible open standard — point it at any compatible endpoint:

| Env var | Purpose | |---|---| | LLM_BASE_URL | e.g. https://api.openai.com/v1, https://openrouter.ai/api/v1, http://localhost:11434/v1 (Ollama) | | LLM_API_KEY | Provider API key (optional for keyless local endpoints). | | LLM_MODEL | Default model for all roles. | | LLM_TARGET_MODEL / LLM_JUDGE_MODEL / LLM_QUESTION_MODEL | Per-role overrides. |

LLM_API_KEY and LLM_BASE_URL fall back to OPENAI_API_KEY / OPENAI_BASE_URL when unset, so an existing OpenAI key works with no extra setup.

LLM_MODEL=gpt-4.1 LLM_API_KEY=sk-... axo-eval run https://docs.example.com

License

MIT © Nihith Sistla