@calculator53295/knowledge-arena

Zero-dependency benchmark runner for local models across offline retrieval strategies and approved public database schemas. It consumes the local-ai-lab plugin API and local Ollama only; no remote provider is configured.

knowledge-arena plan benchmarks/knowledge/matrix.json benchmarks/knowledge/tasks.json
knowledge-arena probe benchmarks/knowledge/tasks.json \
  benchmarks/knowledge/results/corpus-v1.probes.json --execute
knowledge-arena run benchmarks/knowledge/matrix.json benchmarks/knowledge/tasks.json \
  benchmarks/knowledge/results/round-001.trials.json --execute
knowledge-arena score benchmarks/knowledge/results/round-001.trials.json \
  benchmarks/knowledge/results/round-001.scorecard.json

run requires an explicit --execute and refuses to overwrite evidence. Scoring remains disaggregated: source hit (40), expected-term recall (35), and citation validity (25), plus separately measured retrieval, generation, and end-to-end latency. Database trials use only public schema endpoints. Private and restricted aliases are absent from the corpus and cannot be selected. Retrieved passage bodies are transient and never written to evidence. Stored URIs remove the user home prefix, while answer text redacts email addresses and dollar-denominated figures before persistence.

Matrices may set repetitions (1–10) and an integer seed. Each repetition uses temperature: 0 and a stable incremented seed. Scorecards include 95% normal-approximation intervals for model, strategy, dataset, and exact-cell groups; small samples must be interpreted cautiously.