kalmia-optimize

Threshold-driven prompt optimization over your own Kalmia-graded eval suite.

One command in your agent's repo: run your eval suite, let Kalmia grade it, rewrite your prompt file from the graded diagnoses, and repeat - until the pass rate meets your threshold with zero regressions, or the iteration cap runs out. The loop is deterministic code; the only LLM step is the prompt rewrite.

Requirements

• Node 20+
• A Kalmia workspace API key (kal_live_...)
• An Anthropic API key (sk-ant-...) — the prompt rewriter calls Claude with it

Your agent can be written in any language — the optimizer drives your eval command, it does not import your code.

Installation

Run it with no install via npx:

npx kalmia-optimize --eval-cmd "..." --prompt-file system_prompt.txt --threshold 0.9

Or install the command globally:

npm install -g kalmia-optimize
kalmia-optimize --eval-cmd "..." --prompt-file system_prompt.txt --threshold 0.9

Quickstart

export KALMIA_API_KEY=kal_live_...     # workspace API key
export ANTHROPIC_API_KEY=sk-ant-...    # the rewriter calls Claude with your key
export KALMIA_URL=https://your-kalmia-host

kalmia-optimize \
  --eval-cmd "node run-evals.mjs" \
  --prompt-file system_prompt.txt \
  --threshold 0.9

What it does each session

1. Preflight: requires a git repo with a clean tree, verifies both API keys with real calls, and records your starting branch.
2. Baseline (iteration 0): runs your eval command once with no edits. The cases Kalmia grades become the session's expected case set.
3. Loop (up to --max-iterations, default 5): reads the failed criteria, evidence, and diagnoses from Kalmia; one structured Claude call rewrites your prompt file; the variant is committed on a session branch (optimize/<timestamp>); your eval command runs again; Kalmia re-grades.
4. Stop: on success (pass rate >= threshold AND zero regressions AND every expected case definitively scored), at the iteration cap, or when nothing prompt-fixable remains.
5. End state: always checked out on the best variant (your baseline if nothing improved), with a per-iteration report and restore instructions. Your original branch is never touched.

The contract with your eval command

Your eval command must tag its traces with the run identity the CLI hands it:

• EVAL_RUN_ID - use as eval.run_id on every trace of the run
• EVAL_VARIANT - use as eval.variant

This is the same convention the Kalmia evals guide documents. If your runner generates its own run id, the CLI cannot attribute runs and the wait will time out with a pointed message.

Flags and environment

| Flag / env | Required | Meaning | |---|---|---| | --eval-cmd <cmd> | yes | Command that runs your eval suite (executed via shell) | | --prompt-file <path> | yes | The one file the rewriter may edit | | --threshold <rate> | yes | Target pass rate in (0, 1], e.g. 0.9 | | --max-iterations <n> | no | Improvement-iteration cap (default 5; baseline not counted) | | --model <id> | no | Rewriter model (default claude-opus-4-8) | | --kalmia-url <url> / KALMIA_URL | yes | Kalmia base URL | | KALMIA_API_KEY | yes | Workspace API key (kal_live_...) | | ANTHROPIC_API_KEY | yes | Key for the rewriter call | | KALMIA_CRON_SECRET | dev only | Opt-in: lets the CLI nudge a local Kalmia's scoring jobs while it waits. Never needed against production |

Exit codes

| Code | Meaning | |---|---| | 0 | Success: threshold met with zero regressions | | 1 | Ended below threshold (cap reached, or nothing prompt-fixable) | | 2 | Preflight or configuration error - nothing was run | | 3 | Aborted mid-session (eval failure, grading timeout, rewriter failure, Ctrl-C); best variant so far is checked out |

Try it end-to-end

examples/eval-demo-agent/ is a self-contained review agent (deliberately bad by default) plus a step-by-step guide. Point the optimizer at it against a real Kalmia and watch it go from ~50% to a clean, zero-regression pass. See examples/eval-demo-agent/README.md.

Guarantees and limits

• Variant commits are pathspec-limited to your prompt file and bypass commit hooks, so exactly the evaluated bytes are committed - nothing else.
• If your eval command itself modifies the prompt file, the session aborts loudly rather than committing content that was never evaluated.
• A case that regresses, goes unresolved (needs-attention), or goes missing blocks success - a partial run can never fake a win.
• Failures diagnosed as tool/code issues are reported as out of reach; the CLI never pretends a prompt edit fixes them.
• v1 edits prompt files only, runs interactively only, and does not resume crashed sessions (the session branch remains for manual inspection).
• Judge scoring is not perfectly deterministic; the strict zero-regression bar means a flaky case can keep a session below threshold. Review flips in the Kalmia dashboard.