@unotest/judge

LLM-judge service for the @unotest ecosystem. It judges free-form text (a chat reply, generated content) against a natural-language rubric and returns a structured verdict:

{ "verdict": "fail", "reasoning": "the reply promises escalation", "model": "gemini-2.5-flash", "attempts": 2 }

It backs the assertJudge(locator, rubric) assertion of @unotest/web — semantic checks where substring/regex assertions are too brittle for LLM output.

Run

UNOTEST_JUDGE_PROVIDER=vertex \
GOOGLE_CLOUD_PROJECT=my-project \
GOOGLE_CLOUD_LOCATION=us-central1 \
npx @unotest/judge

On start the service checks the provider's credentials (an ADC token exchange, a free GET /models, a claude --version) and refuses to listen if they are dead — a judge that cannot judge should not report itself as ready. npx @unotest/judge --check runs only that check and exits 0/1, for start scripts and CI.

UNOTEST_JUDGE_SKIP_PREFLIGHT=1 waives the startup gate — the service warns and listens anyway. It does not waive the truth: /health keeps probing and keeps reporting the failure, and --check still answers honestly. A green light on a judge that cannot judge is the fault this whole check exists to remove.

The service listens on 127.0.0.1:8790 by default. Point @unotest/web at it with UNOTEST_JUDGE_MODE=remote + UNOTEST_JUDGE_URL=http://127.0.0.1:8790, or skip the HTTP hop entirely with UNOTEST_JUDGE_MODE=local (in-process, requires @unotest/judge installed in the project — a project that runs unotest through npx, with nothing installed locally, can only use remote).

Providers

• vertex — Google Vertex AI via Application Default Credentials. No API keys: auth comes from ADC (gcloud auth application-default login, workload identity, or GOOGLE_APPLICATION_CREDENTIALS). Requires the google-auth-library (a dependency of this package, loaded only when this provider runs) — or a static UNOTEST_JUDGE_ACCESS_TOKEN from gcloud auth print-access-token (those live about an hour and are not refreshed for you — the preflight validates one against Google's free tokeninfo endpoint, so an expired token is caught at startup rather than mid-run). Temperature is pinned to 0.

  Location and model are chosen together. GOOGLE_CLOUD_LOCATION is either global or a region (us-central1), and model availability differs between them — gemini-2.5-flash is regional, newer flash-lite models may be global-only. A model id containing a slash is used as-is (publishers/anthropic/models/claude-haiku-4-5), which is how Vertex addresses third-party models.

• claude — the local Claude Code CLI (claude -p), spawned per judgement without shell interpretation. Auth comes from your Claude Code session, so a subscription works with no API key — this is the provider to reach for when no cloud credentials are at hand. UNOTEST_JUDGE_MODEL is passed as --model (aliases like sonnet work); unset uses the CLI's default. Budget for latency, not for tokens: every verdict spawns the CLI and loads a session, which measured ~6s per call here against ~1s for an HTTP provider — 20 assertions is a couple of minutes. UNOTEST_JUDGE_VOTE costs almost nothing extra (ballots run concurrently: 3 votes measured 6.8s), and the per-call budget defaults to 120s for this provider for the same reason. The prompt goes in on stdin, so a large judged text is fine.

• gemini — the Gemini API with GEMINI_API_KEY. Same wire format as vertex, temperature pinned to 0.

• openai — OpenAI chat completions with OPENAI_API_KEY (response_format: json_object). No sampling params: reasoning models reject temperature, so determinism relies on the strict verdict prompt plus the retry policy.

• anthropic — the Anthropic API (/v1/messages) with ANTHROPIC_API_KEY. No sampling params, same reasoning as openai.

• fake — deterministic, no model. For CI and smoke-testing the wiring. The rubric is a micro-grammar, one constraint per line: must contain: <substring> / must not contain: <substring> (case-insensitive). Anything else fails loudly. Note that a UNOTEST_JUDGE_PREAMBLE set on the client side arrives glued to the rubric, so constraint lines in a preamble apply to every judgement.

All model providers are raw HTTP or a local process — this package ships zero provider SDKs.

Latency

A verdict costs wall-clock time in the scenario that waits for it, so the choice of provider is a choice of pace:

| Provider | Per verdict | Why | | --- | --- | --- | | vertex / gemini / openai / anthropic | one HTTP round-trip (~1s for a flash-class model) | a single JSON POST | | claude | ~6s, measured on a local CLI | every call spawns the binary and loads a session | | fake | none | no model behind it |

UNOTEST_JUDGE_VOTE=N costs almost nothing on top: the N ballots run concurrently, so three votes take about as long as one call (measured 6.8s for claude). UNOTEST_JUDGE_RETRIES, by contrast, is sequential — it only spends a call when a verdict came back fail.

Budgets follow the same shape: UNOTEST_JUDGE_CALL_TIMEOUT_MS defaults to 30s, and to 120s for claude. On the client side @unotest/web has its own whole-request budget (UNOTEST_JUDGE_TIMEOUT_MS, 60s) — with claude plus a slow model the client can give up before the service does, and it now says so when that happens.

Env

| Variable | Meaning | Default | | --- | --- | --- | | UNOTEST_JUDGE_PROVIDER | fake | vertex | claude | gemini | openai | anthropic | required | | UNOTEST_JUDGE_MODEL | model id | vertex/gemini gemini-2.5-flash, openai gpt-5-mini, anthropic claude-haiku-4-5, claude: CLI default | | UNOTEST_JUDGE_RETRIES | extra provider calls on fail; first pass wins | 1 | | UNOTEST_JUDGE_VOTE | odd N; N independent calls, the majority decides. Replaces retries rather than stacking with them | 1 (off) | | UNOTEST_JUDGE_CALL_TIMEOUT_MS | budget for ONE provider call, ms | 30000 (120000 for claude) | | GOOGLE_CLOUD_PROJECT | vertex: ADC project | required for vertex | | GOOGLE_CLOUD_LOCATION | vertex: global or a region (us-central1) | required for vertex | | UNOTEST_JUDGE_ACCESS_TOKEN | vertex: static bearer override (skips ADC) | — | | GEMINI_API_KEY | gemini: API key | required for gemini | | OPENAI_API_KEY | openai: API key | required for openai | | ANTHROPIC_API_KEY | anthropic: API key | required for anthropic | | UNOTEST_JUDGE_CLAUDE_BIN | claude: binary override | claude | | UNOTEST_JUDGE_HOST / UNOTEST_JUDGE_PORT | bind address | 127.0.0.1 / 8790 | | UNOTEST_JUDGE_TOKEN | bearer token required on every /judge call. @unotest/web reads the SAME name to send it | off | | UNOTEST_JUDGE_LOG_LEVEL | silent | error | warn | info | debug; falls back to UNOTEST_LOG_LEVEL | info | | UNOTEST_JUDGE_SKIP_PREFLIGHT | 1 starts even when the credential check fails (/health still reports it) | off |

Logging

At the default info level the service prints one line per judged request — verdict, attempts spent, model, latency:

POST /judge  pass  attempts 3  gemini-2.5-flash  1284ms
POST /judge  ERROR config  Application Default Credentials need re-authentication (invalid_rapt) — run `gcloud auth application-default login`

Errors go to stderr. A run with 21 assertions and UNOTEST_JUDGE_VOTE=3 is 63 billed model calls; this is the only place their cost and latency are visible.

UNOTEST_JUDGE_LOG_LEVEL=debug additionally prints, per call, the effective prompt (rubric including any UNOTEST_JUDGE_PREAMBLE the client prepended), each ballot with its reasoning, and the provider's raw reply.

The judged text is application content — support answers, user data, whatever the scenario put under the assertion. Debug logging writes it to stdout verbatim. It is opt-in for that reason; do not enable it on a shared runner without deciding that is acceptable.

HTTP API

• POST /judge — body {"rubric": "...", "text": "..."} → a JudgeVerdict (above). Errors: {"error": "...", "code": "unauthorized" | "bad-request" | "provider-error" | "config" | "internal"}. config means the operator must fix this service's setup (bad env, expired ADC) and the message says what to run.
• GET /health — {"ok": true} when the provider's credentials work, otherwise 503 + {"ok": false, "code": "config", "error": "..."} with the same message and code a verdict call would have failed with. The check is cached for a few seconds, so polling is cheap, and it never calls the model.

Wire types are shared through @unotest/protocol (JudgeRequest, JudgeVerdict, JudgeHealthResponse, JUDGE_ROUTES), so the service and @unotest/web's client cannot drift.

Faults that nobody chose are re-sent: rate limits (429), backend 5xx, a dropped connection, a spawn the OS could not grant. Up to three attempts, 500ms then 1500ms apart, Retry-After honoured over that curve when the backend names one. The policy sits on the provider contract, so it covers every provider — including claude, which speaks to a process rather than to HTTP.

What is never re-sent: a rejected credential (401/403) comes back as config with the variable to fix, an unknown model is not a hiccup, and a claude usage limit is reported rather than retried — the retry window is seconds while a subscription limit lasts far longer, so re-spawning a ~6s CLI would only make the same failure slower.

With UNOTEST_JUDGE_VOTE=N, a ballot that still errors is dropped rather than failing the request, as long as a majority of the N calls voted.

License

MIT © Ivan Volkov