dsh-rerun-grader-judge

Grader provider scoring an attempt with a separately pinned judge model against a fixed rubric.

This is the least trustworthy grader in the box and is built to admit it. Reach for command, file, and invariant first; use this only where the question genuinely has no objective form.

Three properties keep it from quietly corrupting a trend:

• The judge route is pinned in THIS plugin's config, never taken from the variant. The harness under test must not grade itself, and two variants must not be scored by two different judges.
• Its identity is stamped into every verdict's detail, alongside the vote spread — a wide spread means the rubric is ambiguous, not that the attempt was borderline.
• k votes, reduced by median. One sample from a stochastic grader is a coin flip wearing a number.

Off by default. When you enable it, keep a small human-labelled subset and check agreement periodically; when agreement drops, every score it produced is void.

- id: rerun-grader-judge
  name: 'dsh-rerun-grader-judge'
  config:
    provider: deepseek-official
    model: deepseek-v4-pro
    votes: 3
    redact: ['sk-[A-Za-z0-9]{16,}']

Model Experience

This package DOES send model requests — the only one in rerun that does. They go to a route pinned in its own config, entirely separate from the variants' routes, and carry the rubric, the task prompt, the attempt's tool sequence, and its workspace diff. Nothing it sends reaches the harness under test.

It is also the one place rerun hands workspace contents to a third party; redact patterns are applied to everything before it leaves the process.

KV Cache effect

None on the harness under test. Its own requests share a constant system prompt prefix, so a provider that caches will see the rubric-and-instructions prefix repeat across votes.

Known Limitations and Deferred Work

• The judge drifts. Nothing in this package detects it. Human-agreement sampling is a documented discipline, not a mechanism — the deferred work is a judge-agreement check that fails a run when agreement drops below a threshold.
• No redaction rules ship by default. A rule the package guessed at would give false confidence; the deployment knows what its own tasks contain.
• A failed judge call scores zero, which is indistinguishable in the aggregate from a genuinely bad attempt. The detail says which, but the number does not.