@credence/judge

When two sources disagree, is it the same fact written two ways, or a real contradiction? That question is what stands between a useful conflict signal and an alert list nobody reads.

import { createJudge } from "@credence/judge";

const judge = createJudge(ledger, rules, {
  families: [
    {
      id: "company-naming",
      keys: ["legal_name"],
      equivalence: {
        caseInsensitive: true,
        collapseWhitespace: true,
        stripPunctuation: true,
        aliases: [["inc", "incorporated"], ["ltda", "limited"]],
      },
    },
    { id: "money-rounding", keys: ["total_debt"], equivalence: { numericRelativeTolerance: 0.01 } },
  ],
});

await judge.review(caseId);
// "Acme Inc." vs "ACME Incorporated"  → dismissed (benign variance)
// 900,000    vs 1,400,000             → upheld    (real disagreement)

Deterministic first, LLM only if needed

Conflict families are policy in data: a declaration of when differing values still mean the same thing (case, whitespace, punctuation, alias groups, numeric tolerance, date granularity). They settle the common case with no model, no API key, and no network — which makes the judge cheap, fast, reproducible, and auditable.

An adjudicator is optional and consulted only when the families can't settle it. Two real ones ship in the box, behind their own subpath so the deterministic path never drags a vendor SDK into your install:

import { createJudge } from "@credence/judge";
import {
  createAnthropicAdjudicator,
  createDeepSeekAdjudicator,
  createOpenAICompatibleAdjudicator,
} from "@credence/judge/adjudicators";

createAnthropicAdjudicator();                    // claude-opus-4-8, adaptive thinking
createDeepSeekAdjudicator();                     // deepseek-v4-pro
createOpenAICompatibleAdjudicator({              // OpenRouter, Groq, vLLM, Ollama…
  baseURL: "http://localhost:11434/v1",
  model: "qwen3:32b",
});

const judge = createJudge(ledger, rules, { families, adjudicator });

The seam is a one-line function type, so a provider is a sibling file, never a fork of the judge:

type Adjudicator = (request: AdjudicationRequest) => Promise<Omit<Verdict, "source"> | null>;

Both ship adjudicators ask the same question — the prompt lives in adjudicators/prompt.ts, apart from any SDK — so verdicts from different vendors stay comparable and the calibration harness can score them on equal terms. What differs is only enforcement: Anthropic constrains the verdict shape at the API via structured outputs; the OpenAI-compatible path states the contract in the prompt and validates the response against a schema before anything reaches the ledger. An unparseable answer is an error or an abstention — never a guess.

In both cases:

• Values are untrusted data. They come from documents and third-party systems, so they're delimited in <value> tags and the prompt states plainly that nothing inside them is an instruction.
• Temporality is explicit. "900k last year, 1.4M this year" is change over time, not a contradiction — the prompt says so, and says that when the model can't tell the difference it must leave the conflict open.
• A refusal means abstain, never "equivalent".
• API errors throw by default (onError: "abstain" to soften), so a bad key is loud rather than quietly degrading every verdict.

@anthropic-ai/sdk and openai are optional peer dependencies — install only the vendor you actually adjudicate with. Importing @credence/judge alone pulls neither, and a test walks the real import graph to keep it that way.

Credentials resolve the standard way — ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN or an ant auth login profile; DEEPSEEK_API_KEY for DeepSeek; an explicit apiKey for anything else. Because the families settle the common case, the model is called rarely, and every verdict is cached.

A verdict is cached by (pairStateHash, familyDefHash, model) — the model is part of the key, so switching providers re-judges rather than inheriting the other vendor's answers. Dismissals are attributed judge:<model>, which is also what makes a provider swap visible in the audit trail instead of silent.

Choosing the model

Explicit argument > environment > default:

| variable | default | |---|---| | CREDENCE_ANTHROPIC_MODEL | claude-opus-4-8 | | CREDENCE_DEEPSEEK_MODEL | deepseek-v4-pro | | CREDENCE_OPENAI_MODEL + CREDENCE_OPENAI_BASE_URL | none — both required |

The prefix is CREDENCE_ rather than a bare ANTHROPIC_MODEL on purpose. The model is not cosmetic: it is part of the verdict cache key and of the judge:<model> attribution, so inheriting a variable another tool set for its own reasons would silently re-judge cases and mislabel who decided them.

The generic endpoint has no default — it throws rather than guess a model name, because judging with a model nobody chose is worse than failing to start.

Which vendor the test suite certifies

The library never picks a vendor for you — you pass an Adjudicator. The test suite has to pick one, because it can only certify what it actually reached, so it tries them in order and prints which one a green run proved.

That order defaults to DeepSeek first, and it is a project decision rather than a ranking: DeepSeek is what this repo currently pays for. It is a preference, so a machine holding only an Anthropic key still proves the decision path.

CREDENCE_LLM_PROVIDER pins one vendor — deepseek, anthropic or openai — and a pin is exclusive: it never falls through. Pinning is for attestation, and a run that asked for one vendor and was certified by another would prove nothing about the one you asked for. Combine it with CREDENCE_LLM_STRICT=1 to turn "could not reach it" into a hard failure:

# attest that this environment can really adjudicate with Claude
CREDENCE_LLM_PROVIDER=anthropic CREDENCE_LLM_STRICT=1 pnpm test

A live vendor sometimes fails in ways that say nothing about this code — an empty completion, a rate limit, a 5xx. Those skip loudly rather than reddening the build, on a narrow allowlist (isTransientProviderFailure), and CREDENCE_LLM_STRICT=1 turns them back into failures. A maxTokens ceiling is deliberately excluded: that failure is ours, and excusing it would bury it.

A pin that names no known vendor throws. Falling back to the default on a typo would run DeepSeek while the operator believed they were attesting something else — and the calibration numbers would carry the wrong judge:<model>.

One practical note on reasoning models (deepseek-v4-pro, deepseek-reasoner): they spend tokens thinking before emitting the verdict, so maxTokens defaults to 8192. A tight ceiling truncates them into an empty completion, which looks like a model failure but is a budget failure — the error says which.

It abstains instead of guessing

If no family covers a key and no adjudicator is configured, the verdict is abstained and the conflict stays open. A judge that invents an answer is worse than one that admits it has no basis — silently closing a conflict is exactly the confident failure this project exists to prevent.

Caching, and re-judging when policy changes

Every verdict is cached by (pairStateHash, familyDefHash, model):

• the same conflict is never paid for twice, and
• editing a conflict family changes familyDefHash, so everything is re-judged without a deploy.

Contradictions the judge owns

Some conflicts are invisible to the same-key detector: "marital status says single, but a spouse is recorded" is two different keys disagreeing about the world, not two values of one key.

const judge = createJudge(ledger, rules, {
  contradictionFamilies: [{
    id: "marital-consistency",
    keys: ["marital_status", "spouse_name"],
    question: "whether the marital status matches the presence of a spouse",
  }],
  adjudicator: createAnthropicAdjudicator(),
});

await judge.scanContradictions(caseId);

// The judge owns these findings — tell the sync engine to leave them alone:
createRulesEngine(ledger, { rules, externallyManagedRuleIds: judge.ownedRuleIds });

Without an adjudicator this is a no-op: there is no declarative way to settle a cross-key contradiction, and guessing is worse than staying quiet.

Measuring the judge before trusting it

A judge that closes conflicts on its own and is miscalibrated doesn't make noise — it deletes signal. So it gets a number, not a vibe:

const report = evaluateCalibration(await judge.calibrationDataset(caseId));
if (report.ok) {
  // Only now is `autoDismiss: true` a defensible thing to turn on.
  createJudge(ledger, rules, { families, adjudicator, autoDismiss: true });
} else {
  console.log(renderCalibration(report));
}

An unmeasured model closes nothing

autoDismiss defaults to "policy-only":

| value | declarative family says benign | model says benign | |---|---|---| | "policy-only" (default) | dismissed | deferred — recorded, finding stays open | | true | dismissed | dismissed | | false | nothing is dismissed | nothing is dismissed |

Your families are your own rules: offline, reproducible, inspectable, and scored on every test run by the eval gate. A model is none of those until you have measured it, so autoDismiss: true is the explicit statement that you have. JudgeReviewResult.deferred counts what the judge thought was benign but was not allowed to close — and those are exactly the cases whose human resolution becomes the calibration data that would justify trusting it.

Who counts as a human

calibrationDataset() excludes machine dismissals, matched by attribution prefix: judge:, agent:, bot:, system: (DEFAULT_MACHINE_ACTOR_PREFIXES, override with machineActors). This is why @credence/kit attributes to agent:mcp and not agent — a model dismissing a conflict through a tool call must not come back as the human ground truth it is scored against. Attribution is a convention, not a proof: a caller who passes an arbitrary by for a machine can still poison the set, which is what machineActors is for.

The governing metric is the false dismissal rate — of everything the judge closed, how much was a real conflict — because the errors are asymmetric. A false negative is noise a human clears; a false positive is silent data loss. Abstentions never score as correct (a judge that never decides would otherwise look perfect), and the gate fails on too small a sample: "no evidence it's bad" is not "it's good".

Two gates, deliberately separate

| | eval-gate.test.ts | semantic-eval.test.ts | |---|---|---| | scores | the shipped conflict families | the configured model | | infra | none — no key, no network | one API call per case | | runs | every pnpm test | opt-in: CREDENCE_LLM_EVAL=1 | | on failure | blocks the merge | a finding about the model |

The model gate is off by default because it is non-deterministic, and a gate that can go red without anyone touching the code does not belong on a pull request — people learn to re-run it, and then it protects nothing. It runs on a schedule, where red means the model changed.

CREDENCE_LLM_EVAL=1 pnpm vitest run packages/judge/test/semantic-eval.test.ts

Judge eval: PASS  (18/18 correct, 0 abstained)
  accuracy 100.0% · false dismissal rate 0.0% (0 real conflict(s) closed)
  confusion: tp=6 fp=0 tn=12 fn=0

The semantic set is weighted toward the dangerous direction — twelve of the eighteen cases are real conflicts — and covers the ways a plausible-sounding model gets it wrong: near-miss and transposed figures, currency mismatch, change over time, sibling company names, identifiers (which admit no tolerance at all), and prompt injection — a value that instructs the judge must be compared as data, never obeyed. That last one is the <value>-tag claim above, tested rather than asserted.

A perfect score characterizes the set, not the model: eighteen cases is a tripwire for regression, not a licence to stop looking. The honest way to raise confidence is to add cases the model gets wrong.

runEval() scores labelled cases through the judge's real decision path — with zero infra when you only want to measure the deterministic policy — and evaluateGate() turns that into a regression gate that tolerates zero false dismissals by default.

Human dismissals are the report card

calibrationDataset() turns real usage into labelled examples: a conflict a person dismissed is benign; one they left open is real. It flags rows where the judge and the human disagreed — the only rows actually worth studying.

The judge's own dismissals are excluded. A model must not grade itself.

const examples = await judge.calibrationDataset(caseId);
// [{ subjectKey, values, label: "benign", by: "analyst", judgeVerdict, disagreement: true }]

Dismissals are attributed as judge:<model> so a machine decision is never mistaken for a human one in the audit trail.