LLM-as-a-Verifier for dsh — Best-of-N (Bo5) conversation mode

English | 中文文档

Give DeepSeek V4 Flash test-time scaling: V4 Flash + Bo5 self-verification reaches Fable-5-level scores — 88% on Terminal-Bench 2.1, frontier-model accuracy at a fraction of the cost (≈11× cheaper).

An independent dsh-native implementation of the test-time selection method from LLM-as-a-Verifier (arXiv:2607.05391, MIT). Method by the paper's authors; this implementation by Aispin.

The paper's idea, in one minute

Cheap models can generate great answers — they just can't recognize which one is great. LLM-as-a-Verifier closes that gap:

1. Sample N candidates from a cheap model (DeepSeek V4 Flash): slightly different attempts at the same task.
2. Grade with a fine-grained verifier — the same cheap model, asked to grade pairs of candidates on an A–T letter scale. The score is not the sampled letter: it is the expectation over the grade token's logprob distribution, Σ p(token)·φ(letter) — the model's full belief, not one dice throw.
3. Both orderings per pair cancel the verifier's position bias; repeated evaluations alternate slots.
4. Select the best — the paper's core result: V4 Flash sampling 5 candidates + self-verification matches Fable-5-level frontier scores on Terminal-Bench 2.1 (88.0%) at ~1/11 the cost.

This plugin packages that pipeline as a dsh plugin with a conversational twist: every assistant turn becomes Best-of-N automatically — you see one answer, the model produced five.

One plugin, three faces

| Face | Entry | Use | |---|---|---| | Tool | verify tool | On demand — "use the verify tool to compare A/B/C", the agent calls it | | Service | ctx.verifier.verify({ task, candidates }) | For code — orchestration lines, other plugins | | Mode | Best-of-N conversation mode | Invisible — Bo-N sessions sample every turn N ways, verify, replay only the winner |

Install

From npm (the recommended path — resolves every dependency through your profile):

dsh plugin --profile <your-profile> add @aispin/plugin-verifier

Or plain npm:

npm install @aispin/plugin-verifier

Zero-config: the verifier inherits dsh's configured provider state (credentials + settings seams) — if you've configured DeepSeek on the Models page, it just works. Try it locally:

git clone https://github.com/aispin-dev/llm-as-a-Verifier-dsh.git
dsh plugin --profile <your-profile> add /path/to/llm-as-a-Verifier-dsh

Best-of-N: three-state switch (hot)

① settings global (Web UI panel) → ② session preset ("Bo-N mode") → ③ profile config default → off

Two independent tiers in the Web settings panel: the global tier (what the global switch turns on for every session) and the "Bo-N 模式" preset tier (what sessions that selected the Bo-N preset use — defaults to Bo5, set independently). Plus a user-adjustable verify timeout (default 90s — the ranking's own budget, never borrowed by sampling).

Sampling degrade chain: each rollout carries the sampling budget as its own wall-clock cap — a Bo5 whose 2 rollouts overrun degrades to Bo3 and still ranks the survivors (the footer says: 采样 5 路 2 路未完成 · 3 选 1); below 2 survivors the turn fails open to a normal answer.

The Web settings panel offers the tiers with transparent cost cards:

| Tier | Model calls | Tokens | Latency | |---|---|---|---| | Off | 1 | 1× | 1× | | Fast · Bo-3 | ~9 | 2–3× | ~7–15s | | Precise · Bo-5 | ~16 | 3–5× | ~12–30s | | Custom | 2–8 ways | linear | linear |

Every turn's footer meters the real spend: ⚡ Best-of-N · 5 选 1 → 候选 #2 · 20.0/20 · 24.3s · 10.8K tok

What's inside (implementation parity with the paper)

• Fine-grained reward: expected grade over the top-20 logprob distribution, A=20…T=1 grouped band scale, grading at temperature 1.0 (the natural belief distribution — never collapsed)
• PPT pivot tournament (the paper's O(N·k) selection): random Hamiltonian ring (each candidate lands exactly once per slot — bias cancels inside the ring) → top-k pivots → only non-pivot×pivot pairs graded. Live-verified: Bo-5 grading calls 20 → 11 (−45%)
• Prefix-cache prompt layout (paper v0.2.0, −3.4× uncached tokens): criteria at the prompt tail; role + scale + task + candidates form the shared prefix
• Capability-adaptive grading: logprobs endpoints (DeepSeek official) get expected-grade scoring; logprob-less endpoints (MiniMax, various gateway providers) auto-degrade to letter-sampling grading with double evaluation — any OpenAI-compatible endpoint works (autoDegrade: false for strict mode)
• Fail-open discipline: any breakdown degrades to a normal answer with an explanatory footer — never a dead turn

License

MIT © 2026 Aispin. The method is from LLM-as-a-Verifier (arXiv:2607.05391, MIT). Not affiliated with the paper's authors or DeepSeek.

中文文档

给 DeepSeek V4 Flash 测试时扩展能力：V4 Flash + Bo5 自验证达到 Fable 5 级评分——Terminal-Bench 2.1 上 88%，以前沿模型级别的准确率、约 1/11 的成本完成任务。

LLM-as-a-Verifier（arXiv:2607.05391, MIT）测试时选择方法的 dsh 原生独立实现。方法归论文作者，实现归 Aispin。

论文的思想，一分钟讲清

便宜模型能生成好答案——只是认不出哪个是好答案。LLM-as-a-Verifier 补上这一环：

1. 采样 N 个候选（DeepSeek V4 Flash）：同一任务的多个略有差异的尝试。
2. 细粒度验证器评分——同一个便宜模型，对候选两两成对按 A–T 字母量表打分。分数不是采样出的那个字母，而是 grade token 对数概率分布上的期望值 Σ p(token)·φ(letter)——模型的完整信念，不是掷一次骰子。
3. 每对双向各评一次抵消验证器的位置偏置；重复评估交替 A/B 槽位。
4. 选出最佳——论文核心结论：V4 Flash 采样 5 条候选 + 自验证择优，在 Terminal-Bench 2.1 上达到 Fable 5 级前沿评分（88.0%），成本约 1/11。

本插件把这套管线做成 dsh 插件，并加上对话形态：每个回答自动变成 Best-of-N——你看到一条答案，模型实际做了五条。

一个插件，三张面孔

| 面孔 | 入口 | 用法 | |---|---|---| | 工具面 | verify 工具 | 有感——对话里说"用 verify 工具比较 A/B/C"，模型主动调用 | | 服务面 | ctx.verifier.verify({ task, candidates }) | 代码消费（编排线、其他插件） | | 模式面 | Best-of-N 对话模式 | 无感——选中模式的会话，每轮后台 N 路采样 + 择优，只把胜者呈现给用户 |

安装

npm 安装（推荐——依赖经你的 profile 完整解析）：

dsh plugin --profile <your-profile> add @aispin/plugin-verifier

或直接 npm：

npm install @aispin/plugin-verifier

零配置：验证器继承 dsh 已配置的 provider 状态（credentials + settings seam）——在 Models 页面配过 DeepSeek 即可直接用。本地试用：

git clone https://github.com/aispin-dev/llm-as-a-Verifier-dsh.git
dsh plugin --profile <your-profile> add /path/to/llm-as-a-Verifier-dsh

Best-of-N：三态开关（热生效）

① settings 全局（Web 设置面板）→ ② session preset（"Bo-N 模式"）→ ③ profile config 默认 → 关

Web 设置面板的两层独立档位：全局档位（全局开关开启时所有会话用）+ "Bo-N 模式"档位（选中该 preset 的会话用，默认 Bo5，单独设置互不影响）。另有用户可调的评分超时（默认 90 秒——评审阶段的独立预算，不被采样挤占）。

采样降级链：每路采样以采样预算为自身时限——Bo5 有 2 路超时则降级为 Bo3 继续对存活者择优（footer 明示：采样 5 路 2 路未完成 · 3 选 1）；存活不足 2 路才 fail-open 为普通回答。

Web 设置面板的档位卡直接标注消耗透明：

| 档位 | 模型调用 | token | 延迟 | |---|---|---|---| | 关闭 | 1 次 | 1× | 1× | | 快速经济 · Bo-3 | ~9 次 | 2–3× | ~7–15s | | 精准 · Bo-5 | ~16 次 | 3–5× | ~12–30s | | 自定义 | 2–8 路 | 线性 | 线性 |

每轮回答尾部 footer 显示实际开销：⚡ Best-of-N · 5 选 1 → 候选 #2 · 20.0/20 · 24.3s · 10.8K tok

实现要点（与论文对齐）

• 细粒度奖励：top-20 logprob 分布上的期望分（A=20…T=1 分组带量表），评分温度 1.0（读自然信念分布，绝不坍缩）
• PPT 概率枢轴锦标赛（论文 O(N·k) 选择算法）：随机哈密顿环（每候选恰好在 A/B 槽各一次——环内天然消位置偏置）→ top-k 枢轴 → 只补非枢轴×枢轴对。实测 Bo-5 评分调用 20 → 11（−45%）
• 前缀缓存布局（论文 v0.2.0，未缓存 token −3.4×）：criteria 置于 prompt 尾部，角色+量表+任务+候选构成跨调用共享前缀
• 能力自适应评分：有 logprobs 的端点（DeepSeek 官方）用期望分；没有的（MiniMax、部分网关）自动降级采样评分（每对双评补偿方差）——任何 OpenAI 兼容端点都能当评审（autoDegrade: false 切严格模式）
• Fail-open 纪律：任何断裂降级为普通回答并在 footer 说明原因，绝不杀死对话轮

许可

MIT © 2026 Aispin。方法来自 LLM-as-a-Verifier（arXiv:2607.05391, MIT）。与论文作者及 DeepSeek 无隶属关系。