dsh-eval

English | 中文

Agent evaluation over headless dsh profiles: a benchmark YAML document, one headless dsh subprocess per case x trial, trace harvesting from persisted session logs, automatic metric folding, and JSON/markdown run reports.

The package is a bundle and a command-line app. Install it into a profile and run benchmarks with:

dsh plugin --profile eval add dsh-eval
dsh eval run benchmark.yaml
dsh eval report eval-run.json
dsh eval compare eval-v1.json eval-v2.json

dsh eval is a launcher alias for --profile eval; the eval profile composes dsh-base plus this bundle. Each trial spawns the benchmark-configured dsh command (default dsh) against the benchmark-configured profile (default headless) with a private temp workspace, an isolated DSH_HOME, and a patch overlay forcing plain-JSONL persistence and non-interactive workspace-write/never-approval permissions. The harvested primary session log becomes the trial's trace; subagent child logs are not yet included.

Benchmark document

name: skill-regression
model: deepseek-v4
profile: headless
command: [dsh]
trials: 3
timeoutMs: 600000
seed: 42
cases:
  - id: fix-tests-001
    prompt: Fix the failing tests in this workspace.
    workspace: ./fixtures/fix-tests
    expected:
      tool: bash
      check: ./check.sh
pricing:
  deepseek-v4:
    inputUsdPerMTokens: 0.27
    cacheReadUsdPerMTokens: 0.07
    cacheWriteUsdPerMTokens: 0.27
    outputUsdPerMTokens: 1.10

| Field | Default | Meaning | |---|---|---| | name | required | Benchmark name, also the run record's benchmark field. | | model | required | Model id for pricing lookup and run reporting. | | profile | headless | dsh profile each trial spawns. | | command | [dsh] | dsh launcher argv. | | trials | 1 | Trials per case. | | timeoutMs | 600000 | Per-trial timeout; the direct child is killed on expiry. | | seed | 0 | Reserved for future deterministic paired comparisons. | | cases[].id | required | Stable case id. | | cases[].prompt / cases[].promptFile | exactly one | Task text, inline or relative to the benchmark file. | | cases[].workspace | absent | Workspace tree copied into each trial, relative or absolute. | | cases[].expected | absent | Scripted grading: tool is a substring matched against recorded tool-call names; check is a command run in the trial workspace after the agent exits (exit 0 = task success). At least one is required when present. | | pricing | absent | Per-million-token USD prices keyed by model id; absent models report costUsd: null. |

Judge

judge enables LLM-judge scoring of final answers and hallucination:

judge:
  provider: deepseek
  model: deepseek-v4
  rubric: Prefer correct, concise fixes.
  maxScore: 10

| Field | Default | Meaning | |---|---|---| | provider | deepseek | LLM provider route for the judge call. | | model | benchmark model | Judge model id. | | rubric | absent | Task rubric appended to the judging prompt. | | maxScore | 10 | Maximum final-answer score. |

The judge builds a strict-JSON prompt from the case and the merged trace. A trial carries its verdict (finalAnswerScore, hallucination, rationale) when the reply parses; unusable output or a chat failure reads as null verdict fields, never as a failed trial. A run with a configured judge fails before starting when the host exposes no LLM chat seam.

Metrics

The metric fold computes the automatic metrics directly from the session log: turns, steps, toolCalls, toolResults, toolSuccess plus toolSuccessRate, invalidToolCalls (results carrying an internal failure identity), retries (llm/retry), disjoint token buckets and totals, billed context tokens, llmMs/toolMs/ttftMs/latencyMs, and costUsd when pricing is configured. Scripted grading adds per-trial taskSuccess (check-command exit 0) and toolSelectionAccuracy (expected-tool match) and pools them into run-level rates. LLM-judge metrics (final-answer score, hallucination) are deferred.

Child session logs under the trial's DSH_HOME are merged into the trial trace before the fold, so delegated subagent work counts toward steps, tokens, tool calls, retries, and latency. LLM-judge verdicts add per-trial finalAnswerScore and hallucination and pool into run-level mean score and hallucination rate.

Run report

dsh eval run --out run.json writes one JSON document per run: benchmark and model identity, per-trial outcomes with absolute trace paths, aggregate metrics (means for counts and wall times, pooled success rate), and pooled grading rates. dsh eval report run.json renders the run as markdown. Trial workspaces and traces stay under the run's tempRoot (a private temp directory) and are not deleted; remove them when the run is no longer needed.

Comparison

dsh eval compare run-v1.json run-v2.json renders both runs as a markdown table: benchmark/model identity, completed trials, steps, tool/task/tool-selection rates, invalid calls, retries, tokens, cost, and latency, with signed B - A deltas. When both runs complete the same case x trial keys, the table appends paired statistics: trial count, win/lose/tie on the first available metric (task success, tool selection accuracy, or final-answer score), and mean B - A deltas for steps, tokens, and judge score. The runs' seed is recorded provenance for the pairing, not a guarantee of identical model output.

Replay (keyless CI)

A keyed run records every trial's session log under its tempRoot. Copy the recorded trial trees into a replay.dir and rerun without credentials: the spawned harness mounts @deepseek-ai/dsh-llm-replay, which reconstructs each model stream from the recorded assistant/chunk events.

replay:
  dir: ./recorded

The directory holds one tree per trial named <caseId>-<trial>/, with child logs riding alongside: recorded/fix-tests-001-1/session.jsonl. A missing fixture fails that trial, and judge cannot be combined with replay in one run. The npm release of the replay plugin lags the source workspace, so replay runs require a source-mode harness checkout.

Importing external traces

dsh eval import codex|claude-code <session.jsonl> --out run.json imports a Codex or Claude Code session log as a one-trial run with folded metrics:

• one synthesized turn (and step) per user message;
• assistant text, tool calls, and tool results mapped into the trace vocabulary;
• token usage is not part of either external format, so token and cost metrics stay zero.

The imported run drops into the same report/compare pipeline.

Extension points

The runner is the only consumer today: runBenchmark takes a loaded benchmark and returns the run record, and the report module persists and renders it. The judge chat seam is injected by the host (a dsh-llm stream) and stubbed in tests, keeping the suite keyless.

Model Experience

None, as the benchmark runner reads persisted session logs and spawns headless subprocesses without registering prompt sections, tool schemas, or any other model-facing behavior of its own.

KV Cache effect

None; this package neither assembles nor sends a provider request, so no request prefix exists for it to preserve or invalidate.

Known Limitations and Deferred Work

• Judge output is best-effort — a judge chat failure or unparsable reply yields null verdict fields, and judge calls consume model quota outside the trial's measured cost.
• Paired comparison pairs as-is — trials pair by case id and trial index; seed is recorded provenance, not a deterministic guarantee of identical model output, and per-arm leaderboards are not rendered.
• Replay binds by first-call order — the replay plugin keys recorded scripts to live sessions by first-call order, so concurrent subagents replay non-deterministically.
• Imported traces lack token usage — Codex and Claude Code logs do not record provider usage, so imported runs report zero tokens and no cost.
• Direct-child timeout only — a timed-out trial kills the direct dsh process; on Windows its descendants may survive.
• Windows launcher commands — the spawned command runs without a shell, so .cmd/.bat shims need a direct executable or node <path> override (--dsh "node C:/.../apps/cli/lib/bin.js").
• No atomic report write — the run JSON is written in place; a crash mid-write can truncate the report.