dsh-eval-suite

DSH host plugin for SIFT evaluation-suite authoring and execution:

• Authoring: init, add, coverage, inspect, freeze
• Execution: eval, trigger, run, gate, report

eval supports Skill A/B comparison (with/without or v1/v2) and Agent absolute-quality evaluation with --kind agent. run executes a real task and produces transcripts; gate evaluates an existing benchmark; report renders the static HTML report. Session transcript auditing remains in dsh-audit.

inspect is a static check of evals.json: it reports completeness, stability, and concern flags without running evals or making an LLM call. It also does not produce a pass/fail gate verdict.

Usage

/eval-suite init <skill-path>
/eval-suite add <skill-path> --idea "..."
/eval-suite coverage <skill-path>
/eval-suite inspect <evals-path>
/eval-suite freeze <evals-path>
/eval-suite eval <skill-path> --evals <evals-path>
/eval-suite trigger <skill-path> --evals <evals-path>
/eval-suite run <skill-path> --task "..."
/eval-suite gate <benchmark-path>
/eval-suite report <workspace>

Headless tool: eval_suite.

Auth

By default, SIFT uses headless DSH for its own init/analysis/grading LLM calls:

SIFT_LLM_TRANSPORT=dsh

Provider credentials and endpoint configuration are resolved by dsh-env-health and passed to the nested DSH process:

ANTHROPIC_AUTH_TOKEN=<provider auth>
ANTHROPIC_BASE_URL=<provider auth>
ANTHROPIC_MODEL=<provider auth>
DEEPSEEK_API_KEY=<provider auth>
DEEPSEEK_BASE_URL=<provider auth>

HTTP and Claude Code CLI transports remain available as compatibility options, but the default DSH path has no Claude Code CLI dependency.

Template init --no-generate-prompts, manual add --prompt, inspect, and freeze do not require provider auth.

Managed SIFT runtime

When autoInstall is enabled, the plugin requests the shared SIFT runtime through environmentHealth.ensureRuntime('sift'). The installation itself lives in dsh-env-health; concurrent Audit, Static Check, and Eval requests share one task.

- id: dsh-eval-suite
  config:
    siftBin: sift
    autoInstall: true
    transport: dsh