dsh-rerun-grader-command
v0.1.2
Published
Grader provider scoring an attempt by the exit code of a pinned command in the frozen workspace
Maintainers
Readme
dsh-rerun-grader-command
Grader provider scoring an attempt by the exit code of a pinned command run in the final workspace — the objective grader, and the one every task should reach for first.
The command runs under its own environment rather than the operator's shell, and
deliberately WITHOUT the attempt's RERUN_* variables: a task must not be able
to detect, from inside its own test command, which variant produced the
workspace being graded.
graders:
- kind: command
weight: 3
run: node test.mjsModel Experience
None. This package never contributes to a model request: it runs in the orchestrator process, and the model calls it cares about happen inside a separate attempt subprocess running a different build of DSH entirely.
KV Cache effect
None; it neither assembles nor sends a provider request.
Known Limitations and Deferred Work
- Not rescorable. Running a command needs a live workspace, which a rescore no longer has; the original verdict is carried forward and labelled.
- The command is shell-interpreted, so a suite is executable code. See SECURITY.md.
