DSH Plugin Evaluation Platform

@dsh-plugin-evaluation/evaluation-platform is a local Node 20 service for managing isolated DSH evaluation runs. It provides a small HTTP API and a dependency-free browser console; it does not require a database or a separate frontend build.

Install and start

From a published package:

npm install -g @dsh-plugin-evaluation/[email protected]
dsh-evaluation

The default listener is http://127.0.0.1:3000. Set HOST and PORT to change the bind address. Open / for the console, or call /api/v1/health for a machine-readable readiness check. The CLI uses Docker as its only evaluation runtime; it does not execute DSH on the host machine.

If the default port is already in use, the CLI exits without choosing a different port automatically and prints an alternate command. You can choose one explicitly with either form:

PORT=3001 dsh-evaluation
dsh-evaluation --port 3001

Use dsh-evaluation --help to see the available startup options. --host is also available when the bind address needs to be set explicitly.

On the first launch, make sure Docker Desktop is running and click Initialize environment in the console. The service builds the runner image in the background, shows progress, and enables the evaluation form only after the image is ready. If initialization fails, start Docker Desktop and click Retry. The equivalent API flow is:

curl http://127.0.0.1:3000/api/v1/environment
curl -X POST http://127.0.0.1:3000/api/v1/environment/initialize

The web service sends the selected plugin and evaluation plan to a temporary container. The runner image already contains the pinned DSH checkout and build artifacts; each experiment container installs the selected plugin, runs the evaluation, and writes a structured result. Set DSH_EVALUATION_IMAGE to use another runner image, or set DSH_EVALUATION_HOST=fixture only for explicit fixture-based tests. The plugin registry defaults to ~/.dsh-evaluation/registry; override it with DSH_EVALUATION_REGISTRY_ROOT.

Set DSH_EVALUATION_DATA_ROOT to persist run and report records as well as the plugin registry. On restart, completed records are restored before the API serves requests. A run that was queued or active when the previous process exited is restored as interrupted with an explicit recovery error; it is not automatically re-executed.

Release

普通 push 只运行 CI，不会发布 npm。发布新版本时，先更新 package.json 的版本号，再创建并推送同名 tag：

npm version patch
git push origin main --follow-tags

推送 vX.Y.Z tag 后，GitHub Actions 会先运行 npm run verify，确认 tag 与 package.json 版本一致后，再通过 npm Trusted Publishing 发布包。首次使用前，需要在 npm 包设置中将对应 GitHub 仓库和 Publish workflow 配置为 Trusted Publisher。

As a library, the package exports createEvaluationServer, startServer, ManagedDshHost, PluginRegistry, EvaluationOrchestrator, and the versioned API/server building blocks from src/index.js.

Catalog-governed sources

The package owns catalog-governed loading of evaluation profiles and cases. It reads the standards catalog, accepts only HTTPS GitHub sources pinned to a plain semantic version, a vX.Y.Z tag, or a 40-character commit SHA, validates the fetched JSON, and returns an immutable cached snapshot.

Each fetched JSON document is hashed with SHA-256 over its exact UTF-8 bytes. Those hashes are part of immutable provenance and cache identity; a source adapter may provide expected hashes, which are checked before parsing. Paths are rejected before adapter reads, including percent-encoded, double-encoded, backslash, separator, dot-segment, and traversal variants.

The loader treats source files as data. It never imports, evaluates, or runs content from a source repository. Tests use a local fixture adapter keyed by a catalog-approved GitHub repository and ref; this keeps tests offline without weakening the catalog's HTTPS GitHub policy.

Local HTTP API and console

The server uses only Node's built-in http module and serves the versioned /api/v1 surface:

• GET /health, /status, /plugins, /sources, /runs, and /reports
• POST /plugins to register a local plugin ({"path":"/absolute/plugin"})
• POST /runs to start a single managed evaluation
• GET /runs/:runId and POST /runs/:runId/cancel
• GET /reports/:reportId and GET /reports/:reportId/export
• GET /metrics lists the registered metric types

Run requests accept registered pluginIds or absolute local plugin paths, plus either a generic versioned plan or the legacy non-empty prompt/scheme form. A plan declares its input and independent metric checks:

{
  "schemaVersion": 1,
  "id": "order-quality-v1",
  "version": "1.0.0",
  "name": "Order quality",
  "input": { "prompt": "Find order 123 status" },
  "metrics": [
    { "id": "answer", "type": "output-contains", "expected": "运输中" },
    { "id": "safe", "type": "no-secret" },
    { "id": "deadline", "type": "no-timeout" }
  ]
}

The platform is backed by @dsh-plugin-evaluation/[email protected]. Its case lifecycle, workspace isolation, evidence collection, fixture cleanup, and bounded execution are used by every orchestrated run. Built-in metric types are output-contains, output-exact, file-exists, no-timeout, no-secret, and tool-calls. Applications can create a custom metric registry through the library API without changing the execution host. The execution pipeline remains plan-independent: prepare the isolated environment, run the plugin with the plan input, collect the host result, evaluate the plan metrics, and serialize one report shape. Reports expose the plan, checks, and a summary with total/passed/failed check counts. Request bodies are capped at 64 KiB. Responses, run output, errors, reports, and exports redact credential-like values. The browser console is served from /; API traffic is kept under /api/v1.

Sources, plugins, and runtime isolation

Each Docker managed run receives its own temporary input/output directories and container. The runner creates a private DSH profile inside the container and the host enforces bounded timeouts, termination, concurrency limits, and redaction of credential-like values in status, reports, and errors. The fixture host is available only when DSH_EVALUATION_HOST=fixture is explicitly set or when a test injects it.

Development and release checks

The repository has no third-party runtime dependency installation step:

npm test           # unit and HTTP/orchestration tests
npm run check      # Node syntax checks
npm run clean-room # pack, install the tarball in a fresh directory, call health
npm run verify     # all checks above plus package dry-run

npm run clean-room is local and does not publish anything. The package's files allowlist includes only the CLI, browser assets, source, README, design notes, and MIT license.