@calculator53295/bench-task

Zero-dependency validation for Agent Colosseum benchmark task corpora.

Each direct child of a corpus directory is a task folder containing task.json and prompt.md. The JSON record points at an immutable repository snapshot and declares its mechanical verification command:

{
  "id": "example-fix",
  "repoSnapshot": {
    "url": "https://example.invalid/project.git",
    "pinnedSha": "0123456789abcdef0123456789abcdef01234567"
  },
  "promptPath": "prompt.md",
  "verify": {
    "cmd": "npm test",
    "timeoutMin": 10
  },
  "tier": "bugfix",
  "rubricNotes": "Optional evaluator context."
}

repoSnapshot must contain exactly one of url or path. Run the linter with:

bench-task lint tasks/

The package exports validateTask, assertValidTask, and lintCorpus for plain Node.js and TypeScript consumers. The canonical JSON Schema is exported as @calculator53295/bench-task/schema.