@decada/bench — decada-bench

A headless command-line runner for the Decada game-bug-detection benchmark. It is the Hugging Face / Kaggle / "bring-your-own-model" entry point: run the same benchmark the Decada desktop app runs, but without Electron, against any OpenAI-compatible model endpoint.

Install

npm install -g @decada/bench
# then, once, install the browser the benchmark drives:
npx playwright install chromium

Or run without installing:

npx @decada/bench --help

The package is self-contained: the Decada agent/runner internals are bundled in, so there are no private @gameqa/* packages to resolve. Public runtime deps (Playwright, the OpenAI SDK, etc.) install normally from npm.

It wraps two functions from @gameqa/runner:

• runBenchmarkSuite(...) — runs the bundled public practice suite locally and self-scores it.
• runBenchmarkCases(...) — runs the verified hold-out cases; scoring is done on the backend against ground truth the client never sees.

Prerequisite — Chromium

The benchmark drives a real browser via Playwright. Install Chromium once:

npx playwright install chromium

If Chromium is missing, decada-bench prints this instruction and exits with a non-zero code instead of silently downloading.

Building from the monorepo (for maintainers)

The publishable bundle is produced with esbuild, which inlines the private @gameqa/* workspace packages (shared, ai, runner) into a single dist/index.js and copies the PUBLIC benchmark assets into assets/benchmark/. Public deps (Playwright, OpenAI SDK, AWS SDK, ioredis, axios, ffmpeg-static, dotenv) stay external and are declared as real dependencies.

# from the repo root — builds shared/ai/runner first, then bundles the CLI
npm run build:cli

npm run build:cli runs the workspace builds the bundler depends on, then node cli/build.mjs. To publish:

cd cli
npm publish        # prepublishOnly re-bundles via build.mjs; access is public

The benchmark assets are resolved from the packaged assets/benchmark/ copy. Override with the DECADA_BENCH_DIR env var if you need to point at a different suite.

Usage

Practice suite against your own model (PUBLIC, self-scored)

npx decada-bench --model my-model --base-url https://my-host/v1 --api-key sk-...

Streams one line per case (id, outcome, verdict, bug count, steps) and prints a final precision / recall / F1 / accuracy / false-positive-rate / verdict-accuracy summary plus the TP/FP/FN/TN confusion counts.

Run a single case from the suite:

npx decada-bench --model my-model --base-url https://my-host/v1 --api-key sk-... --case counter-frozen

Publishing credentials

Both --submit and --official publish to your Decada account, so they need a credential. Two options:

• Publish key (recommended) — a personal, scoped, revocable API key. Create one in the web app under Settings → API keys (or in the desktop app's Benchmark tab). It can only publish benchmark results — not touch billing, games, or account settings — so it's safe to drop into CI. Pass it via --publish-key or env DECADA_PUBLISH_KEY. The key is shown only once at creation; copy it then.
• Account JWT — your full login token via --token or env DECADA_TOKEN.

Submit to the community leaderboard

DECADA_PUBLISH_KEY=dcb_... npx decada-bench --model ... --base-url ... --api-key ... \
  --submit --backend https://api.decada.tech

Runs the public suite, then POSTs the result to the community board and prints the returned result id. Requires a publish key (--publish-key / DECADA_PUBLISH_KEY) or an account JWT (--token / DECADA_TOKEN).

Verified hold-out (server-scored)

DECADA_PUBLISH_KEY=dcb_... npx decada-bench --official --model ... --base-url ... --api-key ... \
  --backend https://api.decada.tech

Opens a hidden hold-out session on the backend, runs the agent locally against those cases, submits predictions, and prints the server-scored metrics (authoritative — the client never sees ground truth). Requires a publish key or account JWT (as above).

Flags

| Flag | Description | |---|---| | --model <id> | Actor model id (required) | | --base-url <url> | OpenAI-compatible base URL (required) | | --api-key <key> | Provider key (required). Also from env DECADA_MODEL_API_KEY | | --planner-model <id> | Optional dedicated planner model (else planner is off) | | --tier <low\|high> | Agent tier preset (default high; medium is accepted and maps to high) | | --max-steps <n> | Cap agent steps per case | | --case <id> | PUBLIC mode only: run a single case | | --official | Run the verified hidden hold-out (requires a publish key or --token) | | --submit | After a PUBLIC run, submit to the community board (requires a publish key or --token) | | --backend <url> | Backend base URL (default https://api.decada.tech) | | --publish-key <key> | Personal publish key (recommended for CI). Also from env DECADA_PUBLISH_KEY | | --token <jwt> | Decada account JWT (alternative to --publish-key). Also from env DECADA_TOKEN | | -h, --help | Show usage and exit |

Flags accept both --flag value and --flag=value forms. Secrets (--api-key, --publish-key, --token) can be supplied via environment variables (DECADA_MODEL_API_KEY, DECADA_PUBLISH_KEY, DECADA_TOKEN) so they stay out of shell history. A .env file in the working directory is loaded automatically.

Notes

• The model transport is configured entirely through environment variables that @gameqa/runner and @gameqa/shared read; the CLI translates its flags into those vars (mirroring the desktop app's custom-model setup).
• Exit code is 0 on success and 1 on any failure.