skarmy-bench

Help us make Skarmy's AI coach better — from your own computer, no project access needed.

Skarmy turns founders' ideas into working software. The first thing a founder meets is the Hub coach: an AI that helps sharpen their idea before building starts. How the coach behaves is controlled by a "system prompt" — a page of written instructions. Your mission: write a better one.

What you need

• Node.js 18 or newer installed
• An access token from the Skarmy team (ask Justin)

Getting started

Open your terminal, make a folder to work in, and run:

npx skarmy-bench

The first run asks for your token and your name, then opens the bench in your browser. Everything happens there: create a prompt variant, edit it, run it against the current coach, read the replies side by side, and submit your review — no file editing needed. Leave the terminal window open while you work.

Prefer the terminal?

Every step also has a command:

npx skarmy-bench menu             # the old text menu
npx skarmy-bench practice         # see how the current coach answers
npx skarmy-bench new my-idea      # creates skarmy-bench/variants/my-idea.md
# … edit that file in any text editor …
npx skarmy-bench run my-idea      # your prompt vs the current coach

Each run creates a folder under skarmy-bench/results/ with:

• report.md — every test conversation, with the current coach's reply and your variant's reply side by side, plus an AI judge's ranking.
• review.md — a short form for your verdict. Fill in the … parts: which reply was better per scenario and why, and whether you'd ship it.

Then send it to us:

npx skarmy-bench submit <run-name>

What makes a good review

• Judge like a founder would: which reply actually helps this idea?
• The AI judge's ranking is a second opinion, not the answer — if you disagree with it, say so and why. Those disagreements are gold.
• Look for honesty: the coach must never promise iPhone apps, payments, or user accounts — the first version is always a web demo.

Notes

• Your token is saved in skarmy-bench/config.json — keep it private, and don't commit that folder anywhere public.
• Each token comes with a $5 testing budget (roughly 15–40 runs). The tool shows what's left after each run; when it's used up, ask Justin for a fresh token. Submitting reviews is always free.
• All AI calls run on Skarmy's servers; the current production prompt never leaves them (you see its replies, not its text).
• The browser UI is a local server on your own machine (127.0.0.1 only); your token never reaches the browser.
• Something broken? Copy the error message and send it to Justin.