@webywz/llm-roast
v0.1.20
Published
Roast your LLM prompts and answers — fluff, hedges, hallucination vibes, scored 0–100
Maintainers
Readme
llm-roast
A tiny CLI for eval dumps, prompt reviews, and screenshots you can share. Higher Roast Score means more roastable. This is a heuristic scorer, not a fact checker.
给评测 dump、prompt 复盘和可分享截图用的小 CLI。Roast Score 越高越该被吐槽。这是启发式打分,不是事实核查器。
Install · 安装
npm i -g @webywz/llm-roast
npx @webywz/llm-roast examples/answer.mdRequires Node.js 18.18+. Package: @webywz/llm-roast. Binary: llm-roast.
需要 Node.js 18.18+。包名 @webywz/llm-roast,命令仍是 llm-roast。
Quick start · 快速开始
llm-roast prompt.txt
llm-roast answer.md
llm-roast --prompt p.txt --answer a.txt
cat answer.md | llm-roast
llm-roast - < prompt.txt
llm-roast --json answer.md
llm-roast --html --offline answer.md
llm-roast --html --theme fog --offline answer.md
llm-roast --html report.html --theme midnight --offline answer.md
llm-roast --html --theme aurora --offline answer.md
llm-roast --list-themes
llm-roast --list-themes --json
llm-roast --out report.txt answer.md
llm-roast --spicy answer.md
llm-roast --offline --fail-above 60 answer.md
llm-roast --suite "examples/*.md" --json
llm-roast --suite examples --json --fail-above 60
llm-roast --github --offline answer.md
llm-roast --github --quiet --offline answer.md
llm-roast --github --max-annotations 10 --offline answer.md
llm-roast --locale zh --offline answer.md
llm-roast --copy --offline answer.md| Flag | What it does · 作用 |
| --- | --- |
| --prompt <file> | Force prompt-smell rules (- = stdin) · 按 prompt 气味规则打分 |
| --answer <file> | Force answer rules (- = stdin) · 按回答规则打分 |
| --suite <glob\|dir> | Batch mode: roast ≥2 files from glob/directory (JSON output) · 批处理模式:对 glob/目录中 ≥2 个文件打分(JSON 输出) |
| --out <file> | Write report to file · 将报告写入文件 |
| --html [path] | Self-contained HTML roast report (stdout, or write to path; offline, no CDN) · 自包含 HTML 吐槽报告(stdout 或写入路径;离线、无 CDN) |
| --theme <name> | HTML palette only: ember (default, 0.1.16 look), fog, midnight, paper, aurora · 仅 HTML 配色:ember(默认,与 0.1.16 相同)、fog、midnight、paper、aurora |
| --list-themes | Print available HTML theme names (one per line; --json: JSON array) · 打印可用 HTML 主题名(每行一个;加 --json 则输出 JSON 数组) |
| --json | Machine-readable result · 机器可读输出 |
| --spicy | Harsher tone; score unchanged · 文案更狠;分数不变 |
| --offline | Skip the optional API encore · 跳过可选 API 加戏 |
| --fail-above <n> | Exit 1 when Roast Score > n (suite: maxScore > n; 0–100) · 分数 大于 n 时退出码 1(suite: maxScore 大于 n) |
| --locale <en\|zh> | Terminal output locale (default en; JSON keys unchanged) · 终端输出语言(默认 en;JSON 键名不变) |
| --github | Emit GitHub Actions ::notice / ::warning / ::error annotations · 输出 Actions 注解,PR Checks 可见 |
| --max-annotations <n> | Cap annotation lines with --github (default 20) · 限制 --github 注解条数(默认 20) |
| --quiet | Suppress the pretty terminal report; --json / --github still emit · 抑制漂亮终端报告;--json / --github 仍照常输出 |
| --copy | Copy the pretty roast report to the clipboard; skip silently if unavailable · 复制漂亮终端报告到剪贴板;无剪贴板工具则静默跳过 |
| --color / --no-color | Force / disable ANSI color · 强制 / 关闭颜色 |
| -h / -v | Help / version · 帮助 / 版本 |
A bare file is auto-classified from the name (prompt, answer, response, …) or opening lines (you are vs as an AI).
单文件会按文件名或开头几行自动判断是 prompt 还是 answer。
Suite / batch mode · 批处理模式
Roast multiple files in one run with --suite <glob|dir>:
一次对多个文件打分,使用 --suite <glob|dir>:
llm-roast --suite "examples/*.md" --json
llm-roast --suite examples --json
llm-roast --suite "examples/*.{txt,md}" --json --fail-above 60
llm-roast --suite "examples/{answer,bad-answer}.md" --json
llm-roast --suite "fixtures/{good,bad}/*.md" --jsonSuite mode requires ≥2 roastable files (.txt, .md, .markdown). Supports brace expansion like {a,b} for flexible file matching. Output is always JSON with per-file results and aggregated stats:
批处理模式需要 ≥2 个可打分文件(.txt、.md、.markdown)。支持 大括号扩展 如 {a,b} 以灵活匹配文件。输出始终为 JSON,包含每个文件的结果与汇总统计:
{
"version": "0.1.20",
"files": [
{ "path": "examples/answer.md", "score": 73, "label": "yikes", "kind": "answer", "wordCount": 89 },
{ "path": "examples/prompt.txt", "score": 58, "label": "caution", "kind": "prompt", "wordCount": 42 }
],
"stats": {
"count": 2,
"maxScore": 73,
"avgScore": 66,
"failCount": 0
}
}--fail-above in suite mode applies to stats.maxScore (the highest score across all files). Exit code 1 when maxScore > threshold.
批处理模式下的 --fail-above 应用于 stats.maxScore(所有文件中的最高分)。当 maxScore > 阈值 时退出码为 1。
Exit codes · 退出码:
0— Success · 成功1—--fail-abovethreshold exceeded · 超过--fail-above阈值2— Usage error or I/O failure · 用法错误或 I/O 失败
Fixtures: examples/prompt.txt · examples/answer.md.
示例:examples/prompt.txt 与 examples/answer.md。
# GitHub Actions — annotations + CI gate + job summary
- name: Roast gate
run: npx @webywz/llm-roast --offline --github --quiet --fail-above 60 --max-annotations 20 path/to/answer.mdCopy-paste workflow: examples/github-actions.yml.
可直接复制的 workflow:examples/github-actions.yml。
--github prints workflow commands (::warning file=…::…) so Checks show score and findings. High severity → error, medium → warning, low → notice; score bands use the same scale. With GITHUB_OUTPUT set it also writes score, label, findings. With GITHUB_STEP_SUMMARY set it appends a short Markdown job summary (score / label / key findings). Use --max-annotations <n> to soft-cap PR noise (default 20). --quiet suppresses the pretty terminal report only (--json / --github annotations, summary, and outputs still emit). Combine with --json and annotations go to stderr so stdout stays parseable. Works together with --fail-above (annotations still emit, then exit 1 when over the gate).
--github 输出 workflow 命令,PR Checks 能直接看到分数与 findings。高严重 → error,中 → warning,低 → notice;总分按档位映射。在 Actions 里还会写 GITHUB_OUTPUT(score / label / findings),有 GITHUB_STEP_SUMMARY 时再追加一段简洁 Job Summary。--max-annotations <n> 可限制注解条数(默认 20)。--quiet 只抑制漂亮终端报告(--json / --github 注解、summary、outputs 仍照常)。与 --json 联用时注解走 stderr,stdout 仍是干净 JSON。可与 --fail-above 组合:先出注解,超阈再以 1 退出。
--fail-above 100 never fails (score tops at 100). --fail-above 0 fails on any non-zero score.
--fail-above 100 永远不会失败(满分 100);--fail-above 0 只要非零就失败。
Library · 库
import { roastText, roastDocuments } from "@webywz/llm-roast";
const answer = roastText(src, { kind: "answer", spicy: false });
console.log(answer.score, answer.roast, answer.findings);
const both = roastDocuments([
{ text: prompt, kind: "prompt", source: "prompt.txt" },
{ text: reply, kind: "answer", source: "answer.md" },
]);Features · 特性
| Bucket · 桶 | Examples · 例子 |
| --- | --- |
| Fluff | "as an AI language model", "in today's digital age", "delve", "tapestry", "robust", "leverage", "synergy", "great question" |
| Hedge | "might", "could potentially", "it depends", "perhaps", "to some extent" |
| Overconfidence | "always", "never", "guaranteed", "100%", "everyone knows" — extra weight when no nearby source · 附近没有出处时更重 |
| Hallucination vibes | (Author et al., 2024) with no references, example.com docs, precise % with no source, invented APIs, "studies show" |
| Prompt smells | concise and comprehensive, no lists and bullet the points, stacked roles, missing format / audience / length |
- Deterministic copy — same input → same roast text.
--spicyonly changes tone. 文案确定 — 相同输入得到相同吐槽;--spicy只改语气。 - CI gate —
--fail-abovestill prints the roast, then exits1only when the score exceeds the threshold. CI 门禁 — 仍会打印吐槽;仅当分数超过阈值时以1退出。 - GitHub annotations —
--githubemits Actions notices/warnings/errors for PR UI; optional Job Summary +--max-annotations;--quietfor annotations-only CI logs. GitHub 注解 —--github输出 Actions notice/warning/error;可选 Job Summary 与--max-annotations;--quiet只留注解不打漂亮报告。 - HTML export —
--htmlemits a self-contained offline roast (score, findings, spicy/summary). Omit the path for stdout; give a path to write the file (parent dirs created).--theme <name>swaps CSS-variable palettes (emberdefault = 0.1.16 look, plusfog,midnight,paper,aurora); it does not change JSON, pretty, GitHub annotations, or scoring.--list-themesprints those names (one per line;--list-themes --jsonis a JSON array).--htmland--jsoncannot share stdout. HTML 导出 —--html写出离线自包含吐槽报告(分数、findings、spicy/summary)。省略路径打印到 stdout;给出路径则写文件(自动创建父目录)。--theme <name>只换 CSS 变量配色(默认ember与 0.1.16 相同,另有fog/midnight/paper/aurora);不影响 JSON、终端漂亮输出、GitHub 注解或打分。--list-themes打印这些主题名(每行一个;--list-themes --json为 JSON 数组)。--html与--json不能同时占用 stdout。 - Clipboard —
--copywrites the pretty terminal roast (not JSON / HTML / GitHub chrome) to the system clipboard. Missingpbcopy/wl-copy/xclip/xsel/clip→ skip silently. 剪贴板 —--copy把漂亮终端吐槽报告(不是 JSON / HTML / GitHub 装饰)写入系统剪贴板。没有pbcopy/wl-copy/xclip/xsel/clip则静默跳过。 - Optional deepen — with
OPENAI_API_KEY(preferred) orANTHROPIC_API_KEY, one extra paragraph after the heuristic roast. Failures and timeouts stay quiet.--offlineskips it. 可选加深 — 有 OpenAI(优先)或 Anthropic Key 时,启发式之后多一段评语;失败 / 超时静默。--offline跳过。
Links · 链接
- npm ·
@webywz/llm-roast - Source · github.com/webywz/llm-roast
- Related · prompt-diff · eval-smoke
- License · MIT
npm install && npm test && npm run typecheck