@webywz/context-diet
v0.13.0
Published
Slim LLM context before you send it — drop noise, keep the signal, estimate tokens.
Maintainers
Readme
context-diet
Slim LLM context before you send it.
把要喂给模型的上下文先瘦身。
Drop noise, keep signal, estimate tokens — locally, offline, in milliseconds.
丢掉噪声,留下信号,估算 token —— 本地、离线、毫秒级。
English
Demo
npx @webywz/context-diet --demo --preset logs --max-tokens 400Bundled examples/noisy.txt goes in; slimmed body on stdout, human report on stderr. No clone required — examples/ ships in the npm package.
context-diet 0.5.0
source: examples/noisy.txt
chars: 1,666 → 834 (−49.9%)
tokens: ~667 → ~312 (−53.2%)
cut:
boilerplate lines 9
duplicate paragraphs 2
omitted heavy blobs 1
…Features
| | |
| --- | --- |
| Offline heuristics | Strip ANSI/HTML chrome, drop boilerplate, dedupe, omit base64 blobs |
| Presets | logs, email, chat, html, code, json — sensible keep/strip defaults |
| Budget | --max-tokens drops low-value blocks first (shared across files/dirs); --keep pins headings/code/lists |
| Secret redaction | --redact scrubs PEM / JWT / sk- / ghp_ / Slack / AWS / Bearer / Basic → [redacted] |
| CI token gate | --fail-over-tokens <n> exits 1 when combined output tokens still > n (shared multi-file budget; no silent truncate); --sarif <path> for Code Scanning; --tap for TAP 13; JSON budget field for Actions |
| Pipe-friendly | Slimmed body → stdout; report → stderr |
| Bundled examples | --demo / examples/noisy.txt in the published tarball |
| Optional deepen | --deepen LLM second pass only when you opt in with a key |
| HTML report | --html <path> writes a self-contained before/after report (offline, no CDN) |
| Multi-file / dirs | Pass several files or directories; they share one --max-tokens / --fail-over-tokens budget (budget.shared, sources[]). Directories stay top-level unless -R / --recursive |
| Step summary | --github-summary appends a short diet report to $GITHUB_STEP_SUMMARY |
Install
npm i -g @webywz/context-dietNode 18+. No tokenizer files, no GPU, no account.
npx @webywz/context-diet --helpQuick start
context-diet input.txt
npx @webywz/context-diet --demo --preset logs --max-tokens 400
cat noisy.log | context-diet --preset logs --max-tokens 1500 > slim.txt
context-diet notes.md -o slim.md --keep headings,code,lists
context-diet input.txt --dry-run --diff
context-diet --demo --html diet.html -q
context-diet input.txt --json | jq '{tokens: .output.tokens, cuts}'
context-diet dump.txt --redact --fail-over-tokens 2000
context-diet a.md b.md notes/ --max-tokens 2000 --fail-over-tokens 2500 --json
context-diet --demo --fail-over-tokens 1 --sarif over.sarif
context-diet examples/noisy.txt --keep headings|code|listsPresets
| Preset | Keeps by default | Extra strippers |
| --- | --- | --- |
| logs | code | Prefer ERROR / WARN / FATAL when truncating |
| email | headings, lists | Quoted replies (> / On … wrote: / Original Message) |
| chat | code, lists | Reactions, join/leave, lone timestamps |
| html | headings, code, lists | <script> / <style> / comments / tags → prose |
| code | code | Long comments / blank runs; prefer signatures when truncating |
| json | code | JSONC comments out; drop null / ""; stable pretty-compact |
Explicit --keep always wins over the preset default.
What it does
All offline unless you pass --deepen.
| Step | Behavior |
| --- | --- |
| Redact secrets | Optional --redact before other steps |
| Strip ANSI | Terminal color / OSC sequences |
| Strip HTML | Optional / --preset html |
| Omit heavy blobs | Long base64 / data-URIs → [omitted blob: N chars] |
| Collapse whitespace | Never inside code fences |
| Drop boilerplate | Repeated log lines, signatures, DEBUG chrome |
| Drop quoted / chat chrome | Email quotes; reactions / join-leave |
| Dedupe paragraphs | Exact and near-identical (Jaccard ≥ 0.84) |
| Truncate | --max-tokens + --keep / presets |
| Fail over tokens | --fail-over-tokens hard CI gate on combined output tokens after diet (shared across inputs) |
| TAP | --tap TAP version 13 (exit 1 when over --fail-over-tokens) |
| JSON budget | budget.failOverTokens / budget.outputTokens / budget.ok / budget.shared / sources[] in --json |
Token estimates are cl100k-ish (~4 chars/word, 1 CJK ≈ 1 token). Good enough to budget a prompt — not a billed tokenizer.
CLI
context-diet [file|dir ...] [options]| Flag | Meaning |
| --- | --- |
| [file\|dir ...] | One or more paths; dirs expand to top-level files unless --recursive; omit or - for stdin. Multi-input shares one token budget |
| -o, --output <file> | Write slimmed text here instead of stdout |
| --html <path> | Write self-contained HTML report (stats + capped previews) |
| --max-tokens <n> | Cap estimated output tokens (truncates; shared across files/dirs) |
| --fail-over-tokens <n> | Exit 1 if combined output tokens still > n (CI gate; shared; distinct from truncate) |
| --sarif <path> | Write SARIF 2.1.0 when over --fail-over-tokens (GitHub Code Scanning) |
| --tap | TAP version 13 on stdout (CI; exit 1 if over --fail-over-tokens) |
| --redact | Scrub common secret shapes → [redacted] (before diet heuristics) |
| --keep <cats> | Prefer headings, code, lists when truncating (comma or \|) |
| --preset <name> | logs | email | chat | html | code | json |
| --exclude <glob> | Skip files matching glob pattern (repeatable; dirs only) |
| --include <glob> | Only keep paths matching at least one glob (repeatable; dirs only) |
| --max-files <n> | Fail (exit 3) when a dir scan would process > n files after filters |
| -R, --recursive | Also include files in nested directories. Default stays top-level only. Does not follow directory symlinks |
| --demo | Diet bundled examples/noisy.txt |
| --json | JSON report on stdout |
| --dry-run | Report only; no body / no -o write |
| --diff | Show removed lines on stderr (capped) |
| --deepen | Optional LLM rewrite |
| --provider openai\|anthropic | Force provider for --deepen |
| --github-summary [path] | Append diet Markdown to $GITHUB_STEP_SUMMARY or path |
| -q, --quiet | No human report on stderr |
| -h, --help / -V, --version | Help / version |
Default I/O stays pipe-friendly:
pbpaste | context-diet --preset chat --max-tokens 800 | pbcopyDirectory scanning
Directories expand to top-level files unless you pass -R or --recursive. Recursive mode also selects files in nested directories. It does not follow directory symlinks, so a link cannot loop or walk outside the tree. Names starting with . are skipped at every level.
When scanning directories, context-diet automatically skips:
node_modules/and.git/(always)- Patterns from
.gitignore(when found in project root or nearest git root) - Custom patterns via
--exclude <glob>(repeatable) - Hidden names (prefix
.)
You can also filter with include patterns (only files matching at least one --include glob are kept). Globs match the path relative to the directory you passed, and * does not cross /. --exclude '*.tmp' also matches that pattern against any single path segment, so it skips nested *.tmp files. --include '*.ts' matches only a top-level *.ts path; with --recursive, use **/*.ts to keep nested TypeScript files.
context-diet src/ --include '*.ts' --include '*.js'
context-diet src/ --exclude '*.tmp' --exclude 'test-*'
context-diet . --max-tokens 2000
context-diet src/ --recursive --include '**/*.ts' --max-files 100For safety, use --max-files <n> to prevent processing too many files. The limit is applied after include, exclude, gitignore, and (when set) recursive expansion:
context-diet docs/ --max-files 100 # exits 3 if > 100 top-level files
context-diet docs/ --recursive --max-files 100 # exits 3 if > 100 files after the nested walkOptional LLM pass
--deepen is never required. With a key:
export OPENAI_API_KEY=sk-... # or ANTHROPIC_API_KEY
context-diet chapter.md --deepen --max-tokens 1200Defaults: gpt-4o-mini / claude-3-5-haiku-20241022 (OPENAI_MODEL / ANTHROPIC_MODEL). Local result wins if the model returns longer text. Without a key, --deepen exits 2 and does not phone home.
CI (GitHub Actions)
Starter: examples/github-action.yml (includes --github-summary).
- uses: actions/setup-node@v4
with:
node-version: 22
- name: Slim context
run: |
npx --yes @webywz/context-diet --demo --preset logs --max-tokens 400 \
--redact --fail-over-tokens 500 > slim.txtLibrary
import { diet, dietAsync, estimateTokens } from "@webywz/context-diet";
const { text, input, output, cuts } = diet(raw, {
maxTokens: 2000,
preset: "logs",
keep: ["code"],
redact: true,
});
await dietAsync(raw, { deepen: true });
estimateTokens(text);Exit codes
| Code | Meaning |
| --- | --- |
| 0 | Success |
| 1 | Usage / missing input / unreadable file / --fail-over-tokens exceeded / --tap not-ok |
| 2 | --deepen failed (no key, HTTP error) |
| 3 | --max-files limit exceeded |
Develop
npm test && npm run build
node dist/cli.js --demo --preset logs --max-tokens 400Links
| | | | --- | --- | | npm | npmjs.com/package/@webywz/context-diet | | Issues | github.com/webywz/context-diet/issues | | License | MIT © webywz |
Multi-file budget + SARIF (0.9 / 0.10)
Documented in Features and Quick start above. Summary:
context-diet a.md b.md notes/ --max-tokens 2000 --fail-over-tokens 2500 --json
context-diet --demo --fail-over-tokens 1 --sarif over.sarif- Multiple files/dirs share one
--max-tokens/--fail-over-tokensbudget (JSONbudget.shared,budget.maxTokens,sources[]). - Directories expand to top-level files unless
-R/--recursiveis set.--max-filesapplies after that selection (exit3). Directory symlinks are not followed. --fail-over-tokensgates the combined dieted output, not each file alone.--sarif <path>writes SARIF 2.1.0 when output exceeds--fail-over-tokens.
中文
多文件共享预算 + SARIF(0.9 / 0.10)
详见上文 功能 与 快速开始。可传多个文件/目录,共享 --max-tokens / --fail-over-tokens(闸门看合并后输出);JSON 含 budget.shared、sources[]。--sarif <path> 在超 --fail-over-tokens 时写出 SARIF 2.1.0。目录默认只展开顶层文件;-R / --recursive 才包含嵌套文件,且不跟随目录符号链接。
演示
npx @webywz/context-diet --demo --preset logs --max-tokens 400包内自带 examples/noisy.txt(随 npm 包发布):瘦身后的正文在 stdout,人类可读报告在 stderr。无需 clone。
功能
| | |
| --- | --- |
| 离线启发式 | 剥 ANSI/HTML、去样板、去重、省略 base64 大块 |
| 预设 | logs / email / chat / html / code / json |
| 预算 | --max-tokens 优先丢掉低价值块(多文件/目录共享);--keep 钉住标题/代码/列表 |
| 密钥脱敏 | --redact 擦除 PEM / JWT / sk- / ghp_ / Slack / AWS / Bearer / Basic → [redacted] |
| CI token 闸门 | --fail-over-tokens <n>:合并后输出 token 仍 > n 时退出码 1(多文件共享预算;不静默截断);--sarif <path> 写 Code Scanning;--tap 输出 TAP 13;JSON budget 供 Actions |
| 管道友好 | 正文 → stdout;报告 → stderr |
| 示例随包 | --demo / examples/noisy.txt 在发布 tarball 里 |
| 可选 deepen | --deepen 仅在有 key 且你主动开启时走 LLM |
| HTML 报告 | --html <path>:写出自包含前后对比报告(离线、无 CDN) |
| 多文件 / 目录 | 可传多个文件或目录,共享一份 --max-tokens / --fail-over-tokens(budget.shared、sources[])。目录默认只扫顶层,-R / --recursive 才进入嵌套 |
| Step Summary | --github-summary:把精简饮食报告追加到 $GITHUB_STEP_SUMMARY |
安装
npm i -g @webywz/context-diet需要 Node 18+。无 tokenizer 文件、无 GPU、无账号。
快速开始
context-diet input.txt
npx @webywz/context-diet --demo --preset logs --max-tokens 400
cat noisy.log | context-diet --preset logs --max-tokens 1500 > slim.txt
context-diet notes.md -o slim.md --keep headings,code,lists
context-diet input.txt --dry-run --diff
context-diet --demo --html diet.html -q
context-diet input.txt --json | jq '{tokens: .output.tokens, cuts}'
context-diet dump.txt --redact --fail-over-tokens 2000
context-diet a.md b.md notes/ --max-tokens 2000 --fail-over-tokens 2500 --json
context-diet --demo --fail-over-tokens 1 --sarif over.sarif
context-diet examples/noisy.txt --keep headings|code|lists预设
| 预设 | 默认保留 | 额外剥离 |
| --- | --- | --- |
| logs | code | 截断时优先保留 ERROR / WARN / FATAL |
| email | headings, lists | 引用回复(> / On … wrote: / Original Message) |
| chat | code, lists | 表情反应、进出房系统行、孤立时间戳 |
| html | headings, code, lists | <script> / <style> / 注释 / 标签 → 纯文本 |
| code | code | 长注释 / 多余空行;截断时优先签名行 |
| json | code | 去 JSONC 注释;丢 null / "";稳定 pretty-compact |
显式 --keep 始终覆盖预设默认。
做了什么
除 --deepen 外全部离线。
| 步骤 | 行为 |
| --- | --- |
| 密钥脱敏 | 可选 --redact,在其他步骤之前 |
| 剥 ANSI | 终端颜色 / OSC |
| 剥 HTML | 可选 / --preset html |
| 省略大块 | 长 base64 / data-URI → [omitted blob: N chars] |
| 折叠空白 | 代码围栏内不动 |
| 去样板 | 重复日志行、签名、DEBUG 噪声 |
| 去引用 / 聊天 chrome | 邮件引用;反应 / 进出房 |
| 段落去重 | 精确与近似(Jaccard ≥ 0.84) |
| 截断 | --max-tokens + --keep / 预设 |
| Token 闸门 | --fail-over-tokens:饮食后对合并输出 token 的硬失败 CI 闸(多输入共享) |
| TAP | --tap:TAP version 13(超过 --fail-over-tokens 则退出 1) |
| JSON budget | --json 中的 budget.failOverTokens / budget.outputTokens / budget.ok / budget.shared / sources[] |
Token 估算为 cl100k 近似(约 4 字符/词,1 个 CJK ≈ 1 token)。够用来做预算,不是计费 tokenizer。
命令行要点
| 参数 | 含义 |
| --- | --- |
| [file\|dir ...] | 一个或多个路径;目录默认展开顶层文件(--recursive 才包含嵌套);省略或 - 读 stdin。多输入共享一份 token 预算 |
| -o, --output <file> | 把瘦身正文写到文件 |
| --html <path> | 写入自包含 HTML 报告(统计 + 截断预览) |
| --max-tokens <n> | 估算输出 token 上限(会截断;多文件/目录共享) |
| --fail-over-tokens <n> | 合并后输出 token 仍 > n 时退出 1(CI 闸;共享;与截断不同) |
| --sarif <path> | 超过 --fail-over-tokens 时写出 SARIF 2.1.0(GitHub Code Scanning) |
| --tap | 向 stdout 输出 TAP version 13(超过 --fail-over-tokens 则退出 1) |
| --redact | 擦除常见密钥形态 → [redacted](在饮食启发式之前) |
| --keep <cats> | 截断时优先保留 headings / code / lists |
| --preset <name> | logs | email | chat | html | code | json |
| --exclude <glob> | 跳过匹配 glob 模式的文件(可重复;仅目录) |
| --include <glob> | 仅保留匹配至少一个 glob 的路径(可重复;仅目录) |
| --max-files <n> | 当目录扫描过滤后将处理 > n 个文件时失败(退出码 3) |
| -R, --recursive | 同时纳入嵌套目录中的文件。默认仍只扫描顶层。不跟随目录符号链接 |
| --demo | 对包内 examples/noisy.txt 瘦身 |
| --json | stdout 输出 JSON 报告 |
| --dry-run | 只报告,不写正文 |
| --diff | stderr 显示被删行(有上限) |
| --deepen | 可选 LLM 二次压缩 |
| --provider openai\|anthropic | 强制 deepen 提供商 |
| --github-summary [path] | 追加饮食 Markdown 到 $GITHUB_STEP_SUMMARY 或指定路径 |
| -q, --quiet | 不打印人类报告 |
默认 I/O 适合管道:
pbpaste | context-diet --preset chat --max-tokens 800 | pbcopy目录扫描
目录默认只展开顶层文件。传入 -R 或 --recursive 后,才会纳入嵌套目录中的文件。递归不会跟随目录符号链接,因此不会因链接成环,也不会顺着链接走出当前目录树。每一层以 . 开头的名字都会跳过。
扫描目录时,context-diet 自动跳过:
node_modules/和.git/(始终).gitignore中的模式(在项目根或最近的 git 根找到时)- 通过
--exclude <glob>自定义模式(可重复) - 以
.开头的隐藏名字
也可以用 include 模式过滤(仅保留匹配至少一个 --include glob 的文件)。glob 匹配的是相对于所给目录的路径,* 不跨越 /。--exclude '*.tmp' 还会用该模式匹配任意单个路径段,因此会跳过嵌套的 *.tmp。--include '*.ts' 只匹配顶层的 *.ts;在 --recursive 下要用 **/*.ts 才能选中嵌套的 TypeScript 文件。
context-diet src/ --include '*.ts' --include '*.js'
context-diet src/ --exclude '*.tmp' --exclude 'test-*'
context-diet . --max-tokens 2000
context-diet src/ --recursive --include '**/*.ts' --max-files 100为安全起见,使用 --max-files <n> 防止处理过多文件。计数发生在 include、exclude、gitignore 以及(若开启)递归展开之后:
context-diet docs/ --max-files 100 # 顶层文件 > 100 时退出码 3
context-diet docs/ --recursive --max-files 100 # 嵌套展开后 > 100 时退出码 3可选 LLM 二次压缩
--deepen 不是必须的。有 key 时:
export OPENAI_API_KEY=sk-... # 或 ANTHROPIC_API_KEY
context-diet chapter.md --deepen --max-tokens 1200默认模型:gpt-4o-mini / claude-3-5-haiku-20241022。若模型返回更长,保留本地结果。无 key 时 --deepen 退出码 2,不会联网。
退出码
| 码 | 含义 |
| --- | --- |
| 0 | 成功 |
| 1 | 用法错误 / 缺输入 / 文件不可读 / --fail-over-tokens 超限 / --tap not-ok |
| 2 | --deepen 失败(无 key、HTTP 错误) |
| 3 | --max-files 超限 |
链接
| | | | --- | --- | | npm | npmjs.com/package/@webywz/context-diet | | Issues | github.com/webywz/context-diet/issues | | License | MIT © webywz |
