slop-grader

Rule-based slop grader for text files, powered by Jev. Runs every rule against every line in parallel. No skimming, no missed lines.

• How it works
• Features
• FAQ
• Quick Start
• Rulesets
• CLI Reference
• Output Formats
• Development
• Changelog
• Contributing
• Roadmap

How it works

slop-grader runs as a two-step loop: grade text with the CLI, then paste the output to your AI agent to plan the improvements.

1. Grade the document

Run slop-grader on a document like examples/slop.md:

npx @lukstei/slop-grader@latest -r no-ai-slop examples/slop.md

2. Fix with an AI agent

Pass the output to your AI agent:

• The agent distinguishes real violations from false positives and generates concrete replacements (Example with Gemini 3.8 Flash):

  Line 1 — banned_word

  • Original: # 🚀 The Ultimate Paradigm Shift in Modern Data Architecture
  • Fix: # Modern Data Architecture
  • Reason: Removes the banned phrase "paradigm shift" and decorative emoji.

  Line 7 — binary_contrast + faux_insight + colon_reveal

  • Original: What most people get wrong about databases is simple: it's not about speed, it's about trust.
  • Fix: Database design balances speed and trust.
  • Reason: Removes rhetorical framing and fake insight.

• After your review the plan is applied to produce an improved document.

Features

• Parallel exhaustive grading: Checks every rule against every line independently. No skimming.
• Incremental line caching: Re-evaluates only edited lines; unchanged text resolves from cache with zero API calls. Toggle with --no-cache.
• System One efficiency: Typed probabilities via Jev without text generation. Thousands of checks for cents.
• Dynamic batching: Groups lines to token limits to minimize API calls. See evaluation details.
• Context window padding: Evaluates line rules with $\pm 10$ lines of surrounding context. Boundary lines and sparse edits keep full visibility without paying for extra questions.
• Line and document scope: Flags line patterns and rates whole documents on qualitative rubrics.
• Plain Markdown rulesets: Write rules in Markdown; validate offline with --check.
• Built-in rulesets: Ready-to-use rules for AI writing patterns, document scores, tech docs, and grammar.
• Agent and CI ready: Terminal output for agent fix plans; structured JSON for pipelines.
• Multi-provider: Works with TypeSafe AI and OpenRouter out of the box.

FAQ

Evaluation separates line-level checks (spotting specific patterns or phrases) from document-level checks (evaluating tone or overall structure).

Documents have hundreds of lines, but the number of rules is fixed. Sending one API request per line would mean hundreds of calls. Instead, slop-grader dynamically groups lines into batches sized to fit the model's context budget (up to 255 lines per batch) and evaluates each rule across its batch in a single call. Batches include up to 10 lines of surrounding document context padding so boundary lines and sparse edits retain neighboring visibility for cross-line checks. Every batch response is verified for complete answer-to-question parity; any dropped questions halt execution immediately without caching incomplete results.

Document rules run in a single request across the entire text.

Line rules often depend on nearby text: pronoun references, synonym repetition across sentences, or transitions between paragraphs. Evaluating a line in isolation makes these checks impossible.

slop-grader pads each evaluated line with up to 10 lines of surrounding text ($\pm 10$ lines):

1. Region clustering: Lines slated for evaluation are grouped into continuous regions. Targets within 21 lines of each other merge into a single region so the model sees uninterrupted text without repeated lines.
2. Structured omission: When distant lines or sparse edits share a batch, gaps between regions are marked with ...| [omitted lines], preserving uniform line structure for the model.
3. Zero extra question cost: Padding lines exist strictly as non-evaluated context in the prompt state. Questions are generated only for active evaluation targets, so context lines add no question fees and don't count toward rule quotas.

This ensures single-line edits during incremental cache runs retain their full document surroundings rather than floating in isolation.

Line evaluations are cached by the hash of the line's content and the rule's criteria.

When you edit a document and run slop-grader again:

1. Every line is matched against the local cache for each rule.
2. Unchanged lines resolve immediately from the cache with zero API calls and zero cost.
3. Only new or modified lines are batched and sent to the model.
4. If a rule's instructions or criteria change, its cache invalidates automatically.

This makes repeated runs on large files or during editing loops nearly instantaneous. Pass --no-cache to bypass the cache, or --cache-dir <dir> to customize its location. See docs/CACHE.md for technical details on storage layout, hashing, and eviction.

Standard generative LLMs evaluate an entire document in a single prompt against a list of rules. On longer texts, they skip lines, miss rules, and report wrong line numbers.

Running a separate check for every line and rule with a generative LLM is impractical. A 300-line draft tested against 10 rules would require 3,000 text-generation requests, which is slow and expensive.

slop-grader uses a System One model (Jev). System One models answer discrete semantic questions with typed probabilities without generating text. Because these judgments return numbers instead of prose tokens, slop-grader can test every line against every rule separately and in parallel.

Cost scales with the number of rules and non-empty lines.

Checking examples/slop.md (16 text lines) against 53 rules costs roughly $0.0053 (half a cent):

Rules applied:                53 (48 line, 5 document)
Lines evaluated:              16
Questions asked:              773
API calls:                    49
Questions evaluated via API:  773

Checking a 2,100-word article (300 text lines) against 42 rules costs roughly $0.076 (7.6 cents):

Rules applied:                42
Lines evaluated:              300
Questions asked:              12600
API calls:                    84
Questions evaluated via API:  12600

Re-evaluating an edited document only costs for changed lines, unchanged lines resolve from cache with zero API calls.

Because Jev evaluates semantic probabilities instead of generating text tokens, running thousands of parallel checks costs a fraction of standard LLM generation.

Yes. You can write custom rules in Markdown (-r ./my-rules.md) or JSON. Rules ask plain-text questions about a single line or the whole document, evaluated against criteria you define. See Custom rulesets in the Rulesets section for syntax details and validation instructions.

slop-grader works on any plain text, structured file, or code diff:

• Catch AI writing habits, filler adverbs, and rhetorical formulas in drafts and essays.
• Check technical documentation for missing code examples, disorganized steps, or unexplained jargon.
• Scan code review diffs for swallowed errors, hardcoded credentials, and unjustified type assertions.
• Audit CSV spreadsheets and transaction logs for values exceeding approval thresholds.
• Review contracts and legal agreements for uncapped indemnification obligations.
• Check customer support transcripts for unreleased feature commitments or unauthorized discounts.
• Evaluate incident postmortems to confirm they address systemic defenses instead of individual human error.

See the Rulesets section for pre-built rules and example templates.

Quick Start

Requirements

• Node.js 18+
• An API key for your chosen provider:
  • TYPESAFE_API_KEY — typesafe.ai (uses jev provider)
  • OPENROUTER_API_KEY — openrouter.ai (uses openrouter provider)

Run

export TYPESAFE_API_KEY=...
npx @lukstei/slop-grader@latest -r no-ai-slop -r article-scores my-draft.md

Rulesets

Built-in rulesets

List built-in rulesets directly with --list-rulesets (or -l):

npx @lukstei/slop-grader@latest --list-rulesets
# or with structured JSON including rule IDs:
npx @lukstei/slop-grader@latest -l --json

Pass built-in rulesets by name (-r article-scores):

| Ruleset | What it checks | |---|---| | article-scores | Document-level scores: engagement, narrative arc, closing strength | | tech-docs | Technical documentation patterns: structure, task orientation, completeness, code examples, minimizing complexity | | grammar-english | English grammar: spelling and confused words, agreement, verb tenses, prepositions, pronouns, sentence structure, comparatives | | grammar-german | German grammar: spelling and confused words, agreement and inflection, word order, punctuation and typography | | no-ai-slop | Banned words, empty adverbs, puffery, colon reveals, bold lead-in lists, weasel attribution, dramatic fragmentation |

Custom rulesets

Define custom rules in Markdown (-r ./my-rules.md), organized under # Line Rules and # Document Rules sections.

You can be creative and ask any plain-text question about a single line or the whole document. Rules work on any text, git diffs, server logs, legal contracts, or structured text like CSV files:

# Line Rules

## empty_adverb
Does the line use an adverb that adds nothing to the meaning?

### Criteria
- **true**: The adverb could be deleted without changing the sentence.
- **false**: The adverb carries real emphasis or spoken rhythm.

# Document Rules

## narrative_arc
Rate the narrative arc of the document.

### Criteria
- No clear arc — sections feel disconnected
- Loosely organized — a theme but no build
- Clear progression — each section sets up the next
- Tight arc — the ending pays off the opening

Keep rules granular

Test one pattern per rule. Bundling typos, word choice, and punctuation into a single check hurts model precision, hides which condition triggered in the report, and makes rules harder to tune without regressions. A specific rule ID like compound_spacing (instead of a generic spelling) shows immediately what failed.

Inspirations for rulesets

# Line Rules

## suspicious_refund
Does this CSV transaction row show a refund exceeding $500 without a manager approval ID in column 6?

### Criteria
- **true**: The row records a refund over $500 and column 6 lacks an approval ID.
- **false**: The amount is $500 or less, column 6 contains an approval ID, or the row is not a refund.

# Document Rules

## root_cause_depth
Evaluate whether this postmortem addresses systemic engineering safeguards instead of individual human error.

### Criteria
- Blames operator error without addressing missing guardrails
- Identifies immediate triggers but ignores underlying architecture
- Identifies failure modes and plans concrete monitoring or test coverage
- Proposes systemic automated defenses, blast-radius containment, and architectural fixes

# Line Rules

## uncapped_indemnity
Does this clause expose the company to uncapped indemnification for third-party claims?

### Criteria
- **true**: The clause creates an indemnification obligation without liability caps.
- **false**: The obligation falls under the standard aggregate liability limit.

# Line Rules

## unauthorized_promise
Does this agent response promise an unreleased feature date or custom contract concession?

### Criteria
- **true**: Agent commits to an unannounced date or non-standard term.
- **false**: Agent refers customer to public docs or defers to account managers.

# Line Rules

## unjustified_type_cast
Does this line use a type assertion (`as`), non-null assertion (`!`), or loose cast to silence a compiler error without proper narrowing or input validation?

### Criteria
- **true**: Casts away type safety without an upstream type guard, schema validation, or explanatory comment.
- **false**: Type is narrowed safely, or the assertion bridges an external API boundary with runtime checks.

# Line Rules

## hardcoded_secret
Does this line contain a hardcoded API key, bearer token, private key, or password rather than referencing an environment variable or secret manager?

### Criteria
- **true**: Line contains a literal credential, private token, or hardcoded secret string.
- **false**: Line references an environment variable, config placeholder, mock test fixture, or public key.

# Line Rules

## swallowed_error
Does this catch block or fallback expression silence an unexpected error without diagnostic logging or recovery?

### Criteria
- **true**: Catches an exception and returns null or an empty default without logging context.
- **false**: Logs the error with context, rethrows, or implements a documented recovery strategy.

# Document Rules

## commit_intent
Does this commit message or PR description explain the motivation and problem context rather than merely describing code changes?

### Criteria
- Mechanical change list only with no rationale
- Mentions the fix with minimal explanation of the problem
- Explains the failure trigger, bug condition, and rationale clearly
- Details root cause, design tradeoffs considered, and verification evidence

See docs/SYNTAX.md for the complete Markdown rule syntax specification and validation reference.

Validate ruleset syntax offline without an API key:

npx @lukstei/slop-grader@latest --check -r ./my-rules.md

Use the create-slop-grader-rules skill to create and validate custom rulesets with an AI assistant.

Custom JSON rulesets (-r ./my-rules.json) are also supported.

CLI Reference

npx @lukstei/slop-grader@latest [-c|--check] [-l|--list-rulesets] -r <ruleset> [-r <ruleset> ...] [--provider <jev|openrouter>] [--model <model>] [--concurrency <num>] [--json] [--stats] [--debug] [--no-cache] [--cache-dir <dir>] [-h|--help] [-v|--version] [file]

Flags

| Flag | Short | Description | |---|---|---| | --list-rulesets | -l | List built-in rulesets and descriptions. | | --check | -c | Validate ruleset syntax without grading or calling the API. | | --rules <name\|path> | -r | Ruleset to apply. Repeatable. Accepts built-in names, Markdown (.md) files, or JSON file paths. | | --provider <jev\|openrouter> | -p | Override the AI provider. | | --model <model> | -m | Override the default model (jev-1.13.0 for jev, typesafe/jev-1.13 for openrouter). | | --concurrency <num> | | Maximum concurrent API requests (defaults to 5). | | --json | -j | Emit structured JSON instead of the human-readable report. | | --stats | -s | Print execution statistics (rules applied, lines evaluated, questions asked, API calls, questions evaluated via API, cache hits). | | --debug | -d | Log all API calls (timing, request, response) as JSON to stderr. | | --no-cache | | Disable line-level caching (evaluates all lines from scratch). | | --cache-dir <dir> | | Override the cache directory (defaults to OS cache directory). | | --help | -h | Display usage information. | | --version | -v | Display version number. |

Providers and environment variables

| Variable | Description | |---|---| | TYPESAFE_API_KEY | API key for direct Jev access via TypeSafe AI. Automatically selects jev. | | OPENROUTER_API_KEY | API key for OpenRouter. Automatically selects openrouter. | | TYPESAFE_PROVIDER | Explicitly choose jev or openrouter without passing --provider. |

Provider resolution order:

1. --provider (-p) flag
2. TYPESAFE_PROVIDER environment variable
3. Auto-detected from keys (TYPESAFE_API_KEY selects jev; OPENROUTER_API_KEY selects openrouter)

Grading runs on jev-1.13.0 (TypeSafe) or typesafe/jev-1.13 (OpenRouter) by default, overridable via --model (-m).

Output Formats

Human-readable report

By default, slop-grader prints a human-readable report. Clean lines are omitted; only lines crossing the 0.8 confidence threshold appear. Pass --stats (or -s) to append execution metrics (rules applied, lines evaluated, API calls).

If line rules run but find no violations, No line rule violations found. is displayed.

JSON report (--json)

Pass --json (or -j) for structured machine-readable output:

npx @lukstei/slop-grader@latest -r no-ai-slop -r article-scores --json --stats my-draft.txt | jq .

{
  "file": "/abs/path/to/my-draft.txt",
  "rules": ["/abs/path/to/no-ai-slop.md"],
  "violations": {
    "lines": [
      { "lineNum": 1, "text": "Our platform empowers teams...", "rules": ["banned_word"] }
    ],
    "document": {
      "narrative_arc": { "score": 1.4, "max": 3, "confidence": 0.72, "label": "Loosely organized" }
    }
  },
  "stats": {
    "rules": 6,
    "lineRules": 5,
    "docRules": 1,
    "lines": 12,
    "questions": 61,
    "apiCalls": 6
  }
}

violations.lines and violations.document are empty when the file is clean. Useful for CI pipelines and editor integrations.

Development

npm test         # Run tests
npm run verify   # Run typecheck, biome lint, and tests
npm run build    # Build

Changelog

See CHANGELOG for release history and notable changes.

Contributing

See CONTRIBUTING for contribution guidelines, development setup, and coding best practices.

Roadmap

Planned features we'd like to bring to slop-grader. Have an idea? Open an issue.

• [x] Structured JSON output with --json (v0.1.5)
• [x] Execution statistics with --stats (v0.1.6)
• [x] Custom model override with --model (v0.1.7)
• [x] Debug payload logging with --debug (v0.1.7)
• [x] Markdown-authored rulesets (v0.2.0)
• [x] Offline ruleset validation with --check (v0.2.0)
• [x] Document-level scoring rules (v0.2.1)
• [x] Ruleset listing with --list-rulesets (v0.2.3)
• [x] Incremental line-level caching (v0.2.5)
• [ ] Local web UI for in-browser grading (slop-grader --ui)
• [ ] Official GitHub Action for CI and pull requests
• [ ] Inline pull request review comments and score summaries
• [ ] Markdown mode to skip code blocks and syntax
• [ ] Web playground to test rulesets and generate CLI commands
• [ ] Remediation guidance and fix instructions in reports
• [ ] Expanded rulesets for git commits, PR descriptions, and ADRs
• [ ] Legal and policy document rulesets (Terms of Service, Privacy Policies)
• [ ] Standard input support (cat doc.md | slop-grader -)
• [ ] Token usage and estimated API costs in --stats
• [ ] Standalone binaries via Homebrew and APT (no Node runtime required)
• [ ] Interactive CLI prompt mode
• [ ] Agent skill installer (slop-grader init --skill)
• [ ] JSON and YAML field targeting with JSON Pointers
• [ ] YAML support for authoring rulesets
• [ ] Copy reports directly to clipboard (--clipboard)
• [ ] Dry-run mode to inspect payloads without API calls (--dry-run)
• [ ] Custom provider base URLs for proxies and self-hosted models (--base-url)
• [ ] Custom confidence thresholds per rule
• [x] Configurable request concurrency (--concurrency)
• [ ] Verbose operational logs (-v / --verbose)

License

MIT

Star History