slop-scan

Deterministic CLI for finding AI-associated slop patterns in JavaScript and TypeScript repositories.

Scan a repo, surface the hotspots, and compare codebases using normalized slop metrics.

slop-scan is a slop scanner, not an authorship detector. It reports explainable patterns and suspicious density. It does not claim who wrote the code.

Why use it

• Find the hotspots fast — see which files and directories concentrate the most suspicious patterns
• Understand why something was flagged — every finding includes a rule ID and evidence
• Compare repos fairly — normalize by file count, logical KLOC, and function count
• Benchmark heuristics over time — rerun the pinned benchmark set and watch movement

Good fit for

• checking third-party repos that feel vibe-coded
• comparing known AI-generated repos to mature OSS baselines
• finding low-judgment boilerplate in your own codebase
• iterating on deterministic slop heuristics

Install

Install globally with npm:

npm install -g slop-scan

Install it in a project and run it with npm tools:

npm install --save-dev slop-scan
npx slop-scan scan .

For local development in this repo:

bun install

Quick start

Scan the current repo:

slop-scan scan .

Scan the current repo in lint mode:

slop-scan scan . --lint

Scan another repo and get JSON:

slop-scan scan /path/to/repo --json

Recreate the pinned benchmark set from a source checkout:

bun run benchmark:update

Use it like a linter

Use --lint when you want human-readable findings in local runs, CI logs, or PR checks.

slop-scan scan . --lint

Example output:

medium  Found 3 duplicated function signatures  structure.duplicate-function-signatures
  at src/users/normalize.ts:1:1
  at src/teams/normalize.ts:1:1
  at src/accounts/normalize.ts:1:1

JSON output

Use --json when you want full-fidelity output for scripts, CI, or post-processing.

slop-scan scan . --json

Example CI check:

slop-scan scan . --json | jq -e '.summary.findingCount == 0'

The CLI currently exits non-zero for CLI/runtime errors, not for findings.

Delta comparisons

Use delta when you want a machine-readable comparison between two scans.

Compare two paths directly:

slop-scan delta ../main .
slop-scan delta --base ../main --head . --json

Compare two saved reports:

slop-scan scan ../main --json > base.json
slop-scan scan . --json > head.json
slop-scan delta --base-report base.json --head-report head.json

Fail CI only when new or worse occurrence-level findings show up:

slop-scan delta --base ../main --fail-on added,worsened

delta --json emits a generic report format with:

• base/head scan summaries
• occurrence-level change classification (added, resolved, worsened, improved)
• per-path score deltas
• report metadata and config hashes so downstream tools can detect mismatched scan conditions
• stable per-occurrence fingerprints for built-in rules, so grouped findings can match across rescans without relying on rendered message text

What it catches

Current checks focus on patterns that often show up in unreviewed generated code:

• log-and-continue catch blocks
• error-obscuring catch blocks (default-return or generic replacement error)
• empty catch blocks
• async wrapper / return await noise
• pass-through wrappers
• barrel density
• duplicate helper/function signatures across source files
• over-fragmentation
• directory fan-out hotspots
• placeholder comments
• duplicated test mock/setup patterns

What you get back

• raw repo score
• normalized metrics:
  • score / file
  • score / KLOC
  • score / function
  • findings / file
  • findings / KLOC
  • findings / function
• top file hotspots
• top directory hotspots
• grouped lint-style findings with --lint
• full-fidelity findings with evidence in --json

Supported files

Current language support:

• .ts
• .tsx
• .js
• .jsx
• .mjs
• .cjs

Benchmarks

The repo ships with a pinned, recreatable benchmark set comparing known AI-generated repos against older solid OSS repos.

Blended score = geometric mean of the six normalized-metric ratios versus the mature OSS cohort medians, then rescaled so the mature OSS cohort median is 1.00. Higher means a repo is consistently noisier across the benchmark dimensions.

Cohort medians

| Metric | AI median | Mature OSS median | Ratio | | ------------------- | --------: | ----------------: | --------: | | Blended score | 3.48 | 1.00 | 3.48x | | Score / file | 0.99 | 0.19 | 5.17x | | Score / KLOC | 9.51 | 4.42 | 2.15x | | Score / function | 0.23 | 0.09 | 2.49x | | Findings / file | 0.31 | 0.07 | 4.44x | | Findings / KLOC | 2.96 | 1.40 | 2.12x | | Findings / function | 0.08 | 0.03 | 2.99x |

Pinned benchmark snapshot

Ordered by blended score.

| Repository | Cohort | Ref | Blended | Score/file | Score/KLOC | Findings/file | Findings/KLOC | | --------------------------------------------------------------------- | ---------- | --------- | -------: | ---------: | ---------: | ------------: | ------------: | | garrytan/gstack | ai | 6cc094c | 5.94 | 2.34 | 21.71 | 0.52 | 4.85 | | redwoodjs/agent-ci | ai | 4de00d6 | 3.98 | 0.99 | 10.95 | 0.31 | 3.42 | | jiayun/DevWorkbench | ai | ea50862 | 3.77 | 1.00 | 10.76 | 0.44 | 4.69 | | openclaw/openclaw | ai | 44cf747 | 3.50 | 1.08 | 10.93 | 0.32 | 3.29 | | robinebers/openusage | ai | 857f537 | 3.48 | 1.33 | 8.30 | 0.34 | 2.11 | | emdash-cms/emdash | ai | dbaf8c6 | 2.47 | 0.75 | 6.67 | 0.23 | 2.02 | | FullAgent/fulling | ai | d95060f | 2.40 | 0.53 | 9.51 | 0.16 | 2.96 | | cloudflare/vinext | ai | 28980b0 | 2.21 | 0.48 | 9.20 | 0.15 | 2.76 | | vitejs/vite | mature-oss | bdc53ab | 1.65 | 0.26 | 7.95 | 0.08 | 2.45 | | withastro/astro | mature-oss | 2c9bf5e | 1.63 | 0.27 | 5.68 | 0.09 | 2.02 | | modem-dev/hunk | ai | b37663f | 1.32 | 0.38 | 4.71 | 0.13 | 1.55 | | egoist/tsup | mature-oss | b906f86 | 1.03 | 0.21 | 3.61 | 0.08 | 1.42 | | umami-software/umami | mature-oss | 0a83864 | 1.01 | 0.15 | 4.17 | 0.06 | 1.61 | | sindresorhus/execa | mature-oss | f3a2e84 | 0.99 | 0.17 | 4.85 | 0.05 | 1.37 | | antfu-collective/ni | mature-oss | 6d96905 | 0.73 | 0.11 | 4.68 | 0.02 | 0.94 | | mikaelbr/node-notifier | mature-oss | b36c237 | 0.46 | 0.08 | 0.90 | 0.04 | 0.47 | | vercel/hyper | mature-oss | 2a7bb18 | 0.46 | 0.65 | 1.12 | 0.16 | 0.28 |

Full benchmark assets:

• manifest: benchmarks/sets/known-ai-vs-solid-oss.json
• snapshot: benchmarks/results/known-ai-vs-solid-oss.json
• report: reports/known-ai-vs-solid-oss-benchmark.md

Configuration

The analyzer reads slop-scan.config.ts, slop-scan.config.js, slop-scan.config.mjs, slop-scan.config.cjs, or slop-scan.config.json from the scan root. Root .gitignore entries are also respected.

{
  "ignores": ["dist/**", "coverage/**", "**/*.generated.ts"],
  "plugins": {
    "acme": "slop-scan-plugin-acme"
  },
  "extends": ["plugin:acme/recommended"],
  "rules": {
    "structure.over-fragmentation": { "enabled": true, "weight": 1.2 },
    "comments.placeholder-comments": { "enabled": false },
    "acme/no-generated-wrapper": { "enabled": true, "options": { "threshold": 3 } }
  },
  "overrides": [
    {
      "files": ["src/rules/**"],
      "rules": {
        "structure.directory-fanout-hotspot": { "enabled": false },
        "structure.over-fragmentation": { "enabled": false }
      }
    }
  ]
}

Supported today:

• ignores
• plugins.<namespace> as either a package/path string or a plugin object in module configs
• extends: ["plugin:<namespace>/<config>"]
• rules.<id>.enabled
• rules.<id>.weight
• rules.<id>.options
• overrides[].files
• overrides[].rules.<id>.enabled
• overrides[].rules.<id>.weight
• overrides[].rules.<id>.options

Plugins

slop-scan can load third-party rule plugins and plugin preset configs from JSON or module configs.

For plugin setup, naming rules, and authoring examples, see docs/plugins.md. Simple plugin rules can now declare stable delta behavior with helpers like delta.byPath() / delta.byLocations(), and clustered rules can attach lightweight deltaKeys instead of building fingerprints manually.

See also:

• examples/local-plugin/contains-word-plugin.mjs
• examples/local-plugin/slop-scan.config.ts

This repo also commits a root slop-scan.config.json for self-scans and local development. It keeps the scan focused on the tool itself by excluding heavyweight benchmark checkouts and intentionally disables directory-structure rules under src/rules/**.

How it works

slop-scan is built as a pluggable engine:

• language plugins
• fact providers
• rule plugins
• reporters

That keeps the analyzer deterministic and extensible without turning it into one giant loop of ad hoc checks.

Docs

• plugin guide: docs/plugins.md
• benchmark guide: benchmarks/README.md
• pinned benchmark report: reports/known-ai-vs-solid-oss-benchmark.md
• exploratory note on non-JS/TS candidates: reports/exploratory-vite-astro-openclaw-beads.md

Contributing

Issues and pull requests are welcome.

Local validation

bun run format:check
bun run lint
bun test

Stable self-scan

bun run lint includes a stable self-scan.

It runs the last published slop-scan release against this repo using the committed root config in slop-scan.config.json, then compares the result to tests/fixtures/self-scan-stable-baseline.json.

The check currently fails only when the stable release reports either:

• a higher finding count; or
• a higher repo score.

Useful commands:

bun run lint:self
bun run lint:self:update

Use bun run lint:self:update only when you intentionally accept the new stable self-scan baseline.

Pre-commit hook

A Husky pre-commit hook runs:

bun run format:check
bun run lint

Heuristic changes

If you change rule behavior materially, also rerun:

bun run benchmark:update

License

MIT