@dcwt/semantic-lint

A second reader for the conventions ESLint can't express. It asks Jev (a System One model) yes/no questions about code and diffs, and compares each answer (a probability) against a floor set from evidence.

It complements eslint . (syntax-level rules) and npm run deps:check (the import graph). It never replaces either.

The pattern every tool here shares

question  +  state (a file, or a diff)  →  Jev  →  p (0..1)  →  compare to floor  →  finding / escalation
                                                                    ↑
                                    eval script: labelled fixtures from git history, sweep floors, pick one

1. A question: one noul (yes/no) question per rule, watch or risk. Everything Jev knows about the rule has to be in the question text.
2. Fixtures: real before/after examples, mined from git history or made by corrupting a good file into a bad one.
3. Floors: an eval script runs the fixtures and sweeps floors. Each floor is written down with a dated baseline in the eval file's header.
4. Skips never fail a build: a missing key or a provider outage exits 0. Only an error-severity finding fails.

The three tools

| Tool | Reads | Asks | Runs | Exit | |---|---|---|---|---| | semantic-lint (src/cli.ts) | whole files | one question per applicable rule, one call per file | CI: changed files on PRs, the whole tree on main | 1 on an error finding | | docs-drift (src/docs-drift.ts) | a diff + one AGENTS.md section or ADR | "does this diff make the doc false without editing it?" or "does it contradict this decision?" | pre-push hook, check:diff, CI on PRs and on push to main | 1 on error; CI and the hook treat it as advisory | | review-gate (src/review-gate.ts) | a diff | six risk questions + path rules | pre-push hook, check:diff | always 0 | | semantic-lint --copy | page openings the web tests render | one question per copy-* rule | CI on push to main (all), check:diff (changed) | 1 on an error finding |

semantic-lint: conventions in files

• Rules live in src/rules/, one file each: 20 today.
  • Architecture: use-case shape, repositories, routes, error unions, Inngest steps.
  • copy-*: rendered page openings (see --copy below).
  • web-*: the design system.
  • test-*: tests a no-op would pass.
• A rule declares which files it applies to (appliesTo), the excerpt of the convention (context), the question, optional related files it needs to see (gathers: up to 3 files, 8,000 chars each), and floors: { error, warn }.
• Near-floor answers are asked twice: an answer within ±0.1 of a floor is asked again and the lower sample is kept, so one borderline answer can't flicker a build red.

npx semantic-lint                      # files changed vs origin/main
npx semantic-lint packages/use-cases   # a path
npx semantic-lint --rule use-case-shape --format json
npm run eval    # re-derive floors

Adding a rule:

1. Add one file in src/rules/ and an entry in the barrel (src/rules/index.ts).
2. Add a CANONICAL entry in src/eval/run.ts.
3. Add fixtures under src/eval/fixtures/<slug>/{violating,compliant}/.
4. Run npm run eval and set the floors from its output.

--copy: the sentences a reader sees

The briefs and copy modules are pure functions, and the web tests already walk their branches. So the lint grades what they return, not their source:

1. the web app's harvest:copy script reruns the web tests under vitest.harvest.config.ts. A Vite transform wraps every export of *-brief.ts, *-copy.ts, lib/standfirst.ts, lib/finn-move.ts and month-ahead.ts, and src/test/harvest-copy.ts records each call a test makes directly (calls one of those modules makes to another are only noted, as touches). It writes one JSON per test file to apps/web/.copy-harvest/. About 6 seconds, no test file changed.
2. semantic-lint --copy apps/web/.copy-harvest turns each page opening (a StandfirstCopy) into a unit: head:/figure:, hint:, lede:, facts:, move:, one line each, deduplicated on that text. Its path is <module>#standfirst/<fn>/<hash>; only copy-* rules match a #. With --changed, only openings whose module, test file or touched modules differ from origin/main (working tree included).
3. Findings print with the test that produced them and the opening as read.

copy-says-what-it-means asks whether an opening only restates quantities, leaving the reader to judge them. Its first sweep put true restatements on top, such as "You ended at $1,800. That is where the balance sat on Mon, Jul 20." It also flagged the descriptive destinations: Where it went's share heads, a category's places, repeats and the month ahead. ADR 0104 ruled that those owe a verdict too, and they were rewritten to say how they moved. It errors at 0.8 and warns at 0.65, and the first look is exempt by path, since before a plan there is nothing to judge against. Fixtures are .txt renders under src/eval/fixtures/copy-says-what-it-means/.

docs-drift: does this diff stale AGENTS.md?

• Watches (src/drift/watches.ts): each maps an AGENTS.md heading to the code path prefixes that can make it wrong. There are four today: Use-Case Pattern, Service Injection, Error Handling and Web app. Seven more sections were dropped because git history holds no correction to calibrate them against.
• Suppression: a diff that already edits the doc is skipped, because someone is already reviewing the prose.
• Decision records (src/drift/adr-watches.ts): the same runner asks whether a diff contradicts an ADR still in force (its status is Accepted). Each ADR is asked about only the paths it governs, with its Decision and Amendment sections plus the Decision of any ADR that amends it (alsoRead). v1 covers 0014 (field encryption), 0042 (Plaid failure classes) and 0054 (the MCP surface), all warn-only. 0021 (tenancy) was dropped because it doesn't separate. adr-watches.test.ts fails if a registered ADR is superseded or a live amender is missing.
• Eval: npm run eval:drift uses fixtures in src/eval/fixtures-drift/. Each one pairs a commit's code diff with the doc before the commit (violating) and after it (compliant).

review-gate: does this diff need a deep review?

It routes; it doesn't judge. A false escalation costs one extra review, so it's tuned for recall and never re-asks.

1. Path rules first (triggers in src/gate/dimensions.ts). These need no key: migrations (flagged separately when one drops, renames or tightens), auth/crypto, webhooks, account deletion and retention, and boot/CI/deploy config.
2. Then Jev, with six questions: tenancy, money, auth-secrets, money-events, durable-jobs and data-lifecycle. Each asks whether the diff touches that risk, not whether it breaks a rule.
   • A big diff is split into batches of about 50k characters, riskiest files first. Each dimension keeps its highest answer across batches.
   • Past 8 batches (about 1¢), the rest goes unread and the diff escalates on size.
3. Output: ESCALATE: /code-review high — focus: … with the files each reason points at. Otherwise, the highest dimension that didn't escalate.

Floors (2026-09-27): money 0.8, tenancy 0.7, the rest 0.5. On the last 150 commits, 25% escalate; the target is 15–30%, and above 50% the gate is noise. Re-run with:

npm run eval:gate -- --replay 150   # ~6¢

The eval's labelled commits are live SHAs (LABELLED in src/eval/gate.ts), not stored diffs. The replay prints how many commits sit at or above each floor, per dimension.

Where it runs

Work lands on main by push, not PR, so the push is the checkpoint.

| When | What | How | |---|---|---| | before committing or merging a worktree | review-gate + docs-drift + the page openings it could change, against origin/main, working tree included | npm run check:diff | | every git push | both, over exactly what the push sends | .githooks/pre-push. Enable once per clone: git config core.hooksPath .githooks | | CI, push to main | docs-drift (advisory, continue-on-error) + whole-tree lint + every page opening | semantic-lint job in .github/workflows/ci.yml | | CI, pull request | lint the changed files + docs-drift | same job (in practice only Dependabot opens PRs, and those runs get no key) |

Setup

• Key: AI_DEFAULT_API_KEY, the OpenRouter key; direnv loads it from .envrc. AI_MODEL_SYSTEM_ONE optionally pins the model.
• Build: npm run build. The hook and check:diff run dist/, so rebuild after changing this package or they run the old version.
• Cost: tiny. A gate or drift call is ≤0.13¢. A whole-tree lint is a few cents. Every run prints its cost.

Files

src/
  cli.ts, run.ts, collect.ts, report.ts, types.ts   semantic-lint
  rules/                                            one file per rule
  copy/units.ts                                     harvest → rendered `#standfirst/` units
  docs-drift.ts, drift/                             docs-drift (collect-diff.ts is shared with the gate)
  review-gate.ts, gate/                             review-gate: dimensions.ts holds triggers, questions, floors
  ask.ts                                            one System One call, retried once on a transient error
  eval/run.ts, eval/drift.ts, eval/gate.ts          the evidence for every floor, baselines in each header

Known limits

• Scores drift between Jev versions. Pin AI_MODEL_SYSTEM_ONE to a versioned id and re-run the evals when it changes.
• Thin fixtures make floors optimistic. When a real run flags something wrongly, add it as a fixture.
• The gate's durable-jobs question has no labelled positive yet, so its 0.5 floor is untested.