eval-lint

Write lint rules in plain English and run them through ESLint or Oxlint. Each rule is a prompt; an evaluation model (TypeSafe AI's Jev through Vercel AI Gateway by default) answers whether each file violates it and where. Requires Node.js 22.18+; supports ESLint 9 and 10 and Oxlint 1.83+.

Get started

Install eval-lint next to your linter:

npm install --save-dev eval-lint oxlint

Enable a built-in plugin. For Oxlint, in .oxlintrc.json:

{
  "jsPlugins": ["eval-lint/vercel-design"],
  "rules": {
    "vercel-design/no-metadata-badges": "warn",
    "vercel-design/meaningful-monospace": "warn"
  }
}

For ESLint, in eslint.config.ts (or .js):

import vercelDesign from "eval-lint/vercel-design";

export default [{
  files: ["src/**/*.{jsx,tsx}"],
  plugins: { "vercel-design": vercelDesign },
  rules: { "vercel-design/no-metadata-badges": "warn", "vercel-design/meaningful-monospace": "warn" },
}];

Authenticate with Vercel OIDC (no static key to manage), then run:

vercel link && vercel env pull
npx eval-lint src                    # Oxlint (default)
npx eval-lint --engine eslint src    # ESLint

eval-lint runs your linter, evaluates every enabled rule in batched, concurrent requests, and runs the linter again so findings arrive through your normal output, severities, and suppression comments. Add .eval-lint/ to .gitignore: it holds the evaluation cache, which is what lets npx oxlint or your editor's ESLint report the same findings instantly afterwards, and evaluate a freshly saved file on its own.

The runner loads .env.local, then .env, without replacing existing variables. AI_GATEWAY_API_KEY takes precedence over OIDC. Pulled OIDC tokens expire after 12 hours; rerun vercel env pull. Expired credentials fail the run instead of reporting a pass. See Vercel OIDC setup.

Built-in plugins: eval-lint/vercel-design (design.md and product-copy rules), eval-lint/test-quality, eval-lint/naming-cheatsheet, eval-lint/clarity, eval-lint/errors. Each is described below.

Write your own rules

A plugin is a file. Plugins and configs can be .ts; Node 22.18+ runs them directly:

// lint/errors.ts
import { definePlugin } from "eval-lint";

export default definePlugin({
  name: "errors",
  rules: {
    "useful-catch": "Catch blocks must recover, rethrow, or return an explicit failure. Logging alone is not recovery.",
  },
});

Enable it like any plugin, by path:

{
  "jsPlugins": ["./lint/errors.ts"],
  "rules": { "errors/useful-catch": "error" }
}

// eslint.config.ts
import errors from "./lint/errors.ts";

export default [{ files: ["src/**/*.js"], plugins: { errors }, rules: { "errors/useful-catch": "error" } }];

Keep your normal parser, language settings, ignores, and other rules in the linter config. ESLint TypeScript/JSX parsing still needs its usual parser setup; Oxlint handles JS/TS/JSX/TSX directly.

Share and publish plugins

Each plugin exports a normal plugin object. Combine local or npm packages in the linter config:

{
  "jsPlugins": [
    "@repo/lint-errors",
    "@repo/lint-testing",
    "./lint/product-copy.ts"
  ],
  "rules": {
    "errors/useful-catch": "error",
    "testing/descriptive-names": "warn",
    "product-copy/actionable-errors": "warn"
  },
  "overrides": [{
    "files": ["**/*.test.ts"],
    "rules": { "errors/useful-catch": "off" }
  }]
}

To publish a plugin, default-export definePlugin(...), ship JavaScript, and declare eval-lint as a dependency or peer dependency. Its name is the default Oxlint namespace. Consumers can choose an alias using Oxlint's { "name": "alias", "specifier": "package" } or ESLint's plugins: { alias: plugin }.

Severity, overrides, ignores, and suppression belong to linter configuration. Models and credentials belong to the consuming project, never the published rule package.

Rule API

type RuleDefinition = string | {
  instruction: string;
  message?: string;
  threshold?: number;
};

A plain prompt is a complete rule definition. Describe acceptable code; eval-lint asks whether the target code violates it, with true consistently meaning a violation. Rules that do not apply should return false. The prompt is also the diagnostic text by default. Its wording is preserved; eval-lint does not rewrite or automatically decompose it.

Use the object form only when you want a shorter diagnostic or a different sensitivity:

"useful-catch": {
  instruction: "Catch blocks must recover, rethrow, or return an explicit failure. Logging alone is not recovery.",
  message: "Handle the failure explicitly.",
  threshold: 0.8,
}

A plugin can carry frozen scenarios as ordinary source files, annotated in place:

export default definePlugin({ name: "errors", rules, fixtures: "./fixtures" });

// fixtures/payment-retry.js
import { charge } from "./billing.js";

export async function pay(order) {
  // eval-lint: violation useful-catch
  try {
    await charge(order);
  } catch (error) {
    console.error(error);
  }
}

eval-lint examples ./lint/errors.ts strips the marker lines, runs every fixture through the same pipeline the linter uses with every rule of the plugin enabled, and prints each expectation next to what was reported: the probability, the reported line, the change since the last run (▲/▼), and any report nobody expected. A marker annotates the block below it, up to the next blank line, the next marker, or the first line indented less than the block. // eval-lint: rules a, b at the top narrows which rules apply when a fixture legitimately violates something else. Files without violation markers are the valid cases. Fixtures get neutral paths in model state, so labels never influence the answer. The command exits 1 while any expectation disagrees, so it can gate a rule change in CI.

Fixtures can be as large as real files, which matters: many rules only make sense with the surrounding class, imports, or component in view, and a fixture written for one rule also proves the other rules stay quiet on it. The best fixtures are the before and after of a real review: the before file carries a marker on each hunk the review changed, the after file carries none, and the two together prove a rule fires on the mistake and stays quiet on the fix. Start each rule with a few of those and add one whenever a diagnostic surprises you; a prompt edit that fixes one fixture and quietly breaks another shows up as a ▼ on the other line.

A file starting with // eval-lint: holdout is scored like the rest but reported as agree/disagree only: no probabilities, no ▲/▼, no reported lines. Prompts get tuned against whatever numbers are visible, so keep some fixtures the prompt author has never looked at; their agreement rate is the only number that says anything about files the rules have not met.

Both message and threshold are optional. The model never generates diagnostic text. Changing only the message or threshold reuses detection answers; newly reportable findings can require localization. The plugin's name is optional too and defaults to eval-lint; give separately shared plugins distinct names so they compose clearly in linter config.

Evaluation works on complete files in three passes, each pairing "is there a violation?" with "where is it?", the pattern TypeSafe documents for semantic search. Nothing is prepared, classified, or selected ahead of time, and no syntax index is collected: a rule is only its prompt, repeated verbatim in each boolean question because Jev answers noticeably better when the rule text is in the question rather than referenced elsewhere in the state.

1. Screen. Each file's source is sent to the model exactly once, split into consecutive 64-line chunks that together form the whole file, packed many files per request. Per file and rule, one boolean asks whether any chunk violates the rule and one Choice question over the chunk IDs asks which. Large requests are fast but lower every probability a little, so the screen only decides which chunks deserve a second look: chunks holding at least 0.4 (or the threshold, if lower).

2. Verify. Those chunks are re-judged in requests of at most six files, each holding the chunk with one neighboring chunk on either side as context and its lines tagged with IDs. Per chunk, one boolean judges it and one Choice over its line IDs ranks where the violation is. A run small enough for one wave of verification-sized requests (a few files, as lint-staged or a single-file lint produces) skips the screen: every chunk is judged and ranked directly, then confirmed, so such runs take two rounds.

3. Confirm. For each chunk verified at 0.6 or above, the top-ranked lines (up to three) each get their own boolean in the same small request shape. Line evidence then decides: a confirmed line at or above the threshold is a diagnostic at that line, with its own probability; lines that all fail (under 0.5) refute the chunk, which is how a file-level "yes" about several permitted status badges ends up reported nowhere; a hesitant line leaves the chunk's own probability standing, as with a violation spread over the lines of one element. Consecutive confirmed lines merge into one range.

The Choice questions are peaky: two violating chunks in one file come back as 0.99 and 0.03, and two violating lines in one chunk the same way. So the passes repeat, up to three rounds, only for what just changed. A file that reported a chunk is screened again with that chunk excluded from both questions; a chunk that reported a line is asked again about its other lines; a file the screen was sure about whose chunk verification refuted is screened again for the rest of its chunks. Each round costs requests only for the files that had a violation, and the loop stops as soon as a round finds nothing new. On the 655-file Community corpus this reports every font-mono metric in a dashboard instead of the first one, for roughly ten more requests.

Requests are transport batching, not dependency resolution: other files in a request are not evidence for the current question. Model instructions express that boundary; it is not hard context isolation. This release does not implement autofixes, cross-file loading, evidence-sufficiency checks, or an IDE service; an imported component's unseen internals are outside every judgment.

Thresholds resolve in this order:

1. Linter option: "errors/useful-catch": ["warn", { "threshold": 0.98 }]
2. Plugin rule's threshold
3. Runtime threshold, default 0.95

A threshold is a minimum estimated violation probability, not an accuracy guarantee. Start new rules at warn and evaluate labeled examples. Different providers can have different calibration.

Use native file or line suppression at the reported location. For file suppression:

/* oxlint-disable errors/useful-catch */

Use /* eslint-disable errors/useful-catch */ in ESLint. Suppression happens during reporting and does not necessarily prevent evaluation or source transmission. Use file ignores or overrides that turn rules off to exclude source from evaluation.

Vercel design example

eval-lint/vercel-design carries 24 rules from two sources: the UI guidance in Vercel's design.md, in particular its list of generated-design reflexes to reject (badges for metadata, decorative icons, eyebrows, em dashes, decorative gradients, monospace for prose, authoring narration), and the objective copy checklist behind Vercel's internal product-design skill (curly quotes, …, Verb + Noun on destructive actions, Title Case and sentence case positions, no successfully in toasts, banned words and phrases, no interjections or filler adverbs, toast periods, Couldn't vs Failed to, errors that name a recovery step, auth particles, please, third-person product narration, progressive disclosure of toggle-dependent controls).

A skill is guidance injected before an agent acts, and it steers every model the same way for as long as it stays installed, whether or not the model still needs it. These rules are the retroactive form of the same guidance: they only speak when a real violation lands in the code, so they reach the models and codebases that need them and fall silent as output improves, whether through a better model or a codebase whose canonical patterns the agent now copies. That is also why rule text describes the mistake rather than the whole style guide.

Every rule is a plain prompt string calibrated for a 0.80 threshold, which the example configs use. Typography applies to visible text; badges and icons are judged at their usage sites; copy rules apply to rendered strings, including default prop values, and not to comments, identifiers, or test assertions. Component documentation, analysis methodology, product instructions, operational statuses, and interactive controls are legitimate contexts.

node dist/cli.js -- --config examples/vercel-design/.oxlintrc.json examples/vercel-design/reports
node dist/cli.js --engine eslint -- --config examples/vercel-design/eslint.config.ts examples/vercel-design/reports

Import eval-lint/vercel-design in ESLint or add it to Oxlint's jsPlugins. The plugin and its labeled fixtures are bundled in the npm package. Both guideline sources are attributed in the plugin; this is an experimental adaptation, not an official Vercel plugin.

The package ships 34 small handwritten fixtures. The rules were also developed against before/after pairs of real reviewed files from a private codebase, 30 to 1,000 lines each, with a marker on every hunk the review changed and the reviewed result as the valid case; a third of those are holdouts the prompts were never tuned against. Files like that cannot be published, so any fixtures/private/ directory is gitignored and left out of the build, and pnpm examples runs the source plugins so a local checkout still evaluates them. Keep your own review-derived fixtures the same way.

pnpm examples on the full local set runs 988 rule/file checks in about 15 seconds. Across five refreshed runs it reports 27 to 31 disagreements of 1,010 expectations, with the holdout files agreeing on 234 to 237 of 250. The stable misses are curly-quotes on ' and straight apostrophes inside JSX attributes, title-case-labels on sentence-case headings, and no-please-by-default on strings the rule's own exemptions make arguable (the app is at fault, or the ask is inconvenient). About ten expectations hover within 0.1 of the threshold and flip between runs. These are development numbers for a labeled set, not an accuracy guarantee.

The rules inspect visible source and do not render the page, resolve imported styles, or verify responsive layout.

Naming cheatsheet example

eval-lint/naming-cheatsheet adapts kettanaito/naming-cheatsheet into an experimental naming plugin with 11 independently configurable rules:

• english-names, consistent-case, descriptive-names
• no-contractions, no-context-duplication, boolean-polarity
• function-context, action-verbs, boolean-prefixes
• boundary-state-prefixes, singular-plural

The example configs enable them at warning severity with a 0.90 per-rule threshold. Run them from this checkout:

node dist/cli.js -- --config examples/naming/.oxlintrc.json examples/naming/*.js
node dist/cli.js --engine eslint -- --config examples/naming/eslint.config.ts examples/naming/*.js

Import eval-lint/naming-cheatsheet; the plugin, fixtures, and upstream license are included in the npm package. Its linter namespace remains naming. External property reads, established technical abbreviations, and conventional language constructs are explicitly considered in the instructions. The adaptation can only judge context visible in one file. Diagnostics now identify a small source region, which can contain more than the offending identifier.

The plugin carries 30 fixtures at its 0.90 threshold; eval-lint examples eval-lint/naming-cheatsheet currently reports 10 of 11 marked violations and none of 19 valid fixtures flagged, with boolean-polarity at 0.83. This small handwritten development set is not an accuracy guarantee or held-out evaluation. Changing a threshold trades missed violations against false positives; the core default remains 0.95.

Test-quality example

eval-lint/test-quality provides five rules based on what the test code visibly does:

| Rule | Purpose | | --- | --- | | test-quality/no-tautological-assertions | Compare results with independent expectations | | test-quality/meaningful-assertions | Check the claimed behavior, not just execution | | test-quality/exercise-system | Exercise the real operation instead of its configured replacement | | test-quality/reachable-assertions | Ensure assertions execute and failures reach the runner | | test-quality/public-behavior | Avoid private state and internal call-order coupling |

The Oxlint config and ESLint config enable these as warnings for test files. Each rule has a configurable 0.80 threshold. Import eval-lint/test-quality directly; it and its fixtures ship in the npm package. No dependency traversal or test execution is required.

The context-policy Jev evaluation caught all 13 problematic examples and left all 12 valid examples clear, both individually and with all five rules enabled. This run also caught the previously missed assertion inside an uncalled local helper. Legitimate identity checks, dependency stubs, boolean/presence checks, invoked assertion helpers, and public event ordering are represented in the valid cases. This is a small development corpus, not a general accuracy estimate.

The plugin carries 25 fixtures; eval-lint examples eval-lint/test-quality reports all 13 marked violations caught and none of 12 valid fixtures flagged.

Configure AI

No runtime config is required. The default is Jev through Vercel AI Gateway:

// eval-lint.config.ts
import { defineConfig } from "eval-lint";

export default defineConfig({
  model: "typesafe-ai/jev",
  threshold: 0.95,
  concurrency: 4,
  timeoutMs: 10_000,
  maxRetries: 8,
});

Change model to any newer supported Gateway evaluation-model ID. String IDs explicitly use Gateway, independently of AI SDK's global default provider.

For another provider, or a custom endpoint, pass a factory. It receives the fetch eval-lint routes every request through, so the provider shares the run's concurrency limit, server-requested cooldowns, and per-attempt timeout:

import { createTypeSafeAi } from "@ai-sdk/typesafe-ai";
import { defineConfig } from "eval-lint";

export default defineConfig({
  model: ({ fetch }) => createTypeSafeAi({ fetch }).evaluationModel("jev-latest"),
});

Any AI SDK provider accepting a custom fetch works the same way, including custom implementations for internal endpoints; the model must support boolean and choice questions, and ordinary language-model instances are not evaluation models. A preconstructed model instance is not accepted, because a provider built without that fetch would bypass the shared transport; construct it inside the factory instead. API keys never go in the config: Gateway reads AI_GATEWAY_API_KEY or the Vercel OIDC token from the environment, and other providers take theirs through their constructor options.

AI SDK is the sole retry owner. Each HTTP attempt, including retries, passes through one run-scoped p-queue. A 429 or 503 with a valid Retry-After (seconds or HTTP date) or retry-after-ms pauses dispatch for the shared provider until the latest requested deadline. In-flight calls finish; queued calls and retries wait. Responses without a valid delay use SDK backoff. The timeout starts after dequeueing and covers the full response body, excluding queue waits, cooldowns, and SDK backoff. Cancellation stops queued calls, in-flight calls, and cooldown timers.

maxRetries now uses AI SDK's exponential backoff (initially 2 seconds), replacing eval-lint's previous short custom backoff. Eight retries can therefore wait substantially longer during an outage. EVAL_LINT_DEBUG=1 logs HTTP attempt durations/statuses, cooldowns, and failed model attempts without logging credentials or source payloads.

AI SDK's language-model evaluation adapters return prompted probability estimates; they are not guaranteed to be calibrated like native classifiers. See AI SDK evaluation. eval-lint pins its experimental SDK dependencies.

| Runtime option | Default | Purpose | | --- | --- | --- | | model | "typesafe-ai/jev" | Gateway model ID, or a factory receiving { fetch } | | engine | "oxlint" | "oxlint" or "eslint" | | providerOptions | — | Passed directly to AI SDK | | threshold | 0.95 | Default minimum violation probability | | concurrency | 4 | Maximum simultaneous evaluations and HTTP attempts | | timeoutMs | 10000 | Timeout per attempt; Jev normally answers in under two seconds | | maxRetries | 8 | AI SDK retries for transient failures, with SDK exponential backoff | | maxInputBytes | 160000 | Maximum serialized state + questions per detection request in UTF-8 bytes | | maxQuestionsPerCall | 400 | Maximum questions per request | | cacheVersion | "1" | Bump when a model alias or custom backend changes | | cacheTtlMs | 604800000 | Evaluation cache lifetime: seven days |

Jev accepts roughly 64,000 tokens per request, and latency is nearly flat in size and question count (about 0.3 s for a few kilobytes, 0.8 s for 160 KB), so the screen uses large requests; a request the provider rejects as too large is split in half automatically, and a Choice answer the provider cannot rank (a tie) falls back to one boolean per chunk for that batch. Scores fall as more files share a request, which is why verification and confirmation use small requests and why the screen bar is 0.4 rather than the threshold. On a 655-file, 2 MB TSX codebase with four rules, the defaults make about 50 screening and 30 small requests and finish in about 25 s at concurrency 2 and 15 s at concurrency 4, reporting the same reviewed violations as a design that made 382 requests in 99 s, with none of its false positives.

Runtime config discovery checks the working directory for eval-lint.config.ts, .mts, .mjs, then .js. Use --config for an explicit path. Runtime settings are separate from linter config inheritance.

CLI and caching

eval-lint src
eval-lint examples ./lint/errors.ts
eval-lint --engine eslint -- src --format json
eval-lint --config config/eval-lint.config.ts src
eval-lint -- --config .oxlintrc.json src
eval-lint --refresh src
eval-lint --offline src

Linter-specific arguments go after --. --config before the separator selects runtime config; after it selects linter config. The linter must be installed in the consuming project.

--refresh ignores stored evaluations and writes fresh ones. --offline makes no model calls and fails when a required detection result is missing, stale, or invalid; a positive finding whose localization is missing is reported at the start of the file. --quiet before -- hides only eval-lint's summary. Summaries go to stderr, preserving formatter output on stdout.

Exit 0 means the linter succeeded. Normal linter statuses are preserved for diagnostics. Exit 2 means evaluation was incomplete. Errors, exhausted retries, and changed inputs are never successful checks.

--fix, --cache, --stdin, --output-file, and mutation/watch modes are unsupported by this two-pass runner. Run native fixes separately. A linter cache could skip collection; eval-lint manages its own evaluation cache.

Plain eslint or oxlint emits a runner-required diagnostic for enabled AI rules. An ordinary editor linter extension cannot refresh AI checks by itself. This release supports CLI/CI execution, not an editor background service.

Execution

1. The selected linter runs once in collection mode; each enabled rule records the file's source and its prompt.
2. Screen: files without a fresh cached result are split into 64-line chunks and packed into large requests; per file and rule Jev answers whether any chunk violates and which chunk.
3. Verify: candidate chunks are re-judged in small requests with neighboring context; per chunk Jev answers whether it violates and which line.
4. Confirm: the top-ranked lines of each verified chunk are judged individually; line evidence decides what is reported.
5. Steps 2 to 4 repeat, up to three rounds, for files and chunks that just reported something, with the reported chunks and lines excluded from the questions.
6. The linter runs again, reporting findings through their original named rules.

Running the linter directly

The rules also work when eslint or oxlint runs on its own, as an editor does. Linter rules are synchronous, so a rule cannot ask the model inline; instead it looks up its file and rule in the results the last eval-lint run persisted under the cache directory and reports those. When nothing is persisted for that exact source, the rule evaluates it right there, one blocking request per rule and file through the same pipeline, and the answer is cached for next time. Saving a file in an editor therefore costs one short request per enabled rule; a bare run over a whole repository works but is slow and unbatched, which is what eval-lint is for. When the evaluation cannot run (no credentials, no network), the rule reports AI check incomplete: <reason> under its own id rather than passing the file silently.

Both passes use the same configuration and arguments. There is no separate parser, config merger, or ignore implementation. The summary shows rule/file checks, screening and small (verification plus confirmation) requests, retries, cached checks, findings, and fresh token usage. A file larger than a request is split into consecutive chunk groups that are judged separately. Oversized requests are split rather than truncated; a single chunk that exceeds the limits fails explicitly. Set EVAL_LINT_DEBUG=1 to log each request's timing and size, and EVAL_LINT_REPORT=path.json to dump every chunk and window probability for calibrating thresholds against labeled files.

Requests contain source, relative filenames, and rule prompts. Private per-run temporary files coordinate linter workers (one candidate batch per linter process, one results index) and are removed when the run ends. The persistent cache is a single evaluation.json index with one entry per rule and file: hashes, timestamps, and the screen, verification, ranking, and confirmation probabilities, without plaintext source or prompts. Add .eval-lint/ to .gitignore.

Cache keys include the filename, complete source, prompt, provider/model identity, provider options, protocol version, pipeline constants, and cacheVersion. Because entries are per file, editing one file re-evaluates only that file, regardless of how requests were packed: on the 655-file codebase a fully cached run takes about 0.6 s with no model calls, one edited file about 1.7 s with three requests, and a lint-staged style run of a few small files about 0.7 to 1.3 s cold (two model rounds) or about 0.2 s cached. Reverting an edit hits the cache again. Changing only a threshold or message reuses detection answers; lowering a threshold can trigger new localization requests for newly reportable findings. Model aliases can change without changing their IDs: use --refresh or bump cacheVersion when upgrading or recalibrating a backend.

Verification

# From the repository root
pnpm test
pnpm test:live
pnpm examples
pnpm test:package

Core tests exercise actual ESLint registration, validation, CLI behavior, chunk coverage, finding selection, request packing, offline cache misses, and matching ESLint/Oxlint candidates and reported ranges without calling AI. The live suite makes real Gateway requests; there are no mock models. It requires access to TypeSafe/Jev and checks both linters, independent plugins, severity, suppression, per-file cache invalidation, provider failures, and ESLint 9. pnpm examples runs the three example plugins' labeled scenarios (83 examples) through the shipped pipeline and fails while any disagrees with its label. Range tests pair good and bad files with matching names and line positions, preserve CRLF/non-BMP Unicode offsets, match across linters, and support suppression at the reported start. Live cache tests check source edits, prompt changes, message/severity changes, and forced refreshes. Their explicit 0.80 thresholds are fixture-level choices, not a general calibration guarantee; the runtime default remains 0.95.

Use a linked project's existing OIDC token without copying its env file:

EVAL_LINT_TEST_ENV_FILE=/path/to/project/.env.local pnpm test:live

The test loader reads only VERCEL_OIDC_TOKEN from that file. EVAL_LINT_TEST_MODEL selects another Gateway evaluation model. Live tests send only small synthetic fixtures and incur normal provider usage.

The examples/ directory contains two plugins, configs, and passing/failing source:

node dist/cli.js -- --config examples/.oxlintrc.json examples/fixtures
node dist/cli.js --engine eslint -- --config examples/eslint.config.ts examples/fixtures