llm-quorum

Route AI tasks to the right model CLI, with automatic fallback and cross-model consensus review — one command instead of a manual multi-tab workflow.

A CLI that orchestrates the AI model CLIs you already have installed (Claude Code, Codex, Gemini) instead of calling their provider APIs directly: declare which adapter handles which task type, get automatic fallback when one fails, and run the same review prompt across every configured model in parallel to get a consolidated, consensus-scored report.

Why

Anyone using 2+ frontier-model CLIs ends up deciding "which model for which task" by hand, and getting a second opinion on a review means pasting the same diff into each CLI separately and comparing free-text output yourself. Nothing formalizes that workflow.

llm-quorum resolves three parts of it:

• Declarative routing — a policy per task type says which adapter runs first.
• Transparent fallback — if the primary adapter times out, exits non-zero, or its output matches a configured failure pattern, the next adapter in the chain runs instead, and every attempt is reported.
• Cross-model consensus review — the flagship command: the same prompt runs on every adapter in the chain in parallel, findings are normalized to a common schema, and findings ≥2 models agree on are marked high confidence.

This is a different problem from what LiteLLM or OpenRouter solve: those route provider APIs. llm-quorum orchestrates CLI processes — process timeouts, exit codes, and best-effort parsing of non-JSON output — and packages cross-model consensus as the product, not as a side effect of routing.

Useful for developers and consultants who already drive 2+ AI CLIs by hand and want higher-confidence reviews without copy-paste, teams that want to standardize "AI review before merge" into one command, and anyone who wants lightweight multi-provider routing without adopting a full agent framework.

How it works

 prompt / diff
      │
      ▼
 policy lookup (taskType → primary adapter + fallback chain)
      │
      ├─ run <task-type>:    primary adapter
      │                          │ fails (timeout / non-zero exit / pattern match)
      │                          ▼
      │                      next adapter in the fallback chain
      │                          │ ... until one succeeds or all fail
      │                          ▼
      │                      raw output
      │
      └─ review <path>:      EVERY adapter in the chain, run in PARALLEL
                                 │           │           │
                              claude       codex       gemini
                                 │           │           │
                                 ▼           ▼           ▼
                          best-effort JSON parse of each adapter's raw output
                                 │           │           │
                                 └───────────┴───────────┘
                                             ▼
                              consolidate(): group findings by
                              (file, category, line ± tolerance)
                              ≥2 sources agree → high confidence
                              1 source          → single-source
                                             ▼
                              Markdown report (or --json)

run and review use the same policy shape (primary + fallback) but different semantics: run stops at the first adapter that succeeds, review always calls the whole chain because disagreement between models is the signal it's built to surface.

Quickstart

Requirements

• Node.js >= 22
• At least one of the claude, codex, or gemini CLIs installed and authenticated (their own login flow — llm-quorum does not manage credentials). review needs ≥2 to produce any high-confidence findings.

Windows and WSL are supported. Adapter CLIs are spawned directly without shell: true; on Windows, npm-installed .cmd/.bat/.exe shims are resolved through PATHEXT, fixing ENOENT reports for installed CLIs.

1. Build

git clone https://github.com/NestorPVsf/llm-quorum.git
cd llm-quorum
npm install
npm run build
npm link   # optional: makes `llm-quorum` available globally

Without npm link, run the built CLI directly as node dist/cli.js.

2. Check what's available

llm-quorum doctor

name    available  version / error
claude  yes        1.4.2 (Claude Code)
codex   yes        codex-cli 0.144.2
gemini  no         Binary not found: gemini (ENOENT). Install the CLI and make sure it is on PATH.

3. Preview before spending credits

review fans out to every adapter in its chain in parallel — always confirm the plan first with --dry-run, especially on paid CLIs:

llm-quorum review src/router.ts --dry-run

claude (claude) — model=default effort=default timeoutMs=300000
codex (codex) — model=default effort=default timeoutMs=300000
Review content: 3241 characters

4. Run a real review

llm-quorum review src/router.ts

# Consolidated Review

## Summary

- Total findings: 3
- High confidence: 1
- Single source: 2
- Degraded sources: 1

## High-confidence findings

### HIGH — `src/router.ts:87`
- Category: error-handling
- Sources: claude, codex
- Messages:
  - **claude:** classifyFailure() maps every non-ok status without a matched
    pattern to undefined, so a fallback that succeeds after several failed
    attempts and a real timeout are indistinguishable in the returned
    attempts array.
  - **codex:** Same failure class collapses timeout and unmatched pattern
    cases; consider a dedicated status instead of relying on absence.

## Single-source findings

### MEDIUM — `src/consolidate.ts:134`
- Category: readability
- Sources: claude
- Messages:
  - **claude:** findEmbeddedArray scans for every `[` in the raw string;
    on very large model output this re-scans from each candidate start.

## Degraded sources

- **gemini:** exit code 1: rate limit exceeded, please retry later

(Output above is illustrative — actual findings depend on what each CLI returns.)

Commands

| Command | Flags | Behavior | |---|---|---| | run <task-type> -- <prompt...> | --dry-run, -h/--help | Looks up the policy for <task-type>, runs the primary adapter, and falls through the fallback chain in order on failure. Prints the winning adapter's raw output to stdout; failed attempts and the adapter that ultimately served the request go to stderr. | | review <path\|-> | --json, --dry-run, -h/--help | Reads <path> (or stdin with -), sends the same review prompt to every adapter in the "review" policy's chain (primary + fallback) in parallel, and prints a consolidated report — Markdown by default, JSON with --json. Exits 1 after printing if every source is degraded. | | doctor | -h/--help | Probes every registered adapter with <command> --version and prints an availability table. Takes no other arguments. | | config validate [path] | -h/--help | Validates a config file against the schema. Without path, resolves the same way run/review do (see Configuration below); if none is found, reports that defaults would be used instead of failing. |

Global flags (before a command): -h/--help, -v/--version.

run requires a -- separator before the prompt (llm-quorum run review -- "check this diff") — everything after -- is joined with spaces into a single prompt string.

review also accepts -- before its path, which is useful for filenames that begin with a dash: llm-quorum review -- -unusual-name.ts.

Library API

The same routing and review primitives are available from the package entry point:

import { createClaudeAdapter, loadQuorumConfig, route } from "llm-quorum";

const config = loadQuorumConfig({ policies: [{ taskType: "review", primary: { adapter: "claude" }, fallback: [] }] });
const result = await route("review", "Review src/router.ts", config, { claude: createClaudeAdapter() });
console.log(result.result.raw);

Configuration

Config is a .llmquorumrc.json file, resolved in this order:

1. An explicit path (only config validate <path> accepts one).
2. ./.llmquorumrc.json (current working directory).
3. ~/.llmquorumrc.json (home directory).
4. A built-in default: a single "review" policy (claude → codex → gemini) with timeoutMs: 300000. There is no built-in policy for any other task type — run <task-type> fails with "Unknown taskType" if nothing configures it.

Example, matching the schema src/config.ts actually validates with zod:

{
  "policies": [
    {
      "taskType": "review",
      "primary": { "adapter": "claude" },
      "fallback": [{ "adapter": "codex" }, { "adapter": "gemini" }]
    },
    {
      "taskType": "refactor",
      "primary": {
        "adapter": "codex",
        "model": "gpt-5.6-sol",
        "effort": "xhigh",
        "timeoutMs": 600000
      },
      "fallback": [{ "adapter": "claude" }]
    }
  ],
  "defaults": {
    "timeoutMs": 300000,
    "failurePatterns": ["rate limit", "sandbox"]
  }
}

| Field | Type | Required | Notes | |---|---|---|---| | policies | array | Yes | One entry per taskType. run <task-type> and review (which always uses the "review" entry) look up the matching entry. | | policies[].taskType | string | Yes | Free-form identifier matched against the run/review argument. | | policies[].primary | AdapterRef | Yes | Adapter tried first. | | policies[].fallback | AdapterRef[] | Yes (may be empty) | Tried in order if primary fails. | | AdapterRef .adapter | "claude" | "codex" | "gemini" | Yes | Must be one of the three adapters registered in v1 — see Adapters below. | | AdapterRef .model | string | No | Passed as --model (claude, codex) or -m (gemini). | | AdapterRef .effort | string | No | Only honored by the codex adapter (-c model_reasoning_effort=<effort>); accepted by the schema for claude/gemini but silently ignored by those adapters. | | AdapterRef .timeoutMs | number | No | Overrides defaults.timeoutMs for this specific adapter call. | | defaults.timeoutMs | number | No (default 300000) | Used for any adapter call that doesn't set its own timeoutMs. | | defaults.failurePatterns | string[] | No | Case-insensitive substrings checked against an adapter's combined stdout + stderr. A match is classified as pattern_match and triggers fallback in run even when the process exits 0. |

Run llm-quorum config validate at any time to check the config that would actually be picked up, without touching any adapter.

Adapters

| Adapter | Command invoked | --model equivalent | effort | |---|---|---|---| | claude | printf '%s' "$PROMPT" \| claude -p --output-format json [--model <model>] --tools "" | --model | not supported | | codex | printf '%s' "$PROMPT" \| codex exec --json [--model <model>] [-c model_reasoning_effort=<effort>] --sandbox read-only | --model | -c model_reasoning_effort=<effort> | | gemini | printf '%s' "$PROMPT" \| gemini -p "Follow the task instructions provided via standard input." [-m <model>] --approval-mode plan | -m | not supported |

Every adapter is spawned through cross-spawn with an argument array, no shell: true, and piped stdin. The runtime prompt is written to stdin and the stream is ended immediately; it never appears in argv. Gemini's fixed -p instruction only enables its headless mode and contains no runtime content. cross-spawn resolves Windows PATHEXT shims and safely escapes the remaining argv when a .cmd/.bat launcher requires cmd.exe. Prompts are deliberately capped at 200000 UTF-8 bytes per adapter call as a sanity guard against accidental memory use and paid-model cost, independent of OS argv limits. Stdout/stderr are capped at 5000000 bytes per stream. On timeout or output overflow the child receives SIGTERM.

Adding your own adapter. Implement the Adapter interface from src/adapters/types.ts:

interface Adapter {
  name: string;
  command: string;
  run(prompt: string, opts: RunOptions): Promise<AdapterResult>;
  probe(): Promise<ProbeResult>;
}

If your CLI is a straightforward command args… → stdout tool, wrap the provided createCliAdapter({ name, command, buildArgs }) helper instead of implementing run/probe from scratch (see src/adapters/claude.ts for the smallest example). There is no plugin or dynamic-loading system in v1: register the factory in createRegistry() in src/cli.ts, and add the adapter name to the adapter enum in AdapterRef (both the TypeScript type and the zod schema) in src/config.ts so config files can reference it.

Security & privacy

• Prompts and diffs passed to run/review are sent to whatever CLI(s) your policy configures. This is the same trust boundary as running claude, codex, or gemini yourself by hand — llm-quorum does not add its own network calls, telemetry, or remote logging.
• Authentication is entirely delegated to each underlying CLI's own login/API-key flow. llm-quorum never reads, stores, or transmits credentials itself.
• --dry-run on both run and review prints exactly which adapters and models would be invoked, and for review the size of the content that would be sent, without calling anything — use it before a real run, especially against paid CLIs, since review invokes every adapter in the chain in parallel.
• The built-in adapters request the most restrictive verified non-interactive modes their current CLIs expose: Claude runs with tools disabled, Codex uses its read-only sandbox, and Gemini uses read-only plan approval mode.
• These flags do not eliminate prompt-injection risk. The underlying CLI's own configuration still governs behavior outside what its flags can constrain, including permission settings, hooks, MCP servers, and future CLI changes; llm-quorum only passes the prompt through and cannot impose a stronger sandbox than that CLI exposes. Adversarial reviewed content could therefore cause an underlying CLI to take actions beyond returning review text, depending on the user's CLI configuration. Review untrusted content only with every CLI configured as restrictively as possible, or inside a separate isolated/sandboxed environment.
• Findings in a consolidated report reflect each model's interpretation of the reviewed content. Adversarial or prompt-injection text in untrusted content can influence those findings, so do not treat review output as an unconditional security gate or ground truth without human judgment.

Known limitations

• Best-effort output parsing. review handles whole-payload JSON, serialized answer fields, JSONL agent messages, fenced ```json blocks, and embedded balanced arrays. If none succeed, that source is marked degraded with a parseNote; the command succeeds if another source parsed cleanly and exits 1 if every source degraded.
• No cost or token tracking. review's parallel fan-out to every adapter in the chain can be expensive on paid CLIs; there is no built-in accounting. Use --dry-run first and check each CLI's own billing.
• CLI output formats can drift between versions. llm-quorum does not pin or verify a minimum CLI version; doctor reports the version string each adapter returns for --version so a mismatch is visible, but a format change surfaces as a degraded source rather than an error.
• The adapter set is closed in v1. Only claude, codex, and gemini are registered; adding another CLI requires a source change (see Adapters above), not just a config edit.
• effort only affects codex. Setting it for claude or gemini is valid per the schema but has no effect.
• Timeout is a SIGTERM, not a guarantee. A timed-out adapter call is reported as failed, but the underlying process may still be running or have had side effects — llm-quorum does not verify termination.

Troubleshooting

See docs/failure-modes.md for the symptom → cause → action table, including Windows CLI ENOENT and stale WSL uv_cwd failures.

Development

npm install
npm run typecheck   # tsc --noEmit
npm run lint         # biome check .
npm test             # vitest run
npm run build         # tsdown
npm run dev           # tsdown --watch

Built with a TDD workflow: tests in test/ (including test/adapters/mock-child-process.ts, which mocks cross-spawn) are written against each module before the implementation, so the suite never needs a real claude, codex, or gemini binary installed. See CONTRIBUTING.md for setup and conventions, and DECISIONS.md for the reasoning behind the main design choices. Failure modes and how to read them live in docs/failure-modes.md.

License

MIT — see LICENSE.