tradeoff

It doesn't tell you whether your code is right. It tells you what your code means.

An AI engineering second opinion for your git diff.

Most AI tools read your diff and ask "is this code correct?" — linters, scanners and review bots have that covered. tradeoff asks a different question:

"What engineering decision did this change just make, and what did it cost?"

Try it

export ANTHROPIC_API_KEY=sk-ant-...

git add .
npx tradeoff

No signup. No account with tradeoff. Bring your own model key.

A real example

Here is actual output from adding a Redis cache in front of a product lookup — not a mock-up:

tradeoff

Analyzing 2 changed files...

✓ Read 2 changed files
◐ Thinking about consequences...

──────────────────────────────────────

01  CACHE WITHOUT INVALIDATION ON WRITE

Decision
getProduct now reads from Redis first and populates it on miss, but
updateProductPrice (visible in the diff's context, unchanged) still
writes only to the database.

Benefit
Repeated product reads within the TTL window skip the database query
entirely, reducing read load on Postgres.

Trade-off
A price or inventory change made via updateProductPrice is invisible
to callers of getProduct for up to 300 seconds, since nothing evicts
or refreshes the cache entry on write.

Second-order effect
If price updates are used for anything time-sensitive (promotions,
inventory-driven availability, checkout pricing), the system now has
a silent up-to-5-minute window where reads and writes disagree, and
there's no code path anywhere that clears the stale entry — it only
expires on its own schedule.

Question before merge
Why does updateProductPrice not invalidate or overwrite the
corresponding cache key, and is a 300-second staleness window
acceptable for every caller of getProduct?

──────────────────────────────────────

Think before you merge.

Nobody wrote a rule about cache invalidation. It read the diff, noticed the write path sitting three functions below the read path, and asked the question a senior reviewer would have asked.

What tradeoff looks for

Every non-trivial engineering choice is a trade. It tries to name the one you may not have consciously made:

| Change shape | What it reasons about | |---|---| | Caching | staleness, who owns invalidation, behavior when the cache is down | | Async / messaging | dual writes, duplicate delivery, ordering, who observes failures | | New service call | latency coupling, timeouts, partial and cascading failure | | Schema change | deployment ordering, old/new versions coexisting, DDL locks, rollback | | Retries | idempotency, ambiguous failures, load amplification on a struggling dependency | | Validation | where the rule lives, other paths that bypass it, already-persisted data | | Concurrency | lost updates, atomicity, check-then-act races | | Configuration | defaults, environment drift, blast radius of a bad value | | API contracts | backward compatibility, consumers, versioning, rollout order |

Each finding is tagged observed (supported directly by the diff) or inferred (a consequence conditional on something it can't see). It says "if this call is on the request path, B's latency now propagates to the caller" — not "this will slow down production."

What it deliberately does NOT do

It is not a code reviewer, bug detector, security scanner, linter, PR-approval bot, or a replacement for human review. It won't comment on style, naming, or test coverage.

It also won't manufacture insight. On a README typo it says so and stops:

No meaningful engineering trade-off detected.

This is a one-word typo fix in documentation with no code or
architectural changes.

A false positive costs more trust than an honest "there's nothing here."

Privacy

No account required.
No tradeoff server.
No telemetry.
No source code stored by tradeoff.

With a cloud provider (anthropic, and eventually openai), the selected diff is sent directly from your machine to that provider's API — tradeoff itself never sees or stores it, but it does leave your machine.

With --provider ollama, that's not true — the diff never leaves your machine at all. Every request goes to your own local Ollama server; there is no cloud fallback and no other network call. See the Ollama section under Providers below — it's also marked experimental, for reasons unrelated to privacy (the privacy guarantee is solid; reasoning quality is the open question).

Before anything is transmitted (to a cloud provider, or to local Ollama), tradeoff filters out lockfiles, binaries and generated output, then scans for and redacts obvious secrets (private keys, AWS keys, tokens, connection strings, passwords). Redacted values are never printed back to your terminal, and you're told how many were found.

Secret detection is a safety net, not a guarantee — don't rely on it to make an untrusted diff safe.

Providers

| Provider | Status | Environment variable | |---|---|---| | anthropic | Recommended — default, highest measured reasoning quality | ANTHROPIC_API_KEY | | ollama | Supported — experimental (fully local) | none required | | openai | Interface ready, not implemented | — |

Choosing an unimplemented provider gives you a clear error rather than a crash. Adding one means implementing a single analyze() method and registering it in src/providers/factory.ts — nothing else in the codebase changes. Contributions welcome.

Optionally set ANTHROPIC_MODEL to pin a different Anthropic model.

API keys are read from the environment only — never passed as command arguments, where they'd leak into your shell history.

Ollama (fully local) — experimental

ollama serve
ollama pull qwen2.5-coder:14b

OLLAMA_MODEL=qwen2.5-coder:14b tradeoff --provider ollama

With --provider ollama, the diff, the system prompt, and every reasoning round-trip stay on your machine: tradeoff talks only to http://127.0.0.1:11434 (Ollama's default local address). Nothing is sent to Anthropic, OpenAI, or any other network endpoint. No API key, no account, no signup.

Configuration is two environment variables, both optional:

| Variable | Purpose | Default | |---|---|---| | OLLAMA_MODEL | which local model to use | llama3.1:8b (not the model this was evaluated with — see below) | | OLLAMA_HOST | where Ollama is listening | http://127.0.0.1:11434 |

If Ollama isn't installed or isn't running, or the requested model hasn't been pulled, tradeoff tells you exactly what to run (ollama serve / ollama pull <model>) instead of failing with a raw connection error.

"Experimental" here is a measured conclusion, not a hedge. We ran the same blind 10-fixture evaluation suite used to validate the Anthropic provider (see below) against qwen2.5-coder:14b — the only local model actually benchmarked so far. Roughly half the fixtures produced a genuinely useful, non-generic insight; Anthropic passes all ten on the same fixtures. The dominant failure mode isn't wrong reasoning, it's format: the "question before merge" field sometimes turns into a multi-question checklist instead of the single sharp question the prompt asks for, and on about 3 of 10 fixtures the model returns an empty required field, which tradeoff reports as a clear error rather than silently degrading. We tuned Ollama's decoding options (repeat_penalty: 1.3, shipped as the default — see src/providers/ollamaProvider.ts) to eliminate the worst version of this — runaway repeated-question loops — without changing the shared prompt, but that alone didn't close the quality gap.

The model Ollama defaults to, llama3.1:8b, has not been run through this evaluation — set OLLAMA_MODEL=qwen2.5-coder:14b explicitly if you want the one configuration we actually have data on. Larger local models (32B-class and up) weren't tested: they typically need 20GB+ of RAM/VRAM to run well, more than a typical laptop GPU holds, so expect them to fall back to slow CPU inference rather than assume they'll close the gap.

Local mode is real, private, and useful for experimenting or for diffs you don't want leaving your machine. It is not yet a drop-in substitute for the default Anthropic provider — use Anthropic when reasoning quality matters more than staying local.

Usage

tradeoff                              # analyze staged changes (the hero path)
tradeoff --unstaged                   # analyze unstaged changes
tradeoff --commit HEAD                # analyze a specific commit
tradeoff --repo /path/to/other/repo   # analyze a different local repo, in place
tradeoff --lang ar                    # Arabic output
tradeoff --provider ollama            # select a provider
tradeoff --debug                      # full error details

--repo runs read-only git commands against the target directory — it never copies the repository, never changes its working tree, and never touches its staging area. Combine it with the other flags, e.g. tradeoff --repo ../other-project --unstaged.

Arabic

--lang ar renders the reasoning and the interface in Arabic. Universally understood technical terms stay technical rather than being forced into awkward translations:

جاري تحليل 2 ملف معدّل...

01  التخزين المؤقت بدون إبطال عند التحديث

القرار
تمت إضافة كاش Redis لاسترجاع بيانات المنتج (نمط cache-aside)، لكن
updateProductPrice بقيت كما هي دون أي منطق لإبطال المفتاح المخزن
في Redis عند تغيير السعر.

سؤال قبل الدمج
هل هناك مسارات تعتمد على getProduct وتتطلب دائماً القيمة الحالية
للسعر (كالدفع الفعلي)، أم أن التأخير حتى خمس دقائق مقبول؟

Philosophy

A mirror, not a judge.

No scores. No grades. No badges. No streaks. No shaming, and no AI praise. It never says "this code is wrong" — it says "this introduces X; have you considered what happens when Y?"

The reaction we're aiming for is "Hmm — I didn't think about that." That reaction is the product metric.

The moat isn't access to an LLM; anyone can call one. It's being better at asking the right engineering question.

Development

npm install
npm run dev          # run the CLI from source
npm test             # unit tests (no API calls, no key needed)
npm run lint
npm run typecheck
npm run build

npm run dev -- --repo /path/to/real/repo   # try it against a real project before publishing

Dogfooding against real repos

npm run dev -- --repo <path> is the fastest way to try tradeoff against a real project without leaving this directory. For structured feedback while dogfooding, scripts/dogfood.ts (a developer-only script, not part of the published CLI) runs the same pipeline and then asks a few yes/no questions — useful, surprising, obvious, wrong, would-you-use-it-before-a-PR — and appends only those answers plus run metadata (timestamp, repo path, provider, decision count) to a local, gitignored dogfood-results/log.jsonl. It never writes the diff, the model's output text, or any repository file content to disk.

npm run dogfood -- --repo /path/to/real/repo
npm run dogfood -- --repo /path/to/real/repo --provider ollama

The reasoning evaluation suite

The prompt is the product, so it's tested like one. fixtures/ holds ten realistic diffs — caching, a synchronous service call, Kafka publishing, a schema migration, an API contract break, retries, validation, concurrency, configuration, and a README typo that must not produce architectural drama.

npm run eval                          # all fixtures against anthropic — makes real API calls, costs money
npm run eval -- redis-caching         # just one, while iterating on the prompt
npm run eval -- --provider ollama     # all fixtures against your local Ollama model, no API key/cost

The checks themselves are identical and provider-agnostic — the same blind evaluation runs against whichever provider you pick. "Blind" means fixtures carry an expectation (expectedConcepts, buriedInsight), but nothing about it is shown to the model; only the diff and system prompt are sent. This is what makes the pass/fail counts meaningful, so it isn't something a provider addition should ever bypass.

Each fixture declares the concepts a good answer should reach and a buriedInsight describing the non-obvious thing a strong answer finds. The harness also checks that findings are anchored to identifiers that actually appear in the diff — an insight that never names anything from the change is one that could have been written without reading it.

Those automated checks are a floor, not the bar. The report prints full reasoning next to each fixture's buriedInsight for human grading, and dumps raw JSON to eval/results/ so successive prompt revisions can be compared side by side.

If you change src/prompts/systemPrompt.ts, run the full suite. The prompt is global — a change that fixes one fixture can quietly flatten another.

Contributing

Good places to start: an OpenAI provider, more language support, terminal UX, diff parsing, and — most valuable of all — new fixtures. A realistic diff that produces a weak insight is a genuinely useful bug report.

Roadmap

Deliberately small. Reasoning quality comes before feature count:

• tradeoff challenge — ask one hard question first, reveal the reasoning after you've thought about it
• OpenAI provider
• tradeoff share — a clean Markdown snippet of the engineering idea, not a score card
• Reasoning lenses (--lens reliability, --lens data)

Not planned: dashboards, scores, gamification, CI gating.

Built by

Omar Ismail

Software Engineer focused on backend systems, software architecture and distributed systems.

License

MIT