Azure DevOps AI PR Review Bot

A self-contained Node script + ADO pipeline that runs Claude over every pull request and posts findings as inline comment threads anchored to specific file:line locations. When the AI flags a 🔴 HIGH severity issue, the thread is left active and (combined with a standard branch policy) blocks merge until a human resolves it.

Built for solo / small-team workflows where you want a second pair of eyes on every PR without paying for a managed service. Distributed as an npm package (presto-review) — consumer projects run it via npx --yes presto-review with zero local install.

What it actually does

1. PR opens against a long-lived branch (e.g. main)
2. Pipeline triggers via Azure DevOps build validation branch policy
3. Pipeline fetches the diff (git diff origin/<target>...HEAD)
4. Pre-flight secret scan — regex-based credential detection (AWS, Stripe, JWTs, etc.) runs instantly at zero cost
5. Fetches full file context for changed files via ADO Items API (80K budget, 20K/file cap)
6. Fetches PR metadata (title + description) for intent-aware reviewing
7. Calls the Anthropic API with the diff, file context, and a system prompt you control
8. Claude returns a structured review via tool use — severity, file, line, message, confidence score, and optional fix suggestion per finding
9. Post-hoc validation — verifies file paths and line numbers against the actual diff; drops hallucinated findings
10. Self-critique pass — each HIGH finding gets a second API call demanding quoted evidence; ungrounded HIGHs are downgraded to MEDIUM
11. Deduplicates against existing review threads on the PR (±3 line tolerance) — re-pushes only post new findings
12. Posts inline threads per finding (anchored to file:line in diff view) with optional fix suggestions, plus a summary thread with a PR summary
13. 🔴 HIGH found → thread posted as Active → branch policy blocks merge
14. Otherwise → thread posted as Closed → visible-but-non-blocking
15. Webhook notification — optionally sends Slack or Teams alerts when HIGH findings are detected

Cost: roughly $0.02–$0.30 per PR depending on diff size. Always exits 0 (advisory infrastructure must not break your build).

Why this vs. a managed service

| | This tool | CodeRabbit / Greptile / Diamond | |---|---|---| | Cost | API costs only (~$0.02–0.30/PR) | $20–50/dev/mo subscriptions | | Customization | Edit a markdown file | Limited to vendor's knobs | | Vendor lock-in | None | Yes | | Setup time | ~2 min (npx presto-review init) | ~5 min | | Prompt tailored to your codebase | Yes (you write it) | Generic | | Polish | DIY | Professional | | You own the data flow | Yes | No |

Pick this if: you want full control, low cost, and don't mind editing a markdown file to teach the AI about your codebase.

Pick a managed service if: you want polished UX with zero config, you have budget, and your codebase is conventional enough that a generic reviewer works well.

Quick start

Prerequisites: Azure DevOps project + repo, an Anthropic API key, az CLI logged in.

cd your-project
npx --yes presto-review init

That's it. The init command:

1. Detects your ADO org/project/repo from git remote
2. Auto-detects your stack (Next.js, Python/FastAPI, Go, etc.) and picks a matching system prompt template
3. Asks your target branch and blocking preferences
4. Writes pipelines/pr-review.yml and tools/system-prompt.md
5. Registers the pipeline, creates branch policies, and checks variable group setup via az CLI

Commit the two files, open a test PR, and you should see a Claude review within ~60 seconds.

Manual setup (if you prefer)

If you don't have the az CLI or want more control:

Full step-by-step with screenshots-level detail: docs/01-setup-guide.md.

Repository structure

PResto/
├── README.md                          ← you are here
├── LICENSE                            ← MIT
├── package.json                       ← npm package config (presto-review)
├── pr-review.mjs                      ← the script (~1250 lines, zero dependencies)
├── system-prompt.md                   ← default prompt (copy + edit per-project)
├── pipeline-template.yml              ← ADO pipeline (copy + edit per-project)
├── scripts/
│   ├── test.mjs                       ← local test harness (offline + integration)
│   └── release.mjs                    ← automated release script
├── test-fixtures/                     ← sample diffs for testing
├── docs/
│   ├── 01-setup-guide.md              ← step-by-step setup (copy-files approach)
│   ├── 02-how-it-works.md             ← architecture deep dive + design rationale
│   ├── 03-customizing-the-prompt.md   ← how to teach the AI about your codebase
│   ├── 04-severity-and-blocking.md    ← how merge blocking works
│   ├── 05-troubleshooting.md          ← common issues + fixes (battle-tested)
│   ├── 06-consumer-setup-guide.md     ← setup via npx (recommended for consumers)
│   └── 07-releasing.md               ← how to publish new versions to npm
├── pipelines/
│   ├── pr-validation.yml              ← PR validation + dogfood pipeline
│   ├── ci.yml                         ← CI pipeline (development branch)
│   └── release.yml                    ← npm publish pipeline (master branch)
└── prompt-templates/                  ← starter prompts for common stacks
    ├── README.md
    ├── nextjs-typescript-monorepo.md
    ├── python-fastapi.md
    ├── go.md
    └── generic.md

What's in the script that's worth knowing

The script is ~1250 lines, single file, no dependencies beyond Node 20's built-in fetch. Distributed as presto-review on npm. The non-obvious behaviors:

• Reads the diff from stdin — the pipeline does git diff > /tmp/pr.diff && node pr-review.mjs < /tmp/pr.diff. Keeps the script unaware of git mechanics.
• System prompt loaded from file — the script reads system-prompt.md from its own directory (or $SYSTEM_PROMPT_PATH). Lets you tailor per-project without touching code.
• Tool use for structured output — Claude returns findings via a tool call with a strict JSON schema (severity, file, line, message, confidence, fix). No regex parsing of free-text needed (though a fallback path exists).
• Pre-flight secret scanner — regex patterns detect hardcoded credentials (AWS keys, Stripe keys, JWTs, etc.) before the API call. Zero cost, zero hallucination risk.
• Full file context — fetches the complete source of changed files via ADO Items API (80K budget, 20K/file cap) so Claude reviews with full context, not just diff hunks.
• Auto-fix suggestions — Claude can suggest concrete code fixes per finding, displayed in PR comments as fenced code blocks.
• PR summary — every review includes a 1-3 sentence summary of what the PR does, posted in the summary thread.
• Self-critique for HIGHs — a second API call per HIGH finding demands quoted evidence; ungrounded HIGHs get downgraded to MEDIUM.
• Incremental re-review — deduplicates against existing review threads on the same PR (±3 line tolerance). Re-pushes only post new findings.
• Developer reply detection — when re-reviewing, the script detects developer replies to existing threads and injects that context into the prompt so Claude can reconsider.
• Configurable blocking severities — by default only HIGH blocks merge; set BLOCKING_SEVERITIES=HIGH,MEDIUM to also block on MEDIUM findings.
• Webhook notifications — optionally sends Slack or Teams alerts when blocking findings are detected (WEBHOOK_URL env var).
• Smart model routing — automatically uses Haiku for small diffs (≤50 lines) and Sonnet for complex ones. Configurable via MODEL_ROUTING and TRIVIAL_DIFF_LINES.
• Confidence scoring — each finding includes a 0-100 confidence score displayed in PR comments.
• Known false-positive filter — FALSE_POSITIVE_PATTERNS env var suppresses findings matching regex patterns.
• Diff truncated at 100K chars (configurable via MAX_DIFF_CHARS). Bigger PRs get the first ~25K tokens reviewed; the AI is told it was truncated.
• Always exits 0 — if Anthropic is down, your build doesn't break. The merge-block mechanism is the Active thread status, not a build failure.
• HIGH-detection is a configurable regex (HIGH_MARKER env var, default ^[-*]\s+🔴). Anchored to a list bullet to avoid matching the severity legend line.

Costs

Sonnet 4.5 (default) at typical PR sizes:

• ~2K input + 500 output tokens = ~$0.02 / PR
• Larger PRs (50K diff) closer to $0.20-0.30
• Diffs above 100K chars get truncated (no further cost growth)

Set ANTHROPIC_MODEL=claude-haiku-4-5-20251001 in the pipeline env to drop to Haiku for ~5× cheaper reviews. Lower review quality, but fine for low-stakes repos.

Limitations

• Azure Repos Git only. The pr: YAML trigger doesn't work for Azure Repos (documented ADO quirk) — you must rely on build validation branch policies. Should work as-is for GitHub-hosted repos using ADO, but untested.
• No conversation memory. Each PR review is fresh; the AI doesn't remember prior PRs or learn from your feedback. Developer replies to existing threads ARE detected and injected into re-review context, but there's no cross-PR memory. (Feature: keeps it stateless and cheap. Limitation: it'll make the same mistake twice across different PRs.)
• English-language prompt assumes the codebase comments are in English.

License

MIT. Use it, modify it, ship it. No warranty.

Provenance

Extracted from a working production Azure DevOps PR review pipeline on 2026-05-23. The original was built in a single afternoon to address one team's review-discipline concerns. After a week of in-production use, all the lessons learned (Next.js cache invalidation, SYSTEM_PULLREQUEST_* env vars, status code semantics, etc.) have been folded back into this template and the troubleshooting guide.