Context & Problem
Every code review tool that calls itself 'AI' fails in one of two ways. Diff-only tools are fast and cheap, but they miss the issues that matter: the caller this change just broke, the interface this new type is supposed to satisfy, the test that now exercises dead code. Mega-prompt tools go to the other extreme — paste the whole repository into context — which is slow (minutes per review), expensive, and still misses the same issues because the model can't attend to everything in a large context window. The gap between those two failure modes is retrieval: identifying the right 3–5% of a repository that is genuinely load-bearing for understanding a given diff, structuring it so the model can reason rather than scan, and measuring with an eval harness whether the retrieval actually improved review quality. No off-the-shelf tool solves this correctly because it requires real retrieval engineering — AST-aware chunking, hybrid search, cross-encoder reranking — not framework glue. That is why this system exists.
Off-the-shelf AI code review tools either operate only on the diff (missing critical context like callers, type definitions, and tests) or paste the entire repository into a single mega-prompt (slow, expensive, and noisy). The harder, more interesting problem is building an agent that retrieves the right context for each finding, justifies its reasoning, and stays cheap and fast enough to run on every PR. Doing this well requires real retrieval engineering — not LangChain glue — plus evals that catch regressions before they ship, and production-grade observability so you can debug an agent that thinks for itself.
Design Constraints
- budgetHard $0.50 per-review cost cap enforced inside the agent loop via a per-model pricing table — generous enough to cover a real diff but strict enough to make the system viable at scale.
- latencyReview must complete within a time window that feels synchronous to a developer waiting on a PR — ruling out any multi-minute indexing or batch-processing approach.
- platformModel-agnostic from day one: the agent runtime must support Anthropic, OpenAI, Google, Groq, and Ollama behind a single interface so the system is not locked to any one provider's pricing or availability.
Architecture Overview
Designed as a TypeScript + Python monorepo. The core is a hand-written, model-agnostic Agent runtime in packages/agent — a typed, streaming ReAct loop with explicit termination (a validated submit_review stop tool, an iteration cap, an in-loop $0.50 cost cap, timeouts, and cancellation) that runs the same tools across five providers (Anthropic, OpenAI, Google, Groq, Ollama) behind one seam. Retrieval is a composable hybrid pipeline: tree-sitter produces AST-aware code chunks (Python and TypeScript/JavaScript), Voyage's voyage-code-3 embeddings power semantic search over pgvector, BM25 handles lexical recall, Reciprocal Rank Fusion merges the lanes, and Cohere rerank-v3.5 re-scores the survivors with a cross-encoder. A Python indexer handles repository ingestion, including Haiku-generated contextual chunk prefixes. Evals run a versioned golden dataset (v1: 30 synthetic seeded examples; real public-PR curation in progress) through the agent and score outputs with a versioned LLM-as-judge plus deterministic checks — the baseline run is committed to the repo, failure included. Production concerns are first-class: prompt caching on the system prompt and indexed documents, Redis exact-match plus pgvector semantic caching, tier-based model routing backed by a per-model pricing table, and prompt-injection defenses that treat all retrieved code as untrusted. Every run is traced end-to-end in Langfuse and persisted as a replayable event stream.
Interactive Architecture
Open full screen ↗Key Engineering Decisions
Outcomes & Lessons Learned
Build Status
All build phases complete and tested (426 tests in CI). GitHub delivery (packages/github), the acr-review CLI, and the self-review GitHub Actions workflow are merged to main — the agent reviews this repository's own PRs in CI (19 workflow runs as of 2026-07-11).
Eval Target
Target: ≥0.8 judge score with <20% false positives on dataset v2 — the committed v1 baseline (judge 0.635, 67% FP, verdict: below-bar) is the published starting line; later Claude-judged v1 runs also remain below-bar
Cost Cap
Hard per-review spend cap enforced inside the agent loop via a per-model pricing table, alongside prompt caching, semantic caching, and tier-based routing. This is a code-enforced ceiling, not a measured average — the agent now runs on its own PRs in CI, but no measured per-review cost from those runs is published yet.
Retrospective
- The committed below-bar eval baseline (judge score 0.635, 67% false-positive rate) turned out to be the most useful artifact in the repo — it made regressions impossible to miss and gave reviewers a concrete starting point to push back on. Publishing the failure was the right call.
- Hand-writing the agent runtime added ~600 lines of boilerplate, but the payoff was complete visibility into every token spend, tool invocation, and iteration limit. The first time the loop tried to exceed the cost cap mid-run, we caught it cleanly because we owned the loop.
- Retrieval quality gates matter more than model quality at the margins. Improving the chunking strategy (tree-sitter over line-based splitting) dropped false positives more than switching to a frontier model did.
Evidence & Links
Engineering Deep Dives
Articles, architecture discussions, and deep dives related to this system.