After two months of running AI code review on real pull requests, I found seven recurring limits: unverified fixes, weaker reviews on large PRs, anchoring on detailed specs, softer repeat passes, instruction overload, inconsistent rule enforcement, and missing intent. Research on code generation and long-context LLMs explains parts of this pattern, but not the whole pipeline.
AI review still feels like a magic creature. On one side it catches bugs. On the other it makes things worse and hands you false confidence that everything is fine. The second one wins most days, in my experience.
It runs on the pr-review-toolkit plus a set of custom skills I built from the problems I keep seeing in real projects during code review. Those agents and skills feed a pipeline, and the pipeline is not small. Every agent gets its own clean context. The GitHub Action that drives it has turned into a heavy orchestrator, and it needs constant adaptation as new failure modes show up. I documented the complete setup in my AI coding workflow with TDD, reviews, and guardrails.
In short: AI code review works best as the first pass, not as approval. This is the role I trust LLM code review to play: surface obvious defects, then hand the change to a human reviewer who knows the system’s intent.
1. AI-suggested fixes are not verified fixes
We had code suggestions on at the start. The AI would propose a patch right in the GitHub comment. It looked helpful. It was not. Developers are applied patches, then spent time undoing them.
Specific code-generation evaluations show why an unverified patch is risky. One GitHub Copilot study generated implementations for 200 Java methods from 124,800 prompt variants; 75% did not compile, 13% compiled but failed at least one available test, and 12% passed all tests. In a separate security benchmark, Veracode evaluated more than 100 models on 80 single-function tasks across four languages. It reported that 45% of samples failed security tests, with a 72% failure rate for the Java tasks. These studies measure code generation, not review, and their tasks do not reproduce our pull requests. They are still enough reason not to present a suggested patch as verified.
Anthropic made a related design choice when it launched Code Review for Claude Code on March 9, 2026: focus on bugs, verify findings, and leave approval to a human. The company says less than 1% of findings in its internal use were marked incorrect. That is a self-reported dismissal metric, not an independent false-positive evaluation, but the product boundary is sensible. The AI may point at a problem. I do not let it hand over a fix and pretend the job is finished.
2. Large pull requests weaken AI code review
This was the most consistent thing I saw. Small PR: sharp, on topic, catches real issues. Large PR: it forgets instructions, gets lazy, skips obvious problems, stops doing the code-agreement checks I asked for, and sometimes just leaves an LGTM for nothing.
Chroma tested 18 models, including GPT-4.1, Claude 4, and Gemini 2.5, and found that performance became less reliable as input length grew across its controlled tasks. The older “Lost in the Middle” paper found that performance was often highest when relevant information appeared at the beginning or end of a long input and lower when it appeared in the middle. In the current NoLiMa paper, 10 of 12 tested models scored at or below half of their short-context baseline at 32K tokens. None of these is a code-review benchmark, so they do not prove that a long diff causes the failures I see. They do show why a large advertised context window is not the same as reliable use of every token.
I introduced size limits on pull requests. Not mainly because of the AI, but because I find a 1,000-line PR hard to review well as a human too. The AI just made that cost impossible to ignore.
3. Detailed specs can anchor an AI reviewer
This one surprised me. Follow good spec-driven-development practice, attach the plan, the spec, and a long confident PR description, and the AI reviews worse. It starts trusting the code. It tilts toward “this looks well planned” and leaves an approval. A long, detailed description does the same thing.
Anchoring bias is one plausible mechanism, but the evidence is indirect. A 2024 study found that answers from three GPT models shifted toward numerical hints, and explicit instructions to ignore expert anchors did not reliably remove the effect. The study tested numerical estimation, not code review or specifications. A polished spec may act as an anchor, but that is my hypothesis, not the paper’s finding.
Sycophancy offers another possible explanation. Anthropic researchers found that five AI assistants exhibited sycophancy across four text-generation tasks, while humans and preference models sometimes preferred convincing sycophantic responses over correct ones. The authors say the behavior is likely driven in part by preference judgments. They did not test pull requests, so a confident description influencing a reviewer remains a reasonable concern, not a demonstrated causal chain.
4. Repeat AI review passes get softer
I do not have a paper for this one, so I will flag it as my observation, not a fact. After the AI leaves a batch of comments, the next iteration is noticeably weaker. It skips obvious problems it would have caught on a clean pass. It acts like the job is already done. Why work more if you already found something?
I can only reason about why. Each iteration carries the previous comments forward, which adds context and may contribute to the degradation from observation 2. My sycophancy explanation is even more speculative; none of the cited studies tests repeated review rounds. The effect is consistent in my setup, but I do not know its cause. I trust the second and third AI passes less than the first, and I make sure a human owns the later rounds.
Found this useful? Follow me on social media to stay updated.
5. More instructions do not guarantee better AI review
In my pipeline, more context does not automatically help. Past a point, more rules, examples, and guidance make the results worse. The reviewer spends effort matching the checklist and misses problems outside it.
The research supports a narrower claim. LIFBench evaluated instruction following for 20 models across six context-length intervals and found that performance often declined as inputs grew, with substantial variation by model and task. It did not test code review or prove that a long checklist causes worse reasoning. Together with the broader long-context results, it is enough to treat prompt space as constrained. Past a certain size in my pipeline, adding instructions adds noise that competes with the code for the model’s attention.
6. AI code review applies best practices inconsistently
The AI happily approves smelly code in runs. Even when I enforce a standard, sometimes it flags the violation and sometimes it does not. Same code, same rules, different verdict.
That variability is measurable. One study configured five models with temperature zero and repeated eight benchmark tasks; none produced repeatable accuracy across every task, much less identical output strings. The authors explicitly say the underlying cause is unclear, so floating-point behavior, batching, or routing should not be presented as established explanations. “I told it the rule once” does not mean it will apply the rule every time.
The broader code-quality picture raises a separate concern. GitClear analyzed 211 million changed lines from 2020 through 2024 and reported that, between 2021 and 2024, copy-pasted lines rose from 8.3% to 12.3% while lines associated with refactoring fell from about 25% to under 10%. This is a longitudinal, correlational study; it does not prove that AI caused the change. I treat it as a warning signal about review load and maintainability, not a verdict on every generator or reviewer.
7. AI code review cannot infer missing intent
Add a new field to the database, and the AI nods. It has no idea why you did it. It cannot argue that the field is redundant, or that it breaks an invariant somewhere it cannot see, because it never learned your intent. It accepts the change and moves on.
This is partly an input-boundary problem. “Diff-only” no longer describes every AI reviewer: Anthropic reports that its current system can surface a pre-existing bug in code adjacent to the pull request. My pipeline can inspect the repository too. It still cannot see a schema, API contract, or type definition in another service unless I expose it, and it cannot recover intent that exists only in someone’s head or in a private conversation. A reviewer cannot evaluate a constraint it never receives.
How AI code review works best
The perfect AI review does not exist. AI review is a tool with measured limits. Anchoring, sycophancy, long-context degradation, and non-determinism all appear in the research, but none of the cited studies proves that every code-review failure is immutable or architectural. Better context, prompting, tools, and verification can reduce some failures. I do not assume configuration will eliminate them.
The judgment of the human reviewer matters more now, not less. Do not replace classic code review with AI review. The worst outcome I can picture is AI-written code reviewed only by AI: an uncontrollable slop generation factory, two machines nodding at each other while nobody checks the assumptions.
What works is treating AI review as the first cheap pass that clears obvious noise before a human spends real attention. Keep PRs small. Keep auto-fixes off. Do not hand it your spec as gospel. Keep a person on the late rounds. And keep adapting the pipeline, because the failure modes move.
Adoption is a continuous process. Improvement depends on staying patient, careful, and curious about where it breaks. I will ask you what I keep asking myself: where has AI review actually saved you, and where did it quietly hand you false confidence?
Found this useful? Follow me on social media to stay updated.
