Agentic Development

Pull request review: Scaling AI code review without losing trust

AI code review is the practice of examining changes produced by people or coding agents before they merge, to confirm correctness, security, maintainability, and shared understanding. As agents generate patches faster than engineers can assess them, review must do more than spot defects. It needs cl

MMichael SwindellSeptember 10, 202616 min read

AI code review is the practice of examining changes produced by people or coding agents before they merge, to confirm correctness, security, maintainability, and shared understanding. As agents generate patches faster than engineers can assess them, review must do more than spot defects. It needs clear intent, acceptance criteria, test evidence, traceable decisions, and accountable follow-up. A shared control plane such as Levr helps teams attach work, verification, and discussion to the same change. The goal is not to slow automation, but to keep system knowledge and trust growing at the same pace as code output across the team.

At a Glance

Reliable agentic development depends on review systems that preserve context, evidence, and human accountability.

The bottleneck: Coding agents can create changes far faster than maintainers can understand and approve them.
The real purpose of review: Pull request review transfers knowledge and aligns engineers on the system, not just code quality.
The trust risk: A contributor who cannot explain an agent-generated change cannot effectively respond to review feedback.
The operational fix: Require structured intent, automated verification, durable agent observations, and clear ownership.
The practical outcome: Teams can use coding agents at higher volume without treating review as an unpaid cleanup queue.

Why does AI-generated code make review the bottleneck?

AI-generated code makes review the bottleneck because the cost of producing a plausible patch has fallen sharply, while the work of understanding its behavior, risks, and design tradeoffs remains human and time intensive. More pull requests do not automatically create more trustworthy software.

A coding agent can clone a repository, inspect an error, and propose a patch in minutes or less. That is useful for the person encountering the problem. It becomes expensive when the patch reaches a maintainer who must understand the surrounding architecture, check edge cases, validate tests, and decide whether the change belongs in the project.

This gap creates the drive-by pull request problem. A contributor may submit agent-generated code without expecting to answer questions, revise the implementation, or maintain the result. The maintainer then faces an unpleasant choice: take ownership of someone else’s unfinished work or leave the pull request open until it becomes stale.

Parallel agent workflows amplify the issue. Ten agents can create ten branches, but a reviewer cannot parallelize deep technical comprehension at the same rate. If review capacity stays fixed while generated output multiplies, maintainers receive a growing queue of changes with uncertain provenance and uncertain ownership.

Code review has always involved more than finding syntax errors. Google’s code review guidance describes review as a mechanism for improving code and spreading knowledge across a team. That knowledge transfer becomes more important when the implementation arrives faster than the author’s understanding of it.

What breaks when a contributor cannot explain a pull request?

When a contributor cannot explain a pull request, the normal feedback loop breaks down. Reviewers lose confidence that comments will receive informed responses, maintainers inherit hidden context work, and the team cannot tell whether an implementation reflects an intentional decision or a model’s plausible guess.

In a healthy pull request, review comments start a conversation. A reviewer asks why a design was chosen, what alternatives were rejected, whether a failure mode was considered, or how the behavior should evolve later. The author answers because they understand the system and remain responsible for the change.

Agent-generated code can weaken that contract when the human author merely forwards the output. “The agent chose it” is not an architectural rationale. It does not explain the constraints, confirm that tests cover the intended behavior, or establish who will investigate a regression six months later.

This does not mean every contributor must hand-write every line. It means the named owner must be able to defend the intent, interpret feedback, and stay engaged until the change reaches a maintainable state. Automation can produce a patch. Accountability still needs a person or an explicitly governed agent workflow.

Minimum ownership standard for agent-generated pull requests

State the problem: Explain the observed failure or requested behavior in concrete terms.
Describe the intended result: Record the acceptance criteria before implementation details.
Explain the approach: Summarize the relevant design decision and meaningful alternatives.
Provide verification: Link tests, test results, and any manual checks that support the change.
Name an owner: Make clear who will address review comments and post-merge regressions.

How does AI-generated output pull team mental models apart?

AI-generated output pulls team mental models apart when a system changes faster than its creators and reviewers can comprehend it. The codebase, the author’s understanding, and colleagues’ understanding can drift into separate versions of reality, making maintenance progressively harder.

Every engineer carries an internal model of the system. It is never perfectly complete, but routine collaboration keeps models close enough to support safe changes. Small implementation steps, design discussions, tests, and reviews create repeated opportunities to align on how the software works.

A large agent-generated change can disrupt that alignment. The implementation may be internally coherent, yet the person who merged it may understand only the prompt and the final summary. A reviewer then has to reconstruct the reasoning from code, tests, and scattered tool output. Future maintainers begin even further from the original decision.

This is why skipping review is not a harmless optimization. A successful build proves only that selected checks passed. It does not prove that the people responsible for operating the service understand the behavior, the constraints, or the consequences of changing it later.

Keep review focused on the highest-value context. Reviewers do not need a narrated stream of every agent action. They need a compressed explanation of the problem, intended behavior, changed boundaries, validation evidence, unresolved risks, and decisions that may affect later work.

What should a reliable AI code review workflow include?

A reliable AI code review workflow turns each change into an auditable package of intent, implementation, verification, and ownership. It uses automated checks to remove routine work, while reserving human attention for design, risk, product behavior, and the shared understanding that tests cannot create alone. Skipping this step accumulates as verification debt that compounds with every unreviewed change. The mechanics of scoring and evaluating that output are covered separately in how to evaluate AI-powered code review.

Start before the coding agent writes code. Natural-language requests are useful, but vague instructions create ambiguous implementations. Define the outcome with structured acceptance criteria, affected components, constraints, and explicit non-goals. This gives both the agent and the reviewer a stable contract.

1. Define intent before delegation

Write the task in a way that can be checked. For example, “fix login failures” is too broad. A stronger request identifies the failing condition, expected user-visible behavior, affected API or component, compatibility constraints, and tests that must pass.

2. Limit the scope of agent work

Use smaller, coherent tasks when possible. A narrow change is easier to validate and easier to roll back. Large refactors may still be appropriate, but they should be planned in stages with explicit checkpoints instead of arriving as a single, opaque patch.

3. Require machine-verifiable evidence

Automated tests, static analysis, CI/CD results, and build output should travel with the change. Evidence is not a substitute for review, but it lets reviewers spend less time on checks that automation can perform consistently.

The NIST AI Risk Management Framework emphasizes governance, measurement, and ongoing management of AI system risks. In software delivery, that principle maps directly to recording what an agent was asked to do, what it changed, what evidence it produced, and what approval boundary applied.

4. Review design and behavior, not generated verbosity

Ask questions that test understanding: What contract changed? What happens on failure? Why is this dependency safe? Which cases remain untested? Could this break a caller that is not represented in the test suite? These questions expose risks that line-by-line scanning may miss.

5. Keep a durable decision record

Record the final summary after review. Capture why the approach was approved, what follow-up work remains, and what assumptions are worth revisiting. This prevents the next engineer from having to infer intent from a large diff alone.

Give coding agents a workflow with proof attached.

Levr keeps issues, acceptance criteria, tests, execution results, and approval gates connected, so an agent-generated change has project context beyond a prompt.

Explore agent-first workflows with Levr

How can teams capture agent work without saving every raw session?

Teams can capture agent work by retaining raw logs where needed, then creating concise observations that summarize outcomes, failures, decisions, and next steps. This preserves useful learning while preventing long session histories from becoming another unreadable source of review noise.

Long-running agents can sound productive while making little meaningful progress. A status message may report completed turns, files inspected, or attempted actions without proving that the underlying task moved forward. Treat progress language as an observation to verify, not evidence of completion.

Raw logs remain useful for investigation. They show tool calls, prompts, files, failures, and execution history. But raw logs are usually poor interfaces for a reviewer or the next agent run. They are too long, too repetitive, and too close to the agent’s working process.

Instead, generate a short observation at the end of each meaningful run. The observation should answer: What was attempted? What changed? What evidence supports success? What failed? What is unknown? What should the next run or human do?

A practical observation template

Task: The requested outcome and the relevant acceptance criteria.
Result: Files changed, behavior implemented, or reason no safe change was made.
Evidence: Tests run, results, and known gaps in verification.
Decisions: Important implementation choices and assumptions.
Risks: Unresolved failures, ambiguous requirements, or dependencies requiring review.
Next action: The specific follow-up for a person or another coding agent.

Shared memory should be curated rather than blindly accumulated. If every speculative thought enters the permanent context, future agents inherit noise and false confidence. Retain validated observations, label uncertainty, and connect important records to the issue or pull request they explain.

How do you make prompts precise without returning to manual coding?

You make prompts precise by treating natural language as a specification interface rather than a casual conversation. Define behavior, boundaries, examples, and verification conditions clearly, then let the coding agent choose implementation details within those constraints and subject the result to review.

Natural language is flexible, which is both its advantage and its weakness. A request such as “make the dashboard faster” could mean fewer network calls, lower perceived latency, smaller bundle size, different caching behavior, or simply clearer loading states. A coding agent must choose an interpretation unless the task defines one.

Precision does not require writing implementation code by hand. It requires expressing the problem in a form that reduces avoidable ambiguity. Use examples, explicit constraints, expected failure behavior, non-goals, and testable completion conditions.

Turn vague requests into reviewable specifications

Vague: “Improve the search endpoint.”
Specific: “Return results within the existing response shape, preserve current authorization behavior, add pagination validation, and cover empty, invalid, and multi-page results with tests.”

The second request still leaves implementation freedom, but it gives the agent a clearer target and gives the reviewer concrete questions to ask. It also makes it easier to distinguish a completed task from a convincing-looking patch.

One way to run this process in Levr’s agentic workflow is to define intent and acceptance criteria, let agents take scoped tasks, require automated verification, then hold changes at a human review and approval gate. The work item becomes the shared source of context for both people and agents.

Evidence-Backed Review

The Levr Dashboard tracks pull request review progress by centralizing issues, acceptance criteria, and linked test results. Engineering teams can monitor agentic activity and verify that generated code matches project intent before human approval gates.

Agentic Software Development | Control Plane | Manual & Automatic Verification Issue Tracking |Levr

Monitor review health and agent-driven verification progress in the Levr Dashboard.

What mistakes make agent-generated pull requests harder to review?

The most damaging mistakes are treating generated code as free, accepting narrative progress as proof, and sending reviewers unscoped diffs without intent or ownership. These practices hide the real cost of automation by shifting it into reviewer attention, maintenance debt, and incident response.

Submitting before understanding: Do not open a pull request merely because the code compiles. The owner should understand the requested behavior and key design choices.
Confusing activity with progress: A long agent session, many tool calls, or a cheerful summary does not demonstrate a solved problem.
Reviewing only the diff: Reviewers need task context, acceptance criteria, test results, and known limitations, not just changed files.
Accepting oversized changes: Large generated refactors are difficult to reason about and can conceal unrelated behavior changes.
Using “latest” context indiscriminately: Uncurated memory can spread stale assumptions and unverified conclusions across agent runs.
Removing human gates too early: Autonomy levels should match risk. High-impact changes need stronger evidence and explicit approval.

For engineering managers, the key metric is not raw agent output. Track whether review queues are growing, whether changes arrive with sufficient evidence, whether regressions correlate with agent-generated work, and whether engineers can explain the systems they own. Throughput without comprehension is deferred cost.

Technical Deep Dive FAQ

What is AI code review?

AI code review is the process of evaluating software changes created with AI assistance or by coding agents before those changes merge into a shared codebase. It includes automated checks such as tests and static analysis, plus human review of intent, architecture, security, maintainability, and operational behavior. The important distinction is that AI code review is not just asking a model to inspect code. It is a workflow for making generated changes understandable, verifiable, and attributable to an accountable owner.

Why is code review still necessary when coding agents write tests?

Tests prove that selected scenarios passed under selected conditions. They do not fully explain why a design was chosen, whether requirements were interpreted correctly, whether edge cases were omitted, or whether the team understands the resulting behavior. Code review creates shared system knowledge and exposes assumptions that automated tests may not represent. Coding agents can greatly improve test coverage and speed, but human review remains necessary when decisions affect architecture, user behavior, security boundaries, or long-term maintainability.

What is a drive-by pull request?

A drive-by pull request is a contribution submitted with little expectation that its author will participate in follow-up. In an agentic workflow, it often occurs when someone asks a coding agent to patch a problem and submits the result without understanding or maintaining it. The maintainer then carries the burden of interpreting the code, responding to review concerns, and possibly completing the work. Requiring a named owner, intent summary, and test evidence helps prevent this imbalance.

How should teams review pull requests created by coding agents?

Teams should review coding agent pull requests as evidence-backed changes, not as raw generated diffs. Start with the issue, acceptance criteria, scope, and expected behavior. Then inspect the implementation’s key decisions, test coverage, failure handling, and compatibility implications. Require CI/CD evidence and a concise summary of what the agent attempted and what remains uncertain. The assigned owner should be able to answer reviewer questions and make revisions. This keeps review focused on risk and understanding instead of generated verbosity.

How can a coding agent prove that a task is complete?

A coding agent should prove completion with observable evidence tied to acceptance criteria. Useful evidence includes passing relevant tests, build output, static analysis results, changed files, a concise explanation of the implementation, and explicit disclosure of untested areas or blocked steps. Completion should not be based only on the agent’s natural-language status message. For higher-risk work, a human approval gate should confirm that the evidence supports the intended behavior and that the change is safe to merge.

What is observational memory for coding agents?

Observational memory is a curated record of what happened during a coding agent run, written as a concise, reusable summary rather than a full raw log. It can capture the task, relevant context, decisions, files changed, tests run, failures encountered, and recommended next action. This gives later agents and humans useful continuity without forcing them to parse thousands of tool calls. Effective observational memory labels uncertainty clearly and avoids storing speculative conclusions as established facts.

Should engineering teams store all coding agent logs?

Teams should retain logs according to their operational, security, and compliance needs, but they should not expect raw logs to function as everyday project memory. Detailed logs can help debug failures, investigate unexpected behavior, and reconstruct tool activity. For routine collaboration, short summaries linked to issues and pull requests are more useful. Store the evidence needed for traceability, then create a concise observation that explains the outcome, confidence level, unresolved risks, and required follow-up.

How can teams prevent coding agents from creating unreviewable changes?

Prevent unreviewable changes by setting scope limits before execution. Break large work into smaller tasks with explicit acceptance criteria, protected boundaries, and required verification. Ask agents to produce an implementation plan for complex changes before writing code. Require pull requests to include a problem statement, summary of design choices, test results, and known limitations. If a change remains too large to reason about, split it or request a smaller first step. Reviewability is a delivery requirement, not optional documentation.

What should engineering managers measure in agentic code review?

Engineering managers should measure signals that reveal whether agent output is creating sustainable delivery. Monitor review queue age, time to first review, pull request size, rework caused by review feedback, test health, regressions, blocked work, and the proportion of changes with complete acceptance criteria and verification evidence. Attribute activity to human and agent contributors where practical. Avoid using commits or lines changed as primary productivity measures, because those numbers can rise while reviewer load, confusion, and maintenance risk rise with them.

Can natural language replace programming languages for software development?

Natural language can describe goals, constraints, examples, and desired behavior, making it a valuable interface for coding agents. It does not remove ambiguity by itself. Programming languages, tests, schemas, interfaces, and executable checks provide forms of precision that conversational requests often lack. The practical approach is not choosing one over the other. Use structured natural-language specifications to guide agents, then use code, tests, reviews, and quality gates to verify that the implementation matches the intended contract.

Key Takeaways

AI code review works when teams treat generated code as a starting point for evidence and shared understanding, not as a finished contribution.

Higher output requires stronger context and clearer ownership.

Review capacity: Coding agents increase change volume faster than human comprehension scales.
Trust: Every pull request needs an accountable owner who can explain and revise the work.
Precision: Clear acceptance criteria reduce ambiguity before implementation begins.
Evidence: Tests, CI/CD results, and durable observations make agent work easier to verify.
Control plane: Linking issues, tests, changes, and approvals keeps delivery context from fragmenting.

Agentic development becomes sustainable when code, proof, and understanding move together.