Reviewing Large AI Refactors: Why Passing Tests Miss Deleted Intent

AI coding assistants are now good enough to clean up a neglected module in an afternoon, and the result usually reads better than what it replaced. The danger is not sloppy code; it is correct-looking code that quietly drops behavior nobody wrote down. This guide walks through a realistic failure of that kind, explains why the test suite and the reviewer both missed it, and gives you a concrete review process for large AI-assisted refactors.

A refactor that improved the code and broke the system

Consider a module that has been piling up changes for two years: about 4,000 lines spread over four files, the corner of the codebase that engineers avoid touching whenever they can. It has inconsistent error handling, three separate ways of parsing dates, and several functions nobody can say for sure are still called.

A developer hands it to an AI assistant over two sessions and gets back something clearly better: uniform patterns, proper type hints, dead branches removed, about a third fewer lines. The tests pass. The developer reviews the diff, or believes they have, and ships it.

Six days later a partner integration starts losing records. Nothing errors loudly; around four percent of records simply stop arriving. The cause is the refactor, and specifically a change the developer had read and approved.

Where the assistant genuinely excelled

It is worth being precise about what went well, because the takeaway is not "AI is bad at refactoring." The assistant was excellent at every transformation that depends only on the code in front of it:

• collapsing the three date-parsing helpers into a single function
• flattening nested conditionals into guard clauses
• annotating the entire module with type hints
• finding two branches that really were unreachable
• choosing clearer names

For mechanical, local changes that can be verified by reading them, it was faster and more consistent than a human, and it did not lose focus several hundred lines in, which is where manual refactors tend to degrade. When the task is "make this code express the same behavior more clearly," these tools perform very well. The trouble begins at the edge of what "the same behavior" actually includes.

Why load-bearing code gets deleted

Code that carries no visible reason looks exactly like cruft. Here is the approved change:

# before
def normalise_timestamp(raw):
    ts = raw.strip()
    if ts.endswith("Z") and "." in ts:
        ts = ts.split(".")[0] + "Z"          # partner sends 7-digit microseconds
    return datetime.fromisoformat(ts.replace("Z", "+00:00"))

# after
def normalise_timestamp(raw):
    return datetime.fromisoformat(raw.strip().replace("Z", "+00:00"))

The removed branch looks like a textbook refactoring target: a special case, an oddly specific suffix check, a string split. It seems redundant, because fromisoformat already parses ISO timestamps.

The catch is the input. One partner sends timestamps with seven fractional digits. Before Python 3.11, fromisoformat only parsed the formats that isoformat() itself emits, which means no more than six fractional digits, so a seven-digit value raises a ValueError. Python 3.11 and later accept a much broader ISO 8601 subset and are more forgiving here, so whether this change breaks depends on the runtime, which is one more reason to verify rather than assume. The branch existed because of a support ticket from eighteen months earlier, and its short inline comment was not enough to make anyone, human or model, weigh it properly against how removable it looked. Notice too that the old branch did more than it seemed: it discarded the fractional part entirely, so even the "after" version's output differs for inputs that do parse.

The core problem is simple: the model could see the code, but not the incident that produced it. The ticket, the postmortem, the chat thread and the hours spent debugging were all outside the file. Running git blame would have surfaced a commit message like "fix timestamp parsing for partner feed", which was the whole justification, but nobody asked the assistant to look. The reviewer approved the deletion for the same reason the assistant proposed it: both were reading the diff, not the history.

Why the test suite stayed green

The tests checked what the function returned for inputs someone had thought of, and the problematic input was, by definition, one nobody had.

The fixtures contained timestamps with three fractional digits, because that is what people type when writing test data by hand. Seven-digit values only ever showed up in one partner's live feed.

This is what makes large refactors risky, whether or not AI is involved: a passing suite proves only that the behavior your tests describe survived. It says nothing about behavior they do not describe. When you refactor thirty lines by hand, you think about every one of them. When four thousand lines change at once, the test suite ends up doing almost all of the verification, and it was never designed to carry that load.

Coverage metrics did not help. The function had full line coverage: every line executed during the tests. One branch simply never ran with the input that mattered. Line coverage measures which code ran, not which input shapes were exercised.

Sorting AI changes by risk

A practical response is to stop reviewing AI-generated diffs uniformly and to triage them by category. The categories have very different risk profiles:

• Low risk: type annotations, renames, formatting, early returns and similar mechanical rewrites. They are local and easy to verify by reading.
• Moderate risk: consolidating duplicated helpers. Usually fine, but check that the merged version handles every caller's inputs.
• High risk: deleting code that looks unused, removing special cases that look redundant, and unifying behavior that looks inconsistent.

The high-risk group shares a shape: in each case, the model correctly notices that something looks wrong, and the fact that it looks wrong is exactly why it exists.

The last item bites in a quieter way too. Picture one module that formats errors differently from the rest, and a refactor that makes it match. The difference was intentional, because a downstream consumer parses that specific error shape. Aiming for uniformity was a sound instinct; applying it here was a mistake.

Five practices for the next large refactor

These are ordered roughly by how much each would have prevented the incident above.

Keep each diff small enough to review for real

Four thousand lines is not a reviewable change. Past a few hundred lines, most people are skimming for obvious problems rather than reasoning about each hunk, whatever they tell themselves. Limit each session to one module and each pull request to one module's changes. The bottleneck is not the model's throughput; it is the reviewer's attention.

Require a justification for every deletion

Before accepting the output, ask the assistant for a list of everything it removed and, for each item, why it concluded the removal was safe. Forcing the reasoning into text is one of the most valuable steps you can add. Any justification along the lines of "this appeared redundant" should be treated as a flag for manual investigation.

Give the model the history

Tell the assistant to look up git blame for each line it proposes removing and to report the commit message. The reason for an odd branch is very often recorded there, and the model uses that context well when it has it; it just will not go looking unprompted.

Write the "why" comments before refactoring

Add explanatory comments first, not afterwards. A precise comment, naming the partner, the seven-digit fractions and the six-digit parser limit, would likely have stopped this deletion. A vague note about partner timestamps did not. If whoever reads the code next, a person or a tool, cannot tell why a line exists, that line is one cleanup away from disappearing.

Compare old and new against real traffic

Feed a day's worth of real production inputs through both the old and the new implementation and compare what each returns. For pure transformations like parsing and normalization this is cheap, and it catches precisely the class of problem that fixtures miss, because production traffic contains the inputs nobody thought to write down. If replaying traffic is not possible, sampling real payloads into regression fixtures is the next best thing.

Key takeaways

The refactor was still worth doing: the module is better, the work fit into an afternoon rather than a week, and a single regression across four thousand changed lines compares well with many large manual refactors. Calling the outcome "the AI's mistake" misses the point. It drew a reasonable conclusion from what it could see, a special case with no stated reason, and the human reviewer drew the same conclusion from the same incomplete artifact.

• The repository is only part of the system; the reasons behind odd code live in tickets, commit messages, postmortems and people's memories.
• A tool that sees only the code will confidently remove anything whose justification was never written down.
• Green tests and full line coverage do not prove behavior was preserved for inputs nobody modeled.
• Triage AI diffs by risk, demand written justifications for deletions, and feed the model the history.
• Writing down why unusual code exists was good practice long before AI refactoring; it is now the cheapest safeguard you have.