arxiv:2607.28887

To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing

Published on Jul 30

· Submitted by

Gopi Krishnan Rajbahadur on Aug 4

Centre for Software Excellence

Upvote

Authors:

Abstract

Large language models increasingly write and repair production code, yet evidence is mounting that their test-passing patches leave codebases harder to maintain. We identify one concrete source: deletion avoidance, the systematic tendency to retain code that an intended edit requires removing. Across the five leading models on the official SWE-bench Verified leaderboard, deletion recall against the developer patch reaches at most 71.7% even on tasks all five solve, and models reach the right file for over 92% of required deletions but cut the exact line in under 52% of cases. Instead, 29.0% of passing patches wrap the targeted code in a guard or fallback, a pattern we call Guard-and-Go. Such patches pass because the original tests rarely check removal: when we retrofit 34 Verified tasks with tests that fail if the targeted code remains, four frontier models spanning closed and open weights fall from 63.2% to 41.9%. Because real repairs mix removal with addition, we curate CanItDelete, a benchmark of 200 tasks mined from real commits whose entire required edit is deletion. Even with the addition work gone, the best model still fails one task in five, and smaller open models fall to 18.0%. We then ablate GPT-5.6 Sol under four cumulative prompts; success moves little until we supply the exact lines, which nearly eliminate incomplete deletion yet raise success only to 80.5% because the model then deletes beyond the spans or adds code instead. Finally, through a pilot study we show one potential fix: teaching deletion during post-training reduces deletion avoidance and improves broader code-editing performance, suggesting the behavior is undertrained rather than beyond reach.

View arXiv page View PDF Add to collection

Community

gopirajbahadur

Paper submitter 1 day ago

A lot of AI-generated patches feel like slop because frontier models still struggle with something we barely consider a task at all for a software engineer: 𝗱𝗲𝗹𝗲𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗿𝗶𝗴𝗵𝘁 𝗰𝗼𝗱𝗲 𝗮𝗻𝗱 𝘀𝘁𝗼𝗽𝗽𝗶𝗻𝗴.

We built CanItDelete, 200 tasks from real commits where deletion is the entire edit. No rewiring. No replacement logic. It diagnoses retention, partial deletion, over-deletion, and boundary errors.

Claude Opus 4.8 still failed 21% of tasks, GPT-5.6 Sol 26%, and GLM-5.2, Kimi K2 Thinking, MiniMax-M3, and DeepSeek-V4-Pro roughly one in three. Most failures left required code behind.

Giving models the exact lines helped, but exposed another problem. Claude reached 97.7%, while GLM-5.2 still failed 12%, GPT-5.6 Sol 18%, and Qwen3-235B 42%. For some models, better localization traded incomplete deletion for over-editing.

In real repository work, this becomes 𝙂𝙪𝙖𝙧𝙙-𝙖𝙣𝙙-𝙂𝙤. Instead of removing obsolete logic, models keep it live behind a guard or fallback, often as the default path for inputs the guard does not catch. Across five leading SWE-bench Verified submissions, 29% of passing patches did this.

That can pass tests. It is not merge-ready.

When we required the code to actually be gone on deletion-heavy tasks, resolution fell 21.3 points, and a third of accepted patches failed.

The hopeful result is that deletion can be taught. Adding just 0.7% deletion-focused data cut incomplete deletion by 13.9 points and improved SWE-bench Verified by 5.3 points. Over-deletion also rose, showing that completing a removal and stopping at its boundary are distinct skills.

𝗜𝗳 𝗰𝗼𝗱𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁𝘀 𝗮𝗿𝗲 𝘁𝗼 𝗽𝗿𝗼𝗱𝘂𝗰𝗲 𝗺𝗲𝗿𝗴𝗲-𝗿𝗲𝗮𝗱𝘆 𝗰𝗼𝗱𝗲, 𝘄𝗲 𝗻𝗲𝗲𝗱 𝘁𝗼 𝘁𝗿𝗮𝗶𝗻 𝘁𝗵𝗲𝗺 𝗹𝗶𝗸𝗲 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀. 𝗡𝗼𝘁 𝗼𝗻𝗹𝘆 𝘄𝗵𝗮𝘁 𝘁𝗼 𝘄𝗿𝗶𝘁𝗲, 𝗯𝘂𝘁 𝘄𝗵𝗮𝘁 𝗺𝘂𝘀𝘁 𝗻𝗼 𝗹𝗼𝗻𝗴𝗲𝗿 𝗲𝘅𝗶𝘀𝘁.