arxiv:2609.05779

Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

Published on Sep 5

· Submitted by

Andrej Andrejev on Sep 10

Upvote

Authors:

Andrej Andrejev

Abstract

Diff-based code editing underperforms direct generation overall but excels only on short, localized edits such as refactoring and error fixes, a property termed task locality.

Generated by thinkingmachines/Inkling-Small

Large language models used for code editing can be trained and deployed in at least two output regimes: direct generation, where the model emits the entire modified file in one shot, and iterative diff-based generation ("steps"), where the model emits a sequence of localized search/replace edits applied one at a time until it signals completion or a step budget is exhausted. The diff-based regime is attractive because it mirrors how developers edit code and should require far fewer generated tokens per turn. We train two code models - a 100M-parameter model trained from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B - in both regimes on a shared Flutter/Dart dataset, and evaluate all four resulting models on a held-out set of approx 1,790 tasks per model. Direct generation substantially outperforms diff-based generation on every metric we measure - compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality - and the gap persists after controlling for task difficulty via a matched-ID comparison and when restricting to code that compiles on both sides. We then identify a single, architecture-independent mechanism behind the conditions where diff-based generation does win: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in our dataset. We term this task locality and discuss its implications for when an edit-based training regime is and is not the right choice for a code-editing model.

View arXiv page View PDF Project page GitHub 2 Add to collection

Community

bbidpa

Paper author Paper submitter 1 day ago

Hey all, quick behind-the-scenes on this

What I compared: two ways to make a model edit code:

regenerate the whole file in one shot; vs.
write a sequence of small diffs (search/replace edits), like a human dev would.

The models (all trained both ways, so 4 checkpoints total):

🌈 Rainbow-Pony-100M-Flutter-direct / -steps
🔷 Qwen2.5-Coder-0.5B-Flutter-direct / -steps

Why Rainbow-Pony is trained from random weights (no pretraining at all): wanted a model with zero prior exposure to any language or code, so whatever editing behavior shows up can only come from the fine-tuning itself - not leftover knowledge from some giant pretraining corpus. Basically a clean ablation: isolates the training method from the scale/pretraining question. Qwen gets the same two methods, but starting from a model that already knows how to code — so I could check if the pattern holds either way.

Evaluation setup: ~1,790 real Flutter/Dart editing tasks. Picked Flutter/Dart on purpose too - it's relatively rare in typical training data (less contamination risk), and its widgets/functions are full of closing {} braces, which makes structural correctness easy to check automatically.

The result, in short:

🏆 Direct generation wins overall - easier target for the model than staying correct across a chain of dependent edits
✂️ Diffs hold their own (and win in places!) on short, localized changes - a tiny surgical search/replace is way easier to nail than rewriting a whole file just to touch a few lines

Demo 🎮 - Here you throw in your own Flutter/Dart snippet + instruction and watch all 4 models race on the identical task, live: 🤗 Space - Diffs vs. Whole Files

Ask away if you're curious about the setup, dataset, or eval methodology. Have fun.