Why this one
"Plan first" is the most repeated piece of advice in agentic development and the least evidenced. Every harness now ships a plan mode. I have never seen a controlled comparison, and my own experience is suspiciously positive in a way that should make me distrust it - I may just be remembering the plans that worked.
Design
Paired tasks, same repository, same harness, same model, same time budget.
- Arm A - agent receives the task description and proceeds.
- Arm B - agent writes
PLAN.mdfirst, gets it approved unchanged, then implements.
Thirty tasks, drawn from real work: rename a domain concept across layers, split a module, change a serialisation format, extract an interface, fix a bug whose cause is in a different package. Each task is run twice, once per arm, alternating order to blunt warm-cache effects.
Metrics
| Metric | How |
|---|---|
| First-pass success | Tests pass, task genuinely complete, no human edits to logic |
| Review time | Minutes to approve, recorded honestly by a timer |
| Diff size | Lines changed, and files touched |
| Cost | Tokens and wall-clock per task |
| Plan drift | Did the implementation follow the plan; if not, was the deviation flagged |
Review time is the metric I am most interested in and the one everyone leaves out. If plans double the output but halve the review burden, they are a win; if they only add a step, they are theatre.
What would falsify it
- No difference in first-pass success, and review time unchanged or worse.
- Improvement only on tasks that were already easy - the "plans help when you didn't need them" outcome.
- Plans that are wrong in ways the agent then follows loyally, producing confident garbage. This is the failure mode I expect to be most instructive if it shows up.
Confounders I already expect
Task difficulty is not identical across pairs. My approval criterion is not perfectly consistent day to day. And a plan I approve may improve outcomes simply because I engaged with the task earlier, which is a real effect but not the one the advice claims.
Status
Queued. Writing the task set is the actual work and I have not finished it. Publishing the design before the result is deliberate - it makes it harder for me to quietly reframe the conclusion later.