Why this one

"Plan first" is the most repeated piece of advice in agentic development and the least evidenced. Every harness now ships a plan mode. I have never seen a controlled comparison, and my own experience is suspiciously positive in a way that should make me distrust it - I may just be remembering the plans that worked.

Design

Paired tasks, same repository, same harness, same model, same time budget.

  • Arm A - agent receives the task description and proceeds.
  • Arm B - agent writes PLAN.md first, gets it approved unchanged, then implements.

Thirty tasks, drawn from real work: rename a domain concept across layers, split a module, change a serialisation format, extract an interface, fix a bug whose cause is in a different package. Each task is run twice, once per arm, alternating order to blunt warm-cache effects.

Metrics

Metric How
First-pass success Tests pass, task genuinely complete, no human edits to logic
Review time Minutes to approve, recorded honestly by a timer
Diff size Lines changed, and files touched
Cost Tokens and wall-clock per task
Plan drift Did the implementation follow the plan; if not, was the deviation flagged

Review time is the metric I am most interested in and the one everyone leaves out. If plans double the output but halve the review burden, they are a win; if they only add a step, they are theatre.

What would falsify it

  • No difference in first-pass success, and review time unchanged or worse.
  • Improvement only on tasks that were already easy - the "plans help when you didn't need them" outcome.
  • Plans that are wrong in ways the agent then follows loyally, producing confident garbage. This is the failure mode I expect to be most instructive if it shows up.

Confounders I already expect

Task difficulty is not identical across pairs. My approval criterion is not perfectly consistent day to day. And a plan I approve may improve outcomes simply because I engaged with the task earlier, which is a real effect but not the one the advice claims.

Status

Queued. Writing the task set is the actual work and I have not finished it. Publishing the design before the result is deliberate - it makes it harder for me to quietly reframe the conclusion later.