The most useful thing I did this year was stop evaluating models and start evaluating harnesses.

Not because models do not matter. They obviously do, and each generation moves the floor on what is possible. But the floor keeps rising for everyone at the same time, while the harness - the tools, the feedback loop, the permissions, the context management - stays with you, and compounds.

Two agents on the same model can differ by an order of magnitude in what they accomplish on the same repository. That gap is not model quality. It is everything wrapped around it.

What a harness actually contributes

Tool ergonomics. Whether edits are patches or whole-file rewrites changes everything on files past a few hundred lines: a patch that fails to apply is one recoverable error, a whole-file rewrite that silently drops a section is a bug you will find in review or not at all. Whether search returns file names or file names plus matching lines determines how many turns are spent orienting.

Feedback fidelity. The difference between "command failed" and the actual stderr, verbatim, is the difference between an agent that fixes its own mistake and one that flails. This is cheap to implement and surprisingly rare.

Primitives over prompting. "Find where this is used across the repo, then change all call sites" is a terrible instruction and a trivial tool. Every time I find myself writing elaborate prose to make the agent do something mechanical, that is a missing tool.

Permissioning. Not a compliance feature - an engineering one. An agent that can rm -rf outside the workspace forces you to supervise it continuously, which destroys the whole workflow. Explicit, narrow permissions buy the ability to walk away.

Resumability. Sessions die: context limits, crashed processes, a laptop lid. Whether you can pick up with state intact determines whether long tasks are viable at all.

Orchestration. One agent with a polluted context is worse than one clean parent and two scoped children. The ability to spawn, scope, and collect from subagents is not a nice-to-have - it is the main lever on quality for anything non-trivial.

Why this flips the usual advice

The standard advice is to pick the best model and then adapt your workflow to it. In practice model leadership changes every few months, pricing shifts under you, and the behaviour you tuned your prompts against disappears at the next release. What does not disappear is your repository's conventions, your tests, your permission model, and the way your tools return results.

So the durable investment is the harness, and the durable evaluation is: on my codebase, on my tasks, what fraction of work comes back correct and reviewable? Benchmarks on someone else's synthetic tasks answer a different question.

A short evaluation that beats vibes

For any harness you are considering, run the same five tasks:

  1. One small, well-specified change with a failing test already in place.
  2. One multi-file refactor with an ambiguous boundary.
  3. One bug whose cause is three files away from its symptom.
  4. One task with a deliberate trap: a test that must not be "fixed" by weakening it.
  5. One long task that will definitely exhaust the context window, to see what resumption does.

Score: did it finish, was the diff reviewable, did it tell you what it was uncertain about, and what did the run cost. Five tasks, half a day, and you know more than any leaderboard can tell you.

The model is the engine. The harness is the vehicle, the road, and the steering. I have stopped buying engines without driving the car.