Plan together, run the thing
2026-09-15 · #ai #craft #testing
In April I wrote that AI can write a week of code in an afternoon, and that the fix was to bring back planning and testing. Five months on, the argument holds. Almost none of the mechanics do.
That is worth writing down, because the mechanics are the part people copy. Plan first, test always described a supervisor checking over the model’s shoulder. What I do now looks more like working with a peer whose strengths are different to mine. Same disciplines. Different seat.
Planning was mine. Now it’s ours
The April post had three questions to answer before handing over the keyboard: what you’re doing, what it has to fit into, and what you are explicitly not doing. The questions were there to keep me on requirements and decisions, because the model wouldn’t be.
At the frontier, at least, that has changed. There are models from OpenAI and Anthropic - currently Astra and Fable - that can find the unknowns with me and help close the decisions. It is no longer me doing the thinking and the model doing the typing.
I start with a sentence or two about what I want to achieve, and then we work out together what that actually means. Which decisions need making. The known unknowns: the things we will have to test or measure as we go. The unknown knowns: the things in my head that are not in the repo or anywhere the model can reach. And the unknown unknowns, which is where a second mind earns its keep, because those are by definition the ones I was never going to raise.
Plans used to be about implementation. I explained a requirement, the model wrote down how it would build it. Now the work before the work is defining the requirement together, until the model holds the picture in my head and has helped shape the edges of it.
The models are probably smarter than me. They can certainly hold more in their context in an instant than I can read in an afternoon. So we have one rule: one question at a time, and I answer it. A model that asks six questions in one message gets six half-answers. A model that asks one gets a real answer, and its next question is better for it. We keep going until we are aligned on the goal - the what and the why, not the how.
The plan is a record of decisions, not a list of work
The how still gets written down, but it goes at the bottom, and I mostly don’t read it. The model is better at which files and which functions than I am, and it does not need me watching. What sits at the top is a record of what we decided and why, with the alternatives we rejected. That changes what a plan is for. A task list is a commitment. A record of decisions is a hypothesis. When the benchmark shows the clever optimisation saves nothing under real load, nothing is lost by not building it - the decision gets updated and the reasoning stays. More than one feature has ended at “the deliverable is a well-supported decision not to code this”, and those were good days.
A plan is also a thing you can look at. Mine are HTML now, versioned, uploaded to planreview so it can be shared and opened with just a browser. Chat is a terrible place to keep a decision. Three versions of a plan with the changes visible is a shared surface. A scrollback is not.
I told you to live in plan mode. I’ve banned it
The April post leaned on plan mode hard. Spend longer there than in edit mode, I said, because the harness would make the model explore, ask, and surface assumptions before writing anything. My own config now contains the line “never use plan mode.”
The reason is dull. In my harness, plan mode blocks the permission classifier, so everything after it stops to ask for approval. The lesson is not dull. Everything plan mode enforced - explore first, ask before assuming, surface the unknowns - I kept. It lives in a skill that builds the plan artefact, and in a rule that no non-trivial edit starts until the plan is agreed. The discipline was never the mode. The mode was just where I first learned it. If your harness’s plan mode works, use it. If it stops working, keep the habit and drop the button.
Running the thing scales with the risk
“The model writes the code and the unit tests; I run it against the real world.” Still true, in spirit at least. What has changed, sometimes, is the method.
I have been building mbl.tools, a voice-to-text app, and the loop there is exactly as it was in April. The agent makes a change, I run the updated build, I use it as a real user would, I feed back what I noticed. That is the old manual loop, and for a great many changes there is nothing better.
I have also been building a skill that interviews people and fills in governance documents. That one has a much wider spread of use. I still did the very first run by hand, found problems, and had them fixed. But the quality of the output depends on how the skill is interpreted, what answers it gets, and a hundred other things I could not vary by hand in a week of trying. Manual runs were never going to tell me how it would behave in the world. So I built an evaluation harness: fixtures with their own imaginary products and real codebases, synthetic identities standing in for the person being interviewed, each one written to exercise a different kind of use. The evaluations are orchestrated, run and graded by AI. I review the results and the report.
Reviewing the report is where the verdict stays mine. I recently asked for a benchmark across a spread of Excel and Word files and got back a confident write-up. I asked how many files it had actually tested. “One. And Excel only.” The report was well formatted and it was not evidence of anything, so we went again.
So the use changed and the method changed with it. What did not change is the reason. I want to know how the thing will behave in the real world. Did it meet the requirement? Is it fit for purpose? And the question that gets skipped: is it fit for use, does it feel the way I meant it to feel? That last one still needs a human to run it, even if only once.
How the work is distributed
The other big change is who does the work. The model I spend my time talking to mostly does not write the code. That gets handed off, automatically, to smaller models. Cheaper, at least. I have started rating models on what they are good for in my workflow and where they fit: is this one a doer or a thinker? That deserves its own post, and it will get one.
One rule has not moved at all: YOLO is not a security model. Even with a security classifier model checking commands before they run, the rules I set out there still apply. Human execution for irreversible actions. Permission based on blast radius and controls, not vibes.
Confidence is still not correctness
In April the thing to watch for was fluent prose. Now the prose comes with supporting tables and the vocabulary of someone who has read the kernel source, and it arrives at the wrong root cause anyway. I watched a model diagnose swap pressure, be shown it was wrong, and call its own diagnosis a red herring. It then recommended a reboot, having already found the evidence that the problem did not start at boot. Rich detail is not evidence. It is formatting.
The model catches things I miss, and it should. I catch things the model misses, and I have to. It will defend a hand-rolled workaround until I ask whether we are working with the spec or around it. It will invent a safety rule and call it a requirement until I point out I never asked for it. It will put back a paragraph I asked it to delete, because it liked its own framing. None of that is new behaviour and none of it has gone away. It is what a model does when it has no way to check a fact, and it happens most often just after a compaction, when the context that would have caught it has been summarised out. Modern harnesses and the sheer volume of coding use have convinced a lot of people that models no longer hallucinate. They do. It is just better contained when the task has a verifiable output. Learning where the likelihood goes up is what makes you a good engineer in the AI tooling age.
The loop I’m using today
think: two lines of intent, then let the model find the unknowns
plan: one question at a time -> why not how
edit: delegate; a smarter model reviews
run: at the scale the risk demands
ship: small, then loop
Decide the thing together. Run the thing. Then ship it.