Refactor by Numbers

Your coding agent just rewrote a module. It reports the result is cleaner, safer and easier to reason about.

Do you believe it?

Mine told me exactly that last week, and I had no way to tell. The agent that wrote the code is the same agent grading it. It has no memory of the bug it just introduced, no fear of the pager, and no doubt. Doubt is not part of next-token prediction. Ask an agent whether its own code is good and you get a confident yes, because a confident yes is what that sentence usually looks like.

So I stopped asking. I made the tooling answer instead.

The rule: no adjectives, only measurements

I run a quality audit over a module or a commit range. It is a manual command, not part of the agent loop, and it is report-only. It never fixes anything. It runs three passes and joins them:

scope: a module path, or a commit range base..HEAD
   │
   ├─ complexity review     cyclomatic + cognitive, per function
   ├─ coverage + CRAP join  xcresult coverage × complexity
   └─ mutation testing      edit the code, see if a test screams
   │
   ▼
report only. the agent triages, a human approves

The command hands the work to a separate auditor subagent, not the agent that wrote the code. That separation is the whole trick. An implementation agent is trying to finish a task, and “the code is fine” is the shortest path to finished. The auditor has no task to finish and no feature to ship. It runs the tools, writes down what they say, and stops.

The other important property is that the audit both selects the work and grades it. The same tool that picked the targets decides afterwards whether they improved. The agent never gets to be its own referee.

Step 1: point it at the module, read the ranking

I pointed it at Oratio’s Reader module. It measured 430 non-UI functions, matched 426 of them to coverage data, and sorted them by CRAP.

CRAP stands for Change Risk Anti-Patterns. It is one of the few code metrics with a definition you can argue with mathematically rather than aesthetically:

CRAP = complexity² × (1 − coverage)³ + complexity

Read that formula for a second, because it encodes an opinion. Complexity is squared. The uncovered fraction is cubed. Full coverage collapses the whole first term to zero, and CRAP decays to plain complexity. Complex code is not the problem. Complex code nobody tested is the problem.

The ranking was not a matter of taste:

Look at the last two rows together. runFeedLoop is six times more complex than handleImportResult and carries the same risk score, because somebody tested it. That is the whole argument for measuring instead of eyeballing: the scariest-looking function in the module was not the problem.

The formula is easier to feel as terrain. Keep coverage and complexity as the ground and let CRAP be the height, and the module becomes a landscape. A green plain wherever tests exist. A cliff wall where they do not.

Every function stands exactly on this surface, because its score is fully determined by its two coordinates. performSamplePlayback is halfway up the wall. runFeedLoop lives out on the plain, where the terrain stays flat almost no matter how complex the code gets. There are two ways down from the wall: write simpler code, or write tests. The cubed term makes tests the fast way down.

Step 2: read the numbers correctly

Metrics are only objective if you know what they measure.

Cyclomatic complexity counts paths. Cognitive complexity counts how hard those paths are to hold in your head: nesting, jumps, interleaved conditions. On a flat switch they disagree violently, and cognitive is the one telling the truth. A 30-case switch scores terribly on cyclomatic and reads fine.

That distinction saved me from a fake finding later, and it is why the audit reports both.

Step 3: the refactor

The import and entitlement logic came out of a ViewModel and became a state machine: one State, one Event enum, one reducer, side effects as feedbacks.

// before: behaviour spread across methods, mutating stored properties
func handleImportSelection(_ result: Result<[URL], Error>) { ... }
func handleImportResult(_ result: ImportResult) { ... }

// after: one transition function, exhaustively switchable
Reducer { state, event in
    switch event {
    case let .pickedFiles(urls): ...
    case let .importFinished(result): ...
    }
}

That shape is easy to test, which is the point. But “easy to test” is another adjective, so back to the audit.

Step 4: the same tool grades the result

I ran the audit over the refactor twice. Once right after the rewrite, and again after landing the five tests the first pass asked for.

The complexity row is the one I did not expect to publish. Complexity went up. One reducer scores 32 where the old methods scored 7 and 4, so by the naive reading the code got worse. This is where cognitive complexity earns its place in the report. The same reducer scores 20 cognitive, because it is a flat switch, not a nest of conditions. Meanwhile CRAP fell from 56 to 32, because coverage went to 100% and the cubed term vanished.

The third row had no refactor in it at all. addToLibraryButtonState kept every line of its production code. Tests alone took it from 0% to 87.5% covered and its CRAP from 56 to 7.1. The cubed term is not a theoretical opinion. It is an 8x discount for proving your code works. Nothing in the audited range now scores above the reducer’s 32.

The bottom row is the real result. Mutation testing edits the code and checks whether any test notices. Before, eleven mutants survived: I could swap “start the import” for “reopen the file picker” on the paid-tier gate and every test still passed. Those tests ran the code. They asserted nothing about it. After the refactor, the two runs threw fifteen mutants at the new code and the tests killed twelve.

The three survivors are the referee refusing to sign off. Each one is a real gap in the new state machines: a retained starter offer that a second failure would drop, a stale dismissal event that could hide the wrong offer, and a button state no test reads. The report names the exact test that closes each one. That list is the next work item, not a footnote.

That is the difference between code that looks tested and code that is tested, and it is not a difference any agent can talk its way into.

What these numbers do not say

They say the functions are simpler to reason about and the tests actually assert behaviour. They say nothing about whether a state machine was the right abstraction here. Metrics for that question exist, coupling and cohesion measures like instability or LCOM, but this pipeline does not compute them and I have not found a Swift tool for them I would trust. CRAP cannot see that your module boundary is wrong, and mutation testing is happy to prove that a bad design is thoroughly tested.

The measurements also have edges worth admitting. Fifteen mutants across two runs is a budget, not exhaustive coverage. Everything ranked below the survivors was never mutated. And the app-level wiring has no test target, so the coordinators around these screens were never measured at all.

The judgement stays human. What changed is that the judgement now starts from evidence instead of from an agent’s confident summary of its own work.

Ask for the numbers. The agent will not doubt itself, and it does not have to. The tooling can do that for it.