The Buzzsaw
I gave eight agents one file and told them to loop until it was perfect.
They found 21 real defects. They also produced six false ones — and three of those were mine.
That ratio is the whole lesson. The tests were wrong nearly as often as the code, and the only reason anyone knows is that somebody checked.
The pattern
The buzzsaw is a way of driving agents that assumes the builder cannot see its own blind spots. So you pay a separate adversary to be unimpressed, and you let a machine decide when to stop.
Five moves:
- Name a reference. “Make it good” is unfalsifiable. “Put it next to the real thing, blind, and say which is better” is not. Name the actual best-in-class thing you are chasing.
- Fan out builders over disjoint regions. This is the trick that makes parallelism work on a single artifact. Give each agent an explicit list of what it does not own, and warn it that line numbers shift while it works.
- Loop each item instead of making one pass and declaring victory.
- Add a harsh critic, read-only. Its entire job is to refuse to be impressed. It must never edit.
- Write the acceptance gate first. The loop ends when the gate exits zero. Not when the work looks good.
Most people already do some version of one through three. Four and five are where the value is.
The best finding was not a bug
An agent implemented ambient occlusion — the shading in corners that makes 3D geometry look solid. It did the work correctly. Then it verified its own work: 66.3% of surfaces carried the right vertex data, the material flag was set. It reported a win, with evidence.
The blind critic scored it 2 out of 10. The corners rendered perfectly flat.
The cause was upstream: the scene lighting was bright enough to clip the highlight, so multiplying by a shadow value barely moved a pixel that was already clamped to white. Every line of the implementation was correct. The data was in the buffer. It never reached the screen.
The builder measured the attribute. It never measured the pixel.
That is not a coding error. It is a verification error, and no amount of re-reading the source finds it.
The blocker nobody could see
A “click to play” overlay sat at one stacking layer. The toolbar sat below it. Every button in that toolbar was unreachable — clicking the settings icon captured the mouse instead of opening settings, and three panels had no keyboard fallback at all.
The buttons rendered at full opacity the entire time. It is invisible in a screenshot. It is invisible in the source, because the CSS is unremarkable and the click handlers are correctly registered.
It only appears if you put a real cursor on a real button and click.
It had already shipped.
Six times, the failure was the test
This is the part I did not expect, and it is now the reason I bother with the whole apparatus.
A check read a value through a method that did not exist. It got null. The code treated null as “not supported, skip,” and printed a tidy line. The single most important check in the suite never ran once. In a log, a skipped check and a passing check look nearly identical.
Another: I used a common DOM property as a visibility test. It returns null for any fixed-position element, which means correct code reads as broken. That one mistake produced five separate false results across two tools and three agents before anyone noticed it was the same error each time. It nearly caused a correct line of CSS to be changed to satisfy a broken assertion.
And my own test server sent a header that disables caching — while the thing under test was a caching feature. Every offline test I ran before that was measuring a fiction.
One agent found seven assertions in its own suite that were green without being correct. One of them had a body that returned true unconditionally. A placeholder, in a passing suite, reporting success forever.
Another measured a memory leak and had the numbers to prove it. Then it ran a control group with the feature disabled, watched the control climb identically, realized it had been measuring startup warm-up, and retracted its own finding.
The rules
Six, each earned by getting it wrong:
Check the assertion before you fix the implementation. When something reports a failure, the test is a suspect too.
Measure the output, not the intermediate. Verify the pixel, not the buffer. The user cannot see your vertex attributes.
Prove the tests can fail. “83 assertions passing” and “83 assertions that cannot fail” produce identical output. Break each invariant on purpose and confirm exactly one assertion goes red. That is the only evidence a green suite means anything.
A skipped check is more dangerous than a failing one. Any branch that turns “cannot measure” into anything other than a failure is a place where confidence goes to die quietly.
Measure after the system settles, and keep a control. A measurement taken during warm-up, with no baseline, will find a trend that isn’t there.
Never let one bad idiom spread. When the same surprising result shows up in unrelated places, stop and ask whether it is the same mistake wearing different clothes.
What it costs
Two of the eight agents wrote no production code at all. The critics took over an hour each, and spent it clicking every control, throttling the processor, and blind-ranking screenshots.
They also found the blocker, and the shipping bug that made the simulation run at a third of real time while every instrument agreed everything was fine.
The implication
The lesson is not “write more tests.”
It is that the assertion is code too, and nobody reviews it. Implementation gets scrutinized. Test code gets written once, goes green, and is never read again. So it rots silently, and a green suite becomes a claim nobody has audited.
The agents were good. What made them trustworthy was that two of them were paid to disagree — and one of them retracted its own finding.