Tag: testing

The Personless Harness: Brainstem + Copilot

On August 4, 2026

In 1895 you didn’t buy a car. You bought a horseless carriage — the new machine named for the animal it removed. The name is the whole lesson: the carriage didn’t change. The wheels, the seats, the cab — all still there. What changed was what pulled it. The horse went out, the engine went in, and the same carriage could suddenly run at a pace no stable could feed.

Here’s the same swap, happening right now, in software.

Copilot is the carriage. It’s the vehicle everyone already has — the chat window, the buttons, the place work actually happens. A genuinely good carriage.

You are the horse. Today, almost everything an AI assistant does, it does because a person is pulling: typing the prompt, clicking the button, checking the output, deciding what happens next, then pulling again. That works — beautifully, at the pace of one tired human — and it stops the moment that human needs to sleep, eat, or go to a meeting.

The Brainstem is the engine. Think of it as a second, tireless version of you: it knows your tools, remembers how you like things done, and can keep working after you’ve clocked out. It isn’t a smarter chat window — it’s the thing that can sit where you sit and keep going. Same carriage, different thing doing the pulling.

What the swap looks like in practice

This isn’t really a story about software testing. It’s a story about a new way of working — testing just happened to be the first job the engine took from me, because testing is the purest form of “a human clicks through an app after every change to see what broke.”

I had just finished a feature I believed was done. Instead of opening the app myself, clicking the toggle, and watching what happened, I handed the job to an AI agent with a real browser and a checklist, and went to work on something else.

It came back with five problems. One was serious: the page was loading with zero working code behind it, so the whole feature was just a picture of itself. Another: a switch that could be turned on but never back off. A third: a checkbox that had quietly grown too wide and shoved its own label onto three lines. None of that showed up in my usual automated tests, because it all lived in the gap between “the code looks right” and “the thing actually works when a hand touches it.” Until that day, finding it had always meant me sitting there clicking around.

I fixed the five problems, sent the AI agent back in, and it came back clean: thirty-three checks, all passing, no human involved. My status update changed from “try it and tell me what breaks” to “it’s done, and here’s the proof.”

What makes an AI helper trustworthy at this job

A few things separate a helper you can actually trust from one that just looks impressive in a demo:

It uses the real thing. Real clicks, real pop-ups, real downloads — not a shortcut that skips the part a real person would experience.
It expects the annoying stuff. An unexpected confirmation pop-up is a failure to catch, not something to shrug off. And layout gets measured precisely, not eyeballed.
It checks for things that should be missing, too. A feature that’s supposed to stay hidden until turned on, a panel that’s supposed to disappear — humans almost never double-check that something properly vanished. An AI helper can, every time.
It never fakes success. If it couldn’t actually run the check, it says so loudly. A “looks fine” that never really ran is worse than an honest “this failed.”

How much faster, really?

To put a number on it: that 33-check run, done by hand by someone who knows the app well, takes roughly twenty minutes of full attention — and you have to redo the whole pass after every fix. The AI agent finished the same 33 checks in about 19 seconds. That’s roughly sixty times faster on the stopwatch. But the bigger difference is what it costs you: twenty minutes of a person’s full focus, versus zero. I ran it four times in one evening without noticing the cost at all.

The carriage stays; the horse retires

Notice what this swap does not ask of you. You don’t throw out the tools you already use. Copilot, the app, everything you’ve already built — all of it stays. That’s why the horseless carriage caught on so fast: it didn’t ask the world to reinvent the wheel, just to stop feeding the horse.

And notice where the person goes. The engine didn’t remove people from travel — it moved them from pulling to steering. This doesn’t remove you from the work; it moves you to the seat where you decide where things go and read the results. You stop being the horse. You were never at your best there anyway.

Curious how far this can go? The Buzzsaw pushes the same idea further — instead of one AI helper checking the work, I used eight of them, arguing with each other, to catch things a single reviewer never would.

For a while, this will feel like a strange new way to work. Then, like the horseless carriage before it, it’ll just be how work is done — and the idea that a person once had to manually click through every check will sound as old-fashioned as keeping a horse for your commute.

The Buzzsaw

By Kody Wildfeuer

On August 3, 2026

In Technology

I gave eight AI agents one piece of software and told them to keep working on it, together, until it was actually good.

They found 21 real problems. They also flagged six things that turned out to be non-issues — and three of those false alarms were my own instructions leading them astray.

That ratio is the whole lesson: the checks were wrong almost as often as the code was, and the only reason anyone knows that is because somebody double-checked the checks.

This is the same idea behind The Personless Harness, pushed further — instead of one AI helper checking my work, I used eight of them.

The idea

The approach is built around one assumption: the person who built something can’t see their own blind spots. So you bring in a separate, harsher reviewer whose only job is to stay unimpressed, and you let a clear pass/fail test — not a feeling — decide when the work is actually done.

Five moves make this work:

Name a real target. “Make it good” means nothing to a machine. “Compare it directly against the best example of this you know of, and say which one wins” is something you can actually act on.
Split the work into non-overlapping pieces and hand each piece to a different agent, so several can work at once without stepping on each other.
Keep looping on each piece instead of doing one pass and calling it finished.
Bring in a tough, read-only reviewer. Its only job is to stay unimpressed. It’s never allowed to fix anything itself.
Write the finish line before you start. The work is done when it passes that specific, predefined test — not when it merely looks good.

Most people already do some version of the first three. The real value shows up in the last two.

The best find wasn’t even a bug in the code

One agent added a lighting effect that makes 3D shapes look more solid and realistic — shading in corners and crevices, the kind of detail that makes a scene feel real instead of flat. It built the effect correctly. It even checked its own work afterward and reported a win, with evidence, showing the underlying data was correct.

The tough outside reviewer, looking only at what actually rendered on screen, scored it 2 out of 10. The corners looked completely flat.

The real cause was upstream: the scene’s lighting was so bright that the subtle shading effect got washed out and never visibly reached the screen. Every step of the work was technically correct — the data was right there — it just never showed up where a person could actually see it.

That’s not a coding mistake. It’s a case of checking the wrong thing: verifying the ingredients instead of tasting the dish. No amount of re-reading the code would have caught it.

The bug nobody could see just by looking

A “click to play” overlay was sitting on top of the toolbar underneath it. Every button in that toolbar was actually unreachable — clicking what looked like the settings icon just clicked through to the overlay instead, and a few panels had no other way to reach them at all.

The buttons looked completely normal the whole time. You couldn’t see the problem in a screenshot, and you couldn’t see it by reading the code — everything about it looked correctly built. The only way to find it was to actually click on the button, for real, the way a person would.

It had already shipped.

Six times, the “test” was the actual problem

This is the part I didn’t expect, and it’s the whole reason I keep doing this.

One check quietly failed to run at all — it tried to read a value that didn’t exist, got nothing back, and the code politely treated that as “not applicable” instead of “something’s wrong.” The single most important check in the whole test suite never actually ran, not once, and in the log it looked identical to a real passing check.

Another check used a common way of testing “is this visible on screen,” which happens to always give a false negative for one particular kind of element — meaning genuinely correct, working code kept reading as broken. That one mix-up produced five different false alarms across two tools and three agents before anyone realized it was the exact same mistake showing up in different places.

And in a separate case, my own test setup had caching accidentally turned off — while the very feature being tested was a caching feature. Every test I’d run before I caught that was measuring something that wasn’t really there.

One agent went back through its own tests and found seven that were passing without actually checking anything real — including one that was hard-coded to always say “yes,” forever, regardless of what happened. Another agent measured what looked like a memory leak, complete with solid-looking numbers — then ran a proper side-by-side comparison, realized it was just normal startup behavior, and honestly walked back its own finding.

The rules that came out of this

Six rules, each one earned the hard way:

When something fails, suspect the test as much as the code.

Check what a person would actually see, not just the data behind it. Nobody using the software can see your internal values — only what shows up on screen.

Prove your tests can actually fail. A pile of checks that all pass looks identical to a pile of checks that are incapable of ever failing. Deliberately break something on purpose and confirm exactly one check turns red — that’s the only real evidence a passing test suite means anything.

A silently skipped check is more dangerous than an honest failure. Any shortcut that turns “I couldn’t check this” into a quiet pass is a place where false confidence creeps in.

Measure after things settle down, and always compare against a baseline. A measurement taken too early, with nothing to compare it to, will “find” a trend that was never really there.

Don’t let one wrong idea spread quietly. If the same surprising result shows up in several unrelated places, stop and ask whether it’s actually the same underlying mistake, just dressed differently each time.

What it costs

Two of the eight agents wrote no new code at all — their entire job was reviewing. The tough reviewers spent over an hour each, clicking through every control and comparing results side by side.

They also caught the invisible-button bug, and a separate shipping bug that made everything run at a third of the intended speed while every internal measurement insisted everything was fine.

The real lesson

It isn’t “write more tests.”

It’s that the test itself is something someone wrote — and once it goes green, almost nobody ever looks at it again. Meanwhile the actual product code gets read and re-read constantly. So the tests quietly rot, and “all green” quietly becomes a claim nobody has actually double-checked.

The AI agents involved were good. What made the whole process trustworthy was that a couple of them were specifically there to disagree — and one of them was honest enough to take back its own claim once it turned out to be wrong.

It’s the same problem I ran into building The Ark Pattern: the thing that’s supposed to catch drift can quietly drift itself.