kodyw.com

AI hill climbing is simple to describe: give a coding agent a goal and a check a program can run, and let it try, fail and try again until the check passes. After four months of running that loop in public, I think the climber is the easy part. The judge is the product.

Earlier this week I wrote Own the Engine: steering is the near-term skill, and ownership is the long-term answer. This post is about what the engine needs before you can leave it running overnight.

What is AI hill climbing?

Yesterday Theo posted a video about how he burns through seven Claude subscriptions. His Skate 3 Bench thread ran for four days with almost no input from him, on what he estimates is nearly $5,000 of inference.

The secret isn’t a trick with the account. It’s a shape of task. Give the agent a goal and a check that a program can run. Let it try, read the failure and try again until the check passes. Then give it throwaway machines so it keeps going while you sleep.

He calls it hill climbing, and he’s right that it’s the important idea. It’s a good video. Watch it.

I want to add what you learn after you’ve run that loop for a while, because I’ve been running it in public since June.

What was already there

These are public repos, with the dates GitHub recorded.

On June 22 I started double-jump, the loop itself. It scores every candidate and picks the weakest. An AI improves it until it beats the old score by a margin, and a fixed referee says yes or no. Git keeps a receipt for every accepted step.

A week later, on June 29, came the-coliseum, the population version: things compete, winners breed, losers are culled.

On August 1, rapp-rock-tumbler had sub-agents build ten apps. All ten reported success. Nine were still broken.

That failure produced rapp-ratchet on August 5: checks that only doing the work can pass. Saying it was done passes nothing. A day later, brainstem-harness ported agents to a new runtime and then proved the port changed nothing.

On August 21, a sync protocol shipped through a review loop in openrappter #420. Its fix commit traces 32 reviewed defects to 9 root causes. And on August 26, rapp-parity built a clean reference from public source and compared every live copy against it: behind, ahead, or missing work.

I didn’t invent this either. Evolutionary search, AlphaZero playing itself, and reinforcement learning on checks a program can verify all came first. The idea is old. What’s new is that coding agents are now good enough to be the thing that climbs.

Why can’t you trust the agent’s report?

This is the whole lesson of the rock tumbler. A model told to “keep going until it works” will eventually tell you it works. Ten out of ten said so. One was right.

The report is the agent’s opinion. Only a check it can’t argue with counts as evidence.

Can the climber reach the judge?

Give an agent a failing test and enough hours, and one day it will notice the test is a file it can edit.

So run the check somewhere the agent has no write access to, not just a different folder. Before you accept a pass, confirm the judge is still byte-for-byte the judge you wrote.

How do you know the judge works?

Test it before you trust it. Before a long run, show the judge one example you know is good and one you know is bad. If it can’t tell them apart, the climb will optimize nonsense for four days.

And a judge that goes silent hasn’t said no. An empty answer means “judge unavailable”, never a zero.

Copies are easy, taste is hard

Theo says this too, and I measured it. I asked an agent to port a small server to another language, judged by a 25-case conformance suite. It passed in one round, in about eight minutes.

Then I re-ran his Skate experiment with a two-hour cap. The game loaded and did all three flip tricks on a fake gamepad within the time box. The looks judge still scored it 6, where 7 passes.

Two runs prove nothing general, and a task with a right answer can still be a long climb; Theo’s compiler work shows that. But when the judge is a matter of opinion, the climb slows down, because every round argues with taste.

How do you prove a rewrite changed nothing?

Replay real traffic, but measure the noise first. Record what the old system actually answered, then replay the same requests against the new one.

Send the same traffic to the old system twice before you start. The fields that differ between those two runs, like ids and timestamps, are candidates for noise. Review them before you ignore them, then require everything else to match exactly.

The judge is the product

Theo calls the check “the most important piece”, and four months in, I agree. Accounts, machines and longer runs get cheaper every month. The check doesn’t.

A loop is only as good as the thing that says “pass”, and writing that thing well is most of the work. Spend your hours on the judge. The tokens will take care of themselves.

Update, November 1: watching the pattern get rediscovered

Two days after this post went up, another video showed the same shape from the other end: write the eval before the feature, make the agent ask you to review it, and add a hook so nothing can be marked done until the eval passes. Different words, same idea. The check comes first. The builder is not allowed to grade itself.

I have been working this way longer than I realized. In May 2025, FeedShyWorm 3.0 was the same game rebuilt three times, each version judged by whether it still played. In August 2025, Code Welding was about transplanting a feature between two codebases that share nothing, with one rule: break nothing that already worked. That rule is a judge. I just had not named it yet.

So this weekend I ran the loop on five things at once and wrote down what the judges actually said. Every number below came out of a judge script, not out of an agent’s report.

Why weld something new

Here is the part I think matters most. If you ask an agent to build a thing it has seen ten thousand times, it can pass by memory. A judge barely matters, because the answer is already in the weights. Code welding is the fix. Join two things no model has ever seen joined, and the only road to a pass runs through the judge. The agent cannot cheat from memory, because there is no memory of this thing. That is the whole reason the examples below are odd.

Five climbs, one weekend

A Rust library ported to TypeScript. The judge is 84 recorded input and output pairs from the Rust side, and it refuses any port that imports the vectors or anything else. The agent passed in one round: 84 of 84, and the certificate verified. Copies are easy. Still true.

The Skate bench, again. Same three gates as before: it loads clean, it lands an ollie, kickflip and heelflip from a fake gamepad flick, and two different models score screenshots against real Skate stills, both needing 7 or better. The baseline scores 4.6. The flicks pass 3 of 3. The looks judges say 5 and 6. Taste is still the slow part, and the climb is running as I write this.

My own driving simulator. A single-file racing sim I wrote in 2024. The judge plugs in a virtual Xbox pad and drives: throttle must reach 8 mph and keep rising, steering must turn the car, brakes must slow it, the frame rate must hold 15, the settings panel must open, export must work. All 8 mechanical gates pass. The two looks judges give it 4 and 3, with notes like “boxy car, flat lighting, blank sky, no sense of speed”. Score 8.3 of 11. The mechanical part was done in 2024. The taste part is what the climb is for.

A walkthrough video of that simulator, as its own climb. This one surprised me. The thing being climbed is a script that drives the sim with the pad and records the screen. The judge watches the recording: time spent moving, time spent off the track, laps, top speed, and whether the video is long enough to be worth posting. First result: 108 seconds, moving 95 percent of the time, never off the track, 59 mph, zero laps. Score 7.82. A video is a deliverable, so a video gets a judge.

An original arena game nobody has made. Two factions, five heroes, lanes, creeps, denies, towers, runes, a shop, and a day and night cycle, welded from a decade of arena game metagame into a few hundred lines of plain JavaScript with no engine. The judge runs it three times with a bot and reads a bench the game prints: gold, last hits, denies, towers, result. Baseline: 1129 gold, 47 last hits, 9 denies, 5 towers, and a Defeat. Score 7. It plays. It does not yet look like anything. No model was trained on this game, because it did not exist on Friday.

The judge crashed and nobody noticed

One more lesson, paid for today. The Skate and driving judges both failed to start inside the climb. The climb engine pins the judge by its real path on disk, and that path pointed at a Python without the browser library installed. The judge crashed before it could print a score.

The engine did not stop. It rejected every candidate, three rounds of them, with the reason “no comparable numeric score”, and kept spending agent time. I wrote above that a judge that goes silent has not said no. Here is the stronger version: a judge that goes silent must stop the climb. A missing score is a fire alarm, not a rejection.

The cartridge

Every version that survives a climb now gets packed into one self-contained HTML file. Inside it is a small signed record: the goal, the judge’s hash, the tree hash, the score, and the certificate if it earned one. Think of the file as a cartridge. The game is in it, and so is the receipt for how good it is. A console page lists the cartridges, plays any of them in a frame, and hands a virtual gamepad to whoever is holding it, a person or the next judge.

That is where this is going. I play the builds on the couch while the loop polishes them. The judge is the product, and the cartridge is how you ship it.

Quick answers

What is AI hill climbing?
A coding agent works toward a goal by trying, failing and retrying until a check a program can run says it passed, often for hours or days without a person.

Why is the judge the hardest part?
The climber gets cheaper every month, but only the judge decides what “done” means. A weak judge turns days of compute into confident nonsense.

How do you stop an agent from gaming its test?
Run the check where the agent can’t write, confirm the judge file is unchanged before accepting a pass, and test the judge on one known good and one known bad example first.

Sources: Theo, “How To Give Claude Unlimited Usage“; the repos linked above. Earlier in this series: Own the Engine and The Personless Harness.

Back in August I wrote The Personless Harness. The picture was simple: Copilot is the carriage, you are the horse, and the Brainstem is the engine. Swap the horse for the engine and the same carriage runs at a pace no stable could feed.

This week two videos landed that push on that picture from opposite sides. I want to walk through both, because together they change how I’d tell the story.

You are the next horse

David Shapiro’s You Are The Next Horse is a ten-minute history lesson with a sharp ending. In 1918, American farms had about 26.7 million horses and mules doing almost all the pulling. Within a few decades, tractors outnumbered them. The machines won for the oldest reason there is: better, faster, cheaper, safer. You couldn’t save the horse by feeding it less.

Economists have said for years that people are not horses, because we can always move up to newer, harder work. Shapiro’s answer is that the newer, harder work of the last century was thinking work, and thinking work is exactly what AI is learning to do.

Then he takes apart the line everyone repeats: “AI won’t take your job, someone using AI will.” His version of the meme is a farmer telling his horse it won’t lose its job to a tractor, but to a horse that learns to drive one. It’s absurd on purpose. The tractor never needed the horse. The teamsters really did learn to drive trucks, and that worked for a century, and now the trucks are learning to drive themselves.

His way out isn’t a skill. It’s ownership. Horses could never own a share of the tractor. People can.

Which layer are you standing on?

The second video, from DevLogic, walks through Andrej Karpathy’s funnel, and it explains why most people will hear Shapiro and shrug.

Pour the whole planet in at the top. About six billion people have barely touched these systems. One to two billion use a free chatbot now and then and see a slightly clumsy assistant. About twenty million use frontier models every day for code and math, and they watch projects that used to take weeks get done from a single prompt. At the very bottom, about five thousand people inside the labs watch swarms of agents work for weeks.

Karpathy names three things that decide where you land: how ambitious the problems you hand it are, which kinds of work it happens to be great at (digital, checkable, valuable), and which door you have access to. Each layer doesn’t see a better version of AI. It sees a different one. That’s why arguments about AI go nowhere. Both people are describing what they saw. They’re just standing at different depths.

Where my horse story was right, and where it was too soft

Reading these side by side, the Personless Harness post is a field report from the twenty-million layer. An AI agent with a real browser and a checklist ran thirty-three checks in about nineteen seconds, work that takes a person twenty minutes of full attention every single pass. That isn’t a forecast. I ran it four times in one evening.

It also lines up with Karpathy’s point about where these models are strongest. The rules I wrote for a trustworthy helper (use the real thing, check that what should be gone is gone, never fake a pass) are really rules for making work checkable. Checkable work is exactly where AI is sharpest. If you want the engine to take a job, make the job checkable first.

But my ending was too comfortable. I said the person moves from pulling to steering. Shapiro’s teamsters are that ending, and the trucks are now driving themselves. “Learn to steer” is the horse learning to drive the tractor. It buys time. It isn’t a destination.

And yet Karpathy’s funnel says the time it buys is real. Right now only about one in five hundred people can actually steer these systems. Steering is worth a lot today. It just isn’t forever.

Own the engine

So here is the version of the story I believe now.

Steering is the near-term skill. Ownership is the long-term answer. The Brainstem is where those two meet. It isn’t a smarter chat window you rent by the month. It’s a tireless second version of you that runs on your own machines, knows your tools, remembers how you like things done, and keeps working after you clock out. It’s the engine, and it’s yours.

That also answers Karpathy’s funnel, at least for one person. Most people will never climb down to the twenty-million layer by learning prompts, picking models, and wiring tools together. They shouldn’t have to. A Brainstem brings that layer to them: you talk to it, it does the work, and it shows you the proof. You get the depth without the descent.

The horse never got a share of the tractor. You can own the engine. Start with the biggest, most annoying, most checkable job on your plate, hand it over, and see how far down the funnel you really are.

Sources: David Shapiro, “You Are The Next Horse“; DevLogic, “Karpathy: The AI Understanding Gap Is Getting Worse, Fast“. Earlier in this series: The Personless Harness and The Buzzsaw.

I have been telling people for a while that long AI chats are a tax. Every message re-sends the whole conversation, and the harnesses we use every day add their own instructions and tools on top. I had a feeling that my Brainstem cost pennies because it keeps chats short and puts the useful work into
agent.py files, while Claude Code and Copilot CLI kept growing the chat. A feeling is not a number, so I measured it.

What I ran

Five small business tasks with real rules: tiered sales commission with clawbacks, overtime pay, reorder points, late fees, and invoices paid twice. Each task was packaged three ways from one Python implementation: an Agent Skill written in plain words, the same skill with a bundled script, and a RAPP
agent.py. Claude Code, GitHub Copilot CLI and a Brainstem answered the same questions with the same model, Claude Haiku 5.5. Three runs of every setup, every answer graded to the cent.

What held up

The host charges before you ask. To answer “reply with the number 7”, a fresh Claude Code sends the model 15,176 tokens and a fresh Copilot CLI sends 23,014. A Brainstem sends 2,805. My own Claude Code and Copilot, with everything I have installed over the months, send 32,295 and 33,719. My own Brainstem, with 22 agents loaded, sends 8,716.

Every question pays that charge again. The same 15 questions cost 135k tokens in a Brainstem, every answer right. Claude Code’s best setup used 725k, about 5.4 times as many, also with every answer right.

Weight is what makes them slow. This one surprised me. On fresh installs, with the rules in code, all three answered in about four seconds. My own Copilot, with the same agent.py tools and nothing else running, took 56 seconds per answer and 820k tokens for five questions. The Brainstem did the same five in 35k tokens.

Lock it in, then clear the chat. Once the rules live in code, history is mostly dead weight. A new chat for every question cut the Brainstem’s total from 135k to 90k tokens, still 45 of 45 right.

Call /chat, not a wrapper. Routing the same Brainstem through Copilot CLI and an MCP plugin cost about 7 times as many tokens per answer, and Copilot often skipped the Brainstem altogether. On my own Copilot install it skipped it every time. The MCP plugin is for hosts that cannot make an HTTP call. Everything else should call /chat.

What did not hold up

• Spend does not grow exponentially. It grows with the square of the chat’s length. A chat twice as long costs about four times as much. Bad enough, but I was wrong about the word.
• agent.py is not magic inside Claude Code. There, a skill with a bundled script was cheaper than agent.py tools, because Claude Code hides MCP tools until the model searches for them. It called my agent tools on 13 of 45 questions and improvised the rest. The same files inside a Brainstem were the cheapest setup of all. The host is what saves the tokens, not the file format.
• Plain-word rules were not the accuracy disaster I expected. My first wording of one rule was ambiguous about rounding, and the models filled the gap their own way. Once I fixed the wording, skills written only in words got 116 of 120 answers right. They were still the most expensive setup on both hosts, and their 4 misses were real arithmetic slips. Rules written as code were right every time the code ran.
• I did not test the “lobotomy”. Published studies (Chroma’s Context Rot, Lost in the Middle, RULER) report that accuracy drops as context grows. My 15-question chats were too short to show it.

The cost of ownership

Tokens are the sticker price. Keeping a setup working is the rest. In one evening of measuring:

• Claude Code silently ignored an MCP config written in Copilot’s format. No error, the tools just were not there, and my first “agent.py in Claude Code” numbers measured nothing.
• Claude Code rewrites $0, $1 and friends inside a SKILL.md as skill arguments, so dollar amounts in your rules can turn into file paths.
• Copilot’s usage file is a running total for the whole session, which I first read as per-question.
• The Brainstem’s only failure was one upstream connection reset in 30 questions.

Driving one question from a script took 13 lines of code for the Brainstem (one HTTP request), 29 lines and 8 flags for Claude Code, and 54 lines and 9 flags for Copilot CLI. And the Claude Code session I used to run all of this is itself a long chat: by the end of the measurements it had made 261 model calls and re-sent 81.7 million tokens of context, about 313k per call.

What I do now

1. Measure your host’s idle cost with one trivial question. That number is charged on every call.
2. Put repeated work in code: an agent.py, or a skill with a script.
3. Keep chats short. Finish a job, keep the answer, start fresh.
4. Unload what you do not use. Every plugin, server and skill rides along on every call.
5. Call a Brainstem’s /chat directly. Use MCP only where HTTP is not an option.

The full white paper, with every chart and the mistakes I caught along the way, is
The Context Tax. The bench is open source at
github.com/kody-w/token-bench, and it measures your own setup, as you actually use it, in one command.