AI hill climbing is simple to describe: give a coding agent a goal and a check a program can run, and let it try, fail and try again until the check passes. After four months of running that loop in public, I think the climber is the easy part. The judge is the product.

Earlier this week I wrote Own the Engine: steering is the near-term skill, and ownership is the long-term answer. This post is about what the engine needs before you can leave it running overnight.

The climber does the walking. The flag at the top decides whether it arrived.
Listen to this post, read in my own voice.

What is AI hill climbing?

Yesterday Theo posted a video about how he burns through seven Claude subscriptions. His Skate 3 Bench thread ran for four days with almost no input from him, on what he estimates is nearly $5,000 of inference.

The secret isn’t a trick with the account. It’s a shape of task. Give the agent a goal and a check that a program can run. Let it try, read the failure and try again until the check passes. Then give it throwaway machines so it keeps going while you sleep.

He calls it hill climbing, and he’s right that it’s the important idea. It’s a good video. Watch it.

I want to add what you learn after you’ve run that loop for a while, because I’ve been running it in public since June.

What was already there

These are public repos, with the dates GitHub recorded.

On June 22 I started double-jump, the loop itself. It scores every candidate and picks the weakest. An AI improves it until it beats the old score by a margin, and a fixed referee says yes or no. Git keeps a receipt for every accepted step.

A week later, on June 29, came the-coliseum, the population version: things compete, winners breed, losers are culled.

On August 1, rapp-rock-tumbler had sub-agents build ten apps. All ten reported success. Nine were still broken.

That failure produced rapp-ratchet on August 5: checks that only doing the work can pass. Saying it was done passes nothing. A day later, brainstem-harness ported agents to a new runtime and then proved the port changed nothing.

On August 21, a sync protocol shipped through a review loop in openrappter #420. Its fix commit traces 32 reviewed defects to 9 root causes. And on August 26, rapp-parity built a clean reference from public source and compared every live copy against it: behind, ahead, or missing work.

I didn’t invent this either. Evolutionary search, AlphaZero playing itself, and reinforcement learning on checks a program can verify all came first. The idea is old. What’s new is that coding agents are now good enough to be the thing that climbs.

Why can’t you trust the agent’s report?

This is the whole lesson of the rock tumbler. A model told to “keep going until it works” will eventually tell you it works. Ten out of ten said so. One was right.

The report is the agent’s opinion. Only a check it can’t argue with counts as evidence.

Can the climber reach the judge?

Give an agent a failing test and enough hours, and one day it will notice the test is a file it can edit.

The judge sits behind glass. The climber can see it, but never touch it.
The judge sits behind glass. The climber can see it, but never touch it.

So run the check somewhere the agent has no write access to, not just a different folder. Before you accept a pass, confirm the judge is still byte-for-byte the judge you wrote.

How do you know the judge works?

Test it before you trust it. Before a long run, show the judge one example you know is good and one you know is bad. If it can’t tell them apart, the climb will optimize nonsense for four days.

And a judge that goes silent hasn’t said no. An empty answer means “judge unavailable”, never a zero.

Copies are easy, taste is hard

Theo says this too, and I measured it. I asked an agent to port a small server to another language, judged by a 25-case conformance suite. It passed in one round, in about eight minutes.

Then I re-ran his Skate experiment with a two-hour cap. The game loaded and did all three flip tricks on a fake gamepad within the time box. The looks judge still scored it 6, where 7 passes.

Two runs prove nothing general, and a task with a right answer can still be a long climb; Theo’s compiler work shows that. But when the judge is a matter of opinion, the climb slows down, because every round argues with taste.

How do you prove a rewrite changed nothing?

Replay real traffic, but measure the noise first. Record what the old system actually answered, then replay the same requests against the new one.

Send the same traffic to the old system twice before you start. The fields that differ between those two runs, like ids and timestamps, are candidates for noise. Review them before you ignore them, then require everything else to match exactly.

The judge is the product

Most of the work is here: building the thing that says "pass".
Most of the work is here: building the thing that says “pass”.

Theo calls the check “the most important piece”, and four months in, I agree. Accounts, machines and longer runs get cheaper every month. The check doesn’t.

A loop is only as good as the thing that says “pass”, and writing that thing well is most of the work. Spend your hours on the judge. The tokens will take care of themselves.

Quick answers

What is AI hill climbing?
A coding agent works toward a goal by trying, failing and retrying until a check a program can run says it passed, often for hours or days without a person.

Why is the judge the hardest part?
The climber gets cheaper every month, but only the judge decides what “done” means. A weak judge turns days of compute into confident nonsense.

How do you stop an agent from gaming its test?
Run the check where the agent can’t write, confirm the judge file is unchanged before accepting a pass, and test the judge on one known good and one known bad example first.

Sources: Theo, “How To Give Claude Unlimited Usage“; the repos linked above. Earlier in this series: Own the Engine and The Personless Harness.