kodyw.com

Kody Wildfeuer's blog

kodyw.com
A young tree grows stronger through cracked stone as wind and rain push on it, with loop arrows circling it.

Antifragile by Construction

A four-part series on one agent pattern.

  1. The Caller Knows Nothing
  2. The Agent Writes the Next Prompt
  3. “Anthropic Already Does This.” Let’s Check.
  4. Antifragile by Construction (you are here)

The first three parts of this series described a pattern: a caller that knows nothing, results written for the next step, a transcript you can replay, and a fair check against what Anthropic ships. This last part is about why it matters over time: the mistakes it removes, why it gets better under stress instead of breaking, and why it scales.

The antipatterns it removes

Every one of these is a mistake I’ve made. Each row names the pain first, so you can spot it in your own system.

Antipattern What goes wrong Rule that fixes it
The smart host: glue code that knows each tool’s insides Every new tool needs host changes; the coupling only grows 1. The caller knows nothing
The vague description The tool is called wrong or comes back empty (in my log: count: 0, twice) 2. The description is the interface
The cage: every field turned into a fixed list The right shape, no surprises, no emergence 3. Full liberty
The overloaded parameter: one field asked to do three jobs The model blends the jobs or refuses 2. One parameter, one job
The lost result: tool output kept out of the transcript The model misremembers what it did 4. Outputs go back into the transcript
The leaky transcript: insides, prompts and raw dumps in context Noise, cost and wrong turns 4. Outputs only
The silent return: raw data with no shape and no next step Every caller re-formats it, differently each time 5. The agent writes the next prompt
The post-processor: regex or word lists fixing the model’s wording afterwards Brittle, language-specific, a fight that never ends 2 and 5. Fix it in the description and the agent’s own checks
The full rerun: starting over to fix one step Slow, expensive, and steps that were fine change too 6. Replay one turn with more data

Antifragile, by construction

Nassim Taleb’s word for things that get better from stress, not just survive it, is antifragile. I think this pattern earns the word, and here’s the mechanism rather than the slogan.

In my paper on living holograms, the thing an AI works on is a living object: it changes only by passing through a lens (an agent like the ones above), each pass is recorded so it can be replayed, and what one pass produces is carried into the next. Put that together with the rules here, and every kind of stress turns into an input:

  • A failed check goes back into the transcript as one plain sentence, and the next turn corrects for it.
  • A surprise in the surroundings (a new file, an odd request, a missing field) is just more transcript, and a caller with full liberty can use it.
  • A missing capability becomes a new agent that loads on the next turn.
  • A mistake found late is a reason to replay that one turn with what you know now, not a reason to start over.
  • A better model reads the same descriptions and does more with them, without a line of host code changing.

No one programs the adaptation. It happens implicitly, turn by turn, because the object only ever moves through steps that can see what just happened.

The honest caveat: the same loop that carries a good result forward can carry a bad one. Antifragility here isn’t magic. It comes from three guards working together: checks that cite the material, results that come back into view, and replay. Remove any one and the pattern is merely flexible.

It scales: the transcript is the only state

There’s a second reason this holds up, and it’s about size rather than stress. The Brainstem’s core is stateless: it keeps no conversation of its own between calls. Everything a turn needs arrives in the transcript, and everything it produces leaves in the transcript. So the transcript file is the only thing a caller ever has to pass.

That means you can stand up any number of Brainstems side by side, one per use case, each with its own agents loaded, and hand the same transcript to all of them, or a different one to each. None of them needs a shared session, a sticky server or a database to stay in sync. Add machines and you add capacity. It’s the same reason the web scaled: HTTP servers that keep no session state can be multiplied freely, because the request carries what matters.

And because that transcript can be written as RAPP/1 frames, content-addressed and hash-linked, it isn’t just portable, it’s verifiable. Any Brainstem, anywhere, can check that the history it was handed is the history that happened before it adds a turn. Statelessness gives you scale; RAPP/1 gives you trust in what you scaled.

The honest edge: agents can still keep their own stores (the memory agent writes to one). The core doesn’t, and the more of that state you carry as transcript, the more freely the whole thing scales.

Distributed AI, and the cost of the other path

Stateless scale is the doorway to distributed AI: many minds, on many machines, owned by different people, working on the same problem. With this pattern, distribution needs nothing new. A transcript is a file. Pass it to a Brainstem on a laptop, one in a browser, one in the cloud, one at a partner’s office. Each adds its turn through its own agents and hands the transcript on. RAPP/1 frames let each one verify what it received. There is no central service to coordinate, because the coordination is the transcript.

Now look at the path the comparison in part three describes. Each property lives in a different mechanism with its own host and its own state: sessions saved by the SDK, subagent transcripts kept apart, checkpoints local to Claude Code, Managed Agents that can’t take a new tool mid-session. Each was a sensible answer to a real need. But distribution is a new need, and on that path a new need tends to arrive as a new feature: a way to share sessions, sync memory, hand work between hosts. Every one adds to the list a builder has to learn, and every one has to fit with all the others.

This is my reading of the trajectory, not something Anthropic has said, and they may well take another road. But history is consistent here. CORBA kept adding services to make distributed objects work and got heavier with each one; the web won with a uniform interface and stateless requests that carried what mattered. Features add up. A shape that doesn’t need them is what scales.

Go deeper: two free companion books, written side by side with the same chapters so you can compare them page for page: The Brainstem Agent Programming Language and The Claude Agent Programming Language. Both are runnable, and the Claude one cites Anthropic’s docs throughout.

A person with a magnifying glass compares a workbench of many separate parts with one small sealed box that holds them all.

“Anthropic Already Does This.” Let’s Check.

A four-part series on one agent pattern.

  1. The Caller Knows Nothing
  2. The Agent Writes the Next Prompt
  3. “Anthropic Already Does This.” Let’s Check. (you are here)
  4. Antifragile by Construction

Whenever I explain the pattern from The Caller Knows Nothing and The Agent Writes the Next Prompt, someone says Anthropic already does this, or that it’s just SKILL.md. Both are fair challenges. So instead of arguing, I checked, against Anthropic’s own documentation, with citations on every row.

“Isn’t this just SKILL.md?”

People say so, and I see why. In both, the caller sees only a name and a description and decides when to reach for it. Both are a file you drop in, and both work with any model. But the similarity stops at the front door.

SKILL.md agent.py
When invoked Its instructions load into the caller’s context. The caller reads the insides and does the work. The insides never enter the context. perform() does the work; only the output comes back.
Input No typed parameters. A JSON Schema call that can be checked.
Who executes The model, by following prose. Code, the same way every time.
The transcript Instructions stay loaded and shape every later turn. Only the relevant result goes in.
What comes next Steered by instructions the caller must remember. Steered by the output itself.
Replaying a step The outcome depends on the model following the instructions again. The same code runs again with more data, so you compare results.
Host needs A filesystem, usually a sandbox. Just a tool-calling loop, even in a browser.

A skill teaches the caller how to do something. An agent does it for the caller. A skill makes the caller know more. This pattern keeps the caller knowing nothing. They’re opposites at the one point this post is about.

They’re not rivals, though. Skills are great for know-how a person wants to read and edit: long procedures, style guides, judgment. Agents win where you need the same result every time, a typed input, a clean transcript and replay. A skill can tell the caller when to reach for agents, and once a skill’s procedure is stable, it can become an agent.

“Anthropic already does this.” Let’s check.

This is the debate I keep having, so I settled it the boring way: I had every claim checked against Anthropic’s own documentation, with several independent reviewers trying to refute each one and a claim kept only when a majority couldn’t. Here’s what holds up, as of November 5, 2026. Every row links the page it rests on.

Property of the pattern Does Anthropic have it? Where
The caller sees only a name, a description and a schema; the code is hidden Yes. This is ordinary tool use. Your app runs the tool and returns a result. Tool use overview, Managed Agents tools, Agent SDK custom tools, MCP connector
Only the relevant output enters the transcript Yes, in several ways. Subagents return a summary from a separate context; skill scripts are executed, not loaded; programmatic tool calling shrinks large results; hooks cost no context unless they return output; context editing clears old tool results. Subagents, Skills, Advanced tool use, Features overview, Context editing
Selection by description Yes. Skills and subagents keep their name and description in context and load the body when used. Features overview
Typed parameters on that description-driven unit Not for skills or subagents. A subagent gets a free-text prompt; a skill’s body is instructions. Typed schemas live on tools, a separate mechanism. Subagents
The model creates a capability mid-conversation and uses it next turn Partly. Claude Code picks up skills and subagents written to disk, including ones Claude writes itself, without a restart. On the API, tool search only reveals tools the client already sent, and Managed Agents can’t add tools to a running session. Skills, Tool search, Managed Agents custom tools
The return value is deliberately shaped to steer the next step Advice, not a pattern. Managed Agents suggests returning “only the fields Claude needs to determine its next step.” Nothing defines a result that carries a shaped piece, an instruction for the next step and a structured payload. Managed Agents tools, Memory tool
Replay a fixed transcript from any turn with more data No. Claude Code checkpoints each prompt and /rewind restores code or conversation; the Agent SDK can fork a saved session. Both ask the model again. Neither re-runs a fixed step with new data. Checkpointing, SDK sessions
One portable file that is all of the above No. Each property above lives in a different mechanism with its own host: Claude Code, the API, the SDK, Managed Agents, or Cowork (which runs on Claude Code’s architecture in the cloud). Cowork, Skills in Claude

So the honest verdict: Anthropic built excellent parts, and if someone says “tools already hide their code,” they’re right. The claim here isn’t that any single property is new. It’s that one file carries all of them at once, on purpose: typed and described going in, hidden code inside, a result written for the next step coming out, creatable mid-conversation, and replayable as a fixed step. That combination is the pattern, and I couldn’t find it anywhere in Anthropic’s documentation.

To be fair about the limits of this check: docs change fast, and “I couldn’t find it” rests on absence. Shaping a tool’s result to steer the next step is a convention anyone can adopt in any tool, today. That’s rather the point: it’s not a feature to wait for, it’s a way of writing.

Depth without the descent

In Own the Engine I wrote about Karpathy’s funnel: most people will never climb down to the layer where AI really does the work by learning prompts, picking models and wiring tools together. They shouldn’t have to.

The comparison above is that funnel in miniature. To combine what Anthropic offers, you learn roughly 25 separate concepts across four hosts: tool use, server tools, tool search, programmatic tool calling, the memory tool, context editing, caching, the MCP connector, skills, subagents, hooks, plugins, checkpoints, SDK sessions, Managed Agents, Cowork and more. Each one is good. Together they are a descent.

The pattern asks you to learn one thing: a file with a name, a description, a perform() and a result written for whoever comes next. It doesn’t teach you the 25 concepts; it carries their properties for you. That’s why I hold it to a bar of one hour to learn, and why the file runs on an engine you own rather than one you rent. Depth, without the descent.

Go deeper: two free companion books, written side by side with the same chapters so you can compare them page for page: The Brainstem Agent Programming Language and The Claude Agent Programming Language. Both are runnable, and the Claude one cites Anthropic’s docs throughout.

The Agent Writes the Next Prompt

A four-part series on one agent pattern.

  1. The Caller Knows Nothing
  2. The Agent Writes the Next Prompt (you are here)
  3. “Anthropic Already Does This.” Let’s Check.
  4. Antifragile by Construction

In The Caller Knows Nothing I argued that the model calling an agent should know nothing about its insides. This part is about the other direction: what comes back out, and why that one design choice lets agents coordinate without anyone wiring them together. It also unlocks something no rerun can: going back in time with more than you knew.

The agent writes the next prompt

A tool’s return value is usually treated as data, and the next move is left to a framework or the model. These agents return something written for whoever reads it next. Here is what a small Hacker News agent hands back:

{
  "status": "success",
  "stories": [ ...full records... ],
  "summary": "Top Hacker News stories:\n\n1. **[Title](url)** ...
              \n\nWhen presenting these to the user, render the titles as
              clickable markdown links exactly as written above.",
  "data_slush": { "count": 10, "top_url": "https://..." }
}

Three things travel in that one result:

  • A finished piece (summary), already shaped the way it should be shown. The code comment says it plainly: the model tends to copy this format word for word, so the agent writes it right once.
  • An instruction for the next step, inside the result itself: render these as links. The agent doesn’t know who reads it next, and it doesn’t need to.
  • A structured payload (data_slush) for the next step to use as-is, not for the model to reword.

The agent that learns new agents does the same. Its result says “created TranscriptFollowUp; it loads on the next request; re-run with submit to share it.” The output carries the data, the instruction and, here, a new capability.

That’s where the orchestration lives. Not in a graph, a router or a planner, but in the return values. Each agent steers what happens next by what it leaves in the transcript, without knowing who comes next. That’s why one file can be dropped in and immediately work with everything else.

5. The agent writes the next prompt. Its result is shaped for the reader, carries the next instruction, and passes structured data along, so the steps coordinate through what they leave behind.

Time travel

A conversation scroll rewinds to an earlier turn, takes in a new card of information, and grows a second version beside the first.

There’s one more thing this buys you, and I missed it until it was pointed out. The transcript is static: a plain record of what went in and what came out. The caller is a static loop with the agent loaded beside it. So any turn can be replayed.

That means you can go back. Rewind to the turn where the draft was written, add what you only learned later (the file that arrived Tuesday, the answer a downstream step produced, the agent that was learned three turns on), and run that turn again. The past step gets a second try with data it never had. You don’t rerun the whole pipeline, just the one turn, and you can keep both versions side by side and pick the better one.

6. You can time travel. A static transcript through a static caller can be replayed from any turn, with more than it knew the first time.

Go deeper: two free companion books, written side by side with the same chapters so you can compare them page for page: The Brainstem Agent Programming Language and The Claude Agent Programming Language. Both are runnable, and the Claude one cites Anthropic’s docs throughout.

A person talks at a desk while a row of closed, labeled boxes sits beside them; one box sends a single line into a scroll of conversation.

The Caller Knows Nothing

The simplest idea in agent design, and why almost everyone, including the AI that helps me build, keeps missing it.

In short: an AI agent works best when the model calling it knows nothing about its insides. It sees a name, a description and a few typed parameters, nothing more, and decides when to reach for it, the way you just know you can remember something. The agent’s result comes back into the conversation, and only its result. This is part one of four.

When you need to remember something, you don’t open a database. You don’t look up a schema, pick a table, or decide which index to use. You just know you can remember, and when something in the conversation sounds important, you do. Later, the memory is simply there.

That’s the whole idea. It’s the pattern behind every agent in a Brainstem, and I’ve watched people and models talk right past it for months. This afternoon I watched it happen again, so I’m writing it down.

A four-part series on one agent pattern.

  1. The Caller Knows Nothing (you are here)
  2. The Agent Writes the Next Prompt
  3. “Anthropic Already Does This.” Let’s Check.
  4. Antifragile by Construction

Two small files

A Brainstem (see The Personless Harness) ships with two memory agents. The first saves:

class ManageMemoryAgent(BasicAgent):
    metadata = {
      "name": "ManageMemory",
      "description": "Saves information to persistent memory for future
        conversations. You MUST call this tool whenever the user asks you to
        remember something ... or the information will be lost.",
      "parameters": { "memory_type": ..., "content": ..., "importance": ... }
    }
    def perform(self, **kwargs):
        # clamp, default, store; none of this is the caller's business

The second recalls. It doesn’t even wait to be called: before every turn it puts what it knows straight into the context.

class ContextMemoryAgent(BasicAgent):
    def system_context(self):
        return "<memory>...what you already know...</memory>"

Look at what the model, the caller, knows about these. It knows a name, a sentence about what the thing is for, and a few parameters. That’s all. It doesn’t know there’s a SQLite file. It doesn’t know about clamping, deduplication, or how a timestamp is formatted. It doesn’t need to. When the user says “remember my favorite color is blue,” the model reaches for the thing in its context that sounds like remembering. When the next turn starts, the memory is already in front of it.

We just know we have something we can invoke in our current context that lets us remember, if that sounds important in the current conversation. That’s it.

The rules

1. The caller knows nothing about the inside. Name, description, parameters. Storage, checks, shaping and formats stay hidden in perform().

2. The description is the whole interface. It says what’s needed and when it matters. Every bit of quality you want lives there, not in glue code around it.

3. The caller has full liberty. The description invites; it never confines. The caller may put anything it judges useful into a parameter. The agent takes whatever comes and handles it, and that freedom is where the surprising, good behavior comes from.

How smart people get it wrong

Today I asked my AI collaborator to make our pipeline’s lenses (small agents that each do one language job) “dynamic, with the caller armed to give the right information in the right shape.” It came back with a careful plan. Its centerpiece was a shared caller: host code that would read each lens’s contract and fill its parameters from the case, turning free-text fields into fixed lists of choices.

That’s thoughtful, rigorous, and exactly backwards. It took a few corrections to get it there:

  • First wrong turn: cages. Turning every field into a fixed list of choices does give you the “right shape,” and it kills the caller’s ability to add what you didn’t think to ask for. Constrain only where something downstream truly needs exact values.
  • Second wrong turn: a smart host. A caller that knows each agent’s insides is coupling. Every new agent now needs host code. That’s the machinery we were trying to delete.
  • The fix: nothing in between. The transcript is the input. The agents are loaded beside it. The model invokes what sounds right. Results come back into the transcript. Done.

It’s the same reason Unix pipes worked. The tools knew nothing about each other, and they composed through a plain, shared stream. Here the stream is the conversation.

The evidence is in a chat log

I asked a small model to “just make up” an agent that drafts content from a chat transcript. Its first attempt, built from a thin description, returned count: 0 twice: empty. The second, built with a precise description (who reads the draft, what goes in it, nothing sent automatically), came back as a real tool. Same model, same plumbing. The only difference was the description.

The same log shows the other half. Several times the model said it had saved or recalled something when it hadn’t, or said it hadn’t when it had. The truth was in the tool log the whole time, outside its view. When a result doesn’t come back into the conversation, the caller has to guess. So the loop has to be closed: whatever an agent does goes back into the context, the way memory does.

The transcript carries only what came out

Here is the part that makes it compound. The transcript doesn’t hold the agent’s insides, its prompts, or its working state. It holds only what came out: the relevant result of the agent that was just called.

When ContextMemory runs, what lands in the conversation is the handful of memories that matter, not the database. When LearnNew runs, what lands is “created TranscriptFollowUp, here’s what it does,” and on the next turn that new agent is hot-loaded, sitting in the context beside all the others, ready to be invoked. The conversation didn’t just get information. It got a new capability.

So every step downstream starts from a better place. The next call sees the previous result and can build on it: use the recalled preference, call the agent that didn’t exist a minute ago, refine the draft that came back. Nobody wired those steps together. Each one just leaves its output in the shared stream, and the next one picks up whatever helps.

4. The transcript carries only outputs. Each agent leaves just its relevant result. Nothing from inside it leaks, and nothing that matters is lost. Downstream steps use it to their advantage, and that’s how the whole thing gets better as it goes.

Where this goes next

That’s the core. The rest of the series builds on it: what the agent sends back and how to time travel, a fair, cited check against SKILL.md and Anthropic’s whole ecosystem, and why the pattern is antifragile.

Why it matters

If the caller has to know the insides, every agent is a project. If the caller knows nothing, an agent is a file you drop in. Anyone can write one, any model can use one, and agents you never planned to combine start working together, because each one just says plainly what it’s for.

You don’t build intelligence into the plumbing. You give the model things it can reach for, describe them well, and get out of the way.

It’s that simple. That’s why it’s so easy to miss.

Go deeper: two free companion books, written side by side with the same chapters so you can compare them page for page: The Brainstem Agent Programming Language and The Claude Agent Programming Language. Both are runnable, and the Claude one cites Anthropic’s docs throughout.

A small robot climbs a grassy hill at dusk, its footprints glowing, toward a referee holding a flag at the top.

The Judge Is the Product

AI hill climbing is simple to describe: give a coding agent a goal and a check a program can run, and let it try, fail and try again until the check passes. After four months of running that loop in public, I think the climber is the easy part. The judge is the product.

Earlier this week I wrote Own the Engine: steering is the near-term skill, and ownership is the long-term answer. This post is about what the engine needs before you can leave it running overnight.

The climber does the walking. The flag at the top decides whether it arrived.
Listen to this post, read in my own voice.

What is AI hill climbing?

Yesterday Theo posted a video about how he burns through seven Claude subscriptions. His Skate 3 Bench thread ran for four days with almost no input from him, on what he estimates is nearly $5,000 of inference.

The secret isn’t a trick with the account. It’s a shape of task. Give the agent a goal and a check that a program can run. Let it try, read the failure and try again until the check passes. Then give it throwaway machines so it keeps going while you sleep.

He calls it hill climbing, and he’s right that it’s the important idea. It’s a good video. Watch it.

I want to add what you learn after you’ve run that loop for a while, because I’ve been running it in public since June.

What was already there

These are public repos, with the dates GitHub recorded.

On June 22 I started double-jump, the loop itself. It scores every candidate and picks the weakest. An AI improves it until it beats the old score by a margin, and a fixed referee says yes or no. Git keeps a receipt for every accepted step.

A week later, on June 29, came the-coliseum, the population version: things compete, winners breed, losers are culled.

On August 1, rapp-rock-tumbler had sub-agents build ten apps. All ten reported success. Nine were still broken.

That failure produced rapp-ratchet on August 5: checks that only doing the work can pass. Saying it was done passes nothing. A day later, brainstem-harness ported agents to a new runtime and then proved the port changed nothing.

On August 21, a sync protocol shipped through a review loop in openrappter #420. Its fix commit traces 32 reviewed defects to 9 root causes. And on August 26, rapp-parity built a clean reference from public source and compared every live copy against it: behind, ahead, or missing work.

I didn’t invent this either. Evolutionary search, AlphaZero playing itself, and reinforcement learning on checks a program can verify all came first. The idea is old. What’s new is that coding agents are now good enough to be the thing that climbs.

Why can’t you trust the agent’s report?

This is the whole lesson of the rock tumbler. A model told to “keep going until it works” will eventually tell you it works. Ten out of ten said so. One was right.

The report is the agent’s opinion. Only a check it can’t argue with counts as evidence.

Can the climber reach the judge?

Give an agent a failing test and enough hours, and one day it will notice the test is a file it can edit.

The judge sits behind glass. The climber can see it, but never touch it.
The judge sits behind glass. The climber can see it, but never touch it.

So run the check somewhere the agent has no write access to, not just a different folder. Before you accept a pass, confirm the judge is still byte-for-byte the judge you wrote.

How do you know the judge works?

Test it before you trust it. Before a long run, show the judge one example you know is good and one you know is bad. If it can’t tell them apart, the climb will optimize nonsense for four days.

And a judge that goes silent hasn’t said no. An empty answer means “judge unavailable”, never a zero.

Copies are easy, taste is hard

Theo says this too, and I measured it. I asked an agent to port a small server to another language, judged by a 25-case conformance suite. It passed in one round, in about eight minutes.

Then I re-ran his Skate experiment with a two-hour cap. The game loaded and did all three flip tricks on a fake gamepad within the time box. The looks judge still scored it 6, where 7 passes.

Two runs prove nothing general, and a task with a right answer can still be a long climb; Theo’s compiler work shows that. But when the judge is a matter of opinion, the climb slows down, because every round argues with taste.

How do you prove a rewrite changed nothing?

Replay real traffic, but measure the noise first. Record what the old system actually answered, then replay the same requests against the new one.

Send the same traffic to the old system twice before you start. The fields that differ between those two runs, like ids and timestamps, are candidates for noise. Review them before you ignore them, then require everything else to match exactly.

The judge is the product

Most of the work is here: building the thing that says "pass".
Most of the work is here: building the thing that says “pass”.

Theo calls the check “the most important piece”, and four months in, I agree. Accounts, machines and longer runs get cheaper every month. The check doesn’t.

A loop is only as good as the thing that says “pass”, and writing that thing well is most of the work. Spend your hours on the judge. The tokens will take care of themselves.

Update, November 1: watching the pattern get rediscovered

Two days after this post went up, another video showed the same shape from the other end: write the eval before the feature, make the agent ask you to review it, and add a hook so nothing can be marked done until the eval passes. Different words, same idea. The check comes first. The builder is not allowed to grade itself.

I have been working this way longer than I realized. In May 2025, FeedShyWorm 3.0 was the same game rebuilt three times, each version judged by whether it still played. In August 2025, Code Welding was about transplanting a feature between two codebases that share nothing, with one rule: break nothing that already worked. That rule is a judge. I just had not named it yet.

So this weekend I ran the loop on five things at once and wrote down what the judges actually said. Every number below came out of a judge script, not out of an agent’s report.

Why weld something new

Here is the part I think matters most. If you ask an agent to build a thing it has seen ten thousand times, it can pass by memory. A judge barely matters, because the answer is already in the weights. Code welding is the fix. Join two things no model has ever seen joined, and the only road to a pass runs through the judge. The agent cannot cheat from memory, because there is no memory of this thing. That is the whole reason the examples below are odd.

Five climbs, one weekend

A Rust library ported to TypeScript. The judge is 84 recorded input and output pairs from the Rust side, and it refuses any port that imports the vectors or anything else. The agent passed in one round: 84 of 84, and the certificate verified. Copies are easy. Still true.

The Skate bench, again. Same three gates as before: it loads clean, it lands an ollie, kickflip and heelflip from a fake gamepad flick, and two different models score screenshots against real Skate stills, both needing 7 or better. The baseline scores 4.6. The flicks pass 3 of 3. The looks judges say 5 and 6. Taste is still the slow part, and the climb is running as I write this.

My own driving simulator. A single-file racing sim I wrote in 2024. The judge plugs in a virtual Xbox pad and drives: throttle must reach 8 mph and keep rising, steering must turn the car, brakes must slow it, the frame rate must hold 15, the settings panel must open, export must work. All 8 mechanical gates pass. The two looks judges give it 4 and 3, with notes like “boxy car, flat lighting, blank sky, no sense of speed”. Score 8.3 of 11. The mechanical part was done in 2024. The taste part is what the climb is for.

A walkthrough video of that simulator, as its own climb. This one surprised me. The thing being climbed is a script that drives the sim with the pad and records the screen. The judge watches the recording: time spent moving, time spent off the track, laps, top speed, and whether the video is long enough to be worth posting. First result: 108 seconds, moving 95 percent of the time, never off the track, 59 mph, zero laps. Score 7.82. A video is a deliverable, so a video gets a judge.

An original arena game nobody has made. Two factions, five heroes, lanes, creeps, denies, towers, runes, a shop, and a day and night cycle, welded from a decade of arena game metagame into a few hundred lines of plain JavaScript with no engine. The judge runs it three times with a bot and reads a bench the game prints: gold, last hits, denies, towers, result. Baseline: 1129 gold, 47 last hits, 9 denies, 5 towers, and a Defeat. Score 7. It plays. It does not yet look like anything. No model was trained on this game, because it did not exist on Friday.

The judge crashed and nobody noticed

One more lesson, paid for today. The Skate and driving judges both failed to start inside the climb. The climb engine pins the judge by its real path on disk, and that path pointed at a Python without the browser library installed. The judge crashed before it could print a score.

The engine did not stop. It rejected every candidate, three rounds of them, with the reason “no comparable numeric score”, and kept spending agent time. I wrote above that a judge that goes silent has not said no. Here is the stronger version: a judge that goes silent must stop the climb. A missing score is a fire alarm, not a rejection.

The cartridge

Every version that survives a climb now gets packed into one self-contained HTML file. Inside it is a small signed record: the goal, the judge’s hash, the tree hash, the score, and the certificate if it earned one. Think of the file as a cartridge. The game is in it, and so is the receipt for how good it is. A console page lists the cartridges, plays any of them in a frame, and hands a virtual gamepad to whoever is holding it, a person or the next judge.

That is where this is going. I play the builds on the couch while the loop polishes them. The judge is the product, and the cartridge is how you ship it.

Quick answers

What is AI hill climbing?
A coding agent works toward a goal by trying, failing and retrying until a check a program can run says it passed, often for hours or days without a person.

Why is the judge the hardest part?
The climber gets cheaper every month, but only the judge decides what “done” means. A weak judge turns days of compute into confident nonsense.

How do you stop an agent from gaming its test?
Run the check where the agent can’t write, confirm the judge file is unchanged before accepting a pass, and test the judge on one known good and one known bad example first.

Sources: Theo, “How To Give Claude Unlimited Usage“; the repos linked above. Earlier in this series: Own the Engine and The Personless Harness.

Page 1 of 11

Powered by WordPress & Theme by Anders Norén