30 Failed Attempts and $21.87: The Output Cap of My Agent Factory Was Counting the Model's Thinking

Boulder, my agent build factory, builds websites for an operator who is not a developer. Before a site is built, it goes through an identity step: from the operator’s brief, Boulder asks the model for three “directions”, each a named homepage mockup, and someone picks one.

One morning in October the operator’s page showed a site stuck at that step. I asked my agent to investigate the crash and fix its cause. What it found was not a crash.

The problem: five launches, thirty attempts, nothing to show

The day before, the identity step had been launched five times for that site, including once on a copy of it made to try again from scratch. Every launch made the operator wait about ten minutes and ended with no proposal. Thirty attempts in all, thirty failures, and $21.87 of model spend with nothing to show for it.

The first thing I checked was whether the model or the platform was down. They were not. The same day, another site with a short brief had received its three directions normally. So the failure belonged to this site, and the only thing this site had that the others did not was its brief: 3,640 characters, with a colour table. Richer than any brief Boulder had seen so far.

The second thing I checked was what the model had actually returned. Not an error. Answers that stopped in the middle of a mockup, every time.

What broke: the cap counted the thinking

Each call to the model carried a cap of 32,000 output tokens. That number is older than the model Boulder runs on today: when I wrote about Boulder in April it ran on Claude Sonnet; it now runs on Claude Opus 5.5, a model that reasons before it writes. And Anthropic’s documentation says in one line what I had not read into that number: thinking tokens count toward the max_tokens limit for the turn, and max_tokens remains the hard ceiling on total output (Anthropic, extended thinking).

So the cap I thought was “32,000 tokens of mockup” was “32,000 tokens of reasoning plus mockup”. On a short brief the model reasons less and writes less, and both fit. On this brief it reasoned longer and wrote more, the ceiling fell in the middle of the mockup, and a cut mockup is nothing Boulder can read. Every attempt failed the same way, which is why retrying five times changed nothing. A deterministic truncation is not a flaky call, and retrying it only buys more minutes of waiting and more dollars.

The number that convinced me was the replay. I ran the same brief for real, with the cap lifted, and the three directions finished at 34,296, 47,716 and 62,960 tokens. The third one alone needed almost twice the old cap. It also ruled out the tempting middle ground: a 64,000 cap would have left 1.6 % of margin on that third direction, which is no margin at all for the next brief that is a little richer than this one.

The fix: a cap per model family, and a replay before the merge

The fix is small, and that is the point. A mockup now gets 128,000 tokens, the model’s maximum, when the model reasons before it writes. The other models keep 32,000 as before, including Claude Opus 4.8, which Boulder uses as its fallback model. Only tokens the model actually produces are billed, so raising the ceiling costs nothing by itself; what it buys is that an answer which needs 63,000 tokens is no longer cut at 32,000.

Then I checked the fix the way I now check everything an agent writes for me.

The replay first: the same brief, through the real path, at the new cap. All three directions finished, in five to nine minutes each, and Boulder read them like it does in production, for $3.29. That is the whole cost of the step that had burned $21.87 the day before.

The tests second, with mutations: I break the code on purpose and expect a test to fail. One mutant survived. The test that was supposed to cover the mockup path called the low-level model wrapper directly, so it never went through the function that sets the cap. A test that does not take the path it claims to protect proves nothing about it. I routed it through the real function, and the mutant died. I also searched the repository for any other file or document that still stated the old number, so the cap would not come back from a comment or a default somewhere else.

The independent review third: a pull request written by an agent goes through a reviewer that is a different model with none of the author’s context, and nothing of mine gets merged while that reviewer is red. Then the merge, pinned to the exact commit the reviewer and the tests had seen, with nothing else running on the platform at that moment: no site request, no build, no proposal in flight.

And the proof of deployment, because a merged pull request is not a deployed one. The hosting build finished six minutes after the merge. The agent runtime moved to version 666, and the deployed code was identical to the pull request, cap included.

The last step was the relaunch. The operator’s page has a button for it, “Relaunch the proposals”, and my attempt to click it through the dashboard’s embedded browser did nothing: the panel was not displayed, so the click never landed. Rather than guess, I read what the button does and did exactly that: the same write on the site’s record, the same call to the agent. Twenty minutes after the merge, the three directions were on the page. The launch cost $3.35, with no retry. Nobody had chosen a direction: that decision belongs to the operator, or to me, not to the agent.

The second thing the crash exposed: an e-mail nobody sent

While reading the relaunch path I noticed a sentence on the operator’s page: “You will receive an e-mail when the proposals are ready.” I searched the code for what sent that e-mail. Nothing did. The page promised a notification that did not exist, on the one step where a silent failure had just cost a day.

So the same day I opened a second pull request. Boulder already sends its other notifications through one alert topic, so the e-mail joined that path. The design choice was where to send it from: at the single point where a run’s outcome is written on the site’s record, so that every way a run can end is covered, the proposals, an adjustment and a crash alike, instead of one e-mail per happy path.

This pull request took five rounds of the independent reviewer, and every round found something true.

• The retry setting on the publishing client allowed two sends of the same e-mail. One attempt means one attempt; the setting I had written did not say that.
• The alert that reports a failure to load the agent depended on the very module that had just failed to load.
• A preparation launched without a brief ended without writing anything, so the page stayed on “preparing” and nobody was told.
• A site in the middle of being deleted still received the e-mail. And on a site already live, the e-mail asked the operator to choose a direction, when only I can do that at that stage.
• The raw technical error was displayed on the page.
• The one that changed the design, in round four: the web page launches the agent without waiting for its answer, and if that answer is lost in transit, the SDK retries the call and starts a second preparation, so one click could produce two e-mails. The same round found that a site deleted and recreated under the same name during a preparation would receive the old site’s result.

Each fix for a case left the next case open, so the family was closed by one rule instead: only the preparation the record is waiting for may write its result and send the e-mail. The web page writes a preparation id on the record and hands it to the agent; the agent writes only if the record still carries exactly that id, and “no id” is a value like any other. Round five found the last hole in that rule, a preparation launched by the old web version during the deployment, with no id, writing onto a recreated site that expected its own. Exact match, absence included, closed it.

Two limits are written in the pull request as accepted. If a preparation finishes while its site is being deleted, no e-mail goes out; if the deletion then fails, the mail is lost but the proposals are on the record. And if the agent process itself is killed, by its maximum duration or by memory, no code runs anymore, the record stays on “preparing” and no e-mail leaves. That second limit existed before this change, and it needs a supervisor outside the agent, a scheduled check that fails any preparation stuck for more than an hour and sends the failure e-mail. I wrote that as a separate ticket. When I measured, no record was stuck, and the eleven preparations of the previous days had all finished, the longest in 10.9 minutes.

Hosted CI was blocked on my side (a billing problem on the repository), so I replayed the whole agent CI locally on the pull request merged with main. By the fourth round the touched agent files had 98 tests and 34 mutations all caught, and the web routes 104 tests; the final head replayed green with 162 evaluations and 3,921 tests. The merge was pinned to the tested commit, the hosting build succeeded, and the agent runtime moved to version 669 with the new rule in the deployed code.

What I did not verify

The e-mail has not been received by anyone yet. No preparation has run since the deployment, so I scheduled a read-only check for the end of the week that counts the e-mails sent and names the sites, and I first checked that its filter finds the morning’s run, so that “zero” from it means “no e-mail” and not “broken filter”.

I did not test a brief richer than this one. The third direction used 62,960 of 128,000 tokens, about half, which leaves room, but “about half” is a measurement on one brief, not a bound.

I did not see the operator’s page as she sees it, logged in, after the fix. I read the record and the page from my side.

I did not test the fallback model on this brief. It keeps the old 32,000 cap, so if the primary model is unavailable and the fallback also reasons at length, the same brief could hit the same wall there.

And I did not look for the same mistake elsewhere in Boulder. The identity step is the one that writes the longest answers, but the same fixed cap may sit in other calls too.

Recommendation

If you call a model that reasons before it writes, read your output cap as “reasoning plus answer”. A number that was right for a model that answered directly is wrong for this one, and it will be wrong only on your richest inputs, which is the worst way to be wrong: everything passes until the one input that matters.

Set the cap per model family, not once for the application, and let the reasoning models have their maximum. The ceiling does not bill you; the tokens do.

Log the token usage of every attempt, and look at it before you retry. A response that stops at the same place twice is not flaky. Each retry of a deterministic truncation costs the same minutes and the same dollars and returns the same nothing. Thirty identical failures should have been one failure and an alert.

Replay the real input through the real path before you merge the fix, and let the replay choose the number. My instinct said 64,000. The replay said that would leave 1.6 % of margin.

And when a page promises a notification, grep for the sender. A promise on a screen is a feature only if the code behind it exists.