A Tool Call Returned Success and the Work Never Happened

The flag that reports failure is set by the same code that was wrong about what happened. That is the whole of this article, and everything below is the twelve articles it cost me to learn it.

On 25 July 2026 I audited every entry in my publishing calendar that carried a published URL. There were twenty-five. Twelve live posts on Medium turned out to be truncated: between 41% and 72% of the text I had sent had survived, and some had lost their code blocks entirely. One more URL in that same audit answered HTTP 410. Every one of them had been reported as a success, with a valid URL, weeks earlier.

Two rows in that table do more work than the rest. The short announcements survived, and three other platforms came through untouched. Hold on to both; they are what turns an embarrassing bug into something worth writing down.

My publishing server puts drafts on Medium by driving a signed-in Chrome over the debugging protocol, because Medium's write API has been left to rot. The function that created the draft waited for one thing: the browser's address changing to the editing URL of a new story, /p/<id>/edit. When that happened it treated the work as done and closed the tab in a finally block.

In essence, the whole definition of done was this:

Pasting into the editor is instantaneous. Saving is not. Medium writes the editor's contents to its server incrementally, over several background requests, and the tab was being closed while those requests were still in flight. Everything that had not reached the server by then was gone. No error, no banner, no failed request that anyone could have caught — because the request was never made. The page that would have made it no longer existed.

The adapter returned a valid URL and ok.

For weeks I read this as browser automation is unreliable, APIs are reliable. It is a satisfying conclusion. It is also not what the data says, and noticing that is the only genuinely useful thing I did in this whole episode.

dev.to came through completely intact — that one is an API, so it fits the theory. But Stack Overflow and Hacker News also came through intact, and both of those I drive through exactly the same browser, over exactly the same debugging protocol, with the same Puppeteer, in the same run. If the browser were the problem, they would have been truncated too.

The dividing line was never browser against API. It was one submission against many. Stack Overflow and Hacker News submit a form: the whole text goes in one POST, and when the response comes back there is nothing left to arrive. dev.to's API is the same shape — one request carrying everything. Medium accepted the text in instalments, and my adapter stopped watching after the first one.

Which is also why the short announcements survived. At around 1,500 characters they finished saving before the tab closed. The bug was not Medium breaks; the bug was a race, and short texts won it.

The most ordinary reason there is. I had monitoring. It asked whether the published URLs were reachable, and whether they were picking up views. They were reachable. They were picking up views.

A truncated article is a perfectly healthy web page. It returns 200, it renders, it has a title and a hero image and three paragraphs and then it simply stops, and nothing in an HTTP response has an opinion about whether an article ended where its author meant it to.

This is the part that generalises past publishing, and it is the reason I am writing it up rather than quietly fixing it. Availability checks answer did we reach the code that checks, not is the thing correct. If you monitor a deployment pipeline, an ETL job, a backup, or anything at all that an agent does on your behalf, there is a decent chance your check is the same shape as mine was.

Sitting beside the first was a second bug, and this is the one people retell.

The adapter needed to know whether Medium's editor had failed to save, so it looked for the phrases Medium shows on failure. It did this by searching document.body.innerText for a pattern including Something is wrong — that is, by searching the text of the whole page, which on that page includes the article being published.

One of the articles I was publishing contained the sentence if something is wrong you flip back.

A run that had worked perfectly was declared a failure. The retry ran. A duplicate draft appeared.

Twenty seconds of reading the diagnosis is enough to see the rule, and the rule is more general than browsers: never read status out of prose. If the only way to learn an outcome is to hunt for words in running text, you have guessed it rather than learned it, and prose will eventually contain your keyword by accident. The fix was to stop reading the page as text and look only at the elements a page uses to announce status — [role=alert], [role=status], [aria-live].

MCP defines two ways for a tool call to fail, and they are a genuinely good pair. A protocol error means nothing ran: no tool by that name, arguments that did not match the schema, a malformed request. An execution error means the tool ran and the news is bad — the file was not there, the API returned 403 — and it arrives as a successful response carrying a flag, written out in text, because the recipient is a model in the middle of a task and capable of doing something about it.

Neither level can see what happened to me. From the protocol's point of view the call went perfectly: a request was sent, a well-formed response came back, it carried no error flag, it contained a URL.

A protocol can carry a verdict. It cannot audit one. This third level of failure is invisible to it by construction, and no revision of any specification will fix that, because the component reporting the outcome is the component that was wrong about the outcome.

There is a token bill attached to this too, and it is worth stating because it makes the failure expensive as well as embarrassing. On that request the model paid for twenty tool descriptions, paid again for the call itself, received ok, and moved on to the next item in its plan. The spend was real. The work was not done.

Two things in the adapter, and one thing that is not in the adapter at all.

The first instinct was to paste the story in pieces, on the theory that one big paste loses text. I measured it, on the same 13,953-character story, and the theory did not survive:

Chunking bought nothing. On that story it could not finish at all. What makes the write safe is not how the text goes in, it is that the text is read back out of the server afterwards. The adapter now pastes the whole story in one go, triggers a save, reloads the page, and asks which parts of what it sent are actually there:

If anything is missing, the editor is cleared and the whole story is pasted again — never appended, because appending duplicated content whenever a chunk was judged absent but was not. If a rewrite leaves exactly the same gaps as the one before it, the adapter stops and refuses: stuck after N rewrites. And when every probe is present it still checks two more things before handing back a URL — that the number of rendered code blocks matches what was sent, and that the title survived the round trip.

Note what all of that has in common. Not one of those checks trusts the call that did the work.

The second thing in the adapter is the banner detector, which no longer reads prose.

The third thing is not in the adapter at all, and it is the part that matters: the check that the work happened has to live outside the tool that did the work, and it has to ask a different question. Not did the call return but is what is now on the server the same as what I sent.

scripts/check-published-completeness.mjs reads every published article back and diffs it against the source. It is not clever. It renders each published URL in the signed-in Chrome that owns it — Medium blocks anonymous fetches, and Medium drafts are private to their account, so an anonymous HTTP GET cannot answer the question at all — scrolls to the bottom, and compares against the stored body on two signals: the number of code blocks, and whether the text of the last link is present. In these articles the last link lives in the closing further reading block, which makes it the first thing truncation eats. The script exits non-zero if any publication is incomplete.

Two modes, two different questions, and it took the audit to work out that they were different questions. The --anon mode opens every URL in a browser profile signed in to nothing, because an author always sees his own article in full. A paywall, a member-only interstitial, a login wall — none of those are visible from the account that published the piece.

The one piece of real subtlety is the comparison itself, and it is where a naive version of this check will lie to you. Platforms strip markdown and rewrite typography: ` spec.selector ` loses its backticks, a straight apostrophe comes back curly, a hyphen comes back as an em dash. Compare raw strings and you will declare whole articles truncated. So the comparison runs on visible text, normalised on both sides:

Two things I will not pretend about. Repairing the twelve truncated posts is its own problem rather than a footnote: re-uploading a story replays the same autosave race, and the repair script prints FAIL on a timeout in its final check even when the save went through, so the thing to trust is the completeness diff and not the exit code. And there is one difference the checker will show you that is not damage: Medium splits a code block containing a blank line into two blocks, so a source with eight fences becomes a draft with nine. Cosmetic, nothing lost, and you will chase it once before you learn to recognise it.

None of this is specific to publishing. If something in your stack does work on your behalf — a deploy bot, a nightly ETL, a backup, an agent with tools — these are the four questions the audit turned into:

Anonymous statistics about tool calls that never complete get quoted a lot and sourced almost never. I cannot tell you what is behind them. I can tell you what was behind mine.

This is one server, one platform, one bug, and a sample of twenty-five. It is not evidence about the reliability of AI agents in general, it does not carry a cost in money because I never calculated one, and I have no industry percentage to offer you. What it is, is a worked example of a failure mode that has no place in the protocol's own model of failure — and a reason to go and look at whether your monitoring asks the reachable question or the correct one.

A successful response is an opinion.

Is this a bug in MCP? No. MCP's two failure levels are well designed for what they cover. This failure is invisible to any protocol by construction: the component reporting the outcome is the one that was wrong about it, so there is nothing for a protocol to inspect.

Would the `isError` flag have caught it? No. That flag is set by the tool itself. The tool believed it had succeeded — it had a valid URL to prove it. A flag written by a mistaken component carries the mistake.

Isn't this just browser automation being flaky? That was my hypothesis for weeks and the data killed it. Stack Overflow and Hacker News run through the same Chrome, the same protocol and the same Puppeteer in the same run, and both were intact. The dividing line is one submission versus many, not browser versus API.

Would a retry have fixed it? Not on its own, and the retry path is where the second bug lived: a run that had actually succeeded was declared failed by a prose match, the retry fired, and a duplicate draft appeared. A retry without verification just gives you two ways to be wrong.

How do I check whether my agent actually did the work? Read the result back out of the system of record with something that is not the tool that wrote it, and compare it against what you sent — on visible content, not raw markup. That is all my checker does, and it would have caught this on day one.

My own publishing server and its runbook, docs/publishing.md, written up on 25 July 2026 as the incident was analysed; the audit of all 25 calendar entries carrying a published URL, run the same day; scripts/check-published-completeness.mjs and scripts/check-published-health.mjs; the measurement of the two write paths, run on 13 August 2026. The two levels of failure are from the MCP specification at modelcontextprotocol.io, revisions 2025-06-18 and 2026-07-28.

The three articles this comes out of: where MCP and REST came from, what MCP costs in tokens, and what an MCP server is actually made of.