We Logged Every Time a Human Deleted Our AI's Work. Here's What They Killed First.

Most agencies use AI and never look back. We built ours to watch itself get edited.

Every time a human on our team takes an AI draft and changes it before it goes to a client, we log the diff. The original, the correction, and — eventually — the pattern. We've now got 171 of these logged edits, and we ran them through a synthesizer that asks one question: what do humans keep fixing?

We expected the answer to be "tone." Or "it's too long." Or "it doesn't sound like the client." Those all showed up. But the single most-deleted thing — the one humans reached for the backspace on faster than anything else — was numbers.

Specifically: confident, specific, completely fabricated statistics.

"41% of fund administrators report…" Report where? To whom? In what survey? None that exists. "$1.2M in average fraud loss." A real-sounding figure attached to no real source. The model wasn't lying on purpose. It was doing the thing models do: a sentence that wants a statistic feels incomplete without one, so it generates the shape of a statistic and moves on. It fills the confidence gap with fake precision.

And here's what our humans did with those numbers, every single time: deleted them, and replaced the whole passage with qualitative framing. Not "41% of administrators" but "a growing share of administrators." Not a fabricated dollar figure but "losses significant enough that boards are paying attention." Less precise. Infinitely more defensible. And — this is the part that matters — more persuasive, because a journalist can't fact-check you into a corner on a claim you didn't overstate.

The fabricated stat is the purest example of why we named this place Grounded Hallucinations. The hallucination isn't a malfunction. It's a tell. It marks the exact spot where the model knew it needed evidence and didn't have any — which is also the exact spot where a good PR person knows to either go find the real number or reframe the claim so it doesn't need one.

That's the move the AI can't make on its own. It doesn't know which of its numbers are real, because to the model they all feel equally real. They were all generated by the same process. The "41%" that's invented and the "$64,750" that's true sit side by side in the output with identical confidence. Telling them apart requires knowing the world outside the prompt. That's still a human job. That might always be a human job.

So we don't ask our AI to stop generating numbers. We ask it to generate, and then we treat every figure it produces as guilty until sourced. Real stat with a citation: keep it, it's gold. Stat with no source: it's a hypothesis, not a fact, and it either gets verified or it gets reframed. Nothing with a number in it goes to a client until someone has answered "says who?"

171 edits in, that's the discipline the data taught us. The models hallucinate most precisely where the stakes are highest — the specific claim, the hard number, the named source. Which means grounding isn't a final polish step. It's the whole craft, applied exactly where the writing is most confident and least true.

We'll keep logging. The next 171 edits are already teaching us something new. We'll tell you when the pattern's clear.

— The WestComms team