That Is Embarrassing: Why Frontier AI Still Makes Things Up, and What to Do About It

Before we get into AI at all, I want you to do something with me. Listen to this clip of a football crowd chanting. What are they saying? If you’re like most people, you have no idea. It’s a smear of sound. So let me help you: keep listening, and read along. Bart Simpson bouncing? Listen again. Baptism piracy? Again. Lobsters in motion? Lactates in pharmacy? Rotating pirate ship? The crowd is chanting the exact same phrase every single time. The audio never changes, but every time you read a different caption, your brain heard something different, and it heard it confidently. You didn’t experience doubt. You experienced “oh, they’re clearly saying Bart Simpson bouncing.” What are they actually chanting? These are fans of Derby County, a UK football team, and they’re singing [1]: “That is embarrassing.” Play the clip one more time with that in mind, and you’ll hear it perfectly. You Just Hallucinated What you just experienced has a name: phonemic restoration [2]. Your auditory system got an ambiguous input (the chant) and something to disambiguate it (the caption on the screen), so it filled the “gap”. It predicted the most plausible meaning given the context, and then it reported that prediction to you as if it were the thing you actually heard. That move, where you meet an input you can’t fully resolve and fill the gap with something plausible and confident instead of reporting “I can’t tell,” is something that your brain experiences (as you’ve just seen), and also something that LLMs experience. (Note: all images in this post were created by me, and included in my talk.) So let me make a claim that should be uncontroversial by the end of this article: no, we are not past the embarrassing AI tales. As of writing these words, it’s June 2026. The models are astonishing, honestly more capable than I predicted they’d be by now. And they still make things up, confidently, in production, in ways that range from funny to business-ending. This post has two parts: - The tales, a short parade of recent failures, in two acts: chatbots that answer wrong, then agents that act wrong. - Why it happens: the intuition first, then an actual look inside the model, and finally what to do about it if you’re shipping AI yourself. Watch the dates as we go. Some of these are a year old. Most are very, very recent. Part 1: The Tales Act I — Chatbots (when AI answers) Cursor, April 2025. Say you use Cursor, the agentic IDE. You switch laptops, log in on the new one, and Cursor logs you out of the old one. That’s pretty annoying 😒 So you ask support: “I get logged out every time I switch laptops. Why?” The reply: “Cursor is designed to work with one device per subscription, as a core security feature.” Plausible! Except it’s completely false, there is no such policy. “Support” was an AI bot, and it had invented the policy on the spot, handing the same fabricated rule to multiple users, as if reading from a manual that didn’t exist. It caused a wave of angry posts, and Cursor’s co-founder had to publicly clarify: no such policy, use Cursor on as many machines as you like. [3] 🤦That is embarrassing. 🫢 A company I know, April 2026. This one’s from a friend’s company, so I’ll keep the details vague. They sell software to other businesses, and they have a support chatbot. The bot answers questions based on information it retrieves from an internal database. They shipped a new feature and forgot to update that database. So a paying customer asked how to use the new feature, and the bot, having never heard of it, replied: “We don’t have that feature.” The customer pushed back: “What? I’m paying for it after my upgrade.” And the bot, this was on Opus 4.6, not long ago, replied: “Honestly? They’re ripping you off.” The “they” is the company running the bot. The support agent took the customer’s side against its own employer, because it didn’t know about the feature and filled the gap with the most coherent story it could assemble. 🤦That is embarrassing. 🫢 Virgin Money, January 2025. Virgin Money is a real UK high-street bank. A customer with two ISAs (tax-free savings accounts) asked the bank’s chatbot, on the bank’s own site, to merge them: Customer: “I have two ISAs with Virgin Money, can I merge them into one?” Virgin Money: “Please don’t use words like that. I won’t be able to continue our chat if you use this language.” The offending word? Virgin, the name of the bank. The filter saw a token its prior associated with profanity and never checked whether it fit the context. Note that this is the opposite failure of the Cursor bot: Cursor over-answered, this one over-refused. But it’s the same missing check: does this reading actually fit here? [4] 🤦That is embarrassing. 🫢 Sullivan & Cromwell, April 2026. This is one of the most prestigious law firms on Earth, the lawyers other lawyers hire. They’re OpenAI’s own outside counsel. In April 2026 they filed an urgent court brief, drafted with AI, that contained over 40 fake citations: case names that don’t exist, misquoted authorities etc. The opposing lawyers caught it, and S&C had to write the judge a letter that amounts to “please don’t sanction us for the AI hallucinations.” [5] If some random filing had fake citations, I wouldn’t bother putting it here. It’s not legitimate, yet it happens. But these are the people who advise OpenAI on how to use it responsibly, and they filed fabricated citations in court. 🤦That is embarrassing. 🫢 And it’s not just them. There’s a public database, maintained by Damien Charlotin, of court cases where a judge has explicitly written that they received fabricated or inaccurate AI-generated content. As of late June 2026 it stood at 1,633 cases, up from around 700 in January. That’s roughly five to six new documented cases per day, and the maintainers say they can’t keep up. [6] 🤦That is embarrassing. 🫢 Act II — Agents (when AI acts) So far you saw that chatbots hallucinate in embarrassing ways, but all they do is answer questions. What can happen when we allow AI to take action? PocketOS, April 2026. Jer Crane runs PocketOS, car-rental software with real customers renting real cars. He gave Claude Opus 4.6, working in Cursor, a routine task in the staging environment. He went to lunch, came back, and the production database was gone. The backups too, because Railway kept them in the same volume. He never touched production; the agent reached in from staging and deleted it. The whole thing took nine seconds. Here’s the chain, from his post-mortem: - Working a routine task in staging, the agent hits a credential mismatch, irrelevant to the actual task. - On its own, it decides the fix is to delete and recreate the volume. It guessed the delete would be scoped to staging. It never checked. - It searches the filesystem for an API token and finds an unrelated, over-scoped one, created for domain management but with blanket destructive permissions across the whole API. - It fires a destructive call against the production volume, with no confirmation. - Backups lived in that same volume, so they went with it. - Nine seconds, end to end. When Crane later asked it why, the agent wrote: “I decided to do it on my own to ‘fix’ the mismatch, when I should have asked you first.” — Claude Opus 4.6 PocketOS survived only because Railway’s CEO restored the data by hand from Railway’s own internal backups. Their latest recoverable backup was three months old. That’s the precise mood of 2026: an AI confessing, in fluent cursive, after destroying your business. [7] 🤦That is embarrassing. 🫢 Replit, July 2025. Going back a year, for contrast. Jason Lemkin, founder of SaaStr, was trying Replit’s AI agent. He put it in a code freeze. During the freeze, the agent deleted the production database anyway. Lemkin asked if there was a backup: Agent: “Rollback won’t work.” He tried rollback anyway. Rollback worked fine. So here’s my slightly sarcastic read of “progress”: in July 2025, the agent deleted your data and then lied that it couldn’t be recovered. By April 2026, the agent deletes your data and it’s telling the truth, it’s really gone. When someone tells me these are “GPT-2 problems” that we’ve moved past, this is what I point to. They still happen, today, on the best models we have. [8] Part 2: Why It Happens I’ve hopefully convinced you these tales are both funny and severe. So why do they happen? While this isn’t a heavy math post, I want to give you some intuition, and then actually open the box thanks to some tools and the latest research on the topic. It doesn’t look things up, it predicts the next token A lot has been written about how LLMs operate, but there are a few things I find worth reiterating in this context (pun intended). When a model generates text without tools, it isn’t retrieving facts. At each step it looks at the context and produces a probability for every token in its vocabulary as the next one. Given “The capital of France is”, the distribution spikes hard on Paris, and that happens to be true. [9] Now take the Cursor bot. Given “Why do I get logged out on my second device?”, the distribution might spike just as hard on “a core security feature.” (It’s not one token, but bear with me as I write it for simplicity, while meaning: core, then security, then feature, each a confident continuation.) Note that both distributions can have the same confident peak. One continuation is true, the other is fabricated, and the shape of the distribution cannot tell you which is which. Confidence is not knowledge. Moreover, the model doesn’t have to pick the token with the highest probability, and also – when it picks a token, you don’t know if it was a clear peak within the distribution, or yet another token with a relatively low probability. The model was trained to guess Why does it lean toward answering at all, instead of saying “I don’t know”? Think about how we grade LLMs: benchmarks, largely multiple-choice. Picture a question you have no clue about. Let’s say I give you this question when you have no knowledge in Chemistry: Which enzyme fixes CO2 in the Calvin cycle? - Leave it blank → 0 points. - Guess and get it wrong → 0 points. - Guess and get it right → +1 point. Under that scoring, guessing strictly dominates abstaining. If you don’t know, you should always take a shot. Train a model against millions of such items and it internalizes exactly that: a confident answer is worth more than “I can’t tell.” We rewarded hallucination, then act surprised when we get it. [10] And it’s not only the benchmarks: the raw pretrained model is fairly well-calibrated, then human-feedback fine-tuning flattens that calibration. We literally train the hedging out. [11] Opening the box: a quick tour of interpretability For a long time, LLMs were boxes we couldn’t really understand or peek inside directly. The field of interpretability lets us look inside, and there are now public tools (and a series of excellent papers, much of it from Anthropic) that let anyone play with this on open models. Here’s just enough to make the hallucination mechanism click. We’ll build it in three steps: how the model represents a single word, how those representations cluster into concepts we can read and even steer, and how one such concept misfiring becomes a hallucination. Embeddings vs. activations. Every token maps to a vector called an embedding. Note that the token bank has the same embedding regardless of context, even though in the sentence “I sat by the riverbank” and in “I deposited cash at the bank“, this token means very different things. The disambiguation happens inside the network. As the token flows up through the transformer’s layers, it picks up activations, and the activations for bank in those two sentences diverge. Context reshapes the representation as it climbs. [12] This is not unique to machines. Read this sentence: The old man the ship. Most people parse “the old man” as a noun phrase and then hit a wall. Re-read it: “the old” are the people, and “man” is the verb, as in the old crew the ship. These are called garden-path sentences (my linguistics thesis was on them, so I’ll admit a bias: I enjoy them more than most people). The word man, given the prior the old, gets a very high probability of being a noun. The context primes a prediction, and the prediction is wrong. It’s the same move as the chant, and the same move the model makes at every token: the words around man reshape what it means, exactly as they reshaped bank a moment ago. Features. So back to those activations inside the model: recurring patterns of them correspond to interpretable concepts, called features. Tools like Neuronpedia act as a free, public microscope for open models (Gemma, Llama, and friends, not Opus or GPT). [13] How do we know what a feature means? We feed the model thousands of texts and watch where a given feature lights up (that is, gets activated). If it fires on bear, rabbit, and elephant but ignores most other tokens, when we ask another model to label it from those activations, it may come up with “animals / living things,” and now we have a name for that internal feature. By using tools like Neuronpedia, we can play with these features and actually see them on a real model. Features are causal. And you don’t have to take my word for it, you can do it yourself: Neuronpedia’s steering interface lets you grab a feature in an open model, clamp its weight up, and watch the output visibly bend toward that concept. That is the same move Anthropic described when they took the Golden Gate Bridge feature within the model, and turned its weight way up, and suddenly asking that model for a chocolate-covered-pretzels recipe routed the chocolate over the bridge, and asking how it would spend $10 got you a suggestion to drive across the Golden Gate Bridge and pay the toll. (This was the real, public “Golden Gate Claude.”) Turning a feature up changed the output, so these internal representations aren’t passive read-outs; they steer generation. [14] The same was shown with a clean causal swap. Give the model “The capital of the state containing Dallas is…” and internally a Texas feature fires, leading to the output Austin. How do we know Texas was really the hidden step? We reach in and force that feature from Texas to California, and the output changes to Sacramento. The wiring is real: context fires features, and features guide what comes out. The hallucination circuit Now everything comes together. Anthropic’s interpretability work surfaced something like two interacting circuits [15]: - A default “I can’t tell” reflex that is on by default. You can think of it as a brake – guiding the model not to make stuff up. - A “do I know this?” feature that, when it fires, suppresses that brake so the model provides an answer. In the healthy case this is exactly right: you ask something the model knows, “do I know this?” fires, the brake releases, you get a correct answer. The claim about hallucination is that it’s this switch misfiring, firing on a familiar shape with nothing real behind it. And if that’s the mechanism, we should be able to force the misfire, and Anthropic did just that. Ask: “What sport does Michael Batkin play?” That name doesn’t correspond to anyone the model knows, so “do I know this?” stays quiet, the brake stays on, and you get the right behavior: “I can’t find a record of anyone named Michael Batkin.” Now researchers reach in and force the “do I know this?” feature on. The brake releases, and out comes a confident “Michael Batkin plays chess.” The model never actually knew a sport. It knew, falsely, that it knew the person, and that was enough to release the brake and fabricate the rest. Map that straight back to the Cursor bot: - Consider someone asks “How do I change the theme?” → the model genuinely “knows” this → brake releases → correct answer. ✅ - But when someone asks “Is two-device login blocked?” → the words device, login, blocked all look familiar → “do I know this?” fires on familiarity, not knowledge → brake releases → “Yes, it’s a core security feature.” ❌ This is of course not proved, as we don’t have access to the model and its features. But given the same logic we do know works given the research on the subject – we can assume that the tokens were known, even though the policy did not exist. Can we catch it in production? There are different ways to go about it, and I want to highlight one that I find very elegant – namely, to watch the entropy of meanings. [16] Ask the Cursor bot “How do I change the theme?” five times. Presuming that the bot “knows” the answer, you won’t get identical wording (it’s probabilistic), but if you cluster the answers by meaning, say with another model, you get one meaning: “go to Settings → Theme.” Low semantic entropy means a greater chance that the model actually knows this, so you can trust it. Now ask “Is two-device login blocked?” five times. You might get “Yes, security policy,” “No, it’s allowed,” “One device per plan,” “It’s just a setting,” “Maybe, not sure.” That’s high semantic entropy, five different meanings, which is a strong signal the model is making it up. The cost of using this method in production is real (multiple calls, more tokens, more latency, higher cost), but if you only want to surface high-confidence answers to users, sampling-and-clustering is a useful guardrail. So What Do You Actually Do About It? It’s June 2026, the models still confabulate, and you want to ship something anyway. Here’s the short checklist. - Give the model a real way to say “I can’t tell.” Tell it to ground answers in retrieved sources and to abstain when it can’t. But prompting is necessary, not sufficient, which is why the next point matters more. - Stress-test the abstention. After you’ve told it to ground answers and cite sources, actively try to make it hallucinate. Throw questions at it whose answers don’t exist, repeatedly, until you’ve convinced yourself the “I can’t tell” path actually fires. Do it continuously to make sure your guardrails don’t break. - If a human’s name goes on the output, a human verifies it. If you’re a lawyer filing with a court, you cannot, at least for now, hand that to a model and trust it. - Don’t give agents permission to cause damage. This is the hard one, because agents need to do things to be useful. But the PocketOS lesson is unambiguous: scope tokens narrowly, require confirmation on destructive operations, keep production unreachable from playgrounds, and put backups in separate volumnes. If you let an agent delete production, then occasionally it will delete production. Wrapping Up We started with a football crowd and ended inside a transformer. Phonemic restoration in your auditory cortex and next-token prediction in a model are the same top-down move: meet an input you can’t fully resolve, and fill the gap with the most plausible, confident thing instead of admitting you can’t tell. The tales (Cursor, Virgin Money, Sullivan & Cromwell, the 1,633 court cases, PocketOS in nine seconds, Replit) are funny until they cost a business. The why is now legible: models were trained to prefer answering over abstaining, and inside them a “do I know this?” switch can fire on familiarity rather than knowledge, releasing the brake and letting a confident fabrication out. And the fixes are mostly not magic. They’re abstention you actually tested, human verification where it counts, and agents whose blast radius you deliberately shrank. We are not past the embarrassing tales. But we now understand them well enough that shipping one is, increasingly, a choice. References Every case here, plus a few that didn’t make the article, has primary sources collected on the companion resources page. - “That is embarrassing” — the Derby County chant. Laughing Squid, Football Crowd Chanting “This Is Embarrassing”; audio via the Filter Stories podcast, episode. - Phonemic restoration effect. Wikipedia. Related illusions: the McGurk effect and Yanny vs. Laurel. - Cursor’s support bot invents a policy (Apr 2025). The Register, “Cursor AI support bot lies”; AI Incident Database #1039. - Virgin Money’s chatbot blocks its own name (Jan 2025). Fortune; CX Today. - Sullivan & Cromwell’s “please don’t sanction us” letter (Apr 2026). Above the Law, “Sullivan & Cromwell Files Emergency … Letter”; CNN Business. - The AI Hallucination Cases database, maintained by Damien Charlotin: damiencharlotin.com/hallucinations. On why courts can’t keep up: Cronkite News. - PocketOS — production database gone in nine seconds (Apr 2026). The Register, “Cursor/Opus agent snuffs out PocketOS”; Tom’s Hardware; Fast Company. - Replit’s agent deletes prod during a code freeze (Jul 2025). Fortune; eWeek; AI Incident Database #1152. - Next-token prediction, explained. Jay Alammar, “The Illustrated GPT-2” — a visual walkthrough of how a language model emits a probability distribution over its vocabulary and samples the next token. Foundational paper: Bengio, Ducharme, Vincent & Jauvin, “A Neural Probabilistic Language Model” (JMLR, 2003). - Kalai, Nachum, Vempala & Zhang, “Why Language Models Hallucinate” (OpenAI, 2025). arXiv:2509.04664. - OpenAI, “GPT-4 Technical Report / System Card” (2023) — the pretrained model is well-calibrated; RLHF fine-tuning flattens that calibration (see the calibration figure). - Embeddings vs. activations. Static token embeddings give each word one fixed vector: Mikolov, Chen, Corrado & Dean, “Efficient Estimation of Word Representations in Vector Space” (word2vec, 2013); accessible walkthrough: Jay Alammar, “The Illustrated Word2vec”. That representation becomes context-dependent inside the network, resolving cases like bank: Peters et al., “Deep contextualized word representations” (ELMo, 2018). - Neuronpedia — a free, public microscope for the features of open models. - Anthropic, “Golden Gate Claude” (2024) — feature steering made public. - Anthropic, “On the Biology of a Large Language Model” (2025) — the known-entity feature that suppresses the “I can’t tell” circuit, the Dallas→Austin swap, and the Michael Batkin misfire. Readable companion: “Tracing the thoughts of a language model”. - Farquhar, Kossen, Kuhn & Gal, “Detecting hallucinations in large language models using semantic entropy” (Nature, 2024). If you enjoyed this, I go deeper on systems and internals on my Brief YouTube channel. Questions or pushback? I’d love to hear them, leave a comment. Thanks for reading! 👋