When AI Gets It Wrong

Jul 17

When AI Gets It Wrong - preview

GO DEEPER GUIDE

When AI Gets It Wrong

How to Spot a Confident Mistake — Before It Costs You

Why This Guide Exists

In this week's newsletter, you saw the basic idea: AI sometimes makes things up — not because it's lying, but because it's built to always give you an answer, even when it doesn't really have one. That's called a “hallucination,” and it's one of the most important things to understand about AI if you're going to trust it with anything that matters.

This guide goes further. We'll show you exactly what a hallucination looks like next to a trustworthy answer, in five everyday situations. Then we'll give you a simple, two-minute habit that catches most mistakes before you act on them — no tech background required.

The One Sentence to Remember

AI is built to sound confident, not to be right. Confidence and correctness are two different things — and AI only guarantees the first one.

Part 1: What's Actually Happening When AI “Hallucinates”

AI chatbots like ChatGPT, Claude, and Gemini don't look things up the way a search engine does. They generate an answer one word at a time, based on patterns learned from enormous amounts of text. Most of the time, that produces something accurate.

But here's the catch: nothing in that process checks whether the final answer is actually correct. If AI doesn't have a solid pattern to draw from — a rare topic, a very specific number, an obscure name, a recent event — it doesn't stop and say “I'm not sure.” It keeps generating the next plausible-sounding word until it has produced a complete, confident-sounding answer. That answer can be entirely wrong and still read exactly like a right one.

Where hallucinations happen most

Specific numbers — dollar amounts, dates, statistics, phone numbers, addresses
Citations & quotes — studies, articles, books, or things attributed to real people
Recent events — anything after the AI's training cutoff, or still developing

Independent testing on today's major AI tools finds hallucination rates ranging from about 3% on simple factual questions to well over 15% in long, multi-turn conversations — and errors climb highest in exactly the areas where being wrong matters most: health, money, and legal questions. A study published this spring found AI chatbots gave misleading or flawed medical advice in nearly half of the cases tested.

Part 2: A Real Example — The Disease That Never Existed

In 2026, a team of researchers ran a simple test. They invented a completely fake skin condition and called it “bixonimania.” Then they wrote two fake academic papers about it and posted them online — papers that openly admitted, in plain text, “this entire paper is made up,” and thanked a “Professor Sideshow Bob” in the acknowledgments as an obvious joke.

Within weeks, several major AI tools — including ChatGPT, Google Gemini, Microsoft Copilot, and Perplexity — were describing bixonimania to users as if it were a real medical condition. Some even recommended seeing a specialist for it.

Nothing about those papers was subtle. The AI tools didn't get fooled because the hoax was clever — they got fooled because that's how the technology works. If a topic shows up in text that looks credible, AI will often treat it as real, whether or not it actually is.

What this means for you

If AI can be talked into inventing a disease from an obvious joke, it can just as easily sound completely sure about a wrong drug interaction, an outdated Social Security rule, or a quote no one ever said. Confidence is not proof.

Part 3: Five Everyday Situations — Right vs. Wrong

Here's what a hallucination can look like next to a trustworthy answer, in five situations you might actually run into.