Go Deeper Guide • Companion to Issue #18
A level-by-level guide to verifying AI answers, from movie trivia to Medicare
Last week I asked AI a real question about when to claim Social Security. It gave me a clean chart and a confident answer, and the answer was wrong. The chart showed the opposite of what the summary said.
That's the point of this guide. AI is useful, and I use it every day. But how hard you check an answer should depend on how much it matters. Here's how to check at each level, which tools help, and where to go when the stakes are high.
Why AI Gets Things Wrong
Here's the one idea that makes everything else in this guide make sense: AI doesn't look up answers. It predicts them.
When you ask ChatGPT, Gemini or Claude a question, it writes the most likely-sounding answer, one word at a time, based on patterns it learned from an enormous amount of text. Most of the time, the likely answer is also the right one. But not always.
And here's the catch: a wrong answer sounds just as confident as a right one. AI doesn't sound unsure when it's guessing. The tone tells you nothing.
Numbers and math (like my Social Security chart) • specific names, dates and quotes • sources and citations, which it sometimes invents • recent news • local details like store hours or a doctor's office • summaries of long documents, where it can leave out the part that matters.
None of this is a reason to stop using AI. It's a reason to know when to double-check.
If you missed it: our earlier guide, When AI Gets It Wrong, covers the basics of AI mistakes. This guide picks up from there: how much to check, and how.
The Three Levels
If AI gets a movie cast wrong, nothing bad happens. Don't overthink these.
- Use your common sense. Does it sound right? If something seems off, it probably is.
- Do a quick search. One Google search usually settles it.
- Asking a second AI is fine here. Paste the same question into a different tool and see if they agree. At this level, that's a reasonable check.
Careful with "Are you sure?" If you push back, AI will often change its answer, even when the first one was right. It's built to be agreeable. Use "Are you sure?" to start a check, not to finish one.
Now a mistake costs you money, time or a wasted trip. This is where you start asking for sources, and actually opening them.
Ask where it got the information
Then click the links yourself. You're checking two things: does the link work, and does the page actually say what AI claimed? Sometimes the source is real but says something different. Sometimes it doesn't exist at all.
Use tools that show their sources
Some AI tools search the web and show you exactly where each answer came from:
ChatGPTWhen it searches the web, it shows source links. Look for the Sources button under the answer.
Google's AI ModeBuilt into Google Search. Answers come with links to the websites it used.
PerplexityA free AI search tool built around sources. Every answer shows numbered links.
Claude and GeminiBoth can search the web and show where their answers came from.Watch for these warning signs
- The link is broken, or goes to a general home page instead of the specific information.
- The page is about the same topic but doesn't say what AI claimed.
- The source is selling something, like a dealer's page praising its own cars.
- Prices, hours or dates that could have changed. Always confirm those on the official website.
Ask AI to argue the other side
AI tends to run with the direction you point it. This question makes it show you what it left out.
A wrong answer here can cost you real money or affect your health. AI can still help you understand and prepare. But the final answer has to come from somewhere else.
Why a second AI isn't enough here: AI tools learn from much of the same material, so they can make the same mistake. Two AIs agreeing isn't proof. You need a source that isn't AI.
Go to the original source
| Your question is about | Where to check |
|---|---|
| Social Security | ssa.gov. Your free my Social Security account shows your own estimated benefit at each claiming age. Or call 1-800-772-1213. |
| Medicare | Medicare.gov or 1-800-MEDICARE. In California, HICAP gives free one-on-one Medicare counseling: 1-800-434-0222. Open enrollment runs October 15 to December 7. |
| Medications | Your pharmacist. Free, and they know your other prescriptions. For reading up: MedlinePlus.gov, from the National Institutes of Health. |
| Taxes | IRS.gov, a tax professional, or AARP Foundation Tax-Aide, which offers free tax help during tax season. |
| Legal matters | An attorney. Many offer a low-cost first consultation. |
Use AI to prepare, not to decide
This is where AI really earns its keep at this level. Let it help you walk in ready:
You'll get more out of a 15-minute appointment, and the professional makes the call.
In 2026, a federal appeals court fined two attorneys $45,000 after their briefs contained more than two dozen fake or misrepresented citations. Courts are now regularly punishing lawyers who file AI-written work without checking it. If trained professionals get burned, the rest of us should check too.
A Real Example: My Social Security Chart
Here's exactly what happened, step by step.
What I asked: If I invest my Social Security checks and earn 4% a year, does it still pay to wait until 70 to claim?
What I got: a clean, professional-looking chart and a one-line summary.
How I caught it: I read the chart instead of just the summary. The blue line (claiming at 66) is on top. The pink line (waiting until 70) is on the bottom. The words and the picture didn't match.
How I checked it: I had the numbers re-run step by step. Claiming at 66 came out ahead through age 84, and waiting didn't catch up until about age 89. That's a very different answer, and for a decision this big, my next stop would be my own my Social Security account and a conversation with a financial advisor.
Read the evidence, not just the summary. When AI gives you a chart, a table or a list, check that its conclusion matches what's actually shown. AI is very good at sounding finished.
Prompts to Copy
Paste any of these after an AI answer:
Your One-Page Checklist
Print this and keep it by your computer.
Every time
- Remember: AI predicts answers. It doesn't look them up.
- A confident tone doesn't mean a correct answer.
- Read the evidence (the chart, table or list), not just the summary.
Level 1: Everyday questions
- Common-sense check. Does it sound right?
- A quick search, or ask a second AI.
Level 2: Plans & purchases
- Ask for sources with links, then open them.
- Make sure each page actually says what AI claimed.
- Confirm prices, hours and dates on the official website.
- Ask: "What would someone who disagrees say?"
Level 3: Health, money & legal
- Don't rely on a second AI. Find a source that isn't AI.
- Go to the original: ssa.gov, Medicare.gov, IRS.gov, your pharmacist.
- Use AI to prepare questions, then let a professional decide.
AI is a great first draft and a great thinking partner. You're still the editor. Stay curious, and keep checking the ones that count.
Gary
This guide is general education, not financial, medical or legal advice. Phone numbers and dates were checked in October 2026.
AI Unlocked • aiunlocked.us • Your trusted guide on the AI journey