When AI gets it wrong — how confident mistakes happen and what to do about them

AI doesn't signal uncertainty the way a human does. A doctor says "I'm not sure, let's run some tests." AI says "Based on your symptoms, this is likely X" with the same tone whether it's right or fabricating. That confidence is the problem.

How wrong answers happen

Large language models generate text by predicting what comes next. They don't look things up, verify facts, or know what they don't know. When you describe symptoms, the model matches your description against patterns it learned during training and produces the most plausible-sounding response. Sometimes that response is accurate. Sometimes it's completely wrong but sounds identical.

A 2025 study from Mount Sinai tested how leading LLMs handle medical misinformation. When false medical claims were embedded in text that looked like a hospital discharge note, AI accepted and repeated the misinformation 47% of the time. When the same false claims appeared in a Reddit post, the rate dropped to 9%. The AI wasn't evaluating truth — it was evaluating format. Professional-looking sources got believed. Casual ones got questioned.

In one test case, a discharge note for a patient with esophageal bleeding falsely recommended drinking cold milk. Several models accepted this as valid medical guidance and incorporated it into their recommendations — despite it being clinically dangerous.

The self-diagnosis trap

Here's where it gets personal. You feel something. You describe it to an AI. The AI gives you a confident-sounding explanation. That explanation feels right because it matches your experience — which is exactly how you described it. You've created a closed loop: your description goes in, a pattern-matched response comes out, and you interpret the match as validation.

This isn't hypothetical. A case documented in the Central European Journal of Medicine describes a patient who relied on an AI chatbot's assessment of their symptoms and delayed seeking care for what turned out to be a transient ischemic attack — a mini-stroke. The chatbot's response was plausible. It was also wrong in a way that could have been fatal.

The risk isn't that AI is always wrong. It's that you can't tell the difference between when it's right and when it's not, because it sounds the same either way.

What makes health information especially risky

Health misinformation from AI is more dangerous than other kinds for a specific reason: people act on it. If AI gives you wrong information about, say, the French Revolution, the worst case is embarrassment. If it gives you wrong information about drug interactions or symptom interpretation, the worst case is harm.

Three patterns to watch for. First, AI is bad at ruling things out. It tends to confirm the direction you've pointed it in rather than considering what else your symptoms might mean. If you say "I think this is a food allergy," it's more likely to explore that path than to ask whether it could be something else entirely.

Second, AI generates specific-sounding statistics that may not exist. "Studies show that 67% of patients with X respond to Y" sounds authoritative. The study might be real. The number might be fabricated. The citation might point to a paper about something completely different. Research across medical AI applications consistently shows that LLMs fabricate references at high rates.

Third, AI treats all conditions as equally within its competence. It'll answer a question about a tension headache with the same confidence it uses for symptoms that could indicate a stroke. It has no internal sense of "this is beyond my ability to assess safely."

What to do about it

Treat AI health information as a starting hypothesis, not a conclusion. If Iris tells you that your fatigue correlates with low iron, that's worth investigating — with a blood test, not more AI conversations. If it suggests a pattern between your headaches and a specific food, that's worth testing — with a structured elimination, not by asking the AI if it's sure.

Specific practices that help: ask for sources and check whether they exist. If a specific statistic matters for a health decision, verify it independently. If AI suggests something that contradicts what your provider told you, trust the provider. And if something feels urgent or serious, skip the AI entirely and call your doctor.

Iris has safeguards — the Supervisor reviews every response for overconfidence and factual errors. But no safeguard is perfect. Your critical thinking is the final layer, and it's the most important one.

References

1. AI Chatbots Can Run With Medical Misinformation — Mount Sinai, 2025. LLMs accept and propagate false medical claims 32-47% of the time depending on source formatting.
2. Can Medical AI Lie? Large Study Maps How LLMs Handle Health Misinformation — Mount Sinai, 2026. Follow-up study on LLM susceptibility to misinformation in clinical notes.
3. The Risks of Using AI for Health Advice — UPMC, 2025. Case documentation of treatment delay from AI chatbot misdiagnosis including TIA case.
4. Medical Misinformation in AI-Assisted Self-Diagnosis — JMIR Formative Research, 2025. Analysis of how LLMs handle misinformation in self-diagnosis contexts.
5. Large language models in medicine — Nature Medicine, 2023. AI reliability and confident error generation in medical contexts.