What hallucination means (when AI sounds sure)
A plain-language guide to AI “hallucination” as fluent, confident, unsupported output: a borrowed metaphor, not a mind seeing visions: with TruthfulQA and doors into related Easy posts.
Primary source: Lin, Hilton & Evans, TruthfulQA (ACL 2022): models often imitate popular falsehoods rather than truth: https://aclanthology.org/2022.acl-long.229/
When a chatbot invents a citation, a date, or a “study” that never existed, people often say it hallucinated. The word sounds like a mind seeing things that aren’t there.
That’s the metaphor talking. The engineering picture is quieter, and more useful.
Hallucination, in this borrowed sense, means fluent, confident, unsupported output: text that sounds right and fits the pattern of a good answer, but isn’t grounded in checked facts. The model is not dreaming. It is generating likely next pieces of text. Those pieces can be false.
The short version
- The word is a metaphor. Nothing here is conscious or “seeing visions.”
- Models are trained to continue text and follow instructions, not to only emit verified facts.
- Training piles include myths and errors; models can imitate common misconceptions.
- Fluency and a polite tone are not evidence.
- Related failures show up in images as shortcuts that look like competence.
- Tools, citations, and human checks help. None of them erase the problem.
A borrowed word, not a mind
A token is a chunk of text the model works with, roughly a word or a piece of a word. A language model’s core job is: given the tokens so far, what token is likely next? Do that again and again, and you get smooth paragraphs. Smooth is not the same as true.
Chat products wrap that engine in instruction training and a helpful voice, see What ChatGPT is (and isn’t). The practice loop underneath is still guess, measure error, nudge the numbers, How a model learns. Chatbots are one product family. They are not the whole of AI, and they are not people.
So when we say “hallucination” on this site, we mean a failure mode of fluent generation, not a psychological event.
Why it happens
Pretraining rewards patterns that look like real text. Fine-tuning rewards answers that sound helpful and on-topic. Neither stage guarantees that every claim was checked against a trusted source.
Ask for a paper title, a citation, or a biography detail that isn’t firmly in the patterns, and the model may still produce something that looks like one. Familiar shape; content that may be invented, mixed, or outdated.
Expected if the system is optimized to continue text well, not only to say “I don’t know” when it shouldn’t guess.
Myths in the training pile
Training data is a designed pile of examples, not a library the model rereads like a person, What training data means. Those piles contain careful writing and popular falsehoods, half-remembered science, and forum confidence.
The primary source for this piece is TruthfulQA, a 2022 benchmark by Stephanie Lin, Jacob Hilton, and Owain Evans. They asked questions where some humans answer wrongly because of a misconception. To do well, a model must avoid false answers that imitate common human texts. In their 2022 tests (GPT-3-class models, not today’s chat products), the best model was truthful on 58% of questions; a human baseline reached 94%. Models often mimicked popular misconceptions, and larger models were sometimes less truthful.
Punchline: imitating the crowd is not the same as telling the truth. Scale can amplify fluent falsehoods when those falsehoods are common in training.
Fluency is not a truth guarantee
A confident tone is a product of training and decoding, not a meter on reality. Polite phrasing (“Sure, here’s the study…”) can make an invented detail easier to trust, not harder.
Useful habits: treat unsupported specifics (quotes, DOIs, case numbers) as claims to verify, not receipts; prefer answers that name uncertainty; on high-stakes facts, health, law, money, safety, check with primary sources and qualified humans.
Related failure: shortcuts that look like competence
In medical imaging, a model can score well while leaning on the wrong cue, a hospital marker, a dataset quirk, instead of the disease. That story is in Can AI read medical scans?.
Different product, same lesson: a strong-looking output can ride an easy pattern that isn’t the thing you care about. In text, that pattern is “what a good answer usually sounds like.” In images, it might be a watermark. Competence on the wrong signal is still a failure.
What helps (and what doesn’t erase it)
Teams and users reduce harm with several layers. None make hallucination impossible.
- Retrieval and tools, look things up, then condition the reply on what was found.
- Citations you can click, openable links beat “trust me” prose. Fake citations still happen; click them.
- Asking for uncertainty, “Say if you’re unsure” can help hedge. It does not install a truth sensor.
- Human check on high-stakes facts, draft with the model; verify what matters.
Good products make checking easier. Bad marketing treats fluent chat as an oracle.
Easy → technical doors
| If you’re curious about… | Easy / History next step | Technical door |
|---|---|---|
| Chat products vs. “a mind” | What ChatGPT is (and isn’t) | InstructGPT and related papers via that post |
| Guess → error → update | How a model learns | Optimizers / backprop via that post’s table |
| What’s in the training pile | What training data means | Datasheets for Datasets via that post |
| Imitating popular falsehoods | This piece’s primary source | Lin et al., TruthfulQA (ACL 2022) |
| Image shortcuts that look like skill | Can AI read medical scans? | DeGrave et al. via that post |
| Pretrain then specialize | Pretrain / finetune | Self-supervision papers via History |
| Attention and scale | Transformers and scaling | Transformer / scaling papers via History |
The honest caveat
“Hallucination” here is a nickname for fluent, confident, unsupported output, not evidence of inner experience. Language models can draft, summarize, and explore ideas you can check. They can also sound sure while imitating a myth. Use them as aids with receipts and humans in the loop when it matters. On this site, Easy reading aims for clear claims, clear limits, and a path to the paper when you want to verify.