Ask a person something they genuinely don't know, and they'll usually tell you so. Ask a language model the equivalent question, and — left to its own devices — it often won't. Understanding why gets at something fundamental about how these models work.
There's no internal "I don't know" signal
A language model's core job is predicting the next most plausible token, over and over, given everything before it. There's no separate step where it checks "do I actually have reliable information on this?" before answering — plausibility and correctness aren't the same computation, and only one of them is what the model was built to optimize for. A confident-sounding sentence is often just as easy for the model to produce whether it's grounded in something real or not.
What it does instead
When a question falls outside a model's training data — a recent event past its cutoff, a detail specific to your company that was never public, an edge case nobody wrote about — the model still has to produce something. It falls back on the closest pattern it has: a plausible-sounding, stylistically confident answer assembled from adjacent knowledge, even when that answer isn't actually correct. This is what hallucination is, mechanically — not lying, but pattern-completion with nothing solid underneath it.
Why this is a design problem, not a smarts problem
A bigger, more capable model doesn't fix this on its own — it just gets better at making the guess sound more convincing. The actual fix isn't a smarter model, it's giving the model something real to check against before it answers, which is the entire premise behind retrieval-augmented generation: find the relevant source material first, then answer only from that, instead of asking the model to answer from memory alone.
The takeaway
A model with no source to draw from isn't broken when it produces a fluent, wrong answer — it's doing exactly what it was trained to do: continue the pattern as plausibly as possible. Whether that's a problem depends entirely on whether something outside the model — like a retrieval step — is there to catch it first.
Related reading
- What is RAG, explained for non-engineers — the fix for exactly this failure mode.
- Grounded vs. hallucinated answers — how to tell which kind of answer you're looking at.
- Why AI chatbots are bad at saying "I don't know" by default — the related question of why models don't just say so.