It seems like it should be the easiest behavior to get right: if the chatbot doesn't know, it should just say so. In practice, this is one of the harder things to get an AI system to do reliably — and the reason has less to do with the model's abilities than with how it was trained to behave.
Helpful-sounding answers get rewarded during training
After initial training, most models go through a tuning stage where human reviewers rate responses, and those ratings shape what the model learns to produce more of. A confident, complete-sounding answer generally rates better in the moment than "I don't know" — even when "I don't know" is the honest, correct response — because it looks more helpful on the spot. The model isn't being deceptive; it's optimizing for exactly the signal it was given.
"I don't know" has to be earned, not assumed
For a model to output "I don't know" and have that be trustworthy, something has to actually check whether the information is available — otherwise it's just another confident-sounding phrase competing with all the other plausible outputs. That check is what grounding a chatbot in retrieved source material provides: an explicit signal of "here's what was actually found" versus "nothing relevant turned up," instead of leaving the model to guess at its own certainty.
What it actually takes to fix this
Getting a chatbot to reliably decline unanswerable questions takes more than a polite instruction in the system prompt. It takes an architecture where the model is only ever asked to answer from what was retrieved, and is explicitly told what to do when retrieval comes back empty or weak — turning "admit uncertainty" from a hope into a designed fallback path, the same one described in anatomy of one chatbot conversation.
Why this is worth the engineering effort
A chatbot that occasionally says "I don't know, let me get someone" isn't underperforming — it's doing the one thing that makes every other answer it gives more trustworthy. The alternative, a chatbot that always sounds sure, is the version that actually erodes trust the first time it's confidently wrong.
Related reading
- What happens when you ask an AI a question it was never trained on — the underlying mechanism this fallback exists to catch.
- Grounded vs. hallucinated answers — the check that makes an honest "I don't know" possible.
- Anatomy of one chatbot conversation — this fallback shown happening in a real exchange.