"Temperature" is one of those settings that shows up in every AI product's advanced options, usually with no explanation beyond a slider between 0 and 1. Here's what it's actually doing.
The model is always choosing between options
At each step, a language model doesn't just produce "the answer" — it computes a probability for every possible next token and picks one. Most of the time there's an obvious frontrunner ("The capital of France is" is overwhelmingly likely to continue with "Paris"), but there's almost always a long tail of less likely alternatives sitting right behind it.
What temperature changes
Temperature controls how much weight that long tail gets. At temperature 0, the model always takes the single most probable token — the same input produces the same output every time. Turn temperature up, and lower-probability tokens get a real chance of being picked, which makes output more varied and, at the extremes, less coherent.
Why 0 sounds safest but isn't always right
Temperature 0 sounds like "most accurate," but it's really "most predictable." Always picking the single most probable token can produce flat, repetitive phrasing, and it doesn't magically make the model more factually correct — it just removes randomness from how it says whatever it was going to say. A model can be extremely confident and still wrong; temperature has no opinion on that. It's a sampling setting, not a truth setting.
Where this actually matters for a support chatbot
For a chatbot answering from your documentation, low temperature is usually the right call — you want consistent, literal answers, not creative variation on the same factual question asked twice. But what actually keeps answers accurate isn't the temperature slider, it's whether the answer is grounded in a real retrieved passage in the first place. Temperature shapes tone and variety; grounding is what keeps the content honest.
Related reading
- Grounded vs. hallucinated answers — the setting that actually determines whether an answer is trustworthy.
- Tokens and context windows, explained without the jargon — the other model-level mechanic behind every response.
- 300 milliseconds: everything that happens before a chatbot answers — where generation, temperature included, fits in the full request timeline.