Two words show up constantly around AI products — "tokens" and "context window" — and most explanations of them are written for people who already understand them. Here's the plain version.
What a token actually is
A token is a chunk of text an AI model reads or writes at a time — not quite a word, not quite a character. "Chatbot" might be one token. "Unbelievable" might split into two or three. As a rough rule of thumb, a token is about four characters of English text, or roughly ¾ of a word.
Models don't read sentences the way you do — they read a sequence of tokens, one after another, predicting the next one. That's the whole mechanism under the hood. It's also why AI usage is priced in tokens rather than in words, requests, or minutes: tokens are the actual unit of work the model does.
Why this affects cost and speed
More tokens in, more tokens out, more it costs and the longer it takes — because the model is processing more units either way. A one-sentence question costs less to answer than a request to summarize a ten-page document, and a short, direct chatbot answer is cheaper to generate than a long, meandering one. This is part of why grounding a chatbot in specific retrieved passages is more efficient than dumping an entire knowledge base into every request — fewer tokens in, a more focused answer out.
What a context window is
A context window is the maximum number of tokens a model can "see" at once — your message, the conversation history, any retrieved document passages, and the model's own instructions, all counted together against one limit. Once that limit is full, something has to give.
This is the mechanical reason a chatbot can seem to forget something you said earlier in a long conversation: the earliest messages get pushed out of the window to make room for new ones, the same way an overfull inbox eventually starts dropping the oldest emails first. It's not the model being forgetful in a human sense — it's a hard ceiling on how much text fits in one pass.
Why this matters for a support chatbot specifically
A context window has to hold more than the conversation itself. Every retrieved passage from your docs, chunked and pulled in to ground the answer, also counts against the same limit. This is one more reason chunking well matters: retrieving three tightly relevant paragraphs leaves far more room in the context window than retrieving one bloated, unfocused one — and leaves more room for the actual conversation history to stick around.
The takeaway
Tokens are the unit AI does its work in, and a context window is the ceiling on how many can be in play at once. Neither is something you need to manage by hand day to day, but understanding both explains a lot of chatbot behavior that would otherwise look mysterious — why cost scales with conversation length, why very long conversations can lose earlier context, and why retrieving the right passages (not just more of them) is what actually makes answers both accurate and efficient.
Related reading
- What is RAG, explained for non-engineers — the retrieval pipeline that decides what fills the context window.
- What does it actually cost to run an AI chatbot? — how token usage translates into a real bill.