A visitor types a question, hits enter, and an answer starts streaming back well under a second later. It feels instant. It isn't simple — it's just fast. Here's what actually happens in that window, stage by stage, with rough timing for a typical request.
0ms — send. The visitor's message leaves the browser and hits the API. Nothing AI-related has happened yet; this is a normal HTTP request.
~15ms — the question becomes a vector. The incoming text is converted into an embedding — the same kind of numeric representation every stored document chunk already has. This step is small and fast: one short message, one embedding call, not a document to process.
~40ms — the search. That embedding gets compared against every chunk indexed for the organization, scoped so a visitor on one company's widget never sees a hint of another's data. This is vector search doing its job — not scanning everything one by one, but using an index built to find close matches fast. A shortlist of candidate chunks comes back, ranked by how closely each matches the question's meaning.
~90ms — reranking (if enabled). For ambiguous questions, a second, more careful pass re-scores that shortlist before committing to a final set of chunks. It costs a little extra time in exchange for fewer wrong-passage answers on tricky wording.
~110ms — assembling the prompt. The winning chunks, the conversation history so far, and the system instructions all get packed into a single request to the model — all of it competing for space inside one context window. Too many chunks and something useful gets crowded out; too few and the answer has nothing solid to stand on. This is the assembly step every earlier stage was building toward.
~130ms onward — generation, streamed. The model starts producing tokens, and instead of waiting for the full answer to finish before showing anything, they're streamed to the browser one at a time as they're generated. This is why an answer feels instant even though the model may still be "thinking" three-quarters of the way through it — the visitor is already reading the first sentence while the last one is still being written.
Somewhere in there — the honest exit. Not every request makes it cleanly through all of the above. If the search step comes back with nothing close enough to the question, or the retrieved chunks don't actually support a confident answer, the response that gets generated is a fallback or an escalation instead of a guess — the same behavior a grounded chatbot is supposed to have whenever the evidence isn't there. That decision happens inside this same few-hundred-millisecond window, not as some separate slow path bolted on afterward.
Why the timing matters
Every one of these stages costs real milliseconds, and every one exists because skipping it would make the answer worse: skip search and there's nothing to ground the reply in; skip reranking and ambiguous questions retrieve the wrong passage more often; skip streaming and a three-second answer feels three times slower than the same answer delivered a word at a time. The speed isn't a shortcut around the hard part — it's the hard part, done fast enough to feel like nothing happened at all.
Related reading
- How vector search actually finds the right answer — a closer look at the search step this walks past quickly.
- Chunking and embeddings 101 — how the document side of this search was prepared long before the question ever arrived.
- Tokens and context windows, explained without the jargon — the limit every prompt-assembly step has to work within.
- Anatomy of one chatbot conversation — this same pipeline, seen from the visitor's side across a full exchange.