If a grounded chatbot is only as good as the docs behind it, the fastest way to find out where our own docs were thin was to point Lumen Chat at them and start asking questions like a confused new customer.
So that's what we did: crawled our help docs and blog, connected them as a source, and spent an afternoon asking the kind of questions real visitors ask — not the tidy ones we'd write if we were testing our own product to look good.
What it handled fine
Anything covered by a dedicated post answered cleanly, with a citation back to the source. "How do I connect Google Drive," "what happens if a sync fails," "can I customize the widget's color" — all fine, because we'd already written about them.
Where it said "I don't know"
A few questions came back with an honest fallback instead of a made-up answer:
- "Can I use Lumen Chat in a language other than English?" — never documented, because we'd never explicitly decided the answer ourselves. Turns out the underlying models handle it fine; we just hadn't written it down.
- "What happens to my data if I cancel?" — covered vaguely in the terms page, but not in a way the chatbot could retrieve a confident, specific answer from.
- "Does the widget slow down my site?" — a completely reasonable objection nobody had bothered to address head-on anywhere in our docs.
- "Can two team members edit the same document source at once?" — an edge case real enough that a customer would hit it, obscure enough that we'd never written about it.
None of these were embarrassing exactly — but every one was a real question a prospective customer would ask before trusting us with their support inbox, and our own chatbot couldn't answer any of them from what we'd given it.
What we did about it
Each gap became either a short FAQ addition to an existing page or, where it warranted more than a paragraph, its own post. That's the same process we tell customers to follow when their own chatbot comes back with too many fallbacks — we just hadn't turned it on ourselves until now.
The takeaway
This is the argument for grounding in one sentence: it doesn't just stop a chatbot from making things up, it hands you an honest, ranked list of exactly what your documentation is missing — no support ticket backlog required to find out. If you want to see this happening on your own content instead of ours, connect a source and ask your chatbot the questions you're afraid it can't answer.
Related reading
- Anatomy of one chatbot conversation — the same honest-fallback behavior, walked through turn by turn.
- We tried to jailbreak our own chatbot — another week we spent testing this product on itself.