How to stop an AI chatbot making things up: grounding, guardrails and hand-off

The failure people fear most from an AI assistant is the confident wrong answer: a refund policy that does not exist, an opening time from three years ago, a booking that was never made. It is avoidable, but not by picking a better model. It is avoided by four design decisions that are made before the model is chosen.

1. Ground every answer in your content, and only your content

A general-purpose model knows a great deal and none of it is about your business. An assistant should answer from a defined set of sources — your policies, FAQs, product data, help articles — retrieved at the moment of the question, and be instructed to say “I don’t know” when the sources do not cover it. This is retrieval-augmented generation in practice: the model writes, but the facts come from documents you control.

The discipline this forces on the business is healthy: if the assistant cannot answer a question, it is usually because nobody has written the answer down. The monthly report of unanswered questions is a to-do list for your knowledge base.

2. Give it actions through your systems, not through its memory

“Where is my order?” cannot be answered from a document. It needs a lookup in the order system, with the customer verified first. Assistants that can act — check an order, book a slot, create a ticket — do it through tools that call your systems with the permissions you grant. The model decides which tool to use and what to say; the data comes from the system of record. Anything that costs money or cannot be undone gets a confirmation step.

3. Set guardrails that are enforced, not requested

Instructions in a prompt are requests. Guardrails are enforced: topic limits (it will not discuss competitors’ pricing or give medical advice), output checks (a claim about a policy must cite a source document), rate and cost limits per conversation, and a list of situations that always hand over to a person — complaints, anything legal, anything the customer says is urgent. We test these with a list of trick questions and adversarial prompts before launch, and re-run the list when anything changes.

4. Make the hand-off graceful

The best assistants know their edges. When the question is outside scope, the customer is frustrated, or the confidence is low, the assistant says so and hands over to a person with the transcript and a one-line summary — into your inbox, help desk or WhatsApp. The customer never repeats themselves, and the team sees exactly what was said. Measure the hand-off rate; if it is zero, the guardrails are too loose, and if it is half, the knowledge base has gaps.

The pre-launch test we run

• Fifty real questions from your inbox, with the expected answer and source.
• Twenty questions the assistant must not answer, and what it should say instead.
• Ten attempts to make it invent a policy, discount or promise.
• Every action tested end to end in a sandbox: booking, lookup, ticket, hand-off.
• A cost ceiling per conversation and a monthly budget alert.

What good looks like after launch

Resolution rate up, hand-offs meaningful, and a monthly report that lists the questions it could not answer next to the documents that should answer them. That report is how the assistant keeps getting better without anyone touching the prompt. It is also how we run every assistant we build.