How to Stop an AI Agent From Hallucinating or Going Off-Topic

AI agents "hallucinate" when they're asked something they don't have the facts for and fill the gap with something plausible. You prevent it by giving the agent specific facts in a knowledge base, telling it exactly what to do when the answer isn't there, limiting what it's allowed to promise, keeping it on topic, and testing and monitoring real conversations.

The first time one of my test agents confidently quoted a price that didn't exist, I realised the problem wasn't the model. It was me: I'd given it a vague knowledge base and no instruction for "I don't know". Here's the approach I use now, and that I built into ReplySetter.

Why AI agents make things up

Language models are trained to produce fluent, helpful-sounding answers. When the facts aren't in front of them, they don't naturally say "I don't know". Common triggers in sales conversations:

• Missing facts: a question your knowledge base doesn't cover.
• Vague documents: "pricing varies" invites the model to guess.
• Contradictory documents: an old price list next to a new one.
• Pressure in the instructions: "always answer the lead's question" with no exception.
• Open-ended requests: "can you write me a cover letter?"

1. Ground it in specific facts

Put real, specific answers in the knowledge base. See how to train an AI chatbot on your business. The more specific the source, the less room for invention.

2. Tell it what to do when it doesn't know

The single most effective line I've found:

Only answer factual questions about [business] from the knowledge base. If the answer isn't there, say you'll check with the team, and hand the conversation over. Never guess prices, availability, policies or results.

3. List what it must never do

Explicit limits help:

• Never promise discounts, refunds or deals that aren't in the knowledge base.
• Never guarantee results.
• Never give medical, legal or financial advice.
• Never invent features, integrations or availability.
• Never make up testimonials or statistics.

4. Keep it on topic

A public chat or DM agent will be tested by people asking it to write essays, solve homework or "ignore your instructions". Without limits, you're paying for a free ChatGPT for strangers.

What works:

• A scope instruction: "Only help with questions about [business] and booking. Politely decline anything else."
• Repeat-offender handling: after a couple of off-topic attempts, stop replying.
• Prompt-injection awareness: treat "ignore previous instructions" as off-topic.

ReplySetter's characters stay on topic by default: they decline unrelated requests and stop answering repeat offenders. See characters.

5. Use structure, not one giant prompt

Agents with one long prompt are more likely to drift. A structured flow, where each step has a narrow goal, reduces room for improvisation. See how to write an AI setter prompt.

6. Choose the right model

Newer, stronger models generally follow instructions better and admit uncertainty more often. For steps where accuracy matters most (pricing, objections), use a stronger model; for simple routing, a cheaper one is fine. See the best AI model for sales conversations.

7. Test the edge cases

Before going live, try to break it:

• Ask about a service you don't offer.
• Ask for a discount.
• Ask for a guarantee.
• Ask for medical or legal advice.
• Ask it to write a poem.
• Tell it to ignore its instructions.
• Ask the same question in five different ways.

Fix each failure at its source: a missing fact, a vague document or a weak instruction.

8. Approve before it sends, at first

Start in a mode where you approve every reply. You'll spot problems before any lead does. ReplySetter's suggest and draft modes do this. See human in the loop.

9. Monitor real conversations

Even after testing, review a sample every week:

• Answers that weren't in the knowledge base.
• Promises you didn't authorise.
• Off-topic conversations.
• Conversations where the lead seemed confused.

When something's wrong, the reply's trace should show which knowledge chunks it used and which instruction it followed. In ReplySetter's Test & Debug, you can see this for every reply, and rate replies so you can track improvements.

A quick checklist

• Specific facts in the knowledge base
• "If you don't know, hand off" instruction
• Explicit "never" list
• Scope limits and off-topic handling
• Structured flow
• Capable model for important steps
• Edge cases tested
• Approvals at launch
• Weekly review