Last month I was reviewing a client's internal AI assistant — one they'd been using for six months to answer HR policy questions for staff. It looked clean: fast responses, professional tone, it cited the right sections of the employee handbook most of the time.
Then I asked it a question about parental leave.
The model gave a confident, well-formatted answer that was factually wrong in two places. The leave entitlements it quoted had been updated in a policy refresh eight months earlier, and the AI hadn't been updated to reflect that. More troubling was how it answered: no hedging, no "please verify this with HR", just a direct response that any reasonable person would take at face value.
If an employee had made a financial decision based on that answer, there would have been a real cost to a real person. This is the hallucination problem — and it's more common than most vendors admit.
What Is an AI Hallucination?
An AI hallucination is when a language model produces a response that sounds confident and coherent, but is factually incorrect, fabricated, or misleading.
The term comes from hallucination in psychology — perceiving something that isn't there. In AI, the model isn't lying or malfunctioning; it's doing exactly what it was trained to do, which is generate plausible-sounding text. The problem is that plausible and accurate are not the same thing.
Large language models don't have a database of facts they look up. They generate text based on patterns learned from training data. When asked something outside what they were trained on, or when context is ambiguous, they can produce an answer that sounds right but isn't. There's no built-in "I don't know" instinct — you have to engineer it in.
Why This Matters More Than You Think
Most conversations about AI risk focus on dramatic failures: a model producing something offensive, or a security breach. Those get headlines. The quieter risk is more insidious.
Hallucinations in business tools often:
- Go undetected because the answer sounds plausible and the reader lacks the context to verify it
- Compound over time when AI-generated content is used as input to other decisions or documents
- Erode trust suddenly — a team uses the tool for months without issue, then one bad answer causes a significant problem and confidence collapses entirely
- Create compliance exposure in regulated industries where the accuracy of information matters legally
In New Zealand, businesses operating under the Privacy Act 2020, health regulations, or financial services legislation face real risk if AI-generated guidance leads to incorrect decisions. "The AI told me" is not a defence.
The Most Common Hallucination Scenarios
Not all AI use cases carry the same risk. Here's where I see hallucinations cause the most harm in practice.
Using AI on stale or missing information
If your AI assistant hasn't been updated with your latest documents, it will answer based on what it has — or make something up to fill the gap. That's what happened with my client's HR tool. The underlying model wasn't broken; it just hadn't been given the updated policy. This is one of the most common and most avoidable failure modes.
Asking for specifics the model doesn't have
Asking an AI "What was our Q3 revenue?" or "Who signed off on that supplier contract?" when it doesn't have access to that data is an invitation for fabrication. The model may produce a number or a name that sounds plausible because it's trained to produce complete, coherent responses.
Long chains of reasoning
The more steps a model has to reason through, the more error compounds. Asking an AI to analyse a clause in a contract, apply it to a specific scenario, and then produce a recommendation is a three-step process where each step can introduce inaccuracy.
Domain-specific terminology used loosely
Legal, medical, financial, and technical language can be interpreted multiple ways. A model trained on general text may apply the common usage of a term where the precise industry definition is needed. The response will sound confident either way.
How to Design Around Hallucinations
You cannot eliminate hallucinations from AI systems — but you can design processes that catch them before they cause damage.
Here's how I approach this with clients.
1. Match the use case to the risk level
Low-stakes tasks — drafting a first version of a document, summarising a meeting, generating options for discussion — are appropriate for AI with light oversight. High-stakes tasks — compliance advice, customer-facing information, financial summaries — need human review at every step.
Map your AI use cases by consequence. If the output is wrong, how bad is it? Let that answer determine your review process. A mistake in an internal brainstorm costs nothing. A mistake in a customer-facing document or a staff policy answer can cost a great deal.
2. Give the model the information it needs
Most hallucinations happen when the model lacks specific context to answer accurately. Retrieval-augmented generation — where the model is given relevant documents before it responds — significantly reduces hallucination in domain-specific applications. Rather than relying on what the model learned during training, it draws from the actual documents you've provided.
If you're building or buying an AI tool for internal use, ask how it handles knowledge updates. A tool that relies solely on its base training, with no mechanism for document retrieval, is a higher-risk choice for operational questions.
3. Instruct the model to say when it doesn't know
This sounds simple, but it's surprisingly effective. A system prompt that explicitly tells the model to respond with "I don't have enough information to answer that reliably — please check with [person or document]" changes how it behaves on edge cases dramatically.
Many off-the-shelf AI tools don't do this by default. It's worth asking vendors about their approach to uncertainty handling, or configuring it yourself in tools that allow prompt customisation.
4. Build verification into the workflow
For any AI output that influences decisions, design the process so that verification is a required step, not an optional one. This doesn't have to mean a human reads every word — it might mean a confidence score threshold, a structured spot-check on a sample of outputs, or a second AI pass to look for inconsistencies.
The key is that verification is part of the process by design, not something that happens when someone has time.
5. Monitor for drift over time
An AI system that works well at launch can degrade as your business changes and the underlying model or its context doesn't keep pace. Build in periodic review — monthly or quarterly depending on risk level — where someone actually tests the AI against known-good answers.
This is especially important for any AI that gives advice, answers factual questions, or generates content that goes outside your organisation without further review. What passes at month one may fail at month six.
What to Do If Your Team Has Already Lost Trust
One bad hallucination can undo months of adoption. If your team has been burned, rebuild slowly.
Start by being transparent about what went wrong and what's changed. Retrain users on the appropriate use cases and the review process. Consider limiting the tool to lower-stakes tasks until confidence is rebuilt, then expanding scope as the track record is reestablished.
Don't defend the tool when it clearly failed. The appropriate response to a hallucination is to acknowledge it, fix the process, and move on. Teams that see leadership take the failure seriously are far more likely to continue engaging with the tool once the process is improved.
Getting This Right From the Start
The businesses that get the most value from AI are not the ones who deploy it everywhere and hope for the best. They're the ones who choose use cases carefully, design honest review processes, and stay realistic about what AI can and cannot do reliably.
The tools are genuinely useful — but they require the same critical thinking you'd apply to any information source. An AI assistant is not an oracle. It's a capable but fallible system that works best when paired with clear scope, good context, and a process that catches errors before they matter.
If you're early in your AI deployment and want to think through risk before it becomes a problem, that's exactly what an AI consulting engagement can cover. Not selling you a tool, but helping you understand where AI fits, what to watch for, and how to build something your team can actually trust.
If you're already dealing with a hallucination problem in an existing system, get in touch — it's usually fixable, and it's almost never as complicated as it first appears.