Mary Cazanove
Mary Cazanove · COO, co-founder View Mary Cazanove's profile

RAG: preventing LLM hallucinations on business data

RAG: preventing LLM hallucinations on business data

RAG (Retrieval-Augmented Generation) has become the reference architecture for making an AI answer from your data: your procedures, your contracts, your product documentation, your FAQs. The principle fits in one sentence: search your documents, pass the relevant passages to the LLM, and let the model write its answer from those passages.

It is the right architecture. But it does not make hallucinations disappear: it moves them, and makes them harder to spot. An answer invented from your data borrows your tone, your vocabulary, sometimes even the reference of a passage that does not exist.

And a single error going around (a parameter applied wrongly, an invented clause, a procedure that does not exist) is enough for users to lose trust in the whole feature.

This article draws on the RAG projects we run with software publishers: where hallucinations come from, how to measure them before and during production, and which levers to pull to limit them. For a broader look at their causes and consequences, read our article on LLM hallucinations.

RAG in 30 seconds

Two steps, two components:

  1. Retrieval: the user’s question is turned into a search in your index, meaning your documents split into passages and represented as semantic vectors. The passages closest to the question are selected.
  2. Generation: the LLM receives the question and the retrieved passages, and writes an answer.

The key point: the LLM does not “consult” your knowledge base. It reads an excerpt selected for it. Whatever is not in that excerpt, it does not know. That is where everything is decided.

What we see in the field

On our projects, most reported hallucinations did not come from the model. They came from what it was given to read: a passage poorly selected by retrieval, incomplete or contradictory documents, or a question that even a human expert could not have answered without asking for more detail.

This changes how you fix things. Faced with a hallucination, the first reflex is often to switch models or tweak the prompt. The right first question lies elsewhere: was the right information in the context, and was it unambiguous?

Three origins, three remedies

1. Retrieval did not bring back the right information

The relevant passage was not retrieved: the wrong passage was selected, the document was never ingested, or simply the answer does not exist in your data, such as a recent decision nobody has documented yet.

Sometimes the question itself is too vague. A question like “how many days am I entitled to?” has no reliable answer without knowing which type of leave, which contract or which collective agreement applies. A colleague would ask for clarification.

The LLM has no default “I don’t know”. It produces the most plausible continuation, and it fills the gap. This is the most dangerous case: the invented answer is written in the style of your documents, so it looks like an official answer.

2. The sources contradict each other

At an HR software publisher we work with, answers on certain topics varied widely from one time to the next, and were sometimes wrong. The analysis showed the model was not to blame: on those specific points, several documents in the knowledge base contradicted each other.

Unable to decide, the model did what it could with what it received, and produced an inconsistent answer. No prompt or model tuning could fix a problem that lay in the documents themselves.

Analyzing these hallucinations revealed a clear need: agents whose role is to check the consistency of the knowledge base, in particular by spotting contradictions between documents, so that a human can resolve them before they reach the user.

3. Generation distorts information that is actually there

The information is in the context. The model paraphrases it, mixes up two passages, or invents a detail: an article number, a date, a threshold, a name. On dense texts (legal, finance, HR), this is common.

Why? An LLM is trained to produce the most plausible next word, and plausibility is not faithfulness. Staying faithful to your context is not a native goal of the model: it is a constraint you impose on it, and one you have to check.

Measuring before and during production

Without measurement, your users are the ones who discover hallucinations. By then it is too late to quantify the problem, compare solutions or commit to contractual guarantees. And measurement does not stop at launch: it continues for as long as the agent is in service.

1. A test set grounded in reality

Our recommendation: start from real usage rather than ideal questions. A good test set combines two families of questions.

  • A core set of control questions, almost identical from one RAG project to the next, that checks the limits are respected. When the information does not exist in the knowledge base, the agent must answer “I don’t know, because this information is not in my document base” instead of making something up.
  • Real user queries, exactly as they are typed: with typos, vague wording, business jargon. Your support tickets and question logs are the best source. This is where the gap between demo and production shows.

Each of these cases calls for its own technical solution: a typo, a vague question and a jargon term are not handled the same way. A RAG tested only on clean, well-phrased questions tells you nothing about its real behavior.

2. The metrics to track

The open-source framework Ragas offers metrics that have become standard for evaluating a RAG. We add one measure that is not part of it, and that we consider the most important.

  • Context precision and recall (Ragas): did retrieval bring back the right passages, and all those that were needed? These metrics evaluate retrieval.
  • Faithfulness (Ragas): is every claim in the answer supported by the retrieved context? It evaluates generation.
  • Response relevancy (Ragas): does the answer actually address the question asked?
  • Refusal rate: on control questions with no answer in the knowledge base, did the agent refuse or invent?

3. How to compute them

  • LLM-as-judge: an LLM, often more capable than the one being evaluated, scores each answer against the context, automatically and reproducibly. This is what lets you rerun the test set as often as needed.
  • Human review: manually review 10 to 20% of cases, refusals and edge cases first. This is your reference.
  • On every change: rerun the test set whenever you change chunking, the index, the model or the prompt. The before/after comparison makes every improvement measurable.

4. Continuously, once in production

  • User feedback: an answer flagged as wrong is the most valuable starting point for tracing a hallucination back to its source. By replaying it with its context, you can tell whether the problem lies in retrieval, in the documents or in generation.
  • The rate of answers with no cited source: if every answer must cite its sources, any answer without one signals a malfunction to investigate.
  • Regular audits on a sample of real conversations.

This feedback has a second benefit: it enriches the test set. Every flagged query (unexpected phrasing, poorly handled question, new topic) joins the real user queries in the test set, which keeps getting closer to actual usage.

In the end, the goal is not just a percentage: it is zero critical errors, meaning no error that leads to a wrong action (a payment, a contract, an HR decision). It is a design goal as much as a measurement goal.

Levers to limit hallucinations

1. Make the knowledge base reliable

This is the lesson from our HR project: answers are never better than the documents behind them. Contradictions, coexisting versions, outdated documents: before optimizing the pipeline, make sure the knowledge base says one thing on each topic. Control agents can do this work continuously and flag inconsistencies to a business owner.

2. Give meaning to every indexed passage

Retrieval relies on vector proximity: it matches the question with the passages whose meaning is closest. But a passage cut off from its document often loses that meaning. “This period is extended to 15 days” says neither which period nor in which case it applies.

A large part of our work consists in making every vectorized piece of content carry as much meaning as possible, in particular by enriching each passage with its surrounding context (document, section, topic) before indexing it. On top of that:

  • Structure-aware chunking: a clause must not be cut in two, a table must stay whole. Split along the document’s structure, not by character count.
  • Business metadata: document type, version, effective date, scope. Retrieval can then filter.
  • Hybrid search: semantic search alone struggles with keyword questions (procedure numbers, clause references). Combining lexical and semantic search covers both.
  • Reranking: initial retrieval is noisy. A reranking step keeps the few passages that really matter.

3. Constrain generation

Here, the prompt is a contract, not a suggestion:

  • “Answer only from the provided context.”
  • “If the answer is not in the context, say so. Do not invent.”
  • Cite the passages used.

And the shift in perspective that changes everything: knowing how to refuse is a feature. An AI that answers “I don’t have this information in our documents, here is where to look” earns its users’ trust: they know that when it does answer, the answer is grounded.

4. Cite sources

Every answer should point to the exact passage that supports it. This is much more than a UX detail:

  • Users can check in seconds. Trust is built on what can be verified.
  • Your support team traces a disputed answer back to its origin in a minute.
  • An answer without a source is an answer nobody can check.

That is the principle of our Chercher feature: every answer is sourced, and the pipeline checks its faithfulness to the documents before returning it.

5. Control the critical scope

  • On sensitive actions (HR, contracts, payments): human validation before acting. The model never acts alone.
  • Full traceability: question, retrieved context, answer, model and version. Every answer is auditable and reproducible.
  • Monitoring: in our experience, a drop in faithfulness usually points to a document problem (poorly updated, poorly ingested, contradictory) rather than an AI problem. The alert should steer you towards your data.

And an AI without your data?

An LLM without RAG knows neither your rules, nor your versions, nor your history. On your questions, it answers with the average of what it saw during training, and that average does not match your reality.

RAG changes the nature of the error: you move from a plausible, generic answer to a sourced, verifiable one. That is the difference between an AI feature you show in a demo and one you can put in a contract.

Key takeaways

  • In the field, most hallucinations come from what the model receives: a poorly selected passage, contradictory documents, a vague question. Identify the origin before touching the model.
  • Measure before and during production, with a test set grounded in reality: control questions on limits and real user queries, typos and jargon included. Rerun it on every change, and enrich it with queries spotted in production.
  • Limit hallucinations by starting with the data: a consistent knowledge base, passages enriched with their context, hybrid search, reranking.
  • Knowing how to refuse is a feature, citing sources is an architecture, and the critical scope stays validated by a human.

The quality of a RAG is decided mostly in retrieval and in the documents. On the Chercher page, see how Agora builds a document search where every answer is sourced and verified.

Bring AI into your software with Agora Software.

Let's talk