Hallucinations, LLMs and conversational AI

What is a hallucination?

Hallucinations are “false or misleading responses that appear plausible”, presented with confidence. Concrete examples: fabricated statistics, invented quotations, fictional historical events, names of places or people created from thin air.

Understanding large language models

LLMs are deep neural networks trained on massive datasets to predict the next word. Their key characteristics:

• Excellence in generating syntactically correct text
• Absence of genuine understanding or semantic knowledge
• Computation-based rather than comprehension-based function
• Training requiring billions of parameters and thousands of specialised servers

The origins of LLM “knowledge”

Why LLMs seem magical

Causes of hallucinations

• LLMs’ inability to distinguish probable sequences from accurate ones
• Insufficient, biased or misleading training data
• Ambiguous or imprecise prompts

Research suggests hallucinations are inherent to LLM technology: “Hallucination is Inevitable: An Innate Limitation of Large Language Models”.

Hallucinations in professional settings

Rightly so, generative AI (that is, LLM-based AI) is increasingly used in professional settings.

In particular, to offer simple conversational access to business knowledge bases (see RAG architectures, Retrieval-Augmented Generation).

But these solutions are not perfect, and their errors can have significant consequences. Let’s take a study in a legal context, for example:

• “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools” by Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning and Daniel E. Ho (Stanford and Yale universities)
• “RAG juridique et hallucinations” by Raphaël d’Assignies

This latter study proposes a typology of legal RAG system errors, and a ranking of the errors encountered during comparative testing:

1. Retrieval errors: the relevant documents are not retrieved
2. Interpretation errors: the model misreads the retrieved documents and draws incorrect conclusions from them
3. Synthesis errors: the model incorrectly combines information from multiple documents, for example mixing facts from unrelated situations
4. Contextualization errors: the model lacks the necessary legal context (a legal subtlety, for example)

Possible consequences of hallucinations

To this end, let’s list a few possible implications in professional settings:

1. Spreading false information: propagation of incorrect information, which can affect strategic decisions or planning, particularly problematic in fields where accuracy is critical, such as journalism, scientific research or medicine. In business software, the same risk shows up on the HR or payroll side: an agent that invents a non-existent leave proration rule produces a real payroll error, not just an anecdote
2. Legal risk and liability: erroneous data generated by an LLM could lead to legal action if the information provided caused damage or financial loss, for example financial advice based on incorrect data
3. Loss of trust: professionals and customers may lose trust in the organization or tool using LLMs if they discover it frequently generates and shares incorrect information
4. Ethical and social impact: spreading false information in sensitive contexts such as public health, politics or social sciences can have serious ethical consequences

To minimize these consequences, it is crucial to put in place mechanisms for verifying and validating the information generated by LLMs, and also to train users to identify and correct potential errors.

Mitigation strategies

Current approaches include:

• Adding semantic capabilities through intent recognition
• Contextual coherence analysis
• Mixture of Experts (MoE) architecture
• Comparing outputs across multiple LLMs
• Human annotation for model refinement
• Retrieval-Augmented Generation (RAG) systems

Key takeaway

LLMs revolutionise conversational AI through fluent text generation, but require robust verification mechanisms and user education, particularly in precision-critical sectors such as law, finance and healthcare. The legal RAG study cited above makes the point precisely: 45% of errors come from naive document retrieval and 39% from faulty reasoning, versus under 3% from pure sycophancy. Most hallucinations don’t come from the LLM “making things up”, they come from the surrounding architecture failing to hand it the right information at the right time.