A Guide to AI Hallucinations

February 12, 2025

As AI technologies such as ChatGPT, Gemini, or other such popular LLMs become more commonly used tools, fixing the errors that can occur in their generated output becomes a more pressing concern. One of these heavily documented errors is AI hallucinations. These NLP-generated (read more about NLP here) hallucinations amuse and baffle all who see them, but also tempt a more important question – why are they occurring? The question of why – amongst others that include the commonly accepted definition of AI hallucinations and the steps towards prevention – will be explored throughout the course of this week’s article.

The standalone word ‘hallucination’ has psychological connotations. Following from this, Merriam-Webster defines a hallucination as a “sensory perception (such as a visual image or a sound) that occurs in the absence of an actual external stimulus and usually arise from neurological disturbance…”. Hallucinations as applied to NLPs are similar to neurological hallucinations in that – like a sensory perception – when they occur, they appear real and ‘factual’ despite being imaginary, or unfaithful to provided source content. As a result, this AI generated output can be difficult to recognize upon initial examination.





There are two main classifications for AI hallucinations, intrinsic or extrinsic hallucinations:

• Intrinsic Hallucinations – generated output that contradicts the factual source content. An example of this is the generated output “Cats are three-legged land mammals" as compared to the source content “Cats are four-legged land mammals”.


• Extrinsic Hallucinations – generated output that cannot be supported or contradicted by available source content. An example of this includes any output that doesn't match what is included in source content but could hypothetically be concluded as a result of it.

These different types of AI hallucinations seem to occur for several reasons, the first of which boils down to the data the LLM is trained on. Following from the full name for LLM – large language models – these models are trained on large amounts of internet data. Not all this data is factual, and it can also contain the societal biases of its authors. All of these inaccuracies are likely to spill into the generated response.

Another of the probable reasons for hallucinations are limitations that have to do with the design of generative models. Generative AI models are designed from advanced autocomplete technology, similar to the feature that can be found on a smartphone. As such, the goal of LLMs is to predict the next likely word based on observed data. These models are designed only to generate the most likely content, and so may generate output that is non-factual, but seems reasonable on the surface.

The last of the reasons for the occurrence of hallucinations also results from the current design of generative models. These models are not designed to understand what verifiable versus inaccurate knowledge looks like. Fixing this problem isn’t as simple as feeding LLMs only accurate data, as the built-in generation the model’s design hinges on ensures that potentially inaccurate data is an inevitability.

Growing awareness of the possibility of hallucinations in AI has helped towards work to reduce inaccuracies in the AI response. Some of the strategies currently employed to prevent these hallucinations include, firstly, advanced prompt engineering. Prompt engineering is the process of creating precise, context-aware prompts, or requests to the generative AI, to produce the most ideal outputs.

In addition, retrieval-augment generation (RAG) techniques can also be put to play in the fight against AI hallucinations. Rather than leaning on just LLMs internal knowledge, RAG dynamically fetches relevant information from external knowledge bases prior to generation of responses. This is in efforts to make the LLM output more accurate.

The last of the strategies for hallucination prevention to be covered is reinforcement learning from human feedback (RLHF). This type of learning hinges on human critique within the training process, ensuring the output generated by LLMS fits human expectations. Outputs from the LLM are ranked by humans according to their perceived quality and refined based on this. This helps the ability of LLM models to generate responses that are far more factual.

AI hallucinations represent one of the most significant hurdles in the development of LLM-dependent AI tools. However, these errors in AI output have been shown to be mitigatable and removable through use of several techniques, such as the ones covered in today's article. Understanding the causes and the types of hallucinations is crucial in approaching solutions that improve LLM accuracy and reliability. With ongoing work in this area, AI technology promises greater capability and a future of possibilities.