Hallucinations
Do hallucinations make AI useless?
Errors matter, but usefulness depends on task, verification, and failure cost.
"AI hallucinates, so it is useless."
Not reliable enough for everything. Useful for some things.
Hallucinations are a real limitation. "Therefore useless" is the unsupported leap.
Why people repeat it
The claim is popular because hallucinations are visible, funny, and sometimes dangerous. A fake citation makes the whole system look like a calculator that occasionally invents Thursday. Still, reliability is not binary.
What the sources support
Fact: The GPT-4 technical report describes GPT-4 as a Transformer model pretrained to predict the next token and notes factual reliability limitations despite improved performance.
Baseline: The comparison is not "perfect truth machine" versus "trash"; it is model output with verification versus workflows where errors are unacceptable.
Evidence conclusion: The evidence proves blind trust is a bad idea. It does not prove every assisted drafting, coding, classification, or summarization use is worthless.
Source: GPT-4 Technical Report
Fact: Nature research on semantic entropy reported hallucination-detection performance around AUROC 0.790, beating several baselines, while also noting limits for some error types.
Baseline: A measurable detection method is different from pretending hallucinations are either solved or impossible to manage.
Evidence conclusion: The evidence supports testing and mitigation in specific workflows, not the claim that the entire technology category has zero utility.
Source: Detecting hallucinations in large language models using semantic entropy
Fact: NIST lists confabulation and information integrity as generative AI risks to govern alongside privacy, security, bias, and misuse.
Baseline: Risk-management frameworks do not treat every risk as a ban; they match controls to use case and harm level.
Evidence conclusion: The conclusive lesson is boring but useful: high-stakes use needs controls, and low-stakes use still needs verification proportional to the task.
Source: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
Source balance
Checked both sides before calling it.
Supports the claim
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile - NIST treats confabulation and information integrity as real generative AI risks.
- Detecting hallucinations in large language models using semantic entropy - Research documents hallucination behavior and methods to detect uncertainty.
Challenges or narrows it
- GPT-4 Technical Report - The model shows useful capabilities alongside documented limitations.
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile - The risk-management framing implies mitigation and bounded use, not total uselessness.
Baseline context
- GPT-4 Technical Report - Provides capability and limitation context.
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile - Provides risk-management categories for high-stakes use.
Assessment: Hallucinations are a confirmed risk, but the claim is misleading because reliability failures constrain use cases rather than making every use useless.
Where critics may still have a point
- For legal, medical, financial, or safety-critical advice, a confident wrong answer can be unacceptable without expert review.
- Retrieval, citations, and system prompts do not automatically solve hallucinations.
- Low-stakes usefulness does not justify high-stakes deployment without measurement.
Not reliable enough for everything. Useful for some things.
Conclusive evidence shows current models can produce confident falsehoods and need verification. It does not show they are useless in bounded workflows where outputs are checked, constrained, or low-cost to correct.
Verdict color: Hallucinations are a real reliability failure and high-stakes blocker, but bounded workflows, source grounding, refusal, review, and low-cost correction change the comparison. The useless label is broader than the evidence.
Sources
- GPT-4 Technical Report - Model capability, training-objective, and limitation framing.
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile - Generative AI risk categories and mitigation framing.
- Detecting hallucinations in large language models using semantic entropy - Hallucination detection research and reliability measurement context.