"AI confidently makes up facts, citations, package versions, or other details, so it cannot be trusted."
What this page actually tests
Bare chatbot output cannot be trusted for factual or high-stakes claims without source verification and domain-appropriate review.
Wording note: Cannot be trusted is too broad when applied to every constrained system and low-stakes task. It is accurate for unchecked factual output, which is the concern tested here.
Quick verdict: Claim confirmed
Unchecked factual output is not trustworthy.
Confirmed. A bare chatbot is not a trustworthy factual source. Grounding, abstention, testing, and expert review can make a bounded workflow trustworthy enough for a defined use.
Why people repeat it
The concern is common because a fluent false citation or technical detail can look credible, and users may not discover the error until it has already affected a decision or publication.
Evidence
What the sources support
Fact: OpenAI's September 2025 hallucination explainer says language models can generate plausible false statements and gives SimpleQA examples where one model had a 75% error rate while another had a 26% error rate with far more abstention.
Baseline: The useful comparison is not AI versus perfect truth. It is ungrounded guessing versus systems that reward abstention, cite sources, retrieve context, and get reviewed.
Evidence conclusion: The evidence proves unchecked answers deserve skepticism. It does not prove every AI-assisted workflow is untrustworthy.
Source: Why language models hallucinate
Fact: A 2026 arXiv audit checked 111 million references across 2.5 million papers and estimated 146,932 hallucinated citations in 2025 alone.
Baseline: Scientific citations are unusually easy to verify compared with ordinary prose, so this is stronger evidence than funny screenshot anecdotes. It is still not a universal error rate for every AI task.
Evidence conclusion: Fake references are a real, measurable failure mode. Any article, legal memo, or research summary that uses AI needs citation checks. Groundbreaking stuff: read the sources.
Source: LLM hallucinations in the wild
Fact: GOV.UK describes RAG as grounding model responses in external knowledge sources, while a hallucination survey notes that retrieval-augmented systems still have limits and open questions.
Baseline: A bare model answering from weights is different from a system retrieving vetted documents, showing citations, and failing closed when support is absent.
Evidence conclusion: Grounding lowers risk but does not guarantee correctness. Trust must be evaluated across retrieval, generation, abstention, verification, and domain review.
Source: AI Insights: RAG Systems; A Survey on Hallucination in Large Language Models
Fact: TechTarget's 2025 explainer notes hallucinations can be subtle, including nonexistent package versions or flawed summaries, and that human oversight and domain expertise are often needed to catch them.
Baseline: High-stakes domains need stronger review than low-stakes brainstorming, drafting, or classification tasks where errors are cheap to spot and fix.
Evidence conclusion: The trustworthy answer is conditional: use AI where verification is built in, and keep it away from solo final authority in expensive-error situations.
Source: Why does AI hallucinate, and can we prevent it?
Source balance
Checked both sides before calling it.
Supports the claim
Why language models hallucinate - OpenAI documents plausible false statements and high error rates in a SimpleQA comparison.
LLM hallucinations in the wild - The citation audit estimates 146,932 hallucinated scientific references in 2025.
A Survey on Hallucination in Large Language Models - The survey frames hallucination as a major reliability concern for real-world information retrieval.
Challenges or narrows it
AI Insights: RAG Systems - GOV.UK describes RAG as grounding model responses in external knowledge sources for more reliable, domain-specific answers.
Why does AI hallucinate, and can we prevent it? - The explainer lists mitigation steps such as RAG, fine-tuning, output filtering, prompt design, and human oversight.
Baseline context
Why language models hallucinate - Compares models that guess with models that abstain more often, which matters for trust.
AI Insights: RAG Systems - Separates generic model behavior from retrieval-grounded domain systems.
Why does AI hallucinate, and can we prevent it? - Separates high-stakes uses from lower-stakes workflows where review and correction are practical.
Assessment: The core concern is confirmed for bare factual output. Trustworthiness can be engineered at the workflow level, but that is an added control system rather than evidence that an unchecked answer deserves trust.
Visual evidence
The comparison behind the verdict.
Outcome composition
SimpleQA: errors versus abstention
Each model's SimpleQA outcomes divided into error, abstention, and the correct remainder.
What this shows: The lower error rate comes with far more abstention, while the derived correct shares are similar. Looking only at errors hides the difference between guessing and declining to answer.
Unit: %
gpt-5-thinking-mini
26 %52 %22 %
OpenAI o4-mini
75 %24 %
ErrorAbstentionCorrect
gpt-5-thinking-mini / Error
26 %
gpt-5-thinking-mini / Abstention
52 %
gpt-5-thinking-mini / Correct
22 %
OpenAI o4-mini / Error
75 %
OpenAI o4-mini / Abstention
1 %
OpenAI o4-mini / Correct
24 %
Source: Why language models hallucinate
OpenAI presents the error and abstention values as example SimpleQA metrics, not universal hallucination rates. Correct shares are the remainder after subtracting error and abstention from 100%.
Where critics may still have a point
For legal, medical, financial, safety, hiring, or academic citation work, unchecked hallucinations can cause real damage.
Citations in an AI answer are not evidence until the linked source exists and says what the answer claims.
RAG can retrieve the wrong thing, miss the right thing, or pass good context to a model that still mangles the answer.
Final verdict: Claim confirmed
Unchecked factual output is not trustworthy.
Conclusive evidence shows LLMs can produce confident falsehoods and fake citations at scale. The evidence also shows mitigation changes risk. AI is not trustworthy as an unchecked oracle; it can be trustworthy enough inside bounded, sourced, reviewed workflows.
Why this verdict: The bounded trustworthiness claim is supported across vendor research, government guidance, and citation studies: unchecked factual output can be confidently false and is not a source that should be trusted without verification.
How this was confirmed: Vendor research, a large citation study, government RAG guidance, and a broad hallucination survey independently support the no-unchecked-trust conclusion. Grounded and reviewed workflows provide the challenge baseline; their improvement is treated as a condition for trust, not proof that bare output is reliable.
