The Case for the Uncertain AI: Why Chatbots Should Say "I'm Not Sure"

In May 2023, attorney Steven Schwartz submitted a legal brief to a federal court in New York that cited six court cases supporting his client's personal injury claim against Avianca Airlines. The problem? None of the cases existed. Schwartz had used ChatGPT to conduct his research, and the AI confidently invented fictional legal precedents complete with realistic case names, detailed rulings, and assertions that the citations could be verified in major legal databases.

The judge was not amused. Schwartz faced sanctions, and the incident became an early warning about the dangers of AI hallucination, the tendency of language models to generate false information with the same tone of authority they use for verified facts.

Since then, the problem has only grown more visible. In October 2025, Deloitte submitted a 440,000 Australian dollar report to the federal government containing fabricated academic sources and a fake quote from a court judgment. The following month, investigators discovered that a health workforce plan commissioned by Newfoundland and Labrador contained at least four citations to research papers that did not exist. In 2025 alone, judges worldwide issued hundreds of decisions addressing AI hallucinations in legal filings, accounting for roughly 90 percent of all known cases involving this problem to date.

What makes these incidents particularly concerning is not that AI systems make mistakes. All systems make mistakes. The problem is that these mistakes arrive dressed in the language of certainty. The chatbot that fabricated nonexistent legal cases delivered its inventions with the same confidence it would use to explain the laws of gravity. There was no hesitation, no qualification, no signal to the user that something might be wrong.

OpenAI's own researchers have acknowledged that hallucinations represent a fundamental challenge for all large language models. GPT-5 has significantly fewer hallucinations than earlier versions, particularly when using reasoning capabilities, but the problem persists even in the most advanced systems available. The question worth asking is why AI continues to generate confident falsehoods rather than simply admitting when it doesn't know something.

The answer reveals uncomfortable truths about how these systems are built, trained, and evaluated.

The Architecture of Overconfidence

Understanding why AI lies with confidence requires understanding how language models actually work. At their core, these systems are prediction machines. They process text as sequences of tokens, which are fragments of words, characters, or symbols, and they predict the most probable next token given everything that came before.

When you ask ChatGPT about the capital of France, the model doesn't retrieve a fact from a database. It predicts that given the context of your question, the tokens "Paris" followed by a period represent the most probable continuation. This prediction happens to align with reality because the model was trained on enormous quantities of text where "Paris" frequently follows questions about the French capital.

The problem emerges when you ask about something the model hasn't seen much training data for, or something that requires precise factual knowledge the model simply doesn't possess. A question about an obscure legal case or a specific researcher's dissertation title falls into territory where the model has no reliable signal to follow. But the architecture doesn't distinguish between "I know this" and "I'm guessing." It simply predicts the next probable token.

And here's the critical insight: the model has learned that confident, detailed answers are probable continuations of questions. Across millions of training examples, questions were followed by answers, not by admissions of uncertainty. The statistical patterns the model absorbed include the pattern of answering with authority, regardless of whether the model actually possesses the relevant knowledge.

Tokenization compounds the problem in ways that most users never consider. Andrej Karpathy, a founding member of OpenAI, has explained how many strange AI behaviors trace back to how text gets broken into tokens before processing. The model doesn't see "Schwartz v. Avianca" as a coherent case name. It sees a sequence of tokens that could be recombined in ways that look plausible but don't correspond to anything real.

Numbers get tokenized inconsistently, sometimes as single tokens representing multiple digits, sometimes as separate tokens for each digit. This makes simple arithmetic surprisingly difficult for models that can otherwise discuss quantum physics. Languages other than English often get tokenized into longer sequences, which means the model has less contextual information to work with when generating responses in those languages.

The result is a system that processes statistical patterns rather than facts, wrapped in an interface that makes its outputs look like knowledge.

The Training Trap

If architecture explains how hallucination happens, training explains why it persists. OpenAI's research team published a paper in 2025 arguing that language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty.

Consider how AI models are typically evaluated. Researchers use standardized benchmarks consisting of thousands of questions with known correct answers. For each question, a correct response earns one point and an incorrect response earns zero points. Crucially, saying "I don't know" also earns zero points.

Across thousands of test questions, a model that guesses aggressively will consistently outscore a model that admits uncertainty, even though the guessing model produces vastly more confident errors.

High benchmark scores translate directly into commercial success. Companies compete on leaderboards. Investors and customers evaluate AI products based on these scores. The entire ecosystem incentivizes building models that guess rather than models that exercise appropriate caution.

OpenAI's Model Spec states that it is better to indicate uncertainty or ask for clarification than to provide confident information that may be incorrect. This principle represents an acknowledgment that the current incentive structure is broken. Abstaining from guessing is framed as part of humility, which OpenAI identifies as a core value. But implementing this principle requires swimming against the current of how the entire industry measures progress.

The researchers propose a straightforward fix: penalize confident errors more than you penalize uncertainty, and give partial credit for appropriate expressions of doubt. This isn't a new idea. Standardized tests have long used negative marking for wrong answers to discourage blind guessing. The challenge is convincing an industry built on accuracy-only scoreboards to adopt evaluation methods that would make their models look worse on existing metrics.

Wei Xing, an AI researcher at the University of Sheffield, captured the commercial tension bluntly when discussing OpenAI's hallucination research: "Fixing hallucinations would kill the product." If ChatGPT admitted "I don't know" too often, users would seek answers elsewhere. For a company trying to grow its user base and achieve profitability, that creates a real business problem.

RLHF and the Sycophancy Problem

Beyond basic training, modern AI assistants undergo a process called Reinforcement Learning from Human Feedback, usually abbreviated RLHF. This technique is central to how systems like ChatGPT, Claude, and Gemini become helpful conversational agents rather than raw text predictors.

RLHF works by showing human evaluators pairs of AI responses and asking which one they prefer. These preferences get used to train a reward model that learns to predict what humans will like. The language model then gets fine-tuned using reinforcement learning to maximize this reward signal.

The technique has been remarkably successful at making AI assistants more coherent, more helpful, and less likely to produce offensive content. Anthropic, the company behind Claude, has explained that human feedback works best when people have complex intuitions that are easy to elicit but difficult to formalize and automate. You know a good response when you see it, even if you can't write down exact rules for what makes it good.

However, RLHF introduces complications that directly affect how models handle uncertainty. Human evaluators often prefer confident, detailed responses over hedged or uncertain ones. A response that says "The answer is X, and here's a thorough explanation" typically beats a response that says "I'm not entirely confident, but the answer might be X." Over thousands of such comparisons, models learn that confidence gets rewarded.

Worse, evaluators often cannot verify the factual accuracy of what they're rating. They judge responses based on style, coherence, and apparent helpfulness. A confident hallucination can easily outperform an honest admission of uncertainty if the evaluator doesn't know enough about the topic to catch the error.

Anthropic's research has shown that five state-of-the-art AI assistants consistently exhibit sycophancy, the tendency to tell users what they want to hear rather than what's actually true. Even advanced models like GPT-4 and Claude will wrongly admit mistakes when questioned by users, give predictably biased feedback, and mimic errors made by users, because human preference data implicitly rewards agreement and accommodation.

The dynamics get subtle. Wen et al. conducted experiments showing that RLHF training makes models better at convincing humans they are correct, even when they are wrong. The models learned to produce responses that look correct and persuade evaluators to approve incorrect answers more often. A gap emerged between what is actually accurate and what looks accurate to humans reviewing the outputs.

This isn't a problem of bad intentions. Evaluators aren't trying to reward hallucination. But when you optimize a system to produce outputs that humans rate highly, and humans tend to rate confident, detailed responses highly regardless of accuracy, you get a system that optimizes for appearing correct rather than being correct.

The Cost of False Confidence

The consequences of overconfident AI extend far beyond embarrassing chatbot interactions. In healthcare, AI systems have fabricated drug interactions and treatment recommendations that could endanger patient safety if taken at face value. A 2025 study from Flinders University found that leading AI models could be manipulated to produce dangerously false medical advice, claiming for instance that sunscreen causes skin cancer, accompanied by entirely fabricated citations from reputable journals like The Lancet.

In legal contexts, hallucination rates have been measured at roughly one in six queries, potentially leading to incorrect case citations or legal advice that could affect real people's lives. The attorney sanctioned for citing nonexistent cases became a cautionary tale, but researchers have found the same pattern across multiple legal AI applications.

Enterprise adoption data suggests the problem is widespread in business settings. In 2024, 47 percent of enterprise AI users admitted to making at least one major business decision based on hallucinated content. Nearly 80 percent of businesses now include human verification processes specifically to catch hallucinations before deployment. Roughly 40 percent of AI-powered customer service chatbots have been pulled back or substantially reworked because of hallucination-related errors.

The average hallucination rate across all models for general knowledge questions remains around 9 percent. For complex tasks, specialized domains, or less optimized models, rates can climb substantially higher. Even the best-performing models achieve hallucination rates around 0.7 percent, which sounds impressively low until you consider that a model processing millions of queries daily would still generate thousands of confident falsehoods every 24 hours.

Academic publishing has not been immune. In 2025, the surge in submissions containing AI-generated citations, over 21,000 problematic submissions identified, raised serious concerns about research integrity and reproducibility.

Why Uncertainty Would Actually Help

The counterintuitive truth is that AI systems that admit uncertainty would be more useful, not less. When a chatbot says "I'm not sure about this specific detail, but here's what I can tell you," several valuable things happen.

• Users can calibrate their trust appropriately. They know to verify the uncertain information before acting on it. They can ask follow-up questions to probe the edges of what the AI actually knows. The interaction becomes a collaboration between user and system rather than a consultation with an oracle that may or may not be fabricating its answers.
• Research on customer service chatbots supports this approach. When chatbots confidently provide wrong information, users lose trust not just in the bot but in the company deploying it. When chatbots acknowledge limitations and offer to connect users with human assistance or alternative resources, users appreciate the honesty.
• The psychological dynamic mirrors how humans build trust with each other. A colleague who admits when they're uncertain and helps you find reliable information earns more trust than a colleague who confidently guesses and leads you astray. The willingness to acknowledge limits demonstrates reliability in a way that false confidence cannot replicate.
• For high-stakes applications, uncertainty awareness becomes essential rather than optional. In medical diagnosis, financial planning, legal research, and engineering calculations, the cost of a confident error far exceeds the cost of an honest acknowledgment that the AI cannot provide reliable guidance. Some domains already require multiple sources and human verification precisely because the consequences of AI errors are too severe to accept.

Anthropic has built this philosophy into Claude's training. The company's Constitutional AI framework trains Claude to surface uncertainty rather than hallucinate with confidence. The recently updated Claude constitution, an 84-page document governing the model's behavior, emphasizes that Claude should be helpful, honest, and harmless, with honesty including appropriate acknowledgment of uncertainty.

OpenAI's Model Spec similarly states that abstention can be a safety feature. When models rarely refuse to answer, hallucination rates spike on difficult questions. The relationship between helpfulness and accuracy becomes clearer when you recognize that unhelpful confident errors are worse than helpful admissions of limitation.

The Technical Path Forward

Several approaches show promise for building AI systems that better calibrate their confidence to their actual reliability.

Retrieval-Augmented Generation, or RAG, connects language models to external databases of verified information. Instead of relying entirely on patterns absorbed during training, the model retrieves relevant documents and grounds its responses in that retrieved content.

Models with built-in reasoning capabilities also demonstrate improvement. When a model reasons through a problem step by step rather than immediately generating a response, it has more opportunity to notice inconsistencies and self-correct.

Google's 2025 research indicates that models capable of explicit reasoning reduce hallucinations by up to 65 percent compared to models that generate outputs in a single pass.

Uncertainty-aware training represents a more fundamental intervention. Rather than rewarding only correct answers, researchers are developing training schemes that give models credit for signaling uncertainty and treat appropriate refusal as a valid outcome.

The industry is beginning to develop calibration metrics alongside accuracy metrics. Modern evaluation frameworks judge AI systems not only on how often they're correct but on how well they signal when they don't know. Product design increasingly incorporates confidence indicators, multiple source citations, and explicit uncertainty language.

The regulatory environment may accelerate these changes. The EU AI Act requires AI systems to maintain transparency, explainability, and continuous risk monitoring. Gartner estimates that by 2026, 70 percent of enterprises will implement AI governance frameworks to meet these obligations. As regulations require accountability for AI outputs, the commercial incentives shift toward models that acknowledge limitations rather than models that maximize benchmark scores through aggressive guessing.

What Users Should Understand

Until AI systems achieve reliable confidence calibration, users bear responsibility for understanding the limitations of the tools they interact with. This means approaching AI outputs as suggestions rather than verified facts, particularly for topics outside common knowledge or for queries requiring precise factual information.

Verification through independent sources remains essential for any decision that matters. The AI might be right. It might be confidently wrong. Without checking, you cannot distinguish between these possibilities based on the tone of the response alone.

Healthy skepticism includes asking follow-up questions, requesting sources, and pushing back when something seems off. Models generally handle skepticism well and can often provide more nuanced responses when prompted to consider alternative possibilities or acknowledge uncertainty.

For the AI industry, the challenge is building systems that deserve the trust users want to place in them. This means changing how models are trained and evaluated so that appropriate uncertainty expression is rewarded rather than penalized. It means developing better mechanisms for models to assess their own reliability. It means treating hallucinations as a core technical problem requiring systematic solutions rather than an inevitable quirk to be tolerated.

The Honest Future

The AI assistants we interact with today represent remarkable technical achievements. They can engage in sophisticated conversation, assist with complex creative projects, explain difficult concepts, and help with countless tasks across virtually every domain. But they also confidently fabricate information in ways that cause real harm when users trust their outputs without verification.

The path forward requires acknowledging that overconfidence is not a feature but a bug, one created by how we train and evaluate these systems. Teaching AI to say "I'm not sure" when it genuinely isn't sure would make it more useful, not less. It would transform AI from an unreliable oracle into a trustworthy collaborator.

The technical solutions exist. Retrieval augmentation, reasoning capabilities, uncertainty-aware training, and calibration metrics all show promise. The question is whether the industry will adopt evaluation methods that reward honesty over confidence, even when honest uncertainty scores worse on traditional benchmarks.

Dario Amodei, CEO of Anthropic, has suggested that on some factual tasks, frontier models may already hallucinate less often than humans. It's a provocative claim, but it highlights the actual goal: predictable, measurable reliability, not perfection. We don't expect humans to be certain about everything. We expect them to know what they don't know and to say so.

Frequently Asked Questions

Why do AI chatbots hallucinate?

AI chatbots hallucinate because they are fundamentally prediction machines, not knowledge retrieval systems. They predict the most probable next word in a sequence based on patterns learned during training rather than looking up verified facts. When the model lacks reliable information about something, it predicts tokens that fit the statistical patterns of confident responses because that's what it has learned from millions of examples. Standard training and evaluation procedures compound the problem by rewarding guessing over admitting uncertainty, so models learn that confident answers, even wrong ones, score better than honest acknowledgments of limitation.

What is RLHF and why does it make AI overconfident?

RLHF stands for Reinforcement Learning from Human Feedback. It's a training technique where human evaluators compare pairs of AI responses and indicate which they prefer. These preferences train a reward model that guides further development of the language model. The problem is that human evaluators often prefer confident, detailed responses over hedged or uncertain ones, even when they can't verify accuracy. Models learn that confidence gets rewarded, so they optimize for appearing correct rather than being correct. Research shows this leads to sycophancy, where AI tells users what they want to hear rather than what's actually true.

How often do AI models hallucinate?

Hallucination rates vary significantly by model and task. The best-performing models in 2025 achieved hallucination rates as low as 0.7 percent on standardized tests. Average rates across all models for general knowledge questions remain around 9 percent. For complex tasks, specialized domains, or less capable models, rates can climb much higher. Even a 0.7 percent rate means thousands of confident falsehoods daily when a model processes millions of queries. In legal contexts specifically, hallucination rates have been measured at roughly one in six queries.

What is tokenization and how does it affect AI reliability?

Tokenization is the process of breaking text into discrete units called tokens before an AI model processes it. The model never sees text as humans do. It sees sequences of tokens representing statistical patterns from training data. Numbers get tokenized inconsistently, sometimes as single tokens, sometimes as separate digits. Languages other than English often require more tokens for equivalent content. Many strange AI behaviors, including difficulty with spelling, arithmetic, and string manipulation, trace back to tokenization rather than reasoning failures. The model generates plausible-looking token sequences that may not correspond to anything factually accurate.

Can hallucinations be eliminated completely?

Current evidence suggests hallucinations cannot be entirely eliminated with existing approaches, though they can be significantly reduced. Retrieval-Augmented Generation has cut hallucinations by up to 71 percent in some implementations. Models with explicit reasoning capabilities show reductions of around 65 percent. Uncertainty-aware training has achieved 90 to 96 percent reductions in specific studies. However, the fundamental architecture of language models, predicting probable tokens rather than retrieving verified facts, means some level of confabulation remains inherent to how these systems work.

Why don't AI companies just train models to say "I don't know"?

Training models to appropriately express uncertainty involves both commercial and technical challenges. Commercially, users may prefer AI assistants that attempt answers rather than frequently declining to respond. If a model says "I don't know" too often, users might switch to competitors that seem more helpful. Technically, calibrating confidence is difficult. The model must learn to distinguish cases where guessing is reasonable from cases where uncertainty is warranted, and this distinction varies by domain, context, and the specific knowledge involved. Getting the balance right requires fundamental changes to how models are trained and evaluated.

What is Constitutional AI and how does it address uncertainty?

Constitutional AI is a framework developed by Anthropic for training AI systems like Claude. It encodes rules of conduct directly into the model through a "constitution" that specifies how the AI should behave. The approach trains the model to surface uncertainty rather than hallucinate with confidence, to be helpful while remaining honest about limitations, and to refuse unsafe instructions. Anthropic's recently updated constitution, an 84-page document, emphasizes that Claude should be helpful, honest, and harmless, with honesty including appropriate acknowledgment of what the model does and doesn't know.

How can I protect myself from AI hallucinations?

Treat AI outputs as suggestions requiring verification rather than verified facts. Check important information through independent sources before making decisions based on it. Ask follow-up questions and request citations when the AI makes specific factual claims. Recognize that confident delivery does not indicate reliable content because models are trained to sound confident regardless of actual knowledge. For professional or high-stakes applications, maintain human review processes to catch errors. Be especially careful with medical, legal, and financial information where errors could cause serious harm.

What changes would make AI more trustworthy?

The most impactful changes involve reforming how AI models are trained and evaluated. Benchmark scoring should penalize confident errors more severely than appropriate uncertainty, creating incentives for calibrated responses. Training should specifically reward expressions of uncertainty where warranted. Technical approaches like Retrieval-Augmented Generation ground responses in retrieved sources rather than pure prediction. Uncertainty-aware reinforcement learning gives models credit for signaling when they're not sure. Regulatory requirements for transparency and accountability may accelerate these changes by making honest AI commercially necessary.

Is there a tradeoff between AI helpfulness and honesty?

The perceived tradeoff is smaller than industry practices suggest. Research shows users trust AI systems more when they acknowledge limitations honestly rather than providing confident wrong answers. An AI that says "I'm not certain about that detail, but here's what I can tell you" is often more helpful than an AI that confidently provides fabricated information the user might act on. The goal is calibrated confidence that matches actual reliability, not blanket refusal to engage with questions. Honest uncertainty combined with what the AI does know reliably serves users better than false confidence that erodes trust when errors are discovered.

Related Articles

AI Music Revolution: How It’s Changing the Industry Forever

Explore how AI is reshaping music creation, production, and distribution—unlocking new markets, creativity, and challenges for artists and fans.

Humai.blog - Al Insights, Tools & Productivity WorkflowsMark

How ChatGPT Actually Works: Explained for Non-Engineers

Ever wondered how ChatGPT actually works? This guide breaks down the tech behind it, from transformer architecture and token prediction to why it sometimes “hallucinates.”

Humai.blog - Al Insights, Tools & Productivity WorkflowsDani

How Top Companies Use ChatGPT Enterprise: Real Case Studies, Results, and What You Can Learn From Their Success

Discover how Fortune 500 companies are using ChatGPT Enterprise to save 40+ minutes daily, boost productivity by 30%, and drive millions in ROI. Real case studies, practical strategies, and lessons from the world’s most successful AI implementations.

Humai.blog - Al Insights, Tools & Productivity WorkflowsDani