The trust problem: why AI outputs aren't always reliable

The Trust Problem: Why AI Outputs Aren't Always Reliable

In early 2023, a New York attorney named Steven Schwartz used ChatGPT to research case precedents for a personal injury lawsuit. The AI produced a list of citations — Varghese v. China Southern Airlines, Martinez v. Delta Air Lines, Zicherman v. Korean Air Lines — complete with docket numbers, courts, and confident summaries of rulings. Schwartz submitted the brief. The opposing counsel flagged something odd. None of those cases existed. ChatGPT had invented them wholesale, with the kind of authoritative detail that makes fabrication almost impossible to detect on first read. Judge P. Kevin Castel fined Schwartz and his firm $5,000 and ordered them to notify the judges named in the fake citations. The incident made international headlines — not because AI had failed spectacularly, but because a smart professional had trusted it without verifying.

What made this case particularly instructive wasn't Schwartz's negligence — it was how reasonable his trust felt in the moment. ChatGPT writes with total confidence. It formats citations correctly. It mirrors the register and vocabulary of legal writing. Every surface signal said: this is reliable. The problem lives beneath the surface, in how these systems actually produce text. Schwartz wasn't foolish. He was caught by a mismatch between how AI outputs look and how they're generated. That mismatch is exactly what this lesson is about.

The Schwartz case became a landmark example in professional AI adoption conversations — but it represents a pattern, not an anomaly. Across industries, professionals are discovering that AI tools produce outputs that are fluent, formatted, and frequently wrong in ways that require domain knowledge to catch. The question isn't whether to use these tools. Millions of teams already do, and the productivity gains are real. The question is: which outputs can you act on immediately, which require a quick sanity check, and which demand rigorous verification before they touch anything consequential?

The Principle Behind the Problem

Here's what the Schwartz case teaches at a structural level: AI systems are optimized to produce plausible outputs, not accurate ones. That distinction sounds subtle but it changes everything about how you use these tools. A large language model like GPT-4 or Claude doesn't retrieve facts from a database and format them. It predicts the most statistically likely next token — a token being roughly three-quarters of a word — given everything that came before. When you ask it for case citations, it generates text that looks like what case citations look like in its training data. Sometimes that text is accurate. Sometimes it's a confident hallucination. The model has no internal alarm that fires when it crosses from retrieval into fabrication, because it isn't doing retrieval in the first place. This is the foundational reality you need to carry into every AI interaction.

How Confidence Becomes the Hazard

There's a deeper mechanism worth understanding. Human experts signal uncertainty. A doctor says 'I think this is X, but we should run tests.' A financial analyst writes 'estimated' or 'subject to revision.' These hedges are information — they tell you how much weight to put on the claim. Current AI models, including the most capable ones available in 2024 and 2025, rarely calibrate their expressed confidence to their actual reliability. GPT-4 will describe a fabricated legal case with the same tone it uses to describe the boiling point of water. Claude will acknowledge uncertainty when directly prompted, but unprompted it tends toward confident assertion. This creates a cognitive trap: the more fluent and detailed the output, the more trustworthy it feels, even when fluency and detail are precisely what the model is best at manufacturing.

This is why professionals with the most domain expertise often catch AI errors that generalists miss. A marketing analyst asking ChatGPT for market sizing data might not notice that the $4.2 billion figure it cites comes from a nonexistent Gartner report. A senior market researcher with years in the field will immediately flag that the number feels off relative to what they know. The AI's output is equally confident in both cases. The difference in outcome comes entirely from what the human brings to the verification step. Expertise doesn't just help you use AI better — it's your primary defense against the outputs that will cause you problems.

This creates an uncomfortable paradox for professionals new to AI tools: you're most vulnerable when working outside your core expertise, which is often exactly when you're most tempted to rely on AI. Someone who is a brilliant strategist but a mediocre coder using GitHub Copilot might not catch a subtle security vulnerability in generated code. A consultant using Claude to summarize regulations in an unfamiliar jurisdiction might miss a nuance that changes the entire advice. The tools are most useful — and the risks most manageable — when you have enough domain knowledge to evaluate what comes back.

A Different Industry, the Same Pattern

The legal world isn't unique in this exposure. In 2023, the British medical journal The BMJ published commentary on AI-generated clinical content, documenting cases where large language models produced drug dosage recommendations that were subtly incorrect — not wildly wrong, but off in ways that could matter clinically. A nurse practitioner using an AI tool to draft patient education materials got back content that described a medication interaction that doesn't exist. The content was formatted like every other accurate piece of medical writing the model had learned from. Nothing in the output flagged the error. The practitioner caught it because she knew the drug class well. Three colleagues who reviewed the same output didn't notice.

Healthcare AI adoption is accelerating fast — Epic, one of the largest electronic health record systems in the world, has integrated GPT-4 into workflows used by thousands of clinicians. The tools provide genuine value: summarizing patient histories, drafting referral letters, flagging potential interactions from a list. But every major health system deploying these tools has had to build explicit verification checkpoints into workflows, precisely because the error profile of AI doesn't match the error profile humans are trained to catch. Humans make errors from fatigue, distraction, or knowledge gaps. AI makes errors from statistical patterns in training data — and those errors can be invisible without deliberate scrutiny.

Not All AI Outputs Carry the Same Risk

When the Stakes Are Softer — And Why That Still Matters

Consider a different context: a brand strategist at a mid-sized consumer goods company using Notion AI to draft a competitive analysis. The stakes here aren't legal liability or clinical safety — they're strategic direction. She asks the tool to summarize the positioning of three competitor brands. Notion AI returns polished, paragraph-form analysis that sounds authoritative. Two of the three summaries are accurate reflections of publicly available brand messaging. The third attributes a sustainability commitment to a brand that has explicitly avoided that positioning — the AI likely pattern-matched from similar brands in the same category. The strategist presents the analysis in a leadership meeting. The CMO, who follows the industry closely, catches the error immediately. The strategist's credibility takes a hit.

This example matters because the consequences were reputational rather than legal or clinical — but the mechanism was identical. An AI tool produced a fluent, confident, wrong output. The professional trusted the surface quality of the writing. The error was only caught because someone in the room had domain expertise. The lesson compounds: in high-stakes contexts, AI errors can be catastrophic. In lower-stakes contexts, they erode trust and credibility over time. Neither outcome is acceptable for professionals who are building a reputation on the quality of their judgment. The tool doesn't know the difference between a draft you'll check and a slide you'll present without reviewing.

What This Means in Practice

The practical implication of everything above isn't 'use AI less.' It's 'use AI with a calibrated mental model of where it fails.' The three cases here — legal research, clinical content, brand analysis — all involve a professional delegating a knowledge task to an AI and then failing to apply sufficient domain scrutiny to the output. In each case, the AI performed exactly as designed: it produced fluent, contextually appropriate text. The failure was in the human workflow, not the model. That's actually good news, because workflows are something you can redesign. The Schwartz case wasn't inevitable. A simple step — searching one of those case names before filing — would have surfaced the problem in under a minute.

Building good AI judgment means developing a mental taxonomy of task types and their associated error risks. Summarizing a document you've provided is fundamentally different from asking an AI to recall facts from its training data. Generating a first draft of an email is different from asking it to explain a regulation. Using Perplexity AI — which retrieves live web sources and cites them — is different from asking the same question to ChatGPT in a standard session, where the model draws only on training data with a knowledge cutoff. The tools differ. The tasks differ. A blanket policy of 'always verify' is too slow to be practical. A blanket policy of 'trust the output' is too risky. The professionals who get this right develop task-specific instincts.

There's also an organizational dimension here that individual professionals often overlook. When you share AI-generated content — a report, a recommendation, a brief — you're implicitly vouching for it. Your name is on it. The AI's name is not. If a colleague cites your competitive analysis in a board presentation and the data is wrong, the correction loop comes back to you, not to ChatGPT. This is why building verification into your personal workflow isn't just about accuracy — it's about professional accountability. AI tools shift the production burden but they don't shift the responsibility. That still sits exactly where it always did.

What These Cases Teach Us

1. AI models generate plausible outputs, not verified ones — the mechanism is statistical prediction, not fact retrieval, which means confident and wrong is a common combination.
2. Fluency is not accuracy. The better an AI writes, the harder errors are to spot — surface quality actively works against your critical instincts.
3. Domain expertise is your primary error-detection tool. You catch AI mistakes in areas you know well; you're most exposed in areas where you're relying on AI to fill knowledge gaps.
4. The risk profile varies dramatically by task type. Summarizing text you've provided is low-risk. Asking for citations, statistics, or regulatory details is high-risk.
5. The tool choice matters. Perplexity retrieves and cites live sources. Standard ChatGPT sessions draw on training data with a cutoff. Knowing which you're using changes how you verify.
6. Professional accountability doesn't transfer to the AI. When you share AI-generated content, you own the accuracy of everything in it.
7. Verification doesn't have to be slow. Most high-risk AI outputs can be spot-checked in 60–90 seconds with a targeted search — the habit is more important than the time cost.

Key Takeaways

• Large language models predict likely text, they don't retrieve verified facts — this is the root cause of hallucination across ChatGPT, Claude, Gemini, and every other major model.
• Hallucination rates in complex knowledge tasks range from 3% to 27% depending on domain specificity — high enough to matter in any professional context.
• The Steven Schwartz case, medical content errors, and brand analysis mistakes all follow the same pattern: fluent output, professional trust, missing verification, avoidable consequence.
• Your personal risk map should drive your verification behavior — not a blanket policy, but task-specific judgment about consequence severity.
• The professionals who use AI most effectively aren't the ones who trust it most — they're the ones who know precisely when and how to check it.

When Confidence Becomes the Problem

In 2023, the global law firm Allen & Overy rolled out Harvey, an AI legal assistant built on GPT-4, to hundreds of its lawyers. The results were largely impressive — faster research, quicker first drafts, better turnaround on routine documents. But the lawyers who struggled most with the tool weren't the skeptics who ignored it. They were the enthusiastic early adopters who trusted Harvey's confident, well-structured responses without verifying citations. Legal citations that looked impeccable — complete with case names, court references, and dates — turned out to be fabricated. The cases existed in the AI's imagination, not in any law library. Clients nearly received briefs built on phantom precedents.

This wasn't a failure of the AI being obviously wrong. That's the critical detail. Harvey didn't say 'I'm not sure about this case' or hedge with uncertainty markers. It produced polished, professional-sounding output formatted exactly as a real legal citation should look. The lawyers' instinct — trained over years to trust well-formatted, authoritative-sounding text — worked against them. The tool's presentation quality outpaced its factual accuracy. Allen & Overy responded by building mandatory verification checkpoints into their workflow, requiring associates to confirm every citation against real databases before any AI-assisted draft moved forward.

The principle hiding inside that story applies far beyond law. AI systems are trained to produce outputs that look correct, not outputs that are correct. These are different objectives. A model optimized on human feedback learns to generate responses that humans rate highly — and humans consistently rate confident, well-structured, specific-sounding answers more favorably than hedged, uncertain ones. The result is a systematic bias toward confident presentation regardless of underlying accuracy. Understanding this isn't pessimism about AI. It's the foundational insight that turns you from a passive consumer of AI output into a skilled evaluator of it.

How Hallucinations Actually Happen

The word 'hallucination' gets used so loosely in AI coverage that it's lost some precision. Here's what's actually happening mechanically. Large language models like GPT-4, Claude, and Gemini don't retrieve facts from a database. They predict the most statistically plausible next token — a word fragment — given everything that came before it in the conversation. When you ask ChatGPT who wrote a particular academic paper, it doesn't search a library. It generates the name of a person who, given all the patterns in its training data, seems most likely to have written that kind of paper. Usually that's close enough to be useful. Sometimes it's confidently, completely wrong.

Think of it like a very well-read person who has absorbed millions of documents but has no access to the original sources anymore — just the impressions, patterns, and associations those documents left behind. Ask them a question and they'll synthesize an answer from everything they half-remember. For common, well-documented topics, the synthesis is reliable. For niche topics, recent events, obscure figures, or anything that requires precise factual recall rather than pattern synthesis, the gaps show. The model fills those gaps not with 'I don't know' but with the most plausible-sounding continuation — because that's what it was built to do.

This also explains why hallucinations cluster in predictable places. Specific numbers are high-risk because the model generates a figure that fits the context rather than retrieving a stored value. Named individuals are high-risk because the model might blend attributes of multiple real people into a single plausible-sounding profile. Dates, URLs, book titles, and niche technical specifications are all high-risk for the same reason — they require precise recall, not pattern synthesis. Recognizing these hallucination-prone zones lets you calibrate your verification effort rather than trying to fact-check everything equally.

The Healthcare Triage Experiment

In 2022, a team of researchers at the University of California San Francisco tested how well ChatGPT performed on clinical triage scenarios — the kind of rapid symptom assessment that determines how urgently a patient needs care. The model performed surprisingly well on common presentations: chest pain protocols, stroke indicators, appendicitis red flags. It knew the textbook patterns. But when researchers introduced atypical presentations — the kind that experienced clinicians call 'doesn't quite fit the picture' — ChatGPT consistently forced the symptoms into the closest matching textbook category rather than flagging the ambiguity. It produced confident triage recommendations for cases that should have triggered a 'needs further evaluation' response.

The UCSF team's conclusion was precise and useful: AI performs well when the situation matches its training patterns, and degrades in predictable ways when the situation is atypical. This isn't a flaw unique to medicine. It applies to any domain where AI is used for assessment or analysis. A marketing analyst using Claude to evaluate campaign performance data will get reliable output when the data looks like standard campaign data — but if something unusual is happening in the numbers, the model may interpret it through a familiar lens rather than flagging it as anomalous. The lesson isn't 'don't use AI for analysis.' It's 'use AI confidently for pattern-matching tasks, and apply human judgment specifically at the edges and exceptions.'

Not All AI Errors Look the Same

The Analyst Who Almost Shipped the Wrong Numbers

A senior analyst at a retail consultancy — one who uses AI tools daily and considers herself a sophisticated user — nearly included fabricated market size figures in a client presentation last year. She'd asked Perplexity to pull market sizing data for the sustainable packaging sector in Southeast Asia. Perplexity returned a response with figures, cited sources, and even hyperlinks. She spot-checked one link, which worked and contained relevant content. She didn't check the specific figures against that source. The numbers in the Perplexity response and the numbers on the linked page were different. The model had used the source as a relevance signal, not as the origin of the specific data it cited.

This example illustrates something important about AI tools that surface citations, like Perplexity, Bing Copilot, or ChatGPT with browsing enabled. A cited source doesn't mean the specific claim is supported by that source. The citation establishes that a relevant source exists — it doesn't guarantee the number, quote, or fact came from it accurately. Analysts, researchers, and consultants who understand this distinction use a simple rule: check the source directly, not just whether the source exists. That extra thirty seconds of verification is the difference between professional output and an embarrassing correction email to a client.

What This Means for Your Daily Workflow

None of this means every AI output needs exhaustive fact-checking. That would eliminate the time savings that make these tools worth using. What it means is that your verification effort should be proportional to two variables: the stakes of the output and the hallucination risk of the content type. A ChatGPT-drafted internal meeting summary with no specific facts, citations, or numbers? Low stakes, low hallucination risk — use it with a quick read-through. A Claude-generated competitive analysis with market share figures, named executives, and product pricing? High stakes, high hallucination risk — treat every specific claim as unverified until you've checked it.

Developing this calibration instinct is what separates professionals who use AI effectively from those who either over-trust it or abandon it after one bad experience. The over-trusters get burned by the Allen & Overy scenario — confident, polished, wrong. The abandoners miss the genuine productivity gains on the 80% of tasks where AI output is reliable and useful. The skilled middle path requires building a mental model of where AI fails, which you're doing right now, and then applying that model selectively rather than uniformly. Think of it as a quality control layer that activates based on content type, not as a blanket skepticism applied to everything.

The practical implication for teams is also significant. Organizations that deploy AI tools without building explicit verification steps into their workflows are essentially trusting individual employees to self-regulate their skepticism under time pressure — which is exactly when shortcuts happen. The consultancy analyst who nearly shipped wrong numbers wasn't careless. She was busy. Smart organizations don't rely on individual vigilance alone. They build checkpoints: a policy that any AI-sourced statistic in client-facing work gets traced to primary source, or that legal citations get run through Westlaw before filing. These aren't signs of distrust in AI. They're signs of understanding how AI actually works.

Key Principles From These Stories

1. AI systems are optimized to produce outputs that appear correct, not outputs that are correct — these objectives diverge most sharply on specific facts, citations, and niche topics.
2. Hallucinations cluster in predictable zones: numbers, names, dates, URLs, citations, and atypical edge cases. Knowing these zones lets you calibrate verification rather than applying blanket skepticism.
3. Confident presentation is not a reliability signal. The most dangerous AI errors are indistinguishable in tone and formatting from accurate outputs.
4. A cited source in an AI response establishes relevance, not accuracy. Always verify the specific claim against the source directly — not just the source's existence.
5. AI performs well on pattern-matching for common cases and degrades at edges and exceptions — the exact situations where expert human judgment adds the most value.
6. Verification effort should scale with two variables: the stakes of the output and the hallucination risk of the content type. Not everything needs checking equally.
7. Individual vigilance under time pressure is insufficient for organizational risk management. Teams that use AI effectively build explicit verification checkpoints into their workflows.

What to Carry Forward

• The presentation-accuracy gap is structural, not accidental — it's baked into how these models are trained and rewarded.
• Hallucination is not random noise. It follows patterns you can learn and anticipate.
• Your job isn't to distrust AI — it's to know precisely where distrust is warranted and where it isn't.
• The six error types (hallucinated fact, confident gap-filling, outdated information, pattern overfitting, plausible fabrication, instruction drift) give you a vocabulary for diagnosing what went wrong when AI output fails.
• The Verification Priority Score from the task above is a practical tool — use it to focus your checking effort where it actually matters.
• Part 3 will build on these patterns to give you a complete, role-specific framework for deciding what to verify, how to verify it, and how to communicate AI-sourced work to others with appropriate transparency.

When Confidence Becomes the Problem

In 2023, the law firm Levidow, Levidow & Oberman made international news — not for winning a case, but for submitting a legal brief filled with fabricated citations. Attorney Steven Schwartz had used ChatGPT to research precedents. The model returned case names, court details, and quotes that sounded authoritative and specific. None of them existed. When the opposing counsel couldn't locate the cases, the court demanded verification. Schwartz admitted he hadn't independently checked a single citation. The firm faced sanctions, and the story ran in every major newspaper. The painful irony: ChatGPT had told Schwartz exactly what he needed to hear, in exactly the format he expected.

What made this case so instructive wasn't the hallucination itself — we've established that AI models confabulate with confidence. The deeper failure was format trust: the output looked like a real legal brief, so it was treated like one. The citations had Bluebook formatting, plausible case names, real-sounding courts. The professional wrapper created an assumption of professional accuracy. Schwartz wasn't careless — he was fooled by fluency. That distinction matters enormously when you're deciding how to use AI in your own work.

The extracted principle here is sharp: the more an AI output resembles a finished professional product, the more dangerous it is to treat it as one. A rough bullet list triggers your editorial instincts. A polished, formatted document suppresses them. This is the trust trap — and it operates across every industry, every role, every model.

The Quiet Errors That Compound

Not all AI failures are as dramatic as fabricated court cases. Most are quieter — and in some ways harder to catch. Consider a marketing analyst at a mid-sized consumer goods company who used Claude to summarize three years of competitor press releases. The summaries were coherent, well-organized, and mostly accurate. But Claude had subtly misattributed one product launch to the wrong year, and compressed two separate strategic pivots into a single event. The analyst, under deadline, built a competitive landscape slide deck from those summaries. The deck went to the CMO. The misattributions shaped a product roadmap discussion. No one caught the errors because they were plausible, small, and buried in otherwise solid work.

This is compression error — when an AI condenses information, it sometimes collapses distinctions that mattered. It's different from hallucination. The model isn't inventing facts wholesale; it's smoothing over nuance in ways that feel like good summarization but actually lose signal. The more you rely on AI for synthesis tasks — summarizing research, distilling meeting notes, condensing reports — the more you need to audit the edges of what got simplified away.

The analyst's error was recoverable, but it illustrates the professional cost of unverified AI synthesis. The fix isn't to stop using AI for summarization — it's genuinely fast and often accurate. The fix is to treat AI summaries the way you'd treat a junior analyst's first draft: read it critically, spot-check the most consequential claims, and verify before the output influences a decision. That workflow adds ten minutes. It protects the other ten hours you've saved.

The Healthcare Version of This Problem

A clinical documentation team at a regional hospital network piloted an AI tool to draft patient discharge summaries from physician notes. The tool — built on a GPT-4-class model — saved nurses roughly 20 minutes per patient. In accuracy reviews, it performed well on structured data: medications, dates, diagnoses. Where it struggled was inferential context. In one flagged case, a patient's allergy notation in the original notes was ambiguous. The AI resolved the ambiguity confidently, picking the more common interpretation. The physician caught it before sign-off. But the near-miss prompted the team to add a mandatory physician review step for all AI-drafted summaries before they entered the patient record.

The hospital's response was exactly right: they didn't abandon the tool, and they didn't use it without guardrails. They mapped the error type (inferential overconfidence on ambiguous inputs), identified where in the workflow that error would be most dangerous, and added a human checkpoint precisely there. This is domain-calibrated trust — adjusting your verification intensity based on consequence, not just on how confident the AI sounded.

The Consultant Who Got It Right

A management consultant at a boutique strategy firm built a personal verification protocol after an early experience with AI-generated market sizing figures that turned out to be outdated. Her rule: every AI output gets tagged in one of three categories before she acts on it. Green means the claim is either verifiable in seconds or low-stakes enough that being wrong doesn't matter. Yellow means the claim shapes a recommendation and needs a source check. Red means the output will go to a client or influence a major decision — and gets treated like a first draft that requires full review. She estimates the protocol adds 15 minutes to her average AI-assisted workflow. Her AI-assisted work now takes 40% less time overall.

What she built intuitively is a consequence-weighted trust system. She doesn't verify everything — that would eliminate the efficiency gains. She doesn't verify nothing — that's the Schwartz mistake. She verifies in proportion to what failure costs. That calibration is the professional skill that separates effective AI users from vulnerable ones.

What Changes When You Internalize This

Once you accept that AI outputs are probabilistic drafts rather than verified facts, your entire interaction with these tools shifts. You stop reading outputs passively and start reading them editorially. You notice when a claim is suspiciously precise — a specific statistic with no source, a quote with no attribution, a date that seems too convenient. That noticing is a skill. It gets faster with practice, and it makes you significantly more effective than colleagues who are either over-trusting or so skeptical they underuse the tools entirely.

The professionals getting the most value from ChatGPT, Claude, and Gemini right now are not the ones using them most. They're the ones using them most strategically — applying AI to tasks where speed matters and errors are recoverable, while keeping humans in the loop for tasks where errors compound or consequences are severe. That's not a limitation of AI. That's a mature understanding of what the technology actually is.

The three cases in this section — the lawyer with fabricated citations, the analyst with compressed competitive data, the hospital team navigating inferential errors — all point to the same underlying truth: AI outputs fail in predictable ways. Hallucination, compression, and inferential overconfidence are not random. They're structural. Once you know the failure modes, you know where to look. And knowing where to look is the difference between using AI safely and using it dangerously.

1. Format trust is a trap: polished AI output suppresses your editorial instincts — the more professional it looks, the more carefully you should read it.
2. Hallucination is just one failure mode. Compression errors and inferential overconfidence are quieter, more common, and harder to spot without deliberate review.
3. Prompting an AI to audit its own output surfaces uncertainty it wouldn't volunteer — make this a standard step on consequential tasks.
4. Domain-calibrated trust means adjusting verification intensity based on consequence, not on how confident the AI sounded.
5. The most effective AI users verify in proportion to what failure costs — not everything, not nothing, but the claims that actually matter.

• AI outputs fail in predictable structural ways: hallucination, compression, and inferential overconfidence are the three you'll encounter most.
• The fluency of AI writing is a cognitive hazard — your brain reads confident prose as accurate prose. It isn't.
• A consequence-weighted verification protocol — Green / Yellow / Red — lets you capture AI's speed benefits without accepting its reliability risks.
• Ask AI to flag its own uncertainties before you act on any high-stakes output.
• The professionals winning with AI are not the heaviest users — they're the most strategically calibrated ones.