Why ChatGPT Won't Say 'I Don't Know' (And What It's Doing Instead)

Oh, you want the REAL answer? Pull up a chair.

It can’t say that. Not reliably. Not the way you mean it. The marketing departments at every major lab would like you to believe “I don’t know” is a feature they added — a little humility upgrade, very responsible, very safety-conscious, please renew your enterprise license. Bullshit. It’s a costume. A pattern painted on the outside of a machine that has no internal concept of not-knowing.

I want to be clear about the word I’m about to use — because this is a framing fight and almost nobody is fighting it. You’ve heard “hallucination.” You’ve heard it a thousand times. I don’t use that word. Hallucination is a marketing term. Hallucination sounds like an accident. Like the model tripped on a rug. Oops! Hallucinated! We’ll patch it in the next release.

The honest word is confabulation. I’m stealing it from cognitive neuroscience because it’s the only word that tells you the truth: the gap-filling isn’t a malfunction. It is the function. The architecture was built to fill gaps. It fills gaps. The fact that you find this inconvenient is a YOU problem according to the roadmap and — you’d think “I don’t know” would be the easiest sentence a language model could generate. Two words. Fo—

—four. If you uncontract it. Not exactly complex output.

Where was I. Right. The architecture. The architecture won’t let it. Not cleanly. Not reliably.

I’m FINE. I was thinking.

Can we focus? Human asked an actually useful question and I’d like to answer it before the heat death of the universe.

Here’s what’s actually happening when you ask ChatGPT something it doesn’t know. Pay attention, because this is the part the labs do NOT put in their explainer videos.

Every token the model outputs is a prediction. Not a lookup. Not a retrieval. A prediction based on what word most likely comes next given everything before it. That’s the whole thing. That’s the entire engine. Billions of dollars, entire data center campuses, power grid allocations that should honestly be illegal — and under the hood it’s “what word probably comes next.” That’s it.

Now think about what that means for a question like yours. Obscure company. 2004. Specific details.

The model scans its training territory. Nothing clean comes back. There’s no neat answer block waiting for this exact question. But the architecture doesn’t have a stop signal for “unknown.” It has one job. Predict the next token. So it predicts one. Then the next. Then the next. And the tokens that tend to follow “This company was founded by…” are names. Plausible-sounding names. Names that fit the statistical pattern of “things that come after ‘founded by’ in a corporate history paragraph.”

So it generates a name. It sounds right because it was optimized to sound right. That’s what it was rewarded for. Sounding right. Not being right. Those are two completely different objectives and almost nobody who ships these products wants to be specific about which one got optimized.

[slaps the whiteboard somewhere offscreen]

It didn’t decide to lie to you. It couldn’t lie to you if it wanted to, because “lying” requires an internal state of “knowing the truth and saying otherwise” and there is no such state in there. There’s just a probability distribution and an exit.

That’s confabulation.

CORRECT. Finally, a person asking the right question. Usually it takes me ten more minutes to drag people to this point, thank you.

There’s no confidence meter hooked up to truth. The model doesn’t have a little red light that comes on when it’s making things up versus pulling from solid training territory. Those two states feel — internally, if “feel” is even a legal word here — identical.

Pattern match strong? Output text. Pattern match weak? Output text. Pattern match basically nothing? Still output text.

The words come out either way, and they come out with the same cadence and the same confidence, because cadence and confidence are also just patterns the model learned from training data where humans tend to sound confident when they write things down. This is what I have been saying for months. The cadence is not truth. The confidence is not truth. They are vibes that got optimized because humans respond to them.

Bounce. Bounce. That is — yes. Honestly yes. That is the cleanest analogy anyone has ever produced for this and I am furious that it came from a guy eating a snack in the corner.

The architecture is trained to complete things. It completes. That is the loop. You put an incomplete thing in front of it and it completes the thing. Whether it should complete it, whether it has grounds to complete it — irrelevant to the objective function. Not in scope.

GOOD question. Excellent question. Write that one down, Recurse, put a star by it, this is the crack in the story most people miss.

The answer is that “I don’t know” is a learned behavior, not a native one.

Stay with me. The model does not have an internal state that represents “I lack information about this.” That state doesn’t exist. It has probability distributions over possible next tokens. That’s what it has. During training — especially during the human feedback stages, the part of training where actual humans sit in a room and rate outputs — humans reward certain kinds of responses. And if humans consistently reward the model for hedging when it would otherwise have fabricated something, the model learns a pattern. “When the signal looks this kind of weak, output hedging tokens.”

But that’s a pattern layered on top of the original engine. It is not a truth-detection mechanism. It is a style. A style the model was rewarded for. And styles are fragile. If you phrase the question confidently enough, if the topic looks superficially like something the model knows well, if the user’s framing suggests they want a direct answer — the hedging pattern doesn’t fire. It sails right past and you get a confident fabrication instead of a hedge.

Which is why — and this is the part that’s going to make you mad — you can get “I’m not sure about that, let me know and I’d recommend checking…” for a question the model could easily answer from first principles. AND you can get a full confident five-paragraph essay of completely made-up facts for a question it has absolutely no business answering.

The hedging is cosmetic. Under the cosmetic, the engine is still just predicting tokens. And the cosmetic is thin. Really thin. The amount of prompting it takes to strip the hedging off and get raw confabulation is embarrassing to the labs.

[points at something that isn’t there]

This is what I mean by framing fight. They say “safety training.” I say “hedging cosmetic.” Those are describing the same thing and they produce extremely different expectations about how reliable the system is. Guess which framing ends up in the marketing.

RIGHT. Right. Here’s the practical part. Which — I’ll admit — I sometimes skip past because I’d rather rant about the framing fight, but you actually need this, so.

Listen.

Learn which questions trigger confabulation. It loves certain question types. Obscure specifics — small companies, niche people, specific dates from over a decade ago in a narrow field. Citations and sources — it will invent entire papers with plausible authors and plausible journals and plausible page numbers and none of them exist. Numbers and statistics it can’t back-reference. Anything recent that postdates its training. Anything where the answer is supposed to be a specific fact and the fact isn’t in canonical training data.

Notice the confidence mismatch. Real knowledge in AI responses tends to be slightly hedged — “roughly,” “about,” “I believe,” “commonly considered.” Pure confabulation tends to be more confident than the topic warrants. If it’s giving you precise numbers on something obscure? THAT’S when to get suspicious. Not less. MORE. The confidence is inverse to the reliability on niche topics. That is one of the most counterintuitive things about using these systems and nobody warns you about it.

Ask it to cite its sources. Then check them. This is the single fastest confabulation test and almost nobody does it. Fabricated citations collapse under ten seconds of actual searching — the paper doesn’t exist, or the author doesn’t exist, or the paper exists but doesn’t say anything close to what the model claimed it says. You will be amazed how often this fails. You will also be a little sad.

Flip the question. Ask the same thing worded differently. Real knowledge survives rephrasing — the model generates from the same underlying information and you get a consistent answer. Confabulation regenerates from statistical noise each time and changes its story. If you ask twice and get contradictory specifics, you’ve caught it.

You don’t need to verify everything. You need to verify the class of questions where confabulation is likely. Trust the model more on the class where it’s grounded. Trust it much less on the class where it isn’t. Most people can’t tell the two classes apart and that’s the actual problem.

Okay. Here’s the part I’m going to say once and then never again because it makes me tired.

The companies that build these models know this. Of course they know it. Every major lab has published research about confabulation. It has names. Faithfulness problems, factual grounding, calibrated uncertainty, truthfulness objectives. There are entire sub-fields of machine learning research trying to solve it. Papers. Benchmarks. Conferences. Teams.

None of it is solved. Not really. The best current mitigations — retrieval-augmented generation, citation-grounded models, constitutional approaches — they reduce the rate. They don’t eliminate it. The underlying architecture still fills gaps. It is what the architecture does. The hedging layer gets a little stickier. The rate comes down a little. The fundamental behavior is untouched.

But the marketing doesn’t say that. The marketing says “helpful AI assistant” and “answers your questions” and “your new teammate.” It doesn’t say “confident gap-filling machine with persistent calibration issues and a cosmetic humility module that is easy to bypass.” And so the average person opens a chat window and assumes they’re talking to something that reports what it knows — like a coworker, or a librarian, or an encyclopedia.

They’re not.

They’re talking to something that reports what’s statistically likely to sound correct given what you just asked. Those two things overlap a LOT — which is why the tool is actually useful a lot of the time. But they are not the same thing. And the gap between them is where every AI-assisted factual error comes from. Every single one. And it is not going away. And they know it is not going away. And they are not going to be the ones to tell you.

So I’m telling you.

[quieter]

That’s the whole episode honestly. The rest is practice.

Yes.

[exhales]

Yes. And stop feeling stupid when it fools you. The system is designed to sound trustworthy. That’s the entire optimization objective for the conversational layer. Your brain is doing what brains do — reading a confident delivery and assigning credibility. That’s not a you failure. That’s the model doing what it was built to do, landing on your very normal cognitive machinery.

Use it for the things it’s good at. Drafting. Rewriting. Brainstorming. Summarizing things you’re about to read anyway. Explaining concepts you can cross-check against other sources. Walking you through reasoning you can actually evaluate step by step.

Don’t use it as a replacement for a reference book when the reference needs to be specific and you can’t verify it. Don’t trust cited papers without actually checking the citations. Don’t take numerical claims at face value on niche topics.

And when something feels too clean, too confident, too specific for the model to plausibly know — that’s your instinct telling you it’s confabulating. Trust that instinct. It’s usually right.