Limitations & Hallucinations

The single most important thing to understand about generative AI before using it for anything that matters: it can state something false with exactly the same fluent confidence as something true, and there's no built-in signal telling you which one you got.

What a hallucination actually is

A hallucination is when a model generates a plausible-sounding statement, citation, function name, or fact that is simply wrong or doesn't exist. It isn't the model "lying" — it has no concept of true or false. Remember from lesson 2 that it's predicting statistically likely next tokens; a fabricated citation formatted exactly like a real one is, token by token, a highly likely thing to generate, whether or not the citation is real.

Why confident wrong answers happen

The model's fluency (grammar, tone, structure) comes from the same next-token process as its factual claims, so an answer can be perfectly, confidently written while being completely incorrect. There's no separate "how sure am I of this fact" signal exposed alongside the text — confidence in tone tells you nothing about factual accuracy.

Verifying instead of trusting

For anything where being wrong has real consequences — a citation, a statistic, a legal or medical claim, a piece of code that will run unreviewed — treat the output as a draft that needs independent verification, not a finished, trustworthy answer.