How Large Language Models Work

You don't need the underlying math to build real intuition for what a large language model (LLM) is actually doing when it answers a prompt. At its core, it's doing one very specific thing, over and over: predicting the next token.

Tokens, not words

An LLM doesn't read or write in whole words. Text is first broken into tokens — sometimes a whole word, sometimes a fragment ("generat" + "ive"), sometimes a single character or punctuation mark. A rough rule of thumb: 100 tokens is about 75 words of English. Everything the model does happens at the token level.

Next-token prediction

Given everything written so far (your prompt, plus whatever the model has already generated in its response), the model calculates a probability for every possible next token, then picks one — usually not always the single most likely one, which is where the "temperature" setting from lesson 5 comes in. That chosen token gets appended, and the whole process repeats: predict the next token, append it, repeat. A full response is just this loop running hundreds or thousands of times.

Where the "knowledge" comes from

During training, the model is shown enormous amounts of text and adjusts billions of internal numeric parameters so its next-token predictions get statistically better at matching real text. Nothing is stored as explicit facts in a lookup table — everything the model "knows" is compressed into these parameters as patterns, which is exactly why it can state something fluently and confidently while being factually wrong (more on this in lesson 7).

Note: unless a system explicitly adds it (a technique called retrieval-augmented generation, outside the scope of this course), a plain LLM has no live connection to the internet or a database while generating a response. It isn't "looking anything up" — it's reconstructing an answer from patterns learned once, during training, which may be months or years out of date.
PreviousWhat Is Generative AI?