Natural Language Processing Basics

Natural language processing (NLP) is the branch of AI concerned with getting a computer to work with human language — text and speech — which is a harder problem than it looks, because language is ambiguous, context-dependent, and doesn't come with a fixed set of rules the way arithmetic does.

Tokenization: turning text into numbers

A neural network, as the previous lesson covered, only ever works with numbers — weighted sums and activation functions. Text has to be converted into numbers before any model can touch it, a process called tokenization. A tokenizer splits text into smaller pieces called tokens — often words, or common word-fragments (subwords) for less common words — and each distinct token is mapped to a number. "unbelievable" might get split into tokens like un, believ, and able rather than staying one whole word, which lets the model handle words it's never seen as whole units, as long as it's seen their pieces before.

Classic NLP tasks

  • Sentiment analysis — is this review positive, negative, or neutral?
  • Named entity recognition — picking out people, places, organizations, and dates from a block of text.
  • Machine translation — converting text from one language to another.
  • Summarization — condensing a long document down to its key points.

Each of these is a supervised learning problem in the sense from the previous lesson: a translation model, for example, is trained on huge numbers of paired sentences (the same sentence in two languages) and learns the statistical relationship between them, the same fundamental "learn a pattern from labeled examples" idea, just applied to language.

What a language model is doing, at a high level

Modern language models — the technology behind today's chat systems — are trained on a task that sounds almost too simple to be useful: given some text, predict the single next token. Trained on enormous amounts of text, repeating this one prediction task billions of times, a model ends up encoding a great deal of statistical structure about grammar, facts, and reasoning patterns, purely as a side effect of getting extremely good at "what token comes next." Generating a whole response is just this next-token prediction repeated over and over, each new token feeding back in as more context for predicting the one after it. (This course's separate Gen AI course goes much deeper into how these models are actually built and used — this lesson is only the conceptual foundation.)

Fluent isn't the same as correct: because a language model is fundamentally predicting plausible-sounding next tokens, it can produce text that reads as completely confident and grammatically perfect while being factually wrong — a failure mode usually called hallucination. Nothing about the underlying mechanism guarantees truth; it guarantees statistical plausibility given the training data. Treat fluent output as a starting point to verify, not a citation.
PreviousNeural Networks Basics