Text Generation Parameters

Almost every LLM API exposes a small set of parameters that control how the next-token prediction from lesson 2 actually gets turned into output. Understanding what each one does removes a lot of guesswork.

Temperature: how much the model gambles

Recall that at each step, the model computes a probability for every possible next token. Temperature controls how strictly it sticks to the single most likely one. A low temperature (near 0) almost always picks the top choice, giving focused, repetitive, predictable output — good for factual or code-generation tasks. A higher temperature (1.0+) allows less-likely tokens through more often, giving more varied, creative, sometimes less coherent output.

Max tokens: a hard ceiling, not a target

max_tokens caps how long a response is allowed to get. It doesn't tell the model to aim for that length — the model just gets cut off mid-generation the instant it's reached, which can leave a response ending mid-sentence if it's set too low for what you asked for.

Top-p (nucleus sampling)

top_p is a second, related way to control randomness: instead of considering every possible token, it only considers the smallest set of top tokens whose combined probability reaches the given threshold (e.g. 0.9 means "the most likely tokens that together make up 90% of the probability"), then samples from just that narrowed set. Most APIs recommend adjusting temperature or top-p, not both at once, since they both affect randomness in overlapping ways.