How Language Models Work

Someone types a question into a chatbot and the reply appears a little at a time, left to right, as if an invisible hand were typing it live. That streaming is not a stage effect. It is the model working the only way it can, producing one small piece of text, then the next, then the next, each piece chosen from everything written so far.

A large language model does exactly one thing. It guesses what comes next. Every trick a chatbot seems to pull, answering a question, translating, writing code, arguing a point, is that single operation run in a loop. This lesson takes the trick apart, because once the mechanism is clear the tool stops looking like magic and starts looking like something that can be aimed.

The one thing a model does

Start with the word model, since it gets thrown around loosely. A model is a mathematical function with a huge pile of adjustable numbers inside it. Text goes in, a prediction comes out, and those numbers are what decide the prediction. A large language model is a model whose input and output are both text. The large part refers to the count of the adjustable numbers, now in the billions, and to the mountain of text the model learned from.

The single operation is this. Given a stretch of text, the model predicts which token is most likely to come next. Then it adds that token to the text and predicts again, reading the slightly longer text back in as the new input. It repeats until it decides to stop. Producing output one step at a time and feeding each step back in has a name: autoregressive generation.

A concrete pass makes it plain. Feed the model the text The capital of France is. It does not look up France in a table of countries. It produces a score for every token it knows, a number for how likely each one is to come next, and the token Paris lands near the top. The model picks a token, appends it, and now the input is The capital of France is Paris, which it reads back in to choose the token after that.

It does not always take the single highest-scoring token. A setting called temperature controls how much randomness goes into the pick. Low temperature makes the model almost always take the top choice, which reads as focused and a little repetitive. Higher temperature lets it reach for less likely tokens, which reads as more varied and more prone to going off the rails. This randomness is why the same prompt can produce a different answer twice in a row.

What a token is

Notice that the last section kept saying token, not word and not letter. The distinction is not fussy. It is how the whole system reads. A token is a chunk of text: often a whole common word, sometimes a piece of a longer word, sometimes a space plus a word, sometimes a lone punctuation mark. Before a model sees any text, the text is broken into tokens, a step called tokenization, and each token is swapped for a number, an ID drawn from a fixed vocabulary of roughly one hundred thousand tokens.

Watching it happen removes the mystery. The library below is tiktoken, the open-source tokenizer for OpenAI's GPT models, and cl100k_base is the specific vocabulary the GPT-4 family uses. This code splits one sentence and prints the pieces.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
text = "The model predicts the next token."
ids = enc.encode(text)
pieces = [enc.decode([i]) for i in ids]

print(pieces)
print(len(ids), "tokens")

$python3 tokens.py

Read the pieces slowly, because each oddity is teaching something. The at the start and the in the middle are two different tokens, since the second one carries a leading space; the model folds a space into the token that follows it. The everyday words model, next, and token are each a single token. The period is its own token. And a longer or rarer word gets no special treatment, it just splits: run the word Tokenization through the same code and it comes apart into Token and ization.

Why chop text up this way instead of feeding whole words or single letters? A fixed vocabulary of word pieces can represent any text at all, including typos, names, code, and other languages, by falling back to smaller pieces when a word is rare, while still spending just one token on the common words that fill most writing. A whole-word vocabulary chokes on anything unusual. Letter-by-letter would make every input enormous and slow.

Two practical numbers fall out of this. In English, one token runs about four characters, or roughly three quarters of a word, so a 1,000-word essay lands on the order of 1,300 tokens; other languages can cost far more tokens for the same meaning. And a token ID is not fed to the model as a bare number. It is first turned into a list of numbers called an embedding, which places the token as a point in a space where related meanings sit near each other. Embeddings are how raw numbers come to stand for meaning, and a later skill builds directly on them.

The context window

A model cannot take in unlimited text. The run of tokens it can hold and work on at once is the context window, measured in tokens: the maximum number the model can read in and generate in a single pass, input and output counted together. Current models range from tens of thousands of tokens to well over a million. Those figures climb every year, so treat any specific number as a snapshot, not a constant.

The window is the model's entire world for one request. There is no notepad off to the side, no long-term store it quietly consults. Whatever sits in the window is everything the model can use, and whatever falls outside it might as well not exist. Asking a model about a long document means the document has to fit in the window as tokens; past the limit, something gets cut.

One more consequence sets up the limits later in this lesson. Every single time the model produces a token, it reads the whole window again from the beginning. It is not remembering the sentence it wrote a moment ago the way a person holds a thought. Those earlier tokens are literally part of the input on every step. The model keeps no state between steps except the growing text itself.

How training turns text into prediction

The natural question is where the ability to predict a good next token comes from. It is learned, in a process worth sketching even in rough form. Real training has more moving parts than this, and treating it as two clean phases is a simplification, but the shape is right.

Phase one is pretraining. The model starts with its billions of numbers set to random values, so its early predictions are noise. It is then fed an enormous amount of text, trillions of tokens pulled from books, code, and web pages. At every position in all that text, the model predicts the next token, its guess is compared against the token that actually came next, and every one of its numbers is nudged a tiny amount in the direction that would have made the true token more likely. That nudging, repeated across trillions of examples, is the training. The method has a name, gradient descent, carried out by a procedure called backpropagation, but the idea underneath is just guess, check, adjust, on a colossal scale.

What comes out is worth stating flatly, because it clears up a lot of confusion. The model ends as a fixed set of numbers that captures patterns in text: grammar, common turns of phrase, the shape of a valid function, facts that showed up over and over. There is no database inside it and no stored copy of the training text. When a model reliably states that water boils at 100 degrees Celsius, that is a pattern baked into the numbers because it appeared so often, not a fact the model looked up.

Two things follow. Once training ends, the numbers are frozen, which fixes a knowledge cutoff: the model knows nothing that happened after its training data was gathered, and nothing about the specific person it is about to talk to. And pretraining alone yields a model that continues text, not one that answers a question helpfully. A second phase, post-training, uses example conversations and human ratings to steer the model toward following instructions and being useful; the common technique is called reinforcement learning from human feedback, or RLHF.

What a model cannot do

Three limits catch beginners hardest, and all three fall straight out of the mechanism above.

A model has no memory between calls. Programs talk to a model through an API, an application programming interface, which is just an agreed way for one program to send a request to another and get a response back; the program sends text and receives generated tokens. Each call stands completely alone. The model carries nothing over from the previous one. A chat that seems to remember what was said ten messages ago works because the application resends the entire conversation as part of every new request. Clear that history and the memory is gone, because there was never any memory in the model to begin with.

A model can be confidently wrong. When a model produces fluent, plausible text that happens to be false, that is called a hallucination, and it is not a glitch, it is the mechanism showing through. The model predicts likely tokens, not true ones, and a fabricated citation or a made-up function name can be built from exactly the same likely tokens as a real one. Nothing inside checks a claim against the world. The wrong answer arrives with the same calm confidence as a right one, which is what makes it dangerous.

A model does not truly understand in the human sense. It has learned patterns in text, not the things the text points at; it has processed the word Paris millions of times and has never seen a city. Whether that adds up to a kind of understanding is a genuine debate among researchers, and this lesson will not settle it. For anyone building on top of these models, the useful stance is the cautious one: treat the output as a statistically likely continuation of the input, not as knowledge the model checked and stands behind.

That last limit is the opening for everything that follows. A model outputs the most likely continuation of its input, so the way to get a reliable answer is to build an input whose most likely continuation is the answer wanted. Shaping that input on purpose is prompt engineering, and it is the next lesson.