About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalization and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem ← You are here
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

Modern language models generate text one token at a time, yet they learn from long sequences of tokens that are already known. This sounds contradictory until we distinguish two things: the information a prediction is allowed to use, and whether several training positions can be evaluated together.

Causal self-attention enforces the first rule. Parallel GPU computation makes the second possible. Together they explain how a model trained on complete sequences can later generate unknown tokens sequentially.

Autoregression in one equation

An autoregressive model predicts the next token from the prefix available so far. If a sequence is $x_1,x_2,\ldots,x_T$, the probability chain rule expresses its joint probability as

Read this as a product of conditional probabilities: the probability assigned to token  depends on the preceding tokens. The right-hand terms are next-token probabilities, not the attention weights that describe how one Query mixes Value vectors within a layer.

A decoder cannot meaningfully condition on an unknown future generated token. During generation, it must use the prefix available now, choose the next token, extend the prefix, and repeat.

How a known training sequence helps

During training, the example sequence is already provided by the dataset. Suppose the tokens are “The”, “cat”, “sleeps”, “on”, “the”. The model can process the sequence with causal masking, producing one set of logits at each input position. Each position is trained to predict the following token, so targets are shifted by one position.

The position reading “The” should learn to predict “cat”; the position reading “The cat” should learn to predict “sleeps”; and so on. The future target is available to the loss calculation after the model has made its prediction, but the predicting position cannot look ahead through attention to obtain the answer prematurely.

Because the full training sequence exists up front, many masked Query rows and their losses can be computed in parallel. This is not the same as parallelizing the actual choice of previously unknown next tokens during free generation.

Logits, probabilities and loss are different quantities

A logit is an unnormalized score for a possible vocabulary token. A vocabulary softmax converts logits into a probability distribution over possible next tokens. The training target identifies which vocabulary token should have been predicted.

A widely used objective is cross-entropy. For a correct target token assigned probability , its contribution to the loss is . This makes intuitive sense: a confident correct prediction has  close to one, so the loss is small; assigning the correct target an extremely small probability gives a large penalty.

For example, $-ln\left(0.9\right)\approx0.105$, whereas $-ln\left(0.1\right)\approx2.303$. Across  scored positions, an illustrative average objective is

where $y_t$ is the correct next-token target at position $t$ and $p_t\left(y_t\right)$ is the model probability assigned to that target. This $p_t$ is not the attention distribution over Keys.

Where Q, K and V learn their roles

The matrices $W_Q$, $W_K$ and $W_V$ are trained parameters. There is no human-written rule that assigns one head the job of recognising verbs or another the job of resolving pronouns. Instead, a prediction error generates gradients that travel backward through the model.

For a parameter , simple gradient descent has the form

where $\eta$ is the learning rate. A change to a Query projection may alter attention scores, which may alter softmax weights, Value mixtures, later hidden states, vocabulary logits and finally the loss. Training adjusts parameters through these linked sensitivities. Real optimisers may be more sophisticated than basic gradient descent, but the principle is the same.

A common confusion about probabilities

An attention head produces weights across Key positions: “How much of each Value vector is mixed into this Query’s contextual result?” A language-model output layer produces probabilities over vocabulary tokens: “Which token might be generated next?” These are both called probabilities or distributions, but they live in different spaces and serve different purposes.

An attention weight of 0.8 for a source position does not mean that the model has an 80% chance of generating the word at that position. It means that, within the particular head and Query row, 80% of that head’s attention weight was assigned to the Key position, before all later layers and output processing.

Takeaway. Autoregression defines the permitted information dependency for text generation. Shifted targets and causal masks let an entire known training sequence supply many supervised next-token examples in parallel. Gradients, not manually assigned linguistic roles, shape the attention projections and FFNs.

Next: Part 12 follows a real inference request through prompt prefill, the first generated token and incremental decoding.