About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough ← You are here
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalization and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

A Query-Key dot product tells us how strongly one Query matches one Key according to learned numerical projections. But it is not a useful mixing weight yet. Scores can be negative or very large, and their totals mean nothing by themselves. Softmax converts each Query’s list of scores into a distribution over the Keys that position is permitted to see.

Softmax: a weighted-average bridge

Imagine three advisers whose contributions should receive weights that sum to 100%. Attention does something similar: it computes the weights from scores rather than assigning them manually. For one Query with scores $s_1,\ldots,s_n$, softmax gives

Every weight is nonnegative and the weights for one Query sum to one. Importantly, these are attention weights across Key positions, not probabilities of predicted words in the vocabulary.

Softmax only cares about score differences. If two scores are 2 and 1, the ratio of their weights is $e^2/e^1=e\approx2.718$. Adding the same constant to all scores does not change that ratio. This gives a practical numerical trick: subtract the largest score before exponentiating, preserving the result while avoiding unnecessarily huge exponentials.

A small, fully traceable example

Use three projected token positions with two features in each head:

The unscaled score matrix is $QK^\mathsf{T}$. Scaling by $\sqrt2$ gives scores for Query 2 of $\left[0,0.7071,0.7071\right]$. Without any mask, softmax would distribute Query 2’s weight across all three Key positions. But a decoder predicting tokens from left to right must not allow position 2 to read future position 3.

A causal mask gives forbidden future positions a score of conceptual negative infinity before softmax. For Query 2, the third score becomes $-\infty$. Its weight will be exactly zero after softmax, while the first two scores are renormalised together.

Subtract the largest permitted score, $0.7071$, from the first two scores. The exponentials are approximately $\left[0.4931,1\right].$. Dividing by their sum produces attention weights $\left[0.3302,0.6698,0\right]$. The resulting Value mixture is

The output coordinates need not lie between zero and one; the weights do, but the Values can contain any learned feature magnitude.

A mask restricts information, not parallel hardware

For a three-token causal sequence, the visibility pattern is triangular. Query 1 may see Key 1; Query 2 may see Keys 1 and 2; Query 3 may see all three. It is tempting to interpret the triangle as forcing the GPU to compute Query 1, then Query 2, then Query 3. That is incorrect when the full input sequence is already known. During prompt prefill or training, those masked Query rows can be evaluated in parallel.

Masking after softmax is also wrong unless we renormalize: forbidden Keys would already have consumed part of the denominator, leaving surviving weights that sum to less than one. Mask first, softmax second, mix Values third.

One equation, six separate actions

The standard attention expression is compact:

Read it in this order: create learned Q/K/V projections; compute Q-to-K dot products; scale scores; apply an optional mask ; softmax each Query row; multiply those weights by Value vectors. Keeping these actions separate is the best antidote to treating the formula as an unexplained black box.

What this example has not yet built

The result $\left[0.3302,1.3395\right]$ is a contextual hidden vector for one attention head. It is not a token, a word, or a final model prediction. A Transformer usually adds attention heads, an output projection, residual connections, normalization, an FFN and more blocks before a vocabulary projection produces next-token scores.

Takeaway. Softmax turns compatibility scores into per-Query weights; causal masking determines which Keys are eligible; the weighted sum of Values creates a new contextual representation. These are distinct operations with distinct tensor shapes.

Next: Part 5 asks what multiple attention heads do and how a model represents token order.