From Token Vectors to GPU Inference

About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis. Each post can be published separately; equations, diagrams and examples are selected and simplified from the accompanying technical masterclass rather than copied wholesale.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbours
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$? ← You are here
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalisation and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

The standard attention formula contains a division by $\sqrt{d_k}$. Why that number? Why not divide by $d_k$, by the model width, or by nothing at all? The answer comes from how the spread of a sum can grow when the sum contains more independent contributions.

We will develop the reasoning using ordinary statistics, make the assumptions explicit, and then show a counterexample that prevents a common misunderstanding.

Begin with one unscaled score

For one Query and Key of width $d_k$, the dot product is a sum of coordinate products:

Think of each $q_rk_r$ as one contribution to the score. If more features are added, the sum may fluctuate more widely. Very large differences in scores can make softmax concentrate its weights on one or two Keys. Scaling aims to moderate this growth.

Two statistical terms matter. The mean is a probability-weighted average. The variance is the average squared deviation from that mean; the standard deviation is the square root of variance. Variance measures spread, not the highest possible value.

For a simple teaching model, assume Query and Key coordinates have zero mean, unit variance, and are independent, including across feature positions. These are mathematical assumptions for this derivation, not guaranteed properties of real learned activations.

Follow the variance through the dot product

Write $Z_r=q_rk_r$. If $q_r$ and $k_r$ are independent, zero-mean, unit-variance quantities, then $E\left[Z_r\right]=0$. Their second moments also factor, giving $E\left[Z_r^2\right]=1$, so $Var\left(Z_r\right)=1$.

With independence across coordinates, variances add:

The standard deviation of the unscaled sum is therefore $\sqrt{d_k}$. Divide the score by that standard deviation and the variance is divided by its square:

For example, width 64 gives an idealised unscaled variance of 64 and a standard deviation of 8. Dividing the score by 8 returns its variance to one under the assumptions.

Why independence makes such a difference

Take four contributions, each with mean zero and variance one. Case A: the four are independent. The variance of their sum is $1+1+1+1=4$. After dividing their sum by $\sqrt4=2$, the variance becomes $4/2^2=1$.

Case B: the four contributions are identical, all equal to the same random variable $Z$. Their sum is $4Z$, so its unscaled variance is $4^2Var\left(Z\right)=16$. After dividing that sum by 2, the result is $2Z$, whose scaled variance is $2^2Var\left(Z\right)=4$.

The two outcomes are not contradictory: the numbers 4 and 16 describe variance before scaling; the numbers 1 and 4 describe variance after scaling. Both cases used the same divisor. Their correlation structure changed the outcome.

Four unit-variance terms	Independent	Identical
Variance of the sum before scaling	4	16
Divide the sum by	2	2
Variance of the sum after scaling	1	4

Covariance is the missing term

Covariance describes whether two quantities tend to fluctuate together. The full variance formula includes every pair of contributions:

With four identical unit-variance terms, there are six distinct pairs, each with covariance one. Thus the unscaled variance is $4+2\left(6\times1\right)=16$. With independent terms, all six covariances are zero and the result is 4. Zero covariance is sufficient for this particular variance calculation; independence is stronger.

Real Transformers learn $Q=XW_Q$ and $K=XW_K$. The projected features can be correlated, have nonzero means, and differ in variance. Dividing by $\sqrt{d_k}$ therefore does not guarantee a variance of exactly one for an actual attention head. It is a useful standard scale correction motivated by a deliberately simplified model.

A final numerical checkpoint

For $q=\left[1,-1,1,-1\right]$ and $k=\left[1,1,1,-1\right],$, the dot product equals 2 and $d_k=4$. The scaled score is $2/\sqrt4=1$. Another pair can legitimately produce 2 or -3 after scaling: the proof is about the distribution of scores under assumptions, not a hard bound that each individual score lies between -1 and 1.

Takeaway. Square-root scaling tempers the typical growth of dot-product magnitude with head width. Its statistical justification is useful precisely because we know what it assumes, and what it does not promise.

Next: Part 4 turns scaled scores into attention weights and calculates a causal three-token example.