From Token Vectors to GPU Inference

About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis. Each post can be published separately; equations, diagrams and examples are selected and simplified from the accompanying technical masterclass rather than copied wholesale.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbours
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$? ← You are here
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalisation and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

When one token needs information from other tokens, two different questions arise: Which positions are worth listening to? And what information should be taken from them? Self-attention separates these jobs with three learned projections: Query (Q), Key (K) and Value (V).

The database analogy is useful here. A Query resembles a search request, a Key resembles the index entry used for comparison, and a Value resembles the record content. But unlike a normal lookup returning one exact row, attention usually mixes information from many permissible positions.

Three projections from the same hidden states

Let  contain the hidden-state vectors entering an attention layer. For $T$ token positions of model width $d_{model}$, $X$ has shape $T\times d_{model}$ if we omit the batch dimension. Three learned weight matrices produce new views of those same token vectors:

These are not three copies of $X$. They are distinct trained transformations, so the representation that is useful for matching can differ from the representation that is useful for carrying information. In an illustrative small model, each projection might return vectors of width 2 even when the model hidden-state width is 4.

We often use these roles as a memory aid: a Query asks, a Key is compared, and a Value contributes to the output. That is a description of the computation, not evidence that each vector has a human-readable linguistic meaning.

What a dot product actually measures

For one Query $q=\left[1,0\right]$ and one Key $k=\left[1,1\right]$, the dot product is

A larger dot product indicates greater learned numerical compatibility under the current projections. The score is not yet a probability. It can be negative, greater than one, or affected by vector magnitude. Later, softmax converts a Query’s scores across Keys into nonnegative weights that sum to one.

For Query token  and Key token , with shared feature width $d_k$, the unscaled score is

The index  runs over features, whereas  and  identify token positions. Confusing those two axes is a common cause of shape errors.

From one score to a whole score matrix

Suppose there are three Query positions and four Key positions, each with two comparison features. Then $Q$ has shape $3\times2$, $K$ has shape $4\times2$, and transposing  changes its shape to $2\times4$. Their product is

The output has three rows, one for each Query, and four columns, one for each candidate Key. The inner dimension of two matches because both vectors occupy the same comparison space. Transposition only swaps axes: it cannot make a two-feature Query compatible with a three-feature Key.

Why Value width can be different

Queries and Keys must have compatible widths because they participate in a dot product. Values enter a different operation. If the attention-weight matrix is $3\times4$ and there are four Values of width 5, multiplying the weights by  produces $\left(3\times4\right)\left(4\times5\right)=3\times5$. Each Query receives a five-feature output.

This distinction is easy to lose in diagrams where Q, K and V happen to have identical shapes. The Value width $d_v$ is an architectural choice; it is not logically required to equal $d_k$.

Why three projections are useful

Imagine choosing expert reviewers for a technical document. The attributes used to decide whose opinion is relevant need not be the same as the information each reviewer contributes. Attention learns this separation: Queries and Keys determine weights, while Values provide the quantities mixed by those weights.

In a trained model, all these representations are distributed, context-dependent vectors. We should not assert that one Query means “question” or one head means “grammar”. Instead, the computation is precise: compare Q with K, transform scores into weights, then weight V.

Takeaway. Attention has two stages that must not be conflated. $QK^\mathsf{T}$ creates a matrix of compatibility scores across token positions. Only after scaling and softmax do those scores become weights for mixing the Value vectors.

Next: Part 3 asks why these dot-product scores are divided by the square root of the head width.