About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information ← You are here
6. Part 6 – Normalization and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

One attention head produces one set of Query-Key comparisons and one Value mixture for each token. Why have several heads? Because a single set of learned projections is not obliged to capture every useful relationship. Multiple heads provide multiple learned comparison spaces that can be combined into a richer representation.

Heads are learned, not manually assigned jobs

It is common to hear that one head “finds grammar” and another “finds factual relations.” That may be an intuitive metaphor, but it is not a definition of multi-head attention. Each head has its own learned projections, and what it does emerges from training. Different heads can become redundant, correlated or specialized; no head is guaranteed a neat human label.

In a conventional arrangement, model width $d_{model}$ is divided among $H$ Query heads. For example, $d_{model}=4,096$ and $H=32$ yields head width $d_k=128$ if the architecture partitions width equally.

Each head calculates attention independently, returning Value features for each token. The outputs are then concatenated along the feature axis, not added as probability matrices. A learned output projection $W_O$ mixes the concatenated features back to model width, allowing a residual connection to add the result to the original hidden state.

A two-head shape example

Suppose a token’s hidden vector has width 8. Two heads each produce four output features, so concatenation produces eight features. The output projection can then map $8\rightarrow8$. For a sequence of length $T$, each head returns a $T\times4$ Value mixture; concatenation returns $T\times8$.

The head outputs are not averaged simply because there are two heads. Nor do we need two GPUs: a head is a logical numerical operation, not a hardware processor.

Changing head count at fixed total projected width does not automatically increase Q/K/V projection parameters. With $H$ heads each outputting $d/H$ Query features, the combined Query weight count is $H\cdot d\left(d/H\right)=d^2$. This cancellation is a useful antidote to the misconception that twice as many heads always means twice as many weights.

Multi-head versus grouped-query attention

Many serving architectures distinguish the number of Query heads $H_q$ from the number of Key/Value heads $H_{kv}.$. In conventional multi-head attention (MHA), the counts are equal. In multi-query attention (MQA), many Query heads share one Key/Value head. Grouped-query attention (GQA) uses fewer Key/Value heads than Query heads, with Query heads grouped around those shared K/V representations.

With 32 Query heads and 8 Key/Value heads, a KV cache needs roughly one-quarter as many cached head entries as 32-KV-head MHA, when all other cache dimensions are fixed. That improves serving-memory requirements, though it does not mean Query-head computations disappear. Part 13 derives the byte formula.

Attention also needs to know where tokens occur

Plain content-only self-attention has no intrinsic sequence-order information. If you permute token rows and apply the same operations without positional signals or order-sensitive masking, the outputs permute correspondingly. That is called permutation equivariance. To distinguish “the dog chased the cat” from “the cat chased the dog,” models need positional handling in addition to token content.

Some approaches add learned or fixed positional vectors. Rotary Position Embeddings (RoPE) instead rotate pairs of Query and Key coordinates using angles associated with their positions. A 2D rotation preserves vector length, but changes its alignment with another vector. Thus token position influences the Query-Key dot product.

For rotation matrix $R\left(\theta\right),$, a Query at position $m$ and Key at $n$ can satisfy

The key result is that the pairwise comparison contains the position difference $n – m$. Production RoPE uses many coordinate pairs and frequencies, not merely one rotation of 90 degrees. RoPE does not itself impose causal visibility or guarantee unlimited context extrapolation.

Distinguish order from permission

Positional information answers how tokens are located relative to one another. A causal mask answers which positions a Query is allowed to consult. They do different jobs. A model can use RoPE with a causal mask, but neither mechanism should be mistaken for the other.

Takeaway. Multiple heads create separate learned attention views and an output projection combines them; positional methods help the model distinguish token order. GQA modifies Key/Value sharing primarily to improve inference memory, a topic we will revisit after understanding decoding.

Next: Part 6 supplies the missing architecture glue: normalization, residual connections and why their placement matters.