About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalization and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention ← You are here
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

The Key/Value cache (KV cache) is one of the main reasons autoregressive Transformers can generate more than one token efficiently. Instead of recomputing every previous token’s Key and Value vectors at every decoding step, the model stores them and reuses them. That sounds like ordinary memorization, and much of the intuition is exactly the same.

The details matter, however: the cache exists for each decoder layer, its capacity depends on the number of KV heads, and its logical sequence order need not match where its physical memory blocks are placed.

Why earlier Keys and Values are reusable

In a causal decoder, position $t$ cannot depend on future positions $t+1, t+2,\ldots$. If the model is evaluated deterministically with fixed parameters and consistent positional handling, appending a new token does not change the hidden states of earlier positions. As a result, earlier layers’ Key and Value vectors remain valid when the sequence grows.

This is the mathematical basis for cache reuse. The assumption has limits: changing a sliding context, shifting positions, using noncausal components or deliberately altering earlier states can invalidate cached entries.

A new Query must generally be able to consult the whole permitted Key/Value history, not just the last Key and Value. If two past Values have equal attention scores, their mixture is an average of both. Keeping only the most recent Value would produce a different answer.

Every layer has its own cache

Transformer layers use different learned projection weights and different hidden states. Layer 5 cannot generally reconstruct its historical Keys from Layer 12’s cache. Each layer therefore needs the historical K/V vectors derived from its own inputs. This is why multiplying the per-layer cache by the number of layers is essential for sizing.

Let $L$ be the number of decoder layers, $T$ retained tokens, $H_{kv}$ KV heads, $d$ features per Key or Value head, and $b$ bytes per stored element. Assuming equal Key and Value head widths, the raw cache size per sequence is

The factor 2 means Keys plus Values, not two copies of the same cache. The formula excludes padding, allocator metadata, fragmentation, replication, temporary buffers and other serving-engine overheads.

Why MHA, GQA and MQA produce different cache sizes

Consider a decoder with $L=32$ layers, $T=8,192$ retained tokens, head width $d=128$, and two-byte values. If it stores 32 KV heads, its raw cache is

.Keeping everything else fixed, GQA with eight KV heads uses 1 GiB. MQA with one KV head uses 0.125 GiB, or 128 MiB. The ratio is 32:8:1 because the stored KV-head count changed. The Query-head count does not need to decrease: 32 Query heads can share eight KV heads, four Queries per KV group.

Lower cache storage may help memory residency, concurrency or bandwidth, but the formula alone does not prove an equivalent speedup or unchanged model quality. Query heads still perform their own attention work using the available shared K/V representations.

How PagedAttention changes placement, not attention mathematics

A growing sequence rarely has a convenient, permanently known maximum length. Allocating one huge contiguous cache area for every request can waste capacity. PagedAttention borrows the idea of virtual-memory paging: the serving engine maintains a logical sequence of cached tokens while the physical KV blocks can be allocated at different locations.

Suppose each physical block holds 16 token positions and the logical block table is $\left[7,2,11,5\right]$. Where does logical token position 35 live, using zero-based indexing? Its logical block index is $\lfloor35/16\rfloor=2$ and its offset is $35\;mod\;16=3$. Block-table entry 2 points to physical block 11, so the token is in physical block 11, offset 3.

That translation changes where the cache resides, not the logical order in which attention reads Keys and Values. If the retrieval produces exactly the same logical vectors with the same allowed positions, the mathematical attention output is unchanged in exact arithmetic. The actual kernel may differ slightly in floating-point rounding or performance.

PagedAttention and FlashAttention solve different problems

The names sound alike, but they address different costs. FlashAttention uses tiled online softmax to reduce temporary attention-intermediate memory traffic. PagedAttention manages the long-lived historical K/V cache using logical-to-physical block mappings. A serving system may combine both ideas. One is primarily about how attention is computed; the other is about how cached history is stored and addressed.

Takeaway. KV-cache sizing is driven by layers, retained positions, KV-head count, head width and bytes per element. GQA and MQA reduce the number of cached KV heads; paging reduces allocation constraints without changing logical attention order.

Next: Part 14 looks behind the forward pass to gradients, multi-GPU execution and the limits of attention visualizations.