About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalization and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention ← You are here
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

Transformers rely heavily on matrix multiplication, which is good news for GPUs. But a common mistake is to predict attention speed from arithmetic counts alone. A GPU can have impressive compute throughput and still spend significant time moving data between high-bandwidth memory (HBM) and smaller on-chip working areas.

We need to distinguish three things: how many operations must be done, how much intermediate data is stored, and how often that data crosses memory boundaries.

How attention grows with sequence length

Consider batch size $B$, number of heads $H$, sequence length $T$, and head width $d$ (here $d$ means per-head width, not the full model width). The $QK^\mathsf{T}$ product compares every Query with every Key. There are $T^2$ Query-Key position pairs per head.

A dot product over  features costs about $2d$ floating-point operations when a multiply and an add count as two operations. Therefore the approximate work for one full-sequence score matrix across the batch is

The probability-by-Value product costs about the same if Value width also equals $d$, giving roughly $4BHT^2d$ for those two matrix products together. These estimates exclude the projections, softmax, normalisation, FFN and other operations.

Doubling context length makes this dense pairwise work about four times larger. Increasing context length fourfold makes it about sixteen times larger. Tokenwise FFN projections, in contrast, scale linearly in token count when their widths are fixed.

Memory can grow quadratically too

One attention score matrix contains $BHT^2$ entries. If each stored entry uses  bytes, its size is

For $B=1, H=32, T=4,096$  and two bytes per entry, a single dense intermediate matrix occupies $1,073,741,824$ bytes, or 1 GiB. If a naïve implementation materializes both full scores and full softmax probabilities in that representation, those two buffers alone approach 2 GiB.

This is not a claim that every modern GPU kernel stores both matrices or that 2 GiB is the entire model memory requirement. It is a controlled illustration of why naïvely materializing attention can become expensive.

Why transferring bytes changes performance

HBM is large and high bandwidth, but it is farther from the arithmetic units than on-chip registers and shared memory. These on-chip resources are small, fast working areas. A straightforward attention implementation might write all scores to HBM, read them to compute softmax, write probabilities, and read them again to mix Values.

Even when the arithmetic is efficient, these round trips can dominate parts of the kernel. Arithmetic intensity measures useful arithmetic per byte moved at a given memory level. A kernel with limited data reuse can be bandwidth-bound; a kernel with ample reuse and enough work may be compute-bound.

A large FLOP number is not a benchmark

For example, with $B=1, H=32, T=4,096$  and $d=128$, the two dense attention matrix products together require roughly 275 billion arithmetic operations. That is an operation count, not 275 billion operations per second and not a prediction of milliseconds. Runtime depends on GPU architecture, data type, kernel selection, memory movement, batching and many other components.

Similarly, two models with the same theoretical FLOP count may differ in wall-clock latency because their tensor shapes map differently to GPU kernels or because they move different amounts of data between memory levels.

Where the FFN fits in the cost picture

A conventional FFN of model width  and intermediate width  needs about  dense-matmul FLOPs. That part scales linearly with token count. Yet its weight matrices may be substantial, and low-batch decoding may fetch large weight tensors for relatively little new-token work.

This explains why an “attention bottleneck” should not be assumed from the model name. In a real workload, attention, FFN matrices, KV-cache reads, normalization, kernel launch overhead or scheduling can each matter.

Takeaway. Count both operations and bytes. Dense attention has a quadratic token-pair term, but actual GPU performance depends heavily on data movement and reuse. The next post explains how FlashAttention changes the execution strategy without changing the dense attention formula.

Next: Part 10 makes the central FlashAttention trick understandable through streaming weighted averages.