About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalization and Residual Connections ← You are here
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

Attention is the famous part of a Transformer, but an attention output alone is not a complete block. Modern models also use normalization to control feature magnitudes and residual connections to preserve and update the running representation. These two mechanisms are not decorative extras. Their position in the computation changes the actual forward values and the gradient paths available during training.

Residual connections: update rather than replace

Suppose a software system starts with state $x$ and computes a correction $f\left(x\right)$. Instead of discarding the original state, it can return $x+f\left(x\right).$ That is the intuition behind a residual connection:

The sum is coordinate-wise. It requires $F\left(x\right)$ to have the same shape as $x$. Attention therefore applies an output projection to turn concatenated head outputs back to $d_{model}$ before adding them to the residual stream. The FFN also contracts its output back to the model width for the same reason.

In deep networks, a residual gives information and gradients a direct additive route across the sublayer. It does not solve every optimisation problem or guarantee that arbitrarily deep models will train, but it changes the architecture from “replace everything at each layer” to “learn useful updates.”

What exactly does a normalizer normalize?

For a standard Transformer token-wise normalizer, the statistics are calculated across features within one token. If the input tensor is $B\times T\times d_{model}$, normalization preserves that shape. It does not average across all tokens, silently inspect future tokens, or change the sequence length.

LayerNorm first subtracts the feature mean $\mu$, then divides by a measure of feature spread. With scale vector $\gamma$, optional shift $\beta$, and stabiliser $\epsilon$:

Here $\odot$ means elementwise multiplication. The variance $\sigma^2$ is calculated with division by feature count rather than the sample-statistics correction. The small positive $\epsilon$ helps prevent division by zero.

RMSNorm is different: it does not subtract the mean. It divides by the root mean square, then applies a learned scale:

RMSNorm can therefore retain nonzero mean direction information that LayerNorm removes. They are related mechanisms, not interchangeable spellings for one operation.

Compare both with one short example

Let a token vector be $x = \left[1,3\right]$, choose $\epsilon = 1$ deliberately to make its influence visible, and take $\gamma=\left[1,1\right]$, $\beta=\left[0,0\right]$.

For LayerNorm, the mean is $\left(1+3\right)/2=2$. Centering gives $\left[-1,1\right]$, whose population-style variance is 1. Dividing by $\sqrt{1+1}=\sqrt2$ yields approximately $\left[-0.7071,07071\right]$.

For RMSNorm, the mean square is $\left(1^2+3^2\right)/2=5$. Dividing the original vector by $\sqrt{5+1}=\sqrt6$ gives approximately $\left[0.4082,1.2247\right]$. These are visibly different outputs even before learned scales or biases are applied. The exaggerated $\epsilon=1$ is for teaching, not a recommended production setting.

Pre-LN versus Post-LN is a genuine design choice

In a pre-normalization (Pre-LN) decoder block, the first normalizer prepares the attention input. The attention update is then added to the unnormalized residual stream. A second normalizer prepares the FFN input, and the FFN update is added to its own residual stream:

In this notation, the Attention function includes the learned Q/K/V projections, the attention operation and the output projection. Post-LN places normalization after each residual addition. The order affects both numerical outputs and backpropagation; you cannot swap the order without changing the model.

Why the gradient path matters

A simple derivative illustrates the benefit of a residual route. If $y=x+F\left(x\right)$, the local derivative contains an identity term:

where $I$ is the identity matrix and $J_F$ is the Jacobian, the matrix collecting local partial derivatives of $F$. In a Pre-LN block, this direct additive path bypasses the sublayer’s internal normalization and transformations. In Post-LN, the output normaliser acts on the sum, so its Jacobian also affects that route.

This is one reason normalization placement matters for deep-network optimization. It is not proof that one design always wins under all settings. Different architectures and training schemes choose different compromises.

Takeaway. Residuals preserve a model-width information stream while sublayers compute updates. LayerNorm and RMSNorm control token-feature statistics in different ways. Pre-LN and Post-LN differ in calculation order and gradient flow, not merely in diagram aesthetics.