About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalization and Residual Connections
7. Part 7 – Trace One Complete Transformer Block ← You are here
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

A common Transformer explanation stops just as it becomes useful. It shows attention as a box, adds an FFN box beside it, and calls the result a Transformer. But how does a token actually travel through those boxes? We can trace one complete, deliberately tiny decoder block while keeping tensor shapes and numerical roles visible.

Our source masterclass calculates a three-token, two-feature Pre-LN decoder block and verifies its outputs with a deterministic NumPy implementation. Here we will preserve the important transitions without reproducing every multiplication in that longer derivation.

The starting configuration

Take one sequence with three token positions: batch size $B=1$, token count $T=3$, model width $d_{model}=2$. There is one Query head and one Key/Value head, with $d_k=d_v=2$. The FFN temporarily expands to three features. This is much smaller than a production model, but the types of operations are the same.

The block input is a hidden-state tensor, not raw token IDs:

Our notation suppresses the leading batch dimension for clarity. The model uses RMSNorm with $\epsilon=1$, a purposely large value chosen only to make the hand calculations easy to inspect. The Query and Key projections are identity matrices in this toy case; the Value projection doubles the second feature. Trained models need not use these special weights.

First normalize, then project Q/K/V

Apply RMSNorm separately to each token row to obtain $N_1$. The first row  has mean square $0.5$, giving $\left[1/\sqrt{1.5},0\right]\approx\left[0.8165,0\right]$. The third row  becomes approximately .

Then project the normalized states:

Every projection preserves the three positions. Query and Key have the same comparison width, so $QK^\mathsf{T}$ creates a $3\times3$ score matrix. The causal mask forbids future positions, and stable softmax normalises each permitted row.

Watch the third token use context

The third Query can consult all three Keys. Its scaled scores in the source example are approximately $\left[0.408248,0.408248,0.707107\right]$. Softmax gives weights $\left[0.298657,0.298657,0.402685\right]$.

Multiplying those weights by the three Value vectors gives the third contextual attention mixture:

The learned output projection $W_O$ mixes features and changes this to $o_3\approx\left[0.528594,1.321486\right]$. Now comes the first residual addition: the original input row $x_3=\left[1,1\right]$ is added to the projected attention result, producing

The distinction matters: the residual adds the projected attention output, not the attention probability row and not the raw Value tensor.

The FFN transforms features, not tokens

A second RMSNorm takes the updated residual $R$, producing $N_2$. For the third token the normalised row is approximately $\left[0.693174,1.052728\right]$. The FFN applies an expansion matrix, an activation and a contraction matrix. In this example, the activation is ReLU: negative intermediate features are set to zero and positive ones pass through.

The FFN uses the same learned weights at every token position, but it computes each row independently. The cross-token communication already happened in attention. Finally, a second residual addition produces $Y=R+F$, preserving the original number of tokens and the external feature width.

Verify the result and interpret it correctly

The source implementation reports the final hidden-state matrix as

There are still three rows and two columns. The entries are learned-feature coordinates after one block, not three generated words. A real decoder ordinarily runs additional blocks and possibly a final normalizer, then applies a vocabulary projection to form logits, from which a token-selection policy can generate text.

The source checks an equally important causal property: change only the third input token and the first two output rows must remain unchanged. That is a more useful correctness test for causal information flow than merely checking that each softmax row sums to one.

Where encoders and decoders differ

An encoder with bidirectional self-attention generally permits a token to use left and right context. A causal decoder cannot read future positions. In encoder-decoder designs, cross-attention draws Queries from decoder states while Keys and Values come from encoder output states. Those are different rules about vector sources and visibility, even though the same underlying dot-product operation appears.

Takeaway. A complete Pre-LN block is a sequence of two updates: normalize → attention → output projection → residual, then normalize → FFN → residual. Its result is another set of hidden states, not yet a token prediction.

Next: Part 8 expands the FFN itself, explaining why it widens, activates and contracts each token vector.