About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalization and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix ← You are here
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

Self-attention answers, roughly, “What information should this token gather from other positions?” The feed-forward network (FFN) tackles a different problem: “How should this token’s newly contextualised features be transformed?” It is tempting to describe the FFN as two linear layers. That description misses the most important ingredient: the nonlinear activation between them.

Expand, activate, contract

A conventional FFN uses two weight matrices and, optionally, two bias vectors:

If each token has width  and the intermediate width is $m$, then $W_1$ has shape $d\times m$ and $W_2$ has shape $m\times d$. A common illustrative choice is $m = 4 d$. The output returns to width $d$, allowing it to be added to the residual stream.

For $B$ sequences of $T$ tokens, the FFN input is $B\times T\times d$ and its intermediate activation is $B\times T\times m$. Neither projection mixes token positions. The same $W_1, W_2$ are shared across positions, but they act on each token row separately.

A two-token calculation exposes the mechanism

Choose inputs $X_1=\left[2,-1\right]$ and $x_2=\left[-1,2\right]$, model width 2, intermediate width 3, and the teaching matrices

After the first projection and bias, the two rows are $\left[1,-2,3\right]$ and $\left[1,4,-3\right]$. ReLU, short for rectified linear unit, applies  to each intermediate coordinate. The activated rows become $max\left(0,z\right)$ and $\left[1,4,0\right]$.

Multiplying by $W_2$ and adding $b_2$ gives final FFN outputs $\left[2.75,-2.50\right]$ and $\left[1.25,4.50\right]$. Notice that changing the second input row did not alter the first row’s FFN calculation. Attention and FFN have fundamentally different information-flow patterns.

Why the activation cannot be omitted

Without an activation, two successive matrix multiplications collapse into one matrix product:

With biases, the composition still collapses into a single affine transformation, a matrix multiplication plus an added vector. An FFN without nonlinearity loses the ability to implement the same family of input-dependent changes in slope. For instance, ReLU has a sharp change at zero that a single affine map cannot reproduce for all inputs.

GELU, the Gaussian Error Linear Unit, offers a smoother alternative. Mathematically it can be written as $GELU\left(x\right)=x\mathrm{\Phi}\left(x\right)$, where $\mathrm{\Phi}$ is the standard normal cumulative distribution function. Unlike ReLU, GELU may preserve a small negative output. The exact shape of the activation therefore changes the result, even if the weight matrices stay fixed.

SwiGLU introduces a learned gate

Some Transformers use a gated FFN such as SwiGLU instead of the conventional two-matrix design. One branch makes candidate features, another produces gate values through the SiLU activation, and their outputs are multiplied coordinate-wise before contraction:

SiLU is $x\sigma\left(x\right)$, where $\sigma$ is the sigmoid function. The gate is a learned numerical modulation, not necessarily a binary switch. SwiGLU uses three matrices, so it must not inherit the conventional FFN parameter formula blindly.

Parameter count and performance intuition

The conventional FFN has $dm+md=2dm$ weights before biases. When $m=4d$, that becomes $8d^2$. A three-matrix gated FFN of intermediate width $g$ has $3dg$ weights. Equating those counts gives  as a rough parameter-matching choice; real models can round dimensions for implementation reasons.

On the compute side, two conventional dense projections require about $4BTdm$ floating-point operations, counting one multiply and one add as two operations. This is a model-work estimate, not elapsed time. During prefill, many token rows can reuse FFN weights in large matrix multiplications. During low-batch single-token decode, reading those weights can become relatively expensive because there is much less reuse.

Can an FFN act like memory?

A useful research interpretation views some FFN intermediate features as learned detectors whose activations scale corresponding output directions. This resembles associative key-value retrieval, but it does not prove that every feature stores one neat fact. Representations are distributed, and residual paths, attention and gating interact. The analogy is useful as intuition, not as a literal database inside the network.

Takeaway. The FFN is a shared position-wise nonlinear feature transformer. Its expansion provides a working feature space; its activation makes the computation more expressive; its contraction restores model width for the residual addition.

Next: Part 9 looks at what all these matrix operations cost when they run on a GPU.