From Token Vectors to GPU Inference

About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis. Each post can be published separately; equations, diagrams and examples are selected and simplified from the accompanying technical masterclass rather than copied wholesale.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbours ← You are here
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalisation and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

The word bank does not always mean a financial institution. Compare “She deposited money at the bank” with “The boat rested by the river bank.” The letters are identical, but the meaning changes because the neighboring words change. A language model therefore needs more than a dictionary entry for each word: it needs a representation that takes surrounding context into account.

This requirement is the starting point for understanding Transformers. Before considering attention formulas, we need three ideas: how text becomes tokens, how tokens become numbers, and how a model passes information between positions.

From text to token IDs

A tokenizer converts text into a sequence of integer IDs. Tokens may represent whole words, word fragments or punctuation; a token is not necessarily a word. An ID is similar to a database primary key: its numerical value identifies an entry, but the number itself is not a semantic description. Token 412 does not intrinsically mean “river” any more than customer ID 412 tells you that customer’s preferences.

An embedding table turns each ID into a learned numerical vector. If three input tokens each map to a vector with four features, their representations form a $3\times4$ matrix. Each row represents one token position, and each column represents one learned feature coordinate. The features are not ordinarily hand-labelled as “subject” or “location”; they are numbers adjusted during training.

The full model often uses a much larger model width, written $d_{model}$. For example, a width of 4,096 means that each position is represented by 4,096 features. It does not mean there are 4,096 tokens or 4,096 Transformer layers.

Context changes the representation

An embedding is a starting representation, not the finished contextual meaning. A Transformer repeatedly updates each token’s hidden state by allowing it to draw information from other permitted positions. In the banking sentence, “deposited” and “money” may influence the vector at “bank”; in the riverside sentence, “boat” and “river” may influence it instead. The model does not literally select a dictionary definition. It computes new vectors through learned operations.

A useful analogy is a request handler that consults related records before deciding what to return. The handler starts with one record but uses information associated with others. In attention, a token similarly combines information from other positions. The analogy helps explain why context is useful, though real attention is continuous numerical computation, not a database query.

Why Transformers were different from recurrent networks

Earlier recurrent models propagated a running state from one sequence position to the next. Position 5 depended on computations at position 4, which depended on position 3, and so on. This makes the sequence direction a chain of computational dependencies.

Self-attention can compare many known token positions using matrix operations in parallel. A complete prompt can be processed together while a causal mask controls which earlier positions each token may consult. That is a substantial advantage on modern accelerators, which excel at batched numerical work.

There is an important limit: parallel attention within an already known sequence does not eliminate the need to generate unknown future tokens one by one. We will return to that distinction in Part 12.

Width, depth and sequence length are not interchangeable

These three dimensions affect both capacity and cost, but in different ways. Width is the number of features in a token’s hidden state. Depth is the number of successive Transformer blocks. Sequence length is the number of token positions being processed.

Consider a model with three layers and eight features per token versus one with six layers and four features per token. The first has wider transformations within each layer; the second applies more consecutive transformations. Neither is automatically superior, and neither tells us how many tokens the model can process at once.

A rough parameter-count intuition follows from simple matrix shapes. A dense map from $d$ features to $m$ features has $dm$ weight values, ignoring biases. Doubling $d$ in a square $d\times d$ projection multiplies its weights by four, while doubling the number of otherwise identical layers roughly doubles the repeated block weights. These relationships become useful when estimating model size later.

What attention contributes, and what it does not

The key idea is context-dependent information exchange. Attention will compute scores between token positions and use those scores to form weighted combinations of numerical Value vectors. It does not directly emit an English answer, and it is only one part of a Transformer block. Normalisation, residual connections and position-wise feed-forward computation are equally important to the complete architecture.

Takeaway. Text becomes token IDs, IDs become learned vectors, and Transformer blocks repeatedly update those vectors using permitted context. Keeping token count, feature width and layer count distinct will make the coming matrix shapes much easier to understand.

Next: Part 2 builds Query, Key and Value vectors and shows why the attention score matrix has the shape it does.