About this series

Transformers power modern language models, but explanations often jump from a colorful attention diagram to dense matrix notation. This series takes the middle route: clear intuition, enough mathematics to verify the mechanism, and practical implications for model architecture and GPU inference. It assumes familiarity with programming, arrays and basic algebra, not previous experience building Transformers.

Reading path. Parts 1–4 build one attention calculation. Parts 5–8 assemble a Transformer block, including positional information and feed-forward networks (FFNs). Parts 9–13 connect the math to memory-efficient GPU execution and text generation. Parts 14–15 close with training, implementation and evidence-based performance analysis.

Editorial conventions.  $T$ denotes token positions, $d_{model}$ the number of hidden features per token, $d_k$ Query/Key width, $B$ batch size and $H$ attention-head count. Dimensions are usually written as rows × columns. Approximate costs are analytic estimates, not measurements of a particular GPU.

1. Part 1 – Why a Word Needs Its Neighbors
2. Part 2 – Q, K and V: The Matching Engine
3. Part 3 – Why Divide Attention Scores by $\sqrt{d_k}$?
4. Part 4 – Softmax, Causal Masks and a Three-Token Walkthrough
5. Part 5 – Multiple Heads and Positional Information
6. Part 6 – Normalization and Residual Connections
7. Part 7 – Trace One Complete Transformer Block
8. Part 8 – The Feed-Forward Network Is Not Just Another Matrix
9. Part 9 – The GPU Cost of Attention
10. Part 10 – FlashAttention Without the Mystery
11. Part 11 – Why Predicting the Next Token Is a Learning Problem
12. Part 12 – Prefill, Decode and the First Generated Token ← You are here
13. Part 13 – KV Cache, Grouped-Query Attention and PagedAttention
14. Part 14 – What Gradients and Distributed GPUs Really Change
15. Part 15 – Build, Verify and Profile Your Own Attention

Overview

You send a prompt to a language model and receive streamed text. Internally, that request usually passes through two distinct computational phases: prefill and decode. Both use Transformer layers, but their tensor shapes and performance bottlenecks are different.

Understanding the distinction is more useful than memorizing one “tokens per second” benchmark, because the two phases place very different demands on GPU compute and memory.

Prefill: many Queries, one known prompt

During prefill, the entire input prompt is known. The model embeds the prompt tokens and processes their hidden states through the decoder stack. A layer computes Q/K/V projections for many positions and applies causally masked attention. It can calculate many Query rows at once while respecting which earlier Keys each row may consult.

For a prompt length $T$, a dense full-sequence head has a $T\times T$ score arrangement before masking. The model also creates Key and Value data that can be reused when later tokens are generated. Prefill typically offers substantial matrix-multiplication work and opportunities for efficient parallelism, though the actual bottleneck depends on sequence length, batching, hardware and kernels.

Decode: one new Query per active sequence

After the model chooses a token, the generated token joins the prefix. In a conventional incremental decode step, each active sequence contributes one new Query. That Query compares against the Key/Value history already held in cache, and the model writes its new K/V entries for use in subsequent steps.

For a single sequence, per-head Query-Key score shape changes from roughly $T\times T$ in full prompt processing to $1\times T$ for a single new Query against a -position history. The dense Query-Key work changes correspondingly from approximately $2T^2d_k$ to $2Td_k$ per head for the respective operations.

That smaller arithmetic count does not guarantee lower elapsed latency. Decode may involve many small launches, less weight reuse at low batch size, and repeated reading of a growing KV cache. As a result, memory bandwidth and scheduling often matter more than they did during large-matrix prefill.

When is the first generated token actually predicted?

This detail is frequently described incorrectly. Suppose a prompt contains three tokens $t_1,t_2,t_3$. During prefill, the last prompt position produces logits for the next token, $t_4$. Selecting from those logits can generate $t_4$ immediately after prefill.

The first incremental decode step then consumes $t_4$ and, using cached K/V from the prompt, produces logits for $t_5$. The next step consumes $t_5$ and produces logits for $t_6$, and so on.

Some serving systems group or schedule operations differently, but the mathematical distinction remains: the first generated token may come from the final prompt logits. It is misleading to insist that every first generated token must arise from a separate prior single-token decode step.

A serving request from start to finish

Imagine a client request with a 2,000-token prompt. The service first tokenizes the text and obtains IDs. The scheduler allocates or reserves suitable cache capacity and executes prefill over the known prompt. Each decoder layer produces and stores the historical Key/Value data required by that layer; the last prompt position produces next-token logits.

After a token-selection policy chooses the first output token, incremental decode begins. Each iteration uses the prior cache, calculates a new token state and extends the cache. The process stops on a completion condition, generation limit, context-policy action or request cancellation. If the engine uses paged cache storage, the logical history may map to multiple physical memory blocks.

Measure the user experience with the right metrics

Time to first token (TTFT) measures how long until the user can receive an initial generated token. It includes request processing and often substantial prefill work. Inter-token latency measures the time between subsequent generated tokens; it is more sensitive to incremental decode behavior, cache movement and scheduler effects. Throughput, such as generated tokens per second at a chosen concurrency, describes how much work the service completes over time.

These are different metrics. Batching may increase total throughput while raising queueing delay for some requests. A kernel speedup might reduce isolated attention time but barely move end-to-end TTFT if tokenization, scheduling or other model components dominate.

One phase should not stand in for the other

A benchmark with a long prompt and one generated token primarily exercises prefill. A benchmark with a short prompt and a long continuation gives much more weight to decode. A useful performance report therefore records both prompt and generated lengths, batch/concurrency, model configuration and latency measures rather than presenting a single unlabeled tokens-per-second result.

Takeaway. Prefill processes many known Queries and establishes the KV cache. The final prefill position can predict the first generated token. Decode then uses one new Query per sequence at each step, reusing and extending cached history. Different shapes imply different bottlenecks.

Next: Part 13 shows exactly what the KV cache stores, why GQA changes its size and how PagedAttention manages the physical blocks.