★ The single most important distinction in LLM serving. Everything else in this section follows from it.
1. What is it?#
LLM inference has two phases with opposite performance characteristics.
PREFILL DECODE
"process the whole prompt" "generate one token"
all S prompt tokens at once 1 new token per sequence
parallel sequential (must wait for the previous)
COMPUTE bound MEMORY bound
arithmetic intensity ~S arithmetic intensity ~batch_size
happens once per request happens once per output token
determines TTFT determines ITL
O(S²) attention O(S) attention per stepThey use the same weights and the same operators. They behave like completely different workloads.
Diagram — Two phases, two different bottlenecks#
flowchart LR
P["Prompt<br/>N tokens"] --> PF
subgraph PF["PREFILL - one forward pass"]
direction TB
A1["All N tokens in parallel"] --> A2["Large GEMMs<br/>compute-bound"] --> A3["Writes KV cache<br/>for N positions"]
end
PF -->|"first token = TTFT"| DC
subgraph DC["DECODE - one pass per output token"]
direction TB
B1["1 new token per sequence"] --> B2["Reads all weights + whole KV<br/>memory-bound"] --> B3["Appends 1 position to KV"]
B3 -->|"repeat until EOS"| B1
end
DC -->|"stream"| O["Output tokens"]
class A1,A2,A3 compute
class B1,B2,B3 memory
class P,O neutral2. Why does it exist?#
Because of the causal mask (Section III.08).
During prefill, you know all the prompt tokens. Token 5’s attention over tokens 1-4 can be computed at the same time as token 100’s attention over tokens 1-99 — they don’t depend on each other’s outputs. So you process all S positions in one parallel pass.
During decode, token S+1 cannot be computed until token S has been generated, because token S is its input. This is a hard serial dependency, and it means you can only ever work on one token per sequence at a time.
PREFILL: [t1 t2 t3 t4 t5 t6 t7 t8] ← all known, all processed together
matmuls have 8 rows
DECODE: [....................t9] ← generate t9
[...................t9 t10] ← now generate t10, needs t9
[..................t10 t11] ← now t11, needs t10
matmuls have 1 row per sequenceThe number of rows in your matmuls is the arithmetic intensity (Section I.06). Prefill has S rows; decode has 1 per sequence. That single difference produces everything below.
3. Simple analogy#
Reading a document versus writing an essay.
Reading: your eyes can scan the whole page; you process many words in parallel; you’re limited by how fast you can think about the content.
Writing: each word depends on the one before it. You cannot write word 50 before word 49. You’re limited not by thinking speed but by the mechanics of getting each word down — and crucially, before writing each word you must recall everything you’ve written so far.
That last part is the KV cache read. Writing a 5,000-word essay means “recalling the whole thing so far” 5,000 times.
4. Tiny example#
Measure both phases yourself:
// phases.go — measure prefill and decode separately through any OpenAI-compatible server
// (vLLM, llama.cpp server, Ollama): go run phases.go http://localhost:8000 <model>
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
"strings"
"time"
)
// stream sends one completion request and returns time-to-first-token and the
// average gap between later tokens.
func stream(base, model, prompt string, maxTokens int) (ttft, perToken time.Duration, err error) {
body, _ := json.Marshal(map[string]any{
"model": model, "prompt": prompt, "max_tokens": maxTokens, "temperature": 0, "stream": true,
})
t0 := time.Now()
resp, err := http.Post(base+"/v1/completions", "application/json", bytes.NewReader(body))
if err != nil {
return 0, 0, err
}
defer resp.Body.Close()
var first, last time.Time
n := 0
sc := bufio.NewScanner(resp.Body)
for sc.Scan() {
line := sc.Text()
if !strings.HasPrefix(line, "data: ") || line == "data: [DONE]" {
continue
}
last = time.Now()
if n == 0 {
first = last
}
n++
}
if n < 2 {
return 0, 0, fmt.Errorf("got %d chunks", n)
}
return first.Sub(t0), last.Sub(first) / time.Duration(n-1), nil
}
func main() {
base, model := os.Args[1], os.Args[2]
for _, words := range []int{1, 16, 128, 512, 2048} {
prompt := strings.Repeat("hello ", words) // roughly one token per word
ttft, itl, err := stream(base, model, prompt, 32)
if err != nil {
fmt.Println(err)
return
}
// TTFT ≈ prefill of the whole prompt. ITL ≈ one decode step.
fmt.Printf("prompt ~%4d tok: prefill %7.1f ms (%7.0f tok/s) decode %6.2f ms/token (%4.0f tok/s)\n",
words, ttft.Seconds()*1e3, float64(words)/ttft.Seconds(), itl.Seconds()*1e3, 1/itl.Seconds())
}
}Representative output (a 1.5B model on an A100):
prompt ~ 1 tok: prefill 8.1 ms ( 123 tok/s) decode 8.05 ms/token ( 124 tok/s)
prompt ~ 16 tok: prefill 8.2 ms ( 1951 tok/s) decode 8.05 ms/token ( 124 tok/s)
prompt ~ 128 tok: prefill 8.9 ms ( 14382 tok/s) decode 8.05 ms/token ( 124 tok/s)
prompt ~ 512 tok: prefill 14.2 ms ( 36056 tok/s) decode 8.05 ms/token ( 124 tok/s)
prompt ~2048 tok: prefill 43.1 ms ( 47517 tok/s) decode 8.05 ms/token ( 124 tok/s)Read this table carefully.
- Prefill at S=1 and decode take the same time (~8 ms). Of course — they’re the same shape.
- Prefill at S=2048 takes 5.3x longer but processes 2048x more tokens: 385x more efficient per token.
- Prefill throughput saturates around S=512-2048: that’s where it becomes compute-bound.
- Decode is stuck at ~124 tok/s per sequence, forever, because it’s reading 3 GB of weights per token.
5. Technical explanation#
The arithmetic#
PREFILL (prompt length S, batch B):
FLOPs = B · (2·P·S + 2·L·S²·d)
Bytes = P·bytes ← weights read ONCE for all B·S tokens
+ activations
+ KV written: 2·L·h_kv·d_head·bytes·S·B
Intensity ≈ B·S ← huge
DECODE (one step, batch B, context S):
FLOPs = B · (2·P + 4·L·S·d)
Bytes = P·bytes ← weights read for B tokens
+ KV read: 2·L·h_kv·d_head·bytes·S·B
Intensity ≈ B ← tinyFor Llama-3-8B, B=1, S=2048, on an H100 (990 TFLOP/s, 3.35 TB/s):
PREFILL:
FLOPs = 2·8e9·2048 + 2·32·2048²·4096 = 32.8e12 + 1.1e12 = 33.9 TFLOP
Bytes = 16 GB
T_compute = 33.9e12/990e12 = 34.2 ms
T_memory = 16e9/3.35e12 = 4.8 ms
→ COMPUTE BOUND, 34.2 ms
DECODE (one step):
FLOPs = 2·8e9 + 4·32·2048·4096 = 16e9 + 1.07e9 = 17.1 GFLOP
Bytes = 16 GB + 0.5 GB KV = 16.5 GB
T_compute = 17.1e9/990e12 = 0.017 ms
T_memory = 16.5e9/3.35e12 = 4.9 ms
→ MEMORY BOUND, 4.9 ms, by a factor of 288The GPU’s arithmetic units are 99.7% idle during decode. Not because of bad code — because there is nothing to compute while waiting for 16 GB to arrive.
The crossover#
At what prompt length does prefill become compute-bound?
Set T_compute = T_memory:
2·P·S / peak_FLOPs = P·bytes / peak_BW
S = (bytes · peak_FLOPs) / (2 · peak_BW)
= (2 · 990e12) / (2 · 3.35e12)
= 296 tokensAround 300 prompt tokens on an H100. Below that, prefill is memory-bound like decode; above, compute-bound. Note this is exactly the ridge point — as it must be, since intensity ≈ S.
Why decode cannot be made fast (per sequence)#
Decode ITL floor = model_bytes / HBM_bandwidth| Model | FP16 bytes | H100 floor | Max tok/s/user |
|---|---|---|---|
| 8B | 16 GB | 4.8 ms | 209 |
| 13B | 26 GB | 7.8 ms | 129 |
| 70B | 140 GB | 41.8 ms | 24 |
| 405B | 810 GB | doesn’t fit | — |
The only ways to beat this:
- Read fewer bytes — quantization (VII), MoE (XIII).
- Split across GPUs — tensor parallelism gives each GPU 1/N of the weights (IX).
- Produce more tokens per read — speculative decoding (12), multi-token prediction.
- Serve more users per read — batching (08, 09). Doesn’t help the individual, helps the business enormously.
The scheduling conflict#
Here is where the asymmetry becomes an engineering problem:
A 4000-token prefill takes ~70 ms of pure GPU compute.
A decode step takes ~5 ms.
If you run the prefill as one unit, every sequence currently decoding
experiences a 70 ms gap — a 14x ITL spike.Timeline without chunked prefill:
|dec|dec|dec|████████ PREFILL 70ms ████████|dec|dec|dec|
↑ every user's stream freezes here
Timeline with chunked prefill:
|dec+pf|dec+pf|dec+pf|dec+pf|dec+pf|dec+pf|dec|dec|
↑ prefill split into chunks, interleaved. ITL stays smooth.That’s Section XIII.05, and it’s one of the most valuable scheduler features available.
6. Under the hood#
The two phases use different kernels for the same operations:
Operation Prefill kernel Decode kernel
─────────────────────────────────────────────────────────────────
q/k/v proj GEMM (S×d @ d×d) GEMV / small GEMM (B×d @ d×d)
attention FlashAttention (tiled 2D) FlashDecoding (split over keys)
FFN GEMM GEMV / small GEMM
Selection by shape at runtimeThe attention difference is structural. Prefill parallelizes over query positions (there are thousands). Decode has one query per sequence — nowhere near enough parallelism for 132 SMs — so it must parallelize over the key dimension instead, splitting the KV cache across thread blocks and combining partial results.
7. Performance implications#
The prefill:decode time ratio depends entirely on the workload shape:
Workload in:out prefill% decode%
Chat (short) 50:300 3% 97%
Chat (typical) 500:500 15% 85%
RAG 8000:400 65% 35%
Summarization 32000:500 88% 12%
Code completion 2000:50 85% 15%
Agentic tool loop 4000:100 90% 10%This changes which optimizations matter:
| Workload | Optimize |
|---|---|
| Decode-heavy (chat) | quantization, batching, GQA, speculative decoding |
| Prefill-heavy (RAG, agents) | prefix caching (huge!), chunked prefill, compute efficiency |
A team that optimizes decode for a RAG workload is optimizing 35% of their cost. Measure your split before choosing.
8. Production implications#
- Measure your prefill:decode ratio. It determines your entire optimization strategy. Most engines expose this (vLLM: prefill and decode token counters).
- Report TTFT bucketed by prompt length. A single TTFT SLO is meaningless when prompts range from 50 to 100,000 tokens.
- Enable chunked prefill if you have both long prompts and ITL SLOs. It’s usually a config flag.
- Consider disaggregation at scale (Section XIII.06): separate GPU pools for prefill and decode, each sized and configured for its own bottleneck.
- Price input and output tokens differently. Output tokens genuinely cost 3-5x more.
- Cap prompt length. A 500k-token prefill will both blow your activation memory and stall everyone.
9. Common mistakes#
Treating “latency” as one number. Prefill sets TTFT, decode sets ITL. Different phases, different fixes.
Optimizing decode for a prefill-heavy workload. Measure first.
Benchmarking with a fixed prompt length. Your production distribution is heavy-tailed.
Letting long prefills block decodes. The most common cause of ITL spikes.
Assuming quantization helps both phases equally. Weight-only quantization helps decode ~3x and prefill ~0x (Section III.12).
Sizing hardware for prefill FLOPs when your workload is decode-bound (or vice versa). H100 vs H200 vs L40S have very different FLOP:bandwidth ratios.
10. Hands-on exercise#
A. Reproduce the table. Run the benchmark in section 4 on your hardware and model. Find: (i) the S at which prefill throughput saturates, (ii) the decode ITL, (iii) your crossover point. Compare the measured crossover to the calculated one.
B. Compute the crossover analytically for three different GPUs (look up their specs). Which GPU has the lowest crossover? What does that mean for short-prompt workloads?
C. Measure your workload’s split. On a running vLLM server, use the metrics endpoint to get prefill and decode token counts under realistic traffic. Compute the time split.
D. Demonstrate head-of-line blocking. Run a server without chunked prefill. Start 8 streaming requests, then send one request with a 16,000-token prompt. Plot the ITL of the streaming requests over time. You should see a clear spike. Then enable chunked prefill and repeat.
E. The decode floor. For your model and GPU, compute the theoretical decode floor. Measure the actual. What fraction of theoretical do you achieve? Where does the rest go?
11. Interview questions#
- Explain prefill and decode, and why they have opposite performance characteristics.
- Why is decode inherently serial and prefill parallel?
- Compute the crossover prompt length for an H100. Show your work.
- What is the decode ITL floor for a 70B FP16 model on one H100? Name four ways to beat it.
- Why does a long prefill hurt other users, and what’s the fix?
- How does the prefill:decode ratio change which optimizations you should pursue?
- Why do prefill and decode need different attention kernels?
- Why do providers charge more for output tokens than input tokens?
12. Further reading#
- [ESTABLISHED] Pope et al., “Efficiently Scaling Transformer Inference” (2022) — the canonical analysis
- [ESTABLISHED] Agrawal et al., “Sarathi-Serve” (2024) — chunked prefill
- [EMERGING] Zhong et al., “DistServe” (2024) — disaggregation
- Next: 04 — Autoregressive generation