★ The transferable skill of this section. Learn this and files 02-08 become a reading list rather than a syllabus.
1. The claim structure of an inference paper#
Almost every inference systems paper makes this shape of claim:
"We achieve Nx [speedup / memory reduction / throughput] over
[baseline] on [model] at [configuration] with [quality impact]."Every bracketed term is a place the claim can be true and irrelevant to you.
2. The seven questions#
Ask these, in order, before believing anything.
Q1 — What is the baseline?#
The single most important question.
WEAK BASELINES (be suspicious)
✗ HuggingFace `generate()` — no continuous batching, no paged KV.
A 20x speedup over this is a 2x speedup over vLLM.
✗ an unoptimized reimplementation
✗ the authors' own previous work only
✗ a production system with default (untuned) settings
STRONG BASELINES
✓ vLLM or TensorRT-LLM, tuned
✓ the current state of the art for that specific technique
✓ multiple baselines, including the obvious simple alternative
RED FLAG: the paper doesn't say what the baseline configuration was.Half of reported speedups in this field are speedups over a weak baseline. This is not usually dishonesty; it’s that building a strong baseline is itself a lot of work.
Q2 — What is the evaluation configuration?#
CHECK
batch size ← most important. Batch 1 results rarely transfer.
sequence lengths ← fixed lengths hide the scheduling problem
arrival pattern ← closed-loop hides queueing (Section X.07)
hardware ← an A100 result may not hold on H100
model size ← 7B results may not hold at 70B
precision ← FP16 results may not hold at FP8
RED FLAGS
✗ batch size 1 only
✗ fixed sequence lengths only
✗ one model, one hardware
✗ no arrival-rate sweep for a serving claim“At batch size 1” is the most common hidden qualifier. Many techniques that look transformative at batch 1 give 1.1x at batch 64 (speculative decoding, CUDA graphs, and most memory-traffic optimizations).
Q3 — What bottleneck does it claim to remove, and does the arithmetic hold?#
Every genuine speedup must reduce one of:
bytes moved
FLOPs computed
serialization (dependency chains)
overhead (launches, syncs, CPU work)
APPLY THE ROOFLINE (Section X.02):
if a paper claims a 3x decode speedup, it must be moving
~1/3 the bytes, or doing 3x the work per byte.
If it claims a large speedup without reducing bytes or improving
arithmetic intensity, ask HOW.
RED FLAG: the mechanism isn't stated in terms of a resource.This question kills a lot of claims quickly. A paper claiming 2x faster decode from a better algorithm, on a workload that is 95% memory-bandwidth-bound, must be moving fewer bytes. If it isn’t, either the baseline was bad or the claim doesn’t hold.
Q4 — What breaks at scale?#
ASK
what happens at batch 128 instead of 1?
what happens at 128k context instead of 4k?
what happens with 100 concurrent tenants?
what happens when the technique's assumption is violated?
(e.g. a prefix-caching technique when no prefixes are shared)
what's the worst case, not the average?
MANY PAPERS OPTIMIZE ONE REGIME AND DEGRADE ANOTHER.
→ that's fine and often stated; check whether YOUR regime is
the one that degrades.Q5 — What is the quality impact, and how was it measured?#
For anything touching precision, sparsity, eviction, or approximation:
✓ perplexity necessary, far from sufficient
✓ standard benchmarks better; check WHICH ones
✓ long-form generation ← the one that's usually missing
✓ reasoning tasks ← degrades faster than knowledge
✓ the specific failure mode does the paper test the case where
its approximation should fail?
RED FLAG: perplexity only, or benchmarks chosen after seeing results.
RED FLAG: no evaluation of long generations for a technique that
compounds error per token.KV eviction papers are the canonical case (Section XIII.04): impressive benchmark numbers, and the failure mode — needing an evicted token — is often not directly tested.
Q6 — What does it cost that isn’t reported?#
COSTS THAT PAPERS UNDER-REPORT
training or fine-tuning required
extra memory (a draft model, extra heads, a codebook)
implementation complexity
kernel support required (does it exist outside the paper?)
interaction with other techniques (does it break CUDA graphs?
prefix caching? continuous batching?)
ongoing maintenance (retraining when the base model updates)
hardware requirements (does it need Hopper? NVLink? RDMA?)“Requires retraining the heads” is a large ongoing cost that turns a one-time integration into a permanent obligation (Section XIII.09).
Q7 — Is there a simpler thing that gets most of the benefit?#
Before adopting a sophisticated technique, ask what the simple
alternative achieves.
sophisticated KV eviction vs FP8 KV cache (2x, free, no risk)
learned request routing vs route by prompt length
disaggregated serving vs chunked prefill
a shared KV store vs prefix-aware routing
a trained speculative head vs n-gram lookup
In every one of those pairs, the simple option gets 60-90% of the
benefit for 5-10% of the effort.This question is the most valuable one and the least often asked.
3. Applying it — a worked example#
PAPER CLAIM (hypothetical, but typical in shape):
"SparseKV: 4x KV cache reduction with 0.2% quality loss.
3.1x throughput improvement on Llama-2-7B."
Q1 BASELINE?
→ paper says "HuggingFace transformers with KV cache"
→ WEAK. No continuous batching, no paged KV.
→ the 3.1x is probably ~1.2x over vLLM. Recompute mentally.
Q2 CONFIGURATION?
→ batch 1, fixed 4k context, A100, FP16, one model
→ batch 1 only. What happens at batch 64? Not reported.
→ 4k context only. The technique targets long context; why not test it?
Q3 MECHANISM?
→ evicts 75% of KV blocks → moves 1/4 the KV bytes
→ but at 4k context and batch 1, KV is 0.5 GB and weights are
14 GB. KV is 3% of the bytes moved.
→ reducing 3% by 75% saves 2.6% of bytes.
→ A 3.1x throughput claim CANNOT come from that.
→ the speedup must be coming from the weak baseline. Confirmed by Q1.
Q4 SCALE?
→ at batch 64 and 32k context, KV would be 64 GB vs 14 GB of weights
→ NOW a 4x KV reduction matters (would save ~58% of bytes)
→ so the technique IS potentially valuable, just not for the
reason the paper's numbers show
Q5 QUALITY?
→ perplexity and 3 benchmarks. No long-generation test.
→ no test of "the answer requires an evicted token"
→ the 0.2% claim is not evidence for the failure mode that matters
Q6 HIDDEN COSTS?
→ requires a custom attention kernel (does it exist?)
→ interacts with paged attention how?
→ per-layer eviction budget is a hyperparameter — tuned per model?
Q7 SIMPLER ALTERNATIVE?
→ FP8 KV cache: 2x reduction, ~0% quality loss, a config flag
→ GQA model selection: 8x, free
→ this technique's 4x is on top of those, or instead of them?
CONCLUSION
Interesting idea, mismeasured. The regime where it matters
(high batch, long context) is not the regime it was evaluated in.
Worth watching; not worth adopting on this evidence.
If I had this problem, I'd try FP8 KV first and measure whether
I still need 4x.That analysis took ten minutes and is more useful than reading the paper’s method section carefully. Do it first; read the method only if the claim survives.
4. Where to find the work#
VENUES THAT MATTER FOR INFERENCE SYSTEMS
OSDI, SOSP systems; where PagedAttention and Orca appeared
MLSys ML systems specifically
ASPLOS, ISCA, MICRO architecture
NeurIPS, ICML, ICLR algorithms (FlashAttention, speculative decoding)
arXiv cs.LG, cs.DC everything, immediately, unreviewed
INDUSTRY SOURCES (often more practically relevant)
NVIDIA developer blog and GTC talks
vLLM, SGLang, TensorRT-LLM release notes and design docs
model technical reports (DeepSeek's are unusually detailed)
MLPerf Inference results
RELEASE NOTES ARE UNDERRATED
when a technique appears in a major engine's release notes,
it has passed a practical validation bar that a paper hasn't.A technique landing in vLLM or TensorRT-LLM is stronger evidence than a paper, because someone has integrated it with continuous batching, paged attention, CUDA graphs, and everything else, and it survived.
5. The adoption timeline#
PAPER → PRODUCTION, typical stages
0-6 months paper, reference implementation, benchmark claims
6-12 months independent replication; the claims are revised downward
someone integrates it with a real engine and finds
the interactions
12-18 months a config flag in a major engine, off by default
18-30 months on by default, or abandoned
EXAMPLES
FlashAttention: paper 2022 → default in engines ~2023. Fast, because
it's exact and has no downside.
PagedAttention: paper 2023 → the vLLM release WAS the adoption.
Fast, because the authors shipped it.
Speculative decoding: papers 2022 → still not universally on by
default in 2026. Slow, because of the batch
interaction and implementation complexity.
KV eviction: papers 2023 → still not standard. Slow, because
the quality risk is hard to bound.
PATTERN: exact techniques with no downside adopt fast.
Approximations with task-dependent quality costs adopt slowly
or not at all.Use this to calibrate. If a technique is an approximation whose quality cost depends on the task, expect a long adoption timeline and be skeptical of “it’s ready.”
6. What to do with a paper you find compelling#
1. APPLY THE SEVEN QUESTIONS. Ten minutes.
2. IF IT SURVIVES: compute what it would give YOU.
- your regime (batch, context, workload)
- your current bottleneck (Section X.01)
- Amdahl (Section VII.01)
Often the answer is "2% for me," and you stop.
3. IF IT'S STILL COMPELLING: check whether an engine implements it.
- if yes: try the flag, measure
- if no: is the reference implementation usable? What's the
integration cost with your stack?
4. IF YOU IMPLEMENT IT: measure against YOUR baseline, on YOUR
workload, including quality.
5. WRITE UP WHAT YOU FOUND. Including if it didn't work.
Negative results save your team months.7. Common mistakes#
Believing the headline number. Check the baseline.
Not checking the batch size. Batch 1 results rarely transfer.
Not applying the roofline. A claim that violates it is a claim about the baseline, not the technique.
Adopting before checking the simple alternative.
Not testing the failure mode the approximation is supposed to have.
Assuming a paper’s implementation is production-quality. It usually isn’t, and shouldn’t be.
Reading the method before evaluating the claim.
8. Hands-on exercise#
A. Analyze three papers. Pick three recent inference papers. Apply the seven questions to each in under 15 minutes. Write a one-paragraph verdict for each.
B. Check a claim against the roofline. For a paper claiming a speedup, compute whether the mechanism can produce it given the roofline. Does the arithmetic work?
C. Reproduce a baseline. Take a paper’s baseline and implement it properly (with continuous batching and paged KV). How much of the reported speedup survives?
D. Track an adoption. Pick a technique from 2023-2024. Trace it: paper → reference impl → engine support → default. How long did each stage take? What changed in the claims along the way?
E. Compute your own benefit. For a technique you find interesting, compute what it would give you specifically, using Amdahl and your measured bottleneck. Is it worth reading further?
F. Write a negative result. Try something that doesn’t work for your workload and write it up properly. This is a genuinely valuable artifact.
9. Interview questions#
- How do you evaluate an inference research claim?
- Why is the baseline the most important question?
- How would you use the roofline model to check a speedup claim?
- What’s the typical timeline from paper to production, and what predicts it?
- A paper claims 3x speedup at batch 1. What do you ask?
- Why do exact techniques adopt faster than approximations?
- What would you do before adopting a technique from a paper?
10. Further reading#
- [FUNDAMENTAL] The RESOURCES.md file in this repository, “How to read a paper in this field”
- [ESTABLISHED] Kwon et al., PagedAttention — an example of a paper done well: strong baselines, clear mechanism, honest evaluation
- [FUNDAMENTAL] Any writing on the reproducibility crisis in ML — the same forces operate here
- Next: 02 — Attention research