Case Studies

SYMPTOM   p95 TTFT 4,100 ms (target 800 ms). p95 ITL 24 ms (fine).
          Throughput 3,900 tok/s. No errors.

STOP. What are your hypotheses?

MEASUREMENTS
  queue_wait p95:        3,650 ms      ← 89% of TTFT
  prefill p95:             380 ms
  tokenize p95:              3 ms
  running batch avg:        62 (max_num_seqs = 64)
  KV usage:                 94%
  arrival rate:             18 req/s
  Little's Law check: L = λW = 18 × 6.2 s = 112 in system, capacity 64
                      → 48 always waiting

DIAGNOSIS
  Genuinely at capacity. Not a tuning problem — an arithmetic one.
  The system can serve ~10 req/s at the current request shape; 18 are arriving.

WRONG FIXES
  ✗ optimize kernels (they're fine)
  ✗ increase max_num_seqs (KV is at 94%; you'd cause preemption)
  ✗ increase TP degree (doesn't help throughput; Section IX.12)

RIGHT FIXES, in order
  1. Check prefix caching. Chat workload, 1,100-token system prompt,
     hit rate was 0% (round-robin routing).
     → enable prefix-aware routing: prefill drops 68%, capacity +40%
  2. Enable FP8: capacity +80%
  3. Add replicas for the remainder.

RESULT  after (1) and (2): 18 req/s served at p95 TTFT 640 ms,
        with no new hardware.

SYMPTOM   p50 ITL 22 ms, p99 ITL 310 ms. Spikes are irregular.
          TTFT and throughput fine.

STOP. Hypotheses?

MEASUREMENTS
  Plotted ITL over time: spikes are ~450 ms and occur 3-8 times/minute.
  Correlated with: arrival of requests with prompt > 12,000 tokens.
  chunked prefill: DISABLED
  max_num_batched_tokens: 32,768

DIAGNOSIS
  Head-of-line blocking. A 12k-token prefill runs as one unit, occupying
  the GPU for ~450 ms, during which every decoding sequence stalls.

FIX
  --enable-chunked-prefill --max-num-batched-tokens 2048
  → the prefill is split into 6 chunks, each riding along with a decode step
  → each step is ~28 ms instead of 22 ms, but there are no 450 ms stalls

RESULT  p50 ITL 26 ms (slightly worse), p99 ITL 51 ms (6x better).
        p95 TTFT for long prompts: 480 → 620 ms (slightly worse).
        Overall throughput: +6%.

SYMPTOM   Predicted 3,000 tok/s from the roofline; measured 980 tok/s.
          Batch 32, 8B model, one H100.

STOP. Hypotheses?

MEASUREMENTS
  nsys: GPU busy 38% of wall clock
  API summary: 4,200 cudaLaunchKernel per step, 1 cudaStreamSynchronize
  py-spy: 41% of time in a custom logits processor (Python)
          which calls .cpu() on the logits every step
  CUDA graphs: disabled (--enforce-eager was in the config)

DIAGNOSIS
  Two compounding problems:
  1. --enforce-eager left in the config from a debugging session
  2. a per-step D2H copy + sync in a custom stopping-criterion callback

FIX
  1. Remove --enforce-eager                     → 980 → 1,510 tok/s
  2. Move the stopping check to the GPU          → 1,510 → 2,740 tok/s

RESULT  2,740 tok/s (91% of the roofline prediction).
        Total effort: 3 hours.

SYMPTOM   Server OOMs 3-4 times daily, always near peak. Restarts cleanly.

STOP. Hypotheses?

MEASUREMENTS
  memory_summary() at failure: reserved 78.1 GB, allocated 71.4 GB
                               → 6.7 GB fragmentation, but that's not the
                                 whole story
  Correlation: every OOM within 2 s of a request with prompt > 28,000 tokens
  gpu_memory_utilization: 0.94
  chunked prefill: disabled
  Reproduced: 30k prompt + 40 decoding sequences → OOM

DIAGNOSIS
  Type 2 (Section X.06): activation spike during unchunked long prefill.
  The KV pool was sized using a profiling pass with max_num_batched_tokens
  tokens, but without chunking the actual prefill processed all 30k at once,
  producing an activation tensor 5x larger than profiled.

FIX
  1. --enable-chunked-prefill (bounds activation to max_num_batched_tokens)
  2. --gpu-memory-utilization 0.90 (headroom)
  3. Cap max input at 32,000 at the gateway with a clear 400

RESULT  Zero OOMs in 30 days. Bonus: p99 ITL improved 40% (same root cause
        as Case 2).

SYMPTOM   Went from TP=4 to TP=8 to increase throughput. Throughput per GPU
          dropped 35%; total throughput up only 30%.

STOP. Hypotheses?

MEASUREMENTS
  nvidia-smi topo -m:
     GPU0-3: NV18 to each other
     GPU4-7: NV18 to each other
     GPU0-4: SYS                       ← across the CPU interconnect
  nsys with NCCL trace: AllReduce time 4.2 ms of a 9.1 ms step (46%)
  At TP=4 (within one NVLink island): AllReduce 0.6 ms of 12.8 ms (5%)

DIAGNOSIS
  The node has two NVLink islands, not one NVSwitch fabric.
  TP=8 forces AllReduce across the CPU interconnect.

FIX
  Revert to TP=4, run TWO instances (DP=2 × TP=4), one per NVLink island.
  Pin each with numactl to its NUMA node.

RESULT  Total throughput +95% vs the original TP=4 single instance,
        vs +30% for TP=8. Same 8 GPUs.

SYMPTOM   Capacity request for 64 additional GPUs. Justification:
          "all 64 existing GPUs at 100% utilization."

STOP. Hypotheses?

MEASUREMENTS
  DCGM_FI_PROF_DRAM_ACTIVE:  0.24
  DCGM_FI_PROF_SM_ACTIVE:    0.97
  avg_running_batch_size:    7  (max_num_seqs = 96)
  kv_cache_usage_ratio:      0.09
  queue depth:               0
  replica count:             16, each receiving ~4 req/s

DIAGNOSIS
  Not capacity-limited at all. Traffic is spread too thinly across too many
  replicas; each reads the full model per step to serve 7 sequences
  (Section IX.12, way 5).

FIX
  1. Consolidate 16 → 5 replicas
  2. Prefix-aware routing across them
  3. Keep 1 spare for headroom

RESULT  avg batch 7 → 26. DRAM_ACTIVE 0.24 → 0.71. Throughput per GPU 3.4x.
        44 of 64 GPUs freed. No purchase.

SYMPTOM   User complaints about "worse answers" starting Tuesday.
          Error rate, latency, throughput all normal. All CI tests green.

STOP. Hypotheses?

MEASUREMENTS
  Deploy log: Tuesday's release changed the serving precision from BF16 to
              INT4 (W4A16, GPTQ, group 128) for cost reasons.
  Offline eval at deploy time: MMLU -0.4%, perplexity +0.08. Accepted.
  Output length distribution: mean 218 → 341 tokens (+56%)   ← the signal
  Regeneration rate: 4.1% → 11.3%
  Structured output validity: 99.2% → 91.7%

DIAGNOSIS
  INT4 quantization degraded instruction-following and long-form coherence
  in a way that MMLU and perplexity did not capture (Section VII.06).
  The model rambles and violates JSON schemas more often.

FIX
  1. Immediate: roll back to BF16 (old checkpoint still cached — good)
  2. Re-evaluate with the Level 3 tests (Section IV.12): long generations,
     instruction following, structured output validity
  3. Deploy FP8 instead: -0.1% MMLU, output length unchanged,
     structured validity 99.1%, and 1.9x throughput.

RESULT  Cost target met with FP8. Added output-length distribution and
        structured-validity to the canary guards.

SYMPTOM   Users report the response appears all at once after ~8 seconds.
          Server-side TTFT metric: p95 210 ms. Server-side E2E: p95 7.9 s.

STOP. Hypotheses?

MEASUREMENTS
  curl -N -w '%{time_starttransfer}' through the production ingress: 7.82 s
  curl -N directly to the pod IP: 0.19 s                    ← the answer
  nginx ingress config: proxy_buffering not set → defaults to ON

DIAGNOSIS
  The ingress buffers the entire SSE response before forwarding.
  Server metrics are correct; the user experience is not.

FIX
  proxy_buffering off;
  proxy_cache off;
  proxy_read_timeout 3600s;
  add_header X-Accel-Buffering no;
  (and verify the CDN and WAF layers too)

RESULT  Client TTFT p95: 7.82 s → 0.26 s.
        Added a synthetic client-side TTFT check through the production
        ingress to monitoring.

1. DECOMPOSE FIRST. Six of eight were resolved by phase decomposition
   or a single correlation, before any deep profiling.

2. CHECK THE CONFIGURATION. Cases 3, 4, 8 were configuration errors.
   Cases 2 and 5 were configuration choices made without measurement.

3. MEASURE FROM THE OUTSIDE. Case 8 is invisible from the inside.

4. THE OBVIOUS METRIC LIES. Case 6: GPU utilization. Case 7: MMLU.

5. CORRELATE ANOMALIES WITH SOMETHING. Cases 2 and 4 were both
   "correlate the spikes with the request shape."

6. THE FIX IS OFTEN A FLAG. Cases 2, 3, 4, 8. Total engineering time
   for those four: about a day. Combined impact: large.

7. AMDAHL. In every case, the right fix addressed the dominant term.
   In Case 3, someone could have spent a month on kernels for 5%.