Versioning, Canary Deployments, and A/B Testing

VERSIONING   identifying exactly what is deployed
CANARY       exposing a change to a small fraction first
A/B TESTING  measuring whether the change is better
ROLLBACK     undoing it quickly when it isn't

ORDINARY SERVICE                     LLM SERVICE
errors are 4xx/5xx                   errors include "the answer got worse"
canary detects errors in minutes     quality regressions take hours-days to detect
rollback is instant                  rollback requires reloading weights (minutes)
"the same version" is unambiguous    version = weights + quantization + engine +
                                       kernels + sampling defaults + template
deterministic outputs                nondeterministic (Section IV.12)

model_version = {
  weights:        checkpoint hash / URI
  quantization:   method, bits, group size, excluded layers, calibration set hash
  engine:         vLLM 0.x.y / TRT-LLM engine hash
  kernels:        attention backend, GEMM backend
  parallelism:    TP degree, PP degree
  precision:      dtype, KV cache dtype
  chat_template:  hash
  sampling:       default temperature/top_p if the server sets them
  tokenizer:      version/hash
  runtime:        CUDA version, driver version
}

BLUE/GREEN
  Stand up the full new fleet, switch traffic, keep the old fleet warm.
  ✓ instant rollback (switch back)
  ✗ 2x GPU cost during the transition — expensive for LLMs
  → use for high-risk changes if you can afford it

ROLLING
  Replace replicas one at a time.
  ✓ no extra capacity
  ✗ slow (each replica takes minutes to start and drain)
  ✗ mixed versions serving simultaneously for a long window
  → the default; be aware of the mixed-version window

CANARY
  Route a small % of traffic to the new version, increase gradually.
  ✓ limits blast radius
  ✓ enables measurement before full rollout
  ✗ needs traffic-splitting infrastructure and enough traffic to be significant
  → the right default for model changes

SHADOW / MIRROR
  Send a copy of production traffic to the new version; discard its responses.
  ✓ zero user risk
  ✓ real traffic distribution
  ✗ 2x inference cost for the mirrored fraction
  ✗ can't measure user-facing outcomes (nobody sees the responses)
  → excellent for performance validation, limited for quality

TIER 1 — must not regress (minutes to detect)
  error rate (5xx, 429, timeouts)
  p95 TTFT, p95 ITL
  throughput per GPU
  OOM / crash rate
  → automated rollback triggers

TIER 2 — quality proxies (minutes to hours)
  output length distribution        ← a good early signal!
  stop-reason distribution (eos vs length vs stop-string)
  refusal rate
  structured-output validity rate (if applicable)
  tool-call success rate (if applicable)
  → alert, investigate

TIER 3 — quality (hours to days)
  offline eval suite on the exact serving config
  LLM-as-judge on sampled production traffic
  human evaluation on a sample
  user-facing signals: thumbs up/down, regeneration rate, conversation length,
    task completion
  → the real decision

DESIGN
  - Randomize by USER or CONVERSATION, not by request.
    (Per-request randomization means a user sees both versions mid-conversation.)
  - Run long enough for significance. LLM quality signals are noisy;
    you typically need thousands of conversations, not hundreds.
  - Pre-register the metric. Deciding what to measure after seeing the data
    is how you fool yourself.

METRICS THAT ACTUALLY WORK
  - Regeneration rate (user asks again) — strong negative signal
  - Conversation length / turns to resolution
  - Explicit feedback (thumbs), though sparse and biased
  - Task completion (if measurable in your product)
  - LLM-as-judge on paired outputs (cheap, correlates reasonably with humans,
    but has its own biases — position bias, length bias)

METRICS THAT MISLEAD
  - Perplexity (doesn't measure what users care about)
  - Public benchmarks (not your distribution)
  - Average response length alone (longer ≠ better)

Fast rollback requires the old version to still be RUNNABLE:
  - keep the old checkpoint in the local node cache
  - keep the old engine artifacts (TRT engines, compile caches)
  - keep the old container image pulled
  - ideally keep old replicas warm during the canary

Rollback time:
  traffic shift only (blue/green or canary):     seconds
  redeploy old version (rolling):                minutes (full cold start)