Model Rollouts

CODE ROLLOUT                        MODEL ROLLOUT
failures are errors (5xx)           failures are "worse answers"
detected in minutes                 detected in hours to days
rollback is instant                 rollback needs the old weights loaded
artifact is MB                      artifact is 10-800 GB
identical behavior guaranteed       nondeterministic by nature
tests are deterministic             evaluation is statistical

STAGE 0 — PREPARE
  □ artifact built and checksummed
  □ quantization done offline, config recorded
  □ engine artifacts (TRT engines / compile cache) built per GPU arch
  □ distributed to all regional storage
  □ node caches pre-populated
  □ full version record created (Section VIII.10)

STAGE 1 — OFFLINE EVALUATION
  □ numerical: logit diff, KL divergence, top-1 agreement vs reference
  □ task: your benchmark suite, on YOUR domain
  □ behavioral: 500+ token generations, instruction following,
    structured output validity, refusal behavior
  □ regression suite: known-tricky prompts from past incidents
  → GATE: all within tolerance, or explicitly waived with justification

STAGE 2 — SHADOW (optional but recommended for engine changes)
  □ mirror 1-5% of production traffic to the new version
  □ discard responses; compare latency, memory, errors, crash rate
  □ run for 2-24 hours
  → GATE: no crashes, latency within 10%, no OOM

STAGE 3 — CANARY
  □ 1% of traffic, one region, non-premium tiers
  □ automated guards active (Section VIII.10)
  □ monitor: error rate, TTFT, ITL, output length distribution,
    stop-reason distribution, structured validity, regeneration rate
  □ 2-4 hours minimum, or until statistical significance
  → GATE: all Tier 1 and Tier 2 signals within bounds

STAGE 4 — PROGRESSIVE ROLLOUT
  1% → 5% → 25% → 50% → 100%
  hold at each step for at least 1 hour (longer for the early steps)
  □ automated rollback guards remain active throughout
  □ human check at 25% and 50%
  → GATE at each step

STAGE 5 — SOAK
  □ 100% for 48-72 hours with the old version still deployable
  □ monitor the slow signals: user feedback, conversation length,
    task completion, support tickets
  → GATE: no degradation → decommission the old version

STAGE 6 — CLEANUP
  □ remove old version's replicas
  □ keep the old artifact for 30 days minimum
  □ update the registry, document what changed

MINUTES (automated guards, auto-rollback)
  error rate, 5xx, timeouts
  p95 TTFT, p95 ITL
  OOM / crash rate
  throughput per GPU

MINUTES-HOURS (automated alerts, human judgment)
  output length distribution        ← the most sensitive cheap signal
  stop-reason distribution (eos vs length vs stop-string)
  structured output validity rate
  refusal rate
  tool-call success rate
  prefix cache hit rate (a template change would move this)

HOURS-DAYS (human review)
  regeneration rate
  conversation length / turns to resolution
  explicit user feedback
  support ticket volume and content
  LLM-as-judge on sampled paired outputs
  human evaluation on a sample

FAST ROLLBACK (seconds) requires:
  □ the old version still RUNNING (traffic shift only)
  □ or: the old artifact in every node cache and engine artifacts built

SLOW ROLLBACK (minutes to an hour):
  □ old artifact in regional storage
  □ redeploy, reload, warm up

DURING A ROLLOUT: keep the old version warm.
  Cost: 2x replicas for the rollout window.
  Value: seconds vs an hour of rollback time.
  → for a 4-hour progressive rollout, that's 4 GPU-hours × replicas.
    Cheap insurance.

1. ROLL BACK FIRST, INVESTIGATE SECOND.
   Do not debug in production while users are affected.

2. PRESERVE EVIDENCE
   Sample requests and responses from the bad version (with consent /
   per your privacy policy). You need them to reproduce.

3. REPRODUCE OFFLINE
   With the exact serving configuration, not the reference implementation.

4. ROOT CAUSE
   Common causes, in order:
     - quantization degraded a capability the eval suite didn't test
     - chat template changed subtly
     - sampling defaults changed
     - a kernel/backend change altered numerics beyond tolerance
     - the checkpoint itself is different from what was evaluated

5. ADD THE TEST
   Whatever you missed goes into the evaluation suite. Permanently.

6. WRITE THE POSTMORTEM
   Including: why the existing gates didn't catch it.

ENGINE VERSION UPGRADE (not a model change)
  → shadow first (crashes and performance are the risk)
  → but ALSO validate numerics: kernel changes alter outputs
  → the model version record must include the engine version

QUANTIZATION CHANGE
  → the highest-risk change type
  → full Stage 1-5, no shortcuts
  → soak for at least 72 hours

CHAT TEMPLATE CHANGE
  → subtle and dangerous; the model is out of distribution if wrong
  → test that the rendered prompt is byte-identical for a corpus
    of conversations before and after
  → also: it invalidates the prefix cache

SAMPLING DEFAULT CHANGE
  → affects everyone who doesn't specify parameters
  → treat as a model change; canary it

EMERGENCY SECURITY PATCH
  → skip stages 2-3, go to 5% canary for 15 minutes, then full
  → accept the risk explicitly, document it, soak afterwards