Multi-Region and Disaster Recovery

1. LATENCY
   The speed of light. A user in Singapore talking to a US-East endpoint
   pays 200+ ms of RTT before any inference happens.
   → for a 800 ms TTFT SLO, that's 25% of the budget, spent on physics.

2. AVAILABILITY
   Survive the loss of a region or an availability zone.

3. DATA RESIDENCY
   Legal or contractual requirements that data not leave a jurisdiction.
   → often the actual driver, and it's non-negotiable.

STATELESS WEB SERVICE            LLM SERVICE
replicas are cheap and small     each replica is 8 GPUs and 140 GB
scale up in seconds              minutes
capacity is fungible             GPUs are scarce and region-constrained
state is in a shared database    KV cache is per-replica and ephemeral
failover = DNS change            failover = DNS change + capacity in the
                                   other region, which you must PAY FOR

PATTERN                    COST      FAILOVER    LATENCY     WHEN
Single region              1.0x      none        poor for    small scale,
                                                 distant     one market
                                                 users
Active-passive             1.5-2x    minutes     poor        compliance-driven
  (warm standby)                     to hours    (until fo)  DR requirement
Active-active,             2.0x      seconds     good        the usual answer
  each sized for full load
Active-active,             1.3-1.5x  degraded    good        cost-conscious;
  each sized for 65%                                          accept degradation
                                                              during failover
Multi-region with          varies    seconds     good        large scale
  overflow routing

LATENCY-BASED DNS / ANYCAST
  Route users to the nearest healthy region.
  ✓ simple, works with standard infrastructure
  ✗ DNS TTL means failover takes minutes
  ✗ doesn't account for regional capacity

GLOBAL LOAD BALANCER (application layer)
  ✓ health-aware, capacity-aware, instant failover
  ✓ can implement overflow ("if EU is at 90%, send 10% to US")
  ✗ another component to run

HYBRID (the usual answer)
  latency-based DNS for the primary route
  + application-layer overflow between regions
  + health checks that remove a region quickly

MUST BE IN EVERY REGION
  model weights (in regional object storage + node caches)
  the serving stack (container images)
  configuration
  the model registry (or a regional read replica)

SHOULD BE REGIONAL, NOT REPLICATED
  KV cache (ephemeral; never replicate)
  in-flight request state
  local caches

MUST BE GLOBALLY CONSISTENT (or carefully partitioned)
  rate limit / quota state    ← the hard one
  billing records
  auth tokens
  audit logs

1. Partition quota by region (user's quota is 1/N per region)
   ✓ simple, no coordination
   ✗ a user hitting one region gets 1/N of their quota

2. Global quota store with eventual consistency
   ✓ correct enough; a user can briefly exceed by the sync window
   ✗ requires a cross-region store; adds latency

3. Local enforcement with periodic reconciliation
   ✓ fast, no cross-region latency in the request path
   ✗ over-consumption is detected after the fact
   → THE USUAL ANSWER: enforce locally with a generous local budget,
     reconcile centrally, and correct on the next window.

Getting 140 GB of weights into every region:

  1. Push to regional object storage (S3 cross-region replication,
     or an explicit pipeline)
  2. Pre-populate node-local caches (DaemonSet, or bake into an AMI/image)
  3. Verify checksums per region

TIMING
  A model release must complete in all regions before you shift traffic.
  For a 140 GB model across 4 regions: hours, mostly transfer time.
  → build this into your release timeline.

COST
  Cross-region egress is expensive. Push once to each region's storage
  and pull locally, rather than pulling cross-region per node.

SCENARIO: loss of region us-east-1

  DETECTION
    health checks fail for > 60 s, or error rate > 50% for > 30 s
    → automated: global LB removes the region

  IMPACT
    45% of traffic must move to us-west-2 and eu-west-1
    us-west-2 goes from 65% → 105% utilization → degraded
    eu-west-1 goes from 60% → 88% utilization → acceptable

  DEGRADATION (expected and accepted)
    p95 TTFT: 780 ms → 1,900 ms for affected users
    free-tier traffic shed: ~15%
    premium tier: unaffected (reserved capacity)

  RECOVERY
    RTO (time to serve degraded): 90 seconds (DNS + LB health check)
    RTO (time to serve normally): 45-90 minutes (spin up additional
        capacity in the surviving regions, IF quota is available)
    RPO: not applicable — inference is stateless; in-flight requests
         are lost

  ACTIONS
    1. automated: LB removes the region
    2. automated: degradation ladder engages (Section XI.04)
    3. manual: request additional capacity in surviving regions
    4. manual: communicate degraded status
    5. on recovery: gradual traffic return (10% steps) to avoid
       thundering herd on cold replicas

Multi-region is expensive. Ways to reduce the cost:

1. ASYMMETRIC SIZING
   Size regions by their traffic, not equally.
   
2. SHARED OVERFLOW CAPACITY
   One region has extra capacity that any region can overflow into,
   rather than every region having its own N+1.

3. SPOT / PREEMPTIBLE for the overflow tier
   Cheaper, and you only need it during failover.
   
4. TIER-AWARE FAILOVER
   Only premium traffic fails over; free tier is shed.
   → dramatically reduces the capacity you must hold.

5. EXTERNAL PROVIDER AS THE DR TARGET
   Instead of holding idle GPUs, contract with an API provider for
   burst capacity. Expensive per token, cheap as insurance.