Multi-Region Deployment

Comprehensive guide to deploying applications across multiple geographic regions for availability, performance, and disaster recovery.

When to Use This Skill

• Designing globally distributed applications
• Implementing disaster recovery (DR)
• Reducing latency for global users
• Meeting data residency requirements
• Achieving high availability (99.99%+)
• Planning failover strategies

Multi-Region Fundamentals

Why Multi-Region?

Reasons for Multi-Region:

1. High Availability
   └── Survive region-wide failures
   └── Natural disasters, power outages
   └── Target: 99.99%+ uptime

2. Low Latency
   └── Serve users from nearest region
   └── Reduce round-trip time
   └── Better user experience

3. Data Residency
   └── GDPR, data sovereignty laws
   └── Keep data in specific countries
   └── Compliance requirements

4. Disaster Recovery
   └── Business continuity
   └── RTO/RPO requirements
   └── Regulatory requirements

Trade-offs:
+ Higher availability
+ Lower latency globally
+ Compliance capability
- Higher cost (2x-3x or more)
- Increased complexity
- Data consistency challenges

Deployment Models

Model 1: Active-Passive (DR)
┌─────────────────┐         ┌─────────────────┐
│  PRIMARY (Active)│         │ SECONDARY (Passive)│
│  ┌─────────────┐│         │  ┌─────────────┐│
│  │     App     ││   ──►   │  │    App      ││
│  │    (Live)   ││  Sync   │  │  (Standby)  ││
│  └─────────────┘│         │  └─────────────┘│
│  ┌─────────────┐│         │  ┌─────────────┐│
│  │     DB      ││   ──►   │  │    DB       ││
│  │  (Primary)  ││  Replic │  │  (Replica)  ││
│  └─────────────┘│         │  └─────────────┘│
└─────────────────┘         └─────────────────┘
    All traffic              Failover only

Model 2: Active-Active (Load Distributed)
┌─────────────────┐         ┌─────────────────┐
│   REGION A       │         │   REGION B       │
│  ┌─────────────┐│    ◄──► │  ┌─────────────┐│
│  │     App     ││  Users  │  │     App     ││
│  │   (Active)  ││  routed │  │   (Active)  ││
│  └─────────────┘│   by    │  └─────────────┘│
│  ┌─────────────┐│ location│  ┌─────────────┐│
│  │     DB      ││    ◄──► │  │     DB      ││
│  │  (Primary)  ││  Replic │  │  (Primary)  ││
│  └─────────────┘│   Both  │  └─────────────┘│
└─────────────────┘  ways   └─────────────────┘
  Serves Region A           Serves Region B

Model 3: Active-Active-Active (Global)
┌──────┐    ┌──────┐    ┌──────┐
│  US  │◄──►│  EU  │◄──►│ APAC │
│Active│    │Active│    │Active│
└──┬───┘    └──┬───┘    └──┬───┘
   │           │           │
   └───────────┼───────────┘
               │
         Global Load Balancer
         routes by location

Region Selection

Selection Criteria

Region Selection Factors:

1. User Location
   □ Where are your users?
   □ Latency requirements per region?
   □ User concentration (80/20 rule)?

2. Compliance Requirements
   □ Data residency laws (GDPR, etc.)
   □ Government regulations
   □ Industry requirements (HIPAA, PCI)

3. Cloud Provider Availability
   □ Not all services in all regions
   □ Service feature parity
   □ Regional pricing differences

4. Network Connectivity
   □ Internet exchange points
   □ Direct connect options
   □ Cross-region latency

5. Disaster Risk
   □ Natural disaster patterns
   □ Political stability
   □ Infrastructure reliability

6. Cost
   □ Compute/storage pricing varies
   □ Data transfer costs (egress)
   □ Support availability

Common Region Pairs

Region Pair Strategy:

Americas:
- Primary: US East (N. Virginia)
- Secondary: US West (Oregon) or US East (Ohio)
- Distance: 2,500-3,000 km
- Latency: ~60ms

Europe:
- Primary: EU West (Ireland)
- Secondary: EU Central (Frankfurt) or EU West (London)
- Distance: ~1,000-1,500 km
- Latency: ~20-30ms

Asia Pacific:
- Primary: Singapore or Tokyo
- Secondary: Sydney or Mumbai
- Distance: 5,000-7,000 km
- Latency: ~100-150ms

Global Triad:
- US East + EU West + Singapore/Tokyo
- Covers most global users
- <100ms to 80%+ of users

Data Replication

Replication Patterns

Pattern 1: Async Replication (Most Common)
Primary ──────► Replica
         lag:
         ms to seconds

+ Lower latency for writes
+ Primary not blocked by replica
- Potential data loss on failover (RPO > 0)
- Replication lag visible

Pattern 2: Sync Replication
Primary ◄─────► Replica
         both
         confirm

+ No data loss on failover (RPO = 0)
+ Strong consistency
- Higher write latency
- Availability coupled to both regions

Pattern 3: Semi-Sync Replication
Primary ──────► At least 1 Replica (sync)
        └────► Other Replicas (async)

+ Guaranteed durability for some replicas
+ Balance of latency and durability
- More complex failure handling

Conflict Resolution

Multi-Primary Conflict Resolution:

Scenario: Same record updated in two regions simultaneously

Resolution Strategies:

1. Last Write Wins (LWW)
   └── Timestamp-based
   └── Simple but can lose data
   └── Clock sync important

2. First Write Wins
   └── First committed wins
   └── Later writes rejected or queued
   └── Good for "create once" data

3. Application-Level Resolution
   └── Custom merge logic
   └── Most flexible
   └── Most complex

4. CRDTs (Conflict-free Replicated Data Types)
   └── Mathematically guaranteed convergence
   └── Counters, sets, maps
   └── Good for specific use cases

Best Practice:
- Design to avoid conflicts where possible
- Partition data by region when appropriate
- Use single-primary for conflict-sensitive data

Failover Strategies

Failover Types

Failover Types:

1. DNS-Based Failover
   ┌─────────────────────────────────────────┐
   │  DNS Health Check                       │
   │  ├── Check primary every 10-30s        │
   │  ├── 3 consecutive failures = unhealthy│
   │  └── Update DNS to point to secondary  │
   └─────────────────────────────────────────┘

   RTO: 60-300 seconds (DNS TTL + propagation)
   Pros: Simple, works with any app
   Cons: Slow failover, DNS caching issues

2. Load Balancer Failover
   ┌─────────────────────────────────────────┐
   │  Global Load Balancer                   │
   │  ├── Continuous health checks          │
   │  ├── Instant routing changes           │
   │  └── No DNS propagation wait           │
   └─────────────────────────────────────────┘

   RTO: 10-60 seconds
   Pros: Fast, reliable
   Cons: Requires GLB, potential single point

3. Application-Level Failover
   ┌─────────────────────────────────────────┐
   │  Client/App Aware                       │
   │  ├── Client retries to alternate region│
   │  ├── SDK handles failover              │
   │  └── No infrastructure dependency      │
   └─────────────────────────────────────────┘

   RTO: 1-10 seconds
   Pros: Fastest, most control
   Cons: Requires client changes

RTO and RPO

Recovery Objectives:

RTO (Recovery Time Objective):
└── Maximum acceptable downtime
└── Time from failure to recovery
└── Drives failover automation investment

RPO (Recovery Point Objective):
└── Maximum acceptable data loss
└── Time between last backup and failure
└── Drives replication strategy

Common Targets:
┌──────────────┬──────────┬──────────┬───────────────────┐
│ Tier         │ RTO      │ RPO      │ Strategy          │
├──────────────┼──────────┼──────────┼───────────────────┤
│ Critical     │ <1 min   │ 0        │ Active-active     │
│              │          │          │ Sync replication  │
├──────────────┼──────────┼──────────┼───────────────────┤
│ High         │ <15 min  │ <1 min   │ Active-passive    │
│              │          │          │ Hot standby       │
├──────────────┼──────────┼──────────┼───────────────────┤
│ Medium       │ <4 hours │ <1 hour  │ Warm standby      │
│              │          │          │ Async replication │
├──────────────┼──────────┼──────────┼───────────────────┤
│ Low          │ <24 hours│ <24 hours│ Backup/Restore    │
│              │          │          │ Pilot light       │
└──────────────┴──────────┴──────────┴───────────────────┘

Traffic Routing

Global Load Balancing

GLB Routing Policies:

1. Geolocation Routing
   └── Route by user's geographic location
   └── Europe users → EU region
   └── Fallback for unmapped locations

2. Latency-Based Routing
   └── Route to lowest latency region
   └── Based on real measurements
   └── Adapts to network conditions

3. Weighted Routing
   └── Split traffic by percentage
   └── Good for rollouts, testing
   └── Example: 90% primary, 10% secondary

4. Failover Routing
   └── Primary region until unhealthy
   └── Automatic switch to secondary
   └── Health check driven

Cloud Implementations:
- AWS: Route 53, Global Accelerator
- Azure: Traffic Manager, Front Door
- GCP: Cloud Load Balancing
- Cloudflare: Load Balancing

Session Handling

Session Affinity in Multi-Region:

Challenge: User session state across regions

Option 1: Sticky Sessions
└── User stays in same region for session
└── Failover loses session
└── Simple but limited DR

Option 2: Centralized Session Store
└── Session in Redis/database
└── All regions access same store
└── Adds latency, single point of failure

Option 3: Distributed Session Store
└── Redis Cluster across regions
└── Session replicated
└── Complex but resilient

Option 4: Stateless (JWT/Token)
└── Session in client-side token
└── No server-side state
└── Best for multi-region

Recommendation:
- Prefer stateless where possible
- If stateful, use distributed store
- Design for session loss on failover

Database Patterns

Database Deployment Options

Option 1: Single Primary + Read Replicas
┌───────────────┐         ┌───────────────┐
│   US-EAST     │         │   EU-WEST     │
│  ┌─────────┐  │   ───►  │  ┌─────────┐  │
│  │ Primary │  │  Async  │  │ Replica │  │
│  │  (R/W)  │  │  Replic │  │  (Read) │  │
│  └─────────┘  │         │  └─────────┘  │
└───────────────┘         └───────────────┘
- Writes go to primary region
- Reads served locally
- Failover promotes replica

Option 2: Multi-Primary (Active-Active)
┌───────────────┐         ┌───────────────┐
│   US-EAST     │◄───────►│   EU-WEST     │
│  ┌─────────┐  │  Bi-dir │  ┌─────────┐  │
│  │ Primary │  │  Replic │  │ Primary │  │
│  │  (R/W)  │  │         │  │  (R/W)  │  │
│  └─────────┘  │         │  └─────────┘  │
└───────────────┘         └───────────────┘
- Writes accepted in both regions
- Conflict resolution required
- Complex but lowest latency

Option 3: Globally Distributed Database
┌─────────────────────────────────────────┐
│  CockroachDB / Spanner / YugabyteDB    │
│  ┌─────┐    ┌─────┐    ┌─────┐        │
│  │ US  │────│ EU  │────│ APAC│        │
│  └─────┘    └─────┘    └─────┘        │
│  Automatic sharding and replication    │
└─────────────────────────────────────────┘
- Database handles distribution
- Strong consistency available
- Higher latency for writes

Testing and Validation

Chaos Engineering for Multi-Region

Multi-Region Chaos Tests:

1. Region Failover Test
   □ Fail primary region completely
   □ Measure failover time
   □ Verify data integrity
   □ Test user experience

2. Network Partition Test
   □ Block inter-region communication
   □ Verify split-brain handling
   □ Test conflict resolution

3. Partial Failure Test
   □ Fail subset of services in region
   □ Test degraded operation
   □ Verify monitoring/alerting

4. Data Replication Lag Test
   □ Introduce artificial lag
   □ Test application behavior
   □ Verify consistency expectations

5. Failback Test
   □ Restore failed region
   □ Test data sync
   □ Test traffic redistribution

Schedule:
- Failover tests: Monthly
- Full DR drill: Quarterly
- Chaos experiments: Weekly

Best Practices

Multi-Region Best Practices:

1. Design for Failure
   □ Assume any region can fail
   □ No single points of failure
   □ Automated failover
   □ Regular testing

2. Data Strategy
   □ Define consistency requirements
   □ Choose appropriate replication
   □ Plan for conflicts
   □ Consider data residency

3. Observability
   □ Cross-region metrics
   □ Distributed tracing
   □ Centralized logging
   □ Region-aware alerting

4. Cost Management
   □ Right-size standby resources
   □ Use reserved capacity wisely
   □ Monitor data transfer costs
   □ Consider traffic patterns

5. Operational Readiness
   □ Runbooks for failover
   □ Regular DR drills
   □ On-call training
   □ Post-incident reviews

Related Skills

• latency-optimization - Reducing global latency
• distributed-consensus - Consistency patterns
• cdn-architecture - Edge caching for multi-region
• chaos-engineering-fundamentals - Testing resilience