High-Availability Cloud Architecture: Engineering Resilience Across Multi-Region Failures

High-Availability Cloud Architecture: Engineering Resilience Across Multi-Region Failures

Availability is a business requirement, not an engineering aspiration. A single-region cloud deployment — regardless of how resilient the architecture within that region — inherits the failure domain of the region itself. AWS us-east-1, GCP us-central1, and Azure East US have all experienced significant outages that impacted thousands of customers.

For systems where downtime translates directly to revenue loss, regulatory penalty, or user safety risk, multi-region architecture is not optional. This guide covers the engineering disciplines required to design, build, and operate cloud infrastructure that survives regional failure events.


Defining Your Reliability Targets

Before designing architecture, define the SLOs that drive it:

MetricDefinitionTypical Enterprise Target
RTO (Recovery Time Objective)Max acceptable downtime after failure1 minute – 4 hours
RPO (Recovery Point Objective)Max acceptable data loss window0 seconds – 15 minutes
Availability% uptime over rolling period99.99% (52min/yr downtime)
MTTRMean time to recovery after incident< 15 minutes

These targets determine architecture choices — there is no universal “best” design, only designs that match the cost/risk tradeoff for a given SLO.


Architecture Patterns

Active-Active (Highest Availability, Highest Complexity)

Traffic is distributed across multiple regions simultaneously. Every region serves live traffic. Failure of one region causes only a capacity reduction, not a full outage.

                        ┌─────────────────┐
                        │  Global Load    │
                        │  Balancer       │
                        │  (Anycast/GTM)  │
                        └────────┬────────┘
                    ┌────────────┼────────────┐
                    │            │            │
          ┌─────────▼──────┐ ┌──▼──────────┐ ┌▼─────────────┐
          │  us-east-1     │ │  eu-west-1  │ │  ap-south-1  │
          │  (Primary)     │ │  (Active)   │ │  (Active)    │
          │                │ │             │ │              │
          │  App Cluster   │ │  App Cluster│ │  App Cluster │
          │  DB Primary    │ │  DB Replica │ │  DB Replica  │
          └────────────────┘ └─────────────┘ └──────────────┘
                    │                │               │
                    └────────────────┴───────────────┘
                             Global Data Sync
                          (CockroachDB / Spanner /
                           DynamoDB Global Tables)

Use when: RTO < 1 minute, RPO = 0, revenue-critical systems.

Active-Passive (Balanced Cost/Resilience)

A primary region handles all traffic. A standby region runs warm with replicated data, ready to receive traffic within minutes.

Use when: RTO < 15 minutes, RPO < 5 minutes, cost-sensitive workloads.

Pilot Light (Cost-Optimized DR)

Minimal infrastructure runs in the DR region (just database replicas and minimal compute). Full stack is provisioned via IaC on failover trigger.

Use when: RTO < 1 hour, RPO < 30 minutes, infrequently accessed systems.


1. Global Load Balancing & Traffic Management

AWS Global Accelerator

Routes traffic via AWS’s private backbone to the nearest healthy region — bypassing the public internet for lower latency and more reliable routing:

resource "aws_globalaccelerator_accelerator" "main" {
  name            = "gurkhax-global-accelerator"
  ip_address_type = "IPV4"
  enabled         = true

  attributes {
    flow_logs_enabled   = true
    flow_logs_s3_bucket = aws_s3_bucket.ga_logs.bucket
    flow_logs_s3_prefix = "flow-logs/"
  }
}

resource "aws_globalaccelerator_endpoint_group" "us_east" {
  listener_arn          = aws_globalaccelerator_listener.main.id
  endpoint_group_region = "us-east-1"
  traffic_dial_percentage = 50

  health_check_path             = "/health"
  health_check_port             = 443
  health_check_protocol         = "HTTPS"
  health_check_interval_seconds = 10
  threshold_count               = 2
}

Route 53 Health-Check-Based Failover

resource "aws_route53_health_check" "primary" {
  fqdn              = "api-us-east.gurkhax.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/health"
  failure_threshold = 2
  request_interval  = 10
}

resource "aws_route53_record" "api" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "api.gurkhax.com"
  type    = "A"

  failover_routing_policy {
    type = "PRIMARY"
  }

  health_check_id = aws_route53_health_check.primary.id
  set_identifier  = "primary"
  alias {
    name    = aws_lb.us_east.dns_name
    zone_id = aws_lb.us_east.zone_id
    evaluate_target_health = true
  }
}

2. Database Replication Strategy

Your database is the hardest component to make truly multi-region. Options by workload:

DatabaseMulti-Region OptionRPOComplexity
PostgreSQLAWS Aurora Global DB< 1 secondMedium
PostgreSQLCockroachDB0 (active-active)High
Key-ValueDynamoDB Global Tables~1 secondLow
NewSQLGoogle Spanner0 (active-active)Medium
RedisRedis Enterprise Geo-Replication~5 secondsMedium

Aurora Global Database configuration:

resource "aws_rds_global_cluster" "main" {
  global_cluster_identifier = "gurkhax-global-db"
  engine                    = "aurora-postgresql"
  engine_version            = "15.4"
  database_name             = "production"
  deletion_protection       = true
}

resource "aws_rds_cluster" "primary" {
  cluster_identifier        = "gurkhax-primary"
  engine                    = "aurora-postgresql"
  engine_version            = "15.4"
  global_cluster_identifier = aws_rds_global_cluster.main.id
  master_username           = var.db_username
  master_password           = var.db_password

  backup_retention_period   = 35  # Days
  preferred_backup_window   = "03:00-04:00"
  deletion_protection       = true

  enabled_cloudwatch_logs_exports = ["postgresql"]
}

3. Stateless Application Design

Applications that store session state in-process cannot survive instance failure, let alone regional failover. Design for statelessness:

  • Session state → Redis (with cross-region replication) or signed JWTs
  • File uploads → S3 / GCS with cross-region replication enabled
  • Ephemeral compute state → never. Refactor to event queues (SQS, Pub/Sub, Kafka)
  • Configuration → externalized to parameter stores (SSM Parameter Store, GCP Secret Manager)

4. Chaos Engineering: Validating Resilience Before the Pager Fires

Multi-region architecture that has never been tested under failure conditions is not resilient — it is aspirationally resilient. Run controlled chaos experiments:

# AWS Fault Injection Simulator — simulate AZ failure
import boto3

fis = boto3.client('fis', region_name='us-east-1')

experiment = fis.start_experiment(
    experimentTemplateId='EXT-XXXXXXXX',  # AZ outage template
    tags={'Environment': 'staging', 'Purpose': 'DR drill'}
)

# Monitor metrics during experiment
# Validate: traffic shifts to secondary region within RTO window
# Validate: no data loss beyond RPO window
# Validate: alerting fires correctly
# Validate: runbooks are accurate

Quarterly DR drills must include:

  1. Full regional failover with real traffic rerouting
  2. Database promotion from replica to primary in DR region
  3. Application cold-start from IaC in DR region (pilot light test)
  4. RTO/RPO measurement against SLO targets
  5. Post-drill retrospective updating runbooks

High-Availability Engineering Checklist

  • RTO and RPO targets defined, documented, and signed off by stakeholders
  • Architecture pattern selected based on RTO/RPO (Active-Active / Active-Passive / Pilot Light)
  • Global load balancer with health-check-based traffic steering configured
  • Database replication with continuous monitoring of replication lag
  • Stateless application design — no in-process session state
  • IaC covers full DR region provisioning — no manual steps in failover runbook
  • Chaos engineering experiments scheduled quarterly
  • Runbooks tested in last 90 days — not just written
  • RTO/RPO validated against actual DR drills, not theoretical estimates
  • Recovery procedures documented for each failure tier (instance, AZ, region)

GurkhaX designs and validates multi-region high-availability architectures for enterprises where downtime is not acceptable. Schedule an infrastructure resilience review with our cloud engineering team.