Showing Posts From

Cloud infrastructure

Arjun Thapa

15 Oct, 2026

Cloud & Infrastructure

High-Availability Cloud Architecture: Engineering Resilience Across Multi-Region Failures

Availability is a business requirement, not an engineering aspiration. A single-region cloud deployment — regardless of how resilient the architecture within that region — inherits the failure domain of the region itself. AWS us-east-1, GCP us-central1, and Azure East US have all experienced significant outages that impacted thousands of customers. For systems where downtime translates directly to revenue loss, regulatory penalty, or user safety risk, multi-region architecture is not optional. This guide covers the engineering disciplines required to design, build, and operate cloud infrastructure that survives regional failure events.Defining Your Reliability Targets Before designing architecture, define the SLOs that drive it:Metric Definition Typical Enterprise TargetRTO (Recovery Time Objective) Max acceptable downtime after failure 1 minute – 4 hoursRPO (Recovery Point Objective) Max acceptable data loss window 0 seconds – 15 minutesAvailability % uptime over rolling period 99.99% (52min/yr downtime)MTTR Mean time to recovery after incident < 15 minutesThese targets determine architecture choices — there is no universal "best" design, only designs that match the cost/risk tradeoff for a given SLO.Architecture Patterns Active-Active (Highest Availability, Highest Complexity) Traffic is distributed across multiple regions simultaneously. Every region serves live traffic. Failure of one region causes only a capacity reduction, not a full outage. ┌─────────────────┐ │ Global Load │ │ Balancer │ │ (Anycast/GTM) │ └────────┬────────┘ ┌────────────┼────────────┐ │ │ │ ┌─────────▼──────┐ ┌──▼──────────┐ ┌▼─────────────┐ │ us-east-1 │ │ eu-west-1 │ │ ap-south-1 │ │ (Primary) │ │ (Active) │ │ (Active) │ │ │ │ │ │ │ │ App Cluster │ │ App Cluster│ │ App Cluster │ │ DB Primary │ │ DB Replica │ │ DB Replica │ └────────────────┘ └─────────────┘ └──────────────┘ │ │ │ └────────────────┴───────────────┘ Global Data Sync (CockroachDB / Spanner / DynamoDB Global Tables)Use when: RTO < 1 minute, RPO = 0, revenue-critical systems. Active-Passive (Balanced Cost/Resilience) A primary region handles all traffic. A standby region runs warm with replicated data, ready to receive traffic within minutes. Use when: RTO < 15 minutes, RPO < 5 minutes, cost-sensitive workloads. Pilot Light (Cost-Optimized DR) Minimal infrastructure runs in the DR region (just database replicas and minimal compute). Full stack is provisioned via IaC on failover trigger. Use when: RTO < 1 hour, RPO < 30 minutes, infrequently accessed systems.1. Global Load Balancing & Traffic Management AWS Global Accelerator Routes traffic via AWS's private backbone to the nearest healthy region — bypassing the public internet for lower latency and more reliable routing: resource "aws_globalaccelerator_accelerator" "main" { name = "gurkhax-global-accelerator" ip_address_type = "IPV4" enabled = true attributes { flow_logs_enabled = true flow_logs_s3_bucket = aws_s3_bucket.ga_logs.bucket flow_logs_s3_prefix = "flow-logs/" } }resource "aws_globalaccelerator_endpoint_group" "us_east" { listener_arn = aws_globalaccelerator_listener.main.id endpoint_group_region = "us-east-1" traffic_dial_percentage = 50 health_check_path = "/health" health_check_port = 443 health_check_protocol = "HTTPS" health_check_interval_seconds = 10 threshold_count = 2 }Route 53 Health-Check-Based Failover resource "aws_route53_health_check" "primary" { fqdn = "api-us-east.gurkhax.com" port = 443 type = "HTTPS" resource_path = "/health" failure_threshold = 2 request_interval = 10 }resource "aws_route53_record" "api" { zone_id = aws_route53_zone.main.zone_id name = "api.gurkhax.com" type = "A" failover_routing_policy { type = "PRIMARY" } health_check_id = aws_route53_health_check.primary.id set_identifier = "primary" alias { name = aws_lb.us_east.dns_name zone_id = aws_lb.us_east.zone_id evaluate_target_health = true } }2. Database Replication Strategy Your database is the hardest component to make truly multi-region. Options by workload:Database Multi-Region Option RPO ComplexityPostgreSQL AWS Aurora Global DB < 1 second MediumPostgreSQL CockroachDB 0 (active-active) HighKey-Value DynamoDB Global Tables ~1 second LowNewSQL Google Spanner 0 (active-active) MediumRedis Redis Enterprise Geo-Replication ~5 seconds MediumAurora Global Database configuration: resource "aws_rds_global_cluster" "main" { global_cluster_identifier = "gurkhax-global-db" engine = "aurora-postgresql" engine_version = "15.4" database_name = "production" deletion_protection = true }resource "aws_rds_cluster" "primary" { cluster_identifier = "gurkhax-primary" engine = "aurora-postgresql" engine_version = "15.4" global_cluster_identifier = aws_rds_global_cluster.main.id master_username = var.db_username master_password = var.db_password backup_retention_period = 35 # Days preferred_backup_window = "03:00-04:00" deletion_protection = true enabled_cloudwatch_logs_exports = ["postgresql"] }3. Stateless Application Design Applications that store session state in-process cannot survive instance failure, let alone regional failover. Design for statelessness:Session state → Redis (with cross-region replication) or signed JWTs File uploads → S3 / GCS with cross-region replication enabled Ephemeral compute state → never. Refactor to event queues (SQS, Pub/Sub, Kafka) Configuration → externalized to parameter stores (SSM Parameter Store, GCP Secret Manager)4. Chaos Engineering: Validating Resilience Before the Pager Fires Multi-region architecture that has never been tested under failure conditions is not resilient — it is aspirationally resilient. Run controlled chaos experiments: # AWS Fault Injection Simulator — simulate AZ failure import boto3fis = boto3.client('fis', region_name='us-east-1')experiment = fis.start_experiment( experimentTemplateId='EXT-XXXXXXXX', # AZ outage template tags={'Environment': 'staging', 'Purpose': 'DR drill'} )# Monitor metrics during experiment # Validate: traffic shifts to secondary region within RTO window # Validate: no data loss beyond RPO window # Validate: alerting fires correctly # Validate: runbooks are accurateQuarterly DR drills must include:Full regional failover with real traffic rerouting Database promotion from replica to primary in DR region Application cold-start from IaC in DR region (pilot light test) RTO/RPO measurement against SLO targets Post-drill retrospective updating runbooksHigh-Availability Engineering Checklist RTO and RPO targets defined, documented, and signed off by stakeholders Architecture pattern selected based on RTO/RPO (Active-Active / Active-Passive / Pilot Light) Global load balancer with health-check-based traffic steering configured Database replication with continuous monitoring of replication lag Stateless application design — no in-process session state IaC covers full DR region provisioning — no manual steps in failover runbook Chaos engineering experiments scheduled quarterly Runbooks tested in last 90 days — not just written RTO/RPO validated against actual DR drills, not theoretical estimates Recovery procedures documented for each failure tier (instance, AZ, region)GurkhaX designs and validates multi-region high-availability architectures for enterprises where downtime is not acceptable. Schedule an infrastructure resilience review with our cloud engineering team.

Read IntelPOST #01

Arjun Thapa

10 Sep, 2026

Cloud & Infrastructure

Zero-Trust Architecture: Hardening Multi-Cloud Kubernetes Clusters

In a perimeter-less cloud-native world, zero-trust is not a product — it is an operating model. Every workload, service account, and inter-pod communication path must be treated as untrusted by default. Kubernetes, despite its orchestration power, ships with permissive defaults that are catastrophically dangerous at enterprise scale without deliberate hardening. This post outlines the GurkhaX zero-trust methodology for multi-cloud Kubernetes environments, covering network segmentation, workload identity, secrets management, and runtime anomaly detection.1. Network Policy: Deny-All as the Baseline Kubernetes has no default network isolation. Without explicit NetworkPolicy objects, every pod can communicate with every other pod across all namespaces. Start with a global deny-all baseline: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: production spec: podSelector: {} policyTypes: - Ingress - EgressThen grant communication only along declared trust paths. For example, allowing the payments service to reach the database tier: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-payments-to-db namespace: production spec: podSelector: matchLabels: app: postgres ingress: - from: - podSelector: matchLabels: app: payments-service ports: - protocol: TCP port: 5432Use Cilium or Calico for L7-aware network policies that can filter on HTTP methods, paths, and gRPC calls — far beyond what the standard CNI spec allows.2. Workload Identity: IRSA, Workload Identity Federation, and SPIFFE Never mount long-lived cloud credentials into pods. Instead, use cloud-native workload identity mechanisms:AWS EKS: IAM Roles for Service Accounts (IRSA) via OIDC federation GKE: Workload Identity Federation binding Kubernetes service accounts to GCP IAM roles Azure AKS: Azure AD Workload Identity (formerly AAD Pod Identity)Annotate your service account to bind the IAM role: apiVersion: v1 kind: ServiceAccount metadata: name: s3-reader namespace: data-pipeline annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/s3-read-only-roleFor multi-cloud environments, deploy SPIRE (SPIFFE Runtime Environment) as your workload identity fabric — issuing short-lived X.509 SVIDs to every workload regardless of cloud provider.3. Service Mesh mTLS: Istio or Linkerd Enforce mutual TLS (mTLS) between all services using a service mesh. This ensures:Every service authenticates before communicating Traffic is encrypted in-transit even within the cluster Authorization policies can be enforced at the L7 layerIstio PeerAuthentication to enforce strict mTLS across a namespace: apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: strict-mtls namespace: production spec: mtls: mode: STRICTAdd AuthorizationPolicy to restrict which services can call which endpoints: apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: payments-authz namespace: production spec: selector: matchLabels: app: payments-service action: ALLOW rules: - from: - source: principals: ["cluster.local/ns/production/sa/api-gateway"] to: - operation: methods: ["POST"] paths: ["/v1/charge", "/v1/refund"]4. Secrets Management: Never Store Secrets in etcd Kubernetes secrets are base64-encoded — not encrypted — by default. Any actor with get secret RBAC privileges can exfiltrate credentials trivially. Mandatory controls:Enable etcd encryption at rest using AES-256 or KMS-provider envelope encryption Inject secrets via HashiCorp Vault Agent Injector or External Secrets Operator integrated with AWS Secrets Manager / GCP Secret Manager Audit secret access using kubectl audit-log with alerting on get/list verbs against secret resources5. Runtime Threat Detection with Falco Deploy Falco to detect anomalous workload behavior at the syscall level: - rule: Unexpected Outbound Connection from Pod desc: A pod is making an unexpected outbound connection condition: > outbound and container and not (proc.name in (allowed_processes)) and not fd.sip in (trusted_ips) output: > Unexpected outbound connection from pod (user=%user.name command=%proc.cmdline connection=%fd.name container=%container.name image=%container.image.repository) priority: WARNINGRoute Falco alerts into your SIEM (Elastic, Splunk, or Chronicle) via Falcosidekick for enrichment and automated response playbooks.Zero-Trust Hardening Checklist Default-deny NetworkPolicy in all namespaces Workload identity via IRSA / GKE WIF / SPIRE — no static credentials Service mesh mTLS enforced in STRICT mode etcd encryption enabled with external KMS provider Secrets injected via Vault or External Secrets Operator Falco deployed with custom rules and SIEM integration Pod Security Standards (Restricted profile) enforced via admission controller RBAC least-privilege audit run quarterlyGurkhaX runs Kubernetes security posture assessments and full zero-trust architecture implementations for enterprise clients. Contact our team to schedule an infrastructure security review.

Read IntelPOST #02