Part 18: Disaster Recovery & Chaos Engineering

Why Disaster Recovery Matters

Every year, organizations lose billions of dollars to unplanned downtime. In 2023, a single 14-hour outage at a major bank cost over $100 million in direct losses and immeasurable reputational damage. A cloud provider's regional failure in 2024 took down thousands of businesses for 6 hours, revealing that most had no cross-region failover strategy. These aren't hypothetical scenarios — they're reminders that failure isn't a question of if, but when.

Disaster Recovery (DR) is the set of policies, tools, and procedures designed to enable the recovery or continuation of vital technology infrastructure following a natural or human-induced disaster. It's not just about backups — it's a comprehensive strategy encompassing prevention, detection, and correction.

Resilience vs Availability vs Fault Tolerance

These terms are often used interchangeably, but they represent distinct concepts in system design:

flowchart LR
    A[Backup Only] --> B[Cold Standby]
    B --> C[Warm Standby]
    C --> D[Hot Standby]
    D --> E[Active-Active]

    style A fill:#fee,stroke:#c00
    style B fill:#ffe,stroke:#a80
    style C fill:#ffd,stroke:#880
    style D fill:#dfd,stroke:#080
    style E fill:#dff,stroke:#088
                            

Moving from left to right increases both cost and recovery speed. Your position on this spectrum should be determined by your business requirements — specifically your RTO and RPO targets.

DR Fundamentals

Before designing a DR strategy, you must understand two critical metrics that drive every decision: Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

RTO and RPO Explained

gantt
    title RTO & RPO Visualization
    dateFormat HH:mm
    axisFormat %H:%M

    section Timeline
    Last Backup           :done, 08:00, 1h
    Normal Operations     :done, 09:00, 3h
    Disaster Occurs       :crit, milestone, 12:00, 0h
    Data Loss (RPO)       :crit, 11:00, 1h
    Recovery Window (RTO) :active, 12:00, 4h
    System Restored       :milestone, 16:00, 0h
                            

Business Impact Analysis (BIA)

A BIA maps each system to its business impact, helping you prioritize DR investments:

# business-impact-analysis.yaml
systems:
  - name: Payment Processing
    tier: 1
    rto: 15m
    rpo: 0s
    revenue_per_hour: $500,000
    dr_strategy: active-active

  - name: Customer Portal
    tier: 2
    rto: 1h
    rpo: 5m
    revenue_per_hour: $50,000
    dr_strategy: hot-standby

  - name: Internal Analytics
    tier: 3
    rto: 24h
    rpo: 1h
    revenue_per_hour: $0
    dr_strategy: warm-standby

  - name: Development Environment
    tier: 4
    rto: 72h
    rpo: 24h
    revenue_per_hour: $0
    dr_strategy: cold-standby

DR Tiers: Cost vs Recovery Speed

DR Planning Checklist

# dr-planning-checklist.yaml
checklist:
  assessment:
    - Identify critical systems and dependencies
    - Define RTO/RPO for each system
    - Map data flows and storage locations
    - Identify single points of failure
    - Document external dependencies (APIs, SaaS)

  design:
    - Select DR tier per system based on BIA
    - Choose DR region(s) with geographic separation
    - Design network connectivity between regions
    - Plan DNS failover strategy
    - Define data replication approach

  implementation:
    - Automate infrastructure with IaC (Terraform/Pulumi)
    - Configure automated backups with verification
    - Set up cross-region data replication
    - Implement health checks and monitoring
    - Create runbooks for each failure scenario

  testing:
    - Schedule quarterly DR drills
    - Test backup restoration regularly
    - Conduct tabletop exercises annually
    - Validate failover automation end-to-end
    - Measure actual RTO/RPO vs targets

Backup Strategies

Backups are the foundation of any DR strategy. Without reliable, tested backups, no recovery plan can succeed. The challenge isn't just creating backups — it's ensuring they're complete, consistent, recoverable, and stored safely.

Backup Types

The 3-2-1 Backup Rule

Cloud Backup Services

# AWS Backup Plan with Terraform
resource "aws_backup_plan" "production" {
  name = "production-backup-plan"

  rule {
    rule_name         = "daily-backup"
    target_vault_name = aws_backup_vault.production.name
    schedule          = "cron(0 2 * * ? *)"  # Daily at 2 AM UTC

    lifecycle {
      cold_storage_after = 30   # Move to cold after 30 days
      delete_after       = 365  # Delete after 1 year
    }

    copy_action {
      destination_vault_arn = aws_backup_vault.dr_region.arn
      lifecycle {
        delete_after = 180
      }
    }
  }

  rule {
    rule_name         = "hourly-backup"
    target_vault_name = aws_backup_vault.production.name
    schedule          = "cron(0 * * * ? *)"  # Every hour

    lifecycle {
      delete_after = 7  # Keep for 7 days
    }
  }
}

resource "aws_backup_vault" "production" {
  name        = "production-vault"
  kms_key_arn = aws_kms_key.backup.arn

  # Prevent deletion even by admins
  force_destroy = false
}

resource "aws_backup_selection" "production_databases" {
  name         = "production-databases"
  iam_role_arn = aws_iam_role.backup.arn
  plan_id      = aws_backup_plan.production.id

  selection_tag {
    type  = "STRINGEQUALS"
    key   = "backup"
    value = "daily"
  }
}

Database Backup Patterns

#!/bin/bash
# PostgreSQL automated backup with WAL archiving and verification

set -euo pipefail

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/postgresql"
S3_BUCKET="s3://company-backups/postgresql"
DB_NAME="production"

# Create logical backup
echo "Starting pg_dump for ${DB_NAME}..."
pg_dump \
  --format=custom \
  --compress=9 \
  --file="${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.dump" \
  --verbose \
  "${DB_NAME}"

# Verify backup integrity
echo "Verifying backup integrity..."
pg_restore \
  --list "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.dump" > /dev/null 2>&1

if [ $? -eq 0 ]; then
  echo "Backup verification passed"
else
  echo "ERROR: Backup verification failed!"
  exit 1
fi

# Upload to S3 with server-side encryption
aws s3 cp \
  "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.dump" \
  "${S3_BUCKET}/${TIMESTAMP}/${DB_NAME}.dump" \
  --sse aws:kms \
  --storage-class STANDARD_IA

# Cross-region copy for DR
aws s3 cp \
  "${S3_BUCKET}/${TIMESTAMP}/${DB_NAME}.dump" \
  "s3://company-backups-dr/postgresql/${TIMESTAMP}/${DB_NAME}.dump" \
  --source-region us-east-1 \
  --region us-west-2

# Cleanup local backups older than 7 days
find "${BACKUP_DIR}" -name "*.dump" -mtime +7 -delete

echo "Backup completed: ${DB_NAME}_${TIMESTAMP}.dump"

Object Storage Cross-Region Replication

# S3 Cross-Region Replication with Terraform
resource "aws_s3_bucket" "primary" {
  bucket = "company-data-primary"

  versioning {
    enabled = true
  }
}

resource "aws_s3_bucket" "replica" {
  provider = aws.dr_region
  bucket   = "company-data-replica"

  versioning {
    enabled = true
  }
}

resource "aws_s3_bucket_replication_configuration" "primary_to_dr" {
  bucket = aws_s3_bucket.primary.id
  role   = aws_iam_role.replication.arn

  rule {
    id     = "replicate-all"
    status = "Enabled"

    destination {
      bucket        = aws_s3_bucket.replica.arn
      storage_class = "STANDARD_IA"

      encryption_configuration {
        replica_kms_key_id = aws_kms_key.dr_region.arn
      }
    }

    source_selection_criteria {
      sse_kms_encrypted_objects {
        status = "Enabled"
      }
    }
  }
}

Multi-Region Failover

Multi-region architectures are the gold standard for high-availability DR. By distributing your application across geographically separated regions, you protect against regional outages, natural disasters, and even cloud provider failures.

Active-Passive Architecture

In active-passive, the primary region handles all traffic while the secondary region stays synchronized but idle. On failure, DNS or a load balancer switches traffic to the secondary.

flowchart TB
    subgraph DNS[DNS / Global Load Balancer]
        GLB[Route 53 / Traffic Manager]
    end

    subgraph Primary[Primary Region - US-East-1]
        ALB1[Application LB]
        APP1[App Servers]
        DB1[(Primary DB)]
    end

    subgraph Secondary[DR Region - US-West-2]
        ALB2[Application LB]
        APP2[App Servers - Scaled Down]
        DB2[(Replica DB)]
    end

    GLB -->|Active| ALB1
    GLB -.->|Failover| ALB2
    ALB1 --> APP1
    APP1 --> DB1
    ALB2 --> APP2
    APP2 --> DB2
    DB1 -->|Async Replication| DB2
                            

# Route 53 Health Check and Failover
resource "aws_route53_health_check" "primary" {
  fqdn              = "primary.internal.example.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 10

  tags = {
    Name = "primary-health-check"
  }
}

resource "aws_route53_record" "app" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "app.example.com"
  type    = "A"

  failover_routing_policy {
    type = "PRIMARY"
  }

  set_identifier  = "primary"
  health_check_id = aws_route53_health_check.primary.id

  alias {
    name                   = aws_lb.primary.dns_name
    zone_id                = aws_lb.primary.zone_id
    evaluate_target_health = true
  }
}

resource "aws_route53_record" "app_secondary" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "app.example.com"
  type    = "A"

  failover_routing_policy {
    type = "SECONDARY"
  }

  set_identifier = "secondary"

  alias {
    name                   = aws_lb.secondary.dns_name
    zone_id                = aws_lb.secondary.zone_id
    evaluate_target_health = true
  }
}

Active-Active Architecture

In active-active, both regions serve traffic simultaneously. This eliminates failover time entirely but introduces data consistency challenges.

flowchart TB
    subgraph Users
        U1[Users - Americas]
        U2[Users - Europe]
    end

    subgraph GLB[Global Load Balancer]
        CF[CloudFront / Front Door]
    end

    subgraph Region1[US-East-1]
        LB1[ALB]
        APP1[App Cluster]
        DB1[(Aurora Global - Writer)]
        CACHE1[(ElastiCache)]
    end

    subgraph Region2[EU-West-1]
        LB2[ALB]
        APP2[App Cluster]
        DB2[(Aurora Global - Reader)]
        CACHE2[(ElastiCache)]
    end

    U1 --> CF
    U2 --> CF
    CF -->|Geo-routing| LB1
    CF -->|Geo-routing| LB2
    LB1 --> APP1
    LB2 --> APP2
    APP1 --> DB1
    APP1 --> CACHE1
    APP2 --> DB2
    APP2 --> CACHE2
    DB1 <-->|Sync Replication| DB2
                            

Database Replication Across Regions

# Aurora Global Database with Terraform
resource "aws_rds_global_cluster" "main" {
  global_cluster_identifier = "production-global"
  engine                    = "aurora-postgresql"
  engine_version            = "15.4"
  database_name             = "production"
  storage_encrypted         = true
}

# Primary cluster
resource "aws_rds_cluster" "primary" {
  cluster_identifier        = "production-primary"
  engine                    = "aurora-postgresql"
  engine_version            = "15.4"
  global_cluster_identifier = aws_rds_global_cluster.main.id
  master_username           = "admin"
  master_password           = var.db_password
  db_subnet_group_name      = aws_db_subnet_group.primary.name
  vpc_security_group_ids    = [aws_security_group.db_primary.id]
  backup_retention_period   = 35
  preferred_backup_window   = "02:00-03:00"
  storage_encrypted         = true
  kms_key_id                = aws_kms_key.primary.arn
}

# Secondary cluster in DR region
resource "aws_rds_cluster" "secondary" {
  provider                  = aws.dr_region
  cluster_identifier        = "production-secondary"
  engine                    = "aurora-postgresql"
  engine_version            = "15.4"
  global_cluster_identifier = aws_rds_global_cluster.main.id
  db_subnet_group_name      = aws_db_subnet_group.secondary.name
  vpc_security_group_ids    = [aws_security_group.db_secondary.id]
  storage_encrypted         = true
  kms_key_id                = aws_kms_key.secondary.arn

  depends_on = [aws_rds_cluster.primary]
}

Infrastructure DR Patterns

One of the most powerful DR strategies in the cloud era is treating your Infrastructure as Code repository as your DR plan. If your entire infrastructure can be rebuilt from a git repository, your DR becomes a matter of running terraform apply in a new region.

IaC as Disaster Recovery

flowchart TD
    A[Disaster Detected] --> B{Automated or Manual?}
    B -->|Automated| C[Trigger DR Pipeline]
    B -->|Manual| D[Ops Team Decision]
    D --> C
    C --> E[Pull IaC from Git]
    E --> F[terraform init - DR Region]
    F --> G[terraform apply]
    G --> H[Restore Data from Backups]
    H --> I[Verify Health Checks]
    I --> J[Update DNS to DR Region]
    J --> K[System Operational]
                            

#!/bin/bash
# DR failover automation script
set -euo pipefail

DR_REGION="us-west-2"
PRIMARY_REGION="us-east-1"
WORKSPACE="dr-failover"

echo "=== DISASTER RECOVERY FAILOVER INITIATED ==="
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "Target region: ${DR_REGION}"

# Step 1: Initialize Terraform for DR region
echo "[1/6] Initializing Terraform..."
cd infrastructure/
terraform workspace select ${WORKSPACE} || terraform workspace new ${WORKSPACE}
terraform init -backend-config="region=${DR_REGION}"

# Step 2: Apply infrastructure
echo "[2/6] Provisioning DR infrastructure..."
terraform apply \
  -var="region=${DR_REGION}" \
  -var="environment=production" \
  -var="is_dr_failover=true" \
  -auto-approve

# Step 3: Restore database from latest backup
echo "[3/6] Restoring database..."
LATEST_SNAPSHOT=$(aws rds describe-db-cluster-snapshots \
  --db-cluster-identifier production-primary \
  --query "reverse(sort_by(DBClusterSnapshots,&SnapshotCreateTime))[0].DBClusterSnapshotIdentifier" \
  --output text \
  --region ${PRIMARY_REGION})

aws rds restore-db-cluster-from-snapshot \
  --db-cluster-identifier production-dr \
  --snapshot-identifier "${LATEST_SNAPSHOT}" \
  --engine aurora-postgresql \
  --region ${DR_REGION}

# Step 4: Wait for database to be available
echo "[4/6] Waiting for database..."
aws rds wait db-cluster-available \
  --db-cluster-identifier production-dr \
  --region ${DR_REGION}

# Step 5: Verify application health
echo "[5/6] Verifying application health..."
for i in {1..30}; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    "https://dr.internal.example.com/health")
  if [ "$STATUS" = "200" ]; then
    echo "Health check passed!"
    break
  fi
  echo "Attempt $i/30: Status $STATUS, retrying..."
  sleep 10
done

# Step 6: Update DNS
echo "[6/6] Switching DNS to DR region..."
aws route53 change-resource-record-sets \
  --hosted-zone-id ${HOSTED_ZONE_ID} \
  --change-batch file://dns-failover.json

echo "=== FAILOVER COMPLETE ==="
echo "DR environment is now serving traffic"

Terraform State Backup and Recovery

# Remote state with cross-region replication
terraform {
  backend "s3" {
    bucket         = "company-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:us-east-1:123456789:key/abc-123"
    dynamodb_table = "terraform-locks"

    # State bucket has CRR enabled to DR region
  }
}

# Backup state to secondary location
resource "null_resource" "state_backup" {
  triggers = {
    always_run = timestamp()
  }

  provisioner "local-exec" {
    command = <<-EOT
      aws s3 cp \
        s3://company-terraform-state/production/terraform.tfstate \
        s3://company-terraform-state-dr/production/terraform.tfstate \
        --region us-west-2
    EOT
  }
}

Kubernetes Cluster DR with Velero

# Velero backup schedule for Kubernetes DR
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: production-daily-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  template:
    includedNamespaces:
      - production
      - monitoring
      - ingress-nginx
    excludedResources:
      - events
      - events.events.k8s.io
    storageLocation: aws-dr-region
    volumeSnapshotLocations:
      - aws-dr-region
    ttl: 720h  # 30 days retention
    defaultVolumesToRestic: true
    hooks:
      pre:
        - exec:
            container: app
            command:
              - /bin/sh
              - -c
              - "pg_dump production > /backup/pre-backup.sql"
---
# Velero backup storage location (DR region)
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: aws-dr-region
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: k8s-backups-dr
    prefix: velero
  config:
    region: us-west-2
    s3ForcePathStyle: "true"

# Velero DR restore commands
# List available backups
velero backup get

# Restore to DR cluster
velero restore create production-restore \
  --from-backup production-daily-backup-20260514020000 \
  --namespace-mappings production:production \
  --restore-volumes=true

# Monitor restore progress
velero restore describe production-restore --details

# Verify restored resources
kubectl get pods -n production
kubectl get pvc -n production

Chaos Engineering Principles

Chaos engineering is the discipline of experimenting on a system to build confidence in the system's capability to withstand turbulent conditions in production. Rather than waiting for failures to happen, chaos engineering proactively injects failures to discover weaknesses before they cause real outages.

The Chaos Experiment Lifecycle

flowchart TD
    A[Define Steady State] --> B[Form Hypothesis]
    B --> C[Design Experiment]
    C --> D[Limit Blast Radius]
    D --> E[Run Experiment]
    E --> F{Steady State Maintained?}
    F -->|Yes| G[Confidence Increased]
    F -->|No| H[Weakness Found]
    H --> I[Fix the Issue]
    I --> J[Re-run Experiment]
    J --> F
    G --> K[Expand Blast Radius]
    K --> C
                            

Principles of Chaos Engineering

Start Small: Blast Radius Control

# Progressive blast radius expansion
chaos_experiments:
  level_1_pod:
    description: "Kill single pod"
    blast_radius: "1 pod in non-critical service"
    risk: low
    approval: team_lead
    rollback: automatic (k8s restarts pod)

  level_2_node:
    description: "Drain and terminate a node"
    blast_radius: "1 node (multiple pods affected)"
    risk: medium
    approval: engineering_manager
    rollback: auto-scaling replaces node

  level_3_az:
    description: "Simulate AZ failure"
    blast_radius: "33% of capacity"
    risk: high
    approval: vp_engineering
    rollback: manual DNS failover

  level_4_region:
    description: "Simulate full region failure"
    blast_radius: "50% of capacity"
    risk: critical
    approval: cto + scheduled maintenance
    rollback: DR failover procedure

Chaos Engineering Tools

The chaos engineering ecosystem has matured significantly, with tools ranging from simple instance terminators to enterprise platforms with sophisticated experiment orchestration.

Litmus Chaos — Kubernetes-Native

# Litmus ChaosEngine - Pod Delete Experiment
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-chaos
  namespace: production
spec:
  engineState: active
  appinfo:
    appns: production
    applabel: app=payment-service
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "30"
            - name: CHAOS_INTERVAL
              value: "10"
            - name: FORCE
              value: "false"
            - name: PODS_AFFECTED_PERC
              value: "50"
        probe:
          - name: payment-health-check
            type: httpProbe
            mode: Continuous
            httpProbe/inputs:
              url: http://payment-service.production:8080/health
              method:
                get:
                  criteria: ==
                  responseCode: "200"
            runProperties:
              probeTimeout: 5
              retry: 3
              interval: 5

# Litmus ChaosExperiment - Network Latency
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-network-latency
  namespace: production
spec:
  definition:
    scope: Namespaced
    permissions:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["create", "delete", "get", "list"]
    image: litmuschaos/go-runner:latest
    args:
      - -c
      - ./experiments -name pod-network-latency
    command:
      - /bin/bash
    env:
      - name: NETWORK_INTERFACE
        value: eth0
      - name: NETWORK_LATENCY
        value: "200"   # 200ms added latency
      - name: JITTER
        value: "50"    # 50ms jitter
      - name: TOTAL_CHAOS_DURATION
        value: "60"
      - name: TARGET_PODS
        value: "payment-service"
      - name: CONTAINER_RUNTIME
        value: containerd

AWS Fault Injection Simulator

{
  "description": "Simulate AZ failure for production EKS cluster",
  "targets": {
    "eks-nodes": {
      "resourceType": "aws:eks:nodegroup",
      "resourceArns": [
        "arn:aws:eks:us-east-1:123456789:nodegroup/production/ng-az-a"
      ],
      "selectionMode": "ALL"
    }
  },
  "actions": {
    "terminate-instances": {
      "actionId": "aws:ec2:terminate-instances",
      "parameters": {},
      "targets": {
        "Instances": "eks-nodes"
      },
      "startAfter": ["wait-30s"]
    },
    "wait-30s": {
      "actionId": "aws:fis:wait",
      "parameters": {
        "duration": "PT30S"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:123456789:alarm:HighErrorRate"
    }
  ],
  "roleArn": "arn:aws:iam::123456789:role/FISRole",
  "tags": {
    "Environment": "production",
    "Team": "platform"
  }
}

Gremlin — Enterprise Chaos Platform

# Gremlin CLI examples

# Install Gremlin agent on Kubernetes
helm repo add gremlin https://helm.gremlin.com
helm install gremlin gremlin/gremlin \
  --namespace gremlin \
  --create-namespace \
  --set gremlin.secret.managed=true \
  --set gremlin.secret.teamID="YOUR_TEAM_ID" \
  --set gremlin.secret.clusterID="production-eks"

# Run CPU stress experiment
gremlin attack cpu \
  --length 60 \
  --cores 2 \
  --percent 80 \
  --target-tags "service=payment,env=production"

# Run network blackhole (drop all traffic to a dependency)
gremlin attack network blackhole \
  --length 120 \
  --hostnames "database.internal" \
  --target-tags "service=api-gateway"

# Run latency injection
gremlin attack network latency \
  --length 60 \
  --ms 500 \
  --jitter 100 \
  --hostnames "cache.internal" \
  --target-tags "service=user-service"

Running Chaos Experiments

Game Days: Planned Chaos Events

A Game Day is a scheduled event where teams deliberately inject failures to test system resilience. It's the DR equivalent of a fire drill — planned, controlled, and educational.

# game-day-plan.yaml
game_day:
  name: "Q2 2026 Regional Failure Simulation"
  date: "2026-06-15"
  duration: "4 hours (10:00 - 14:00 UTC)"
  scope: "Production - Payment processing pipeline"

  participants:
    incident_commander: "Sarah Chen"
    observers: ["CTO", "VP Eng", "Security Lead"]
    responders: ["Platform Team", "Payment Team", "SRE"]

  pre_requisites:
    - All runbooks updated and accessible
    - Monitoring dashboards pre-loaded
    - Communication channels established (#game-day-war-room)
    - Customer support briefed on potential impact
    - Rollback procedures tested in staging

  scenarios:
    - name: "Database primary failover"
      trigger: "Promote Aurora replica to primary"
      expected_rto: "< 60 seconds"
      success_criteria:
        - "Zero failed transactions"
        - "p99 latency < 500ms during failover"
        - "Automated alerts fire within 30s"

    - name: "AZ-a complete failure"
      trigger: "Terminate all instances in us-east-1a"
      expected_rto: "< 5 minutes"
      success_criteria:
        - "Auto-scaling replaces capacity in other AZs"
        - "No customer-visible errors"
        - "Load balancer drains connections gracefully"

    - name: "Cache cluster failure"
      trigger: "Terminate ElastiCache primary node"
      expected_rto: "< 30 seconds"
      success_criteria:
        - "Application falls back to database"
        - "Graceful degradation (slower, not broken)"
        - "Cache rebuilds automatically"

  abort_conditions:
    - "Customer error rate exceeds 5%"
    - "Revenue impact exceeds $10,000"
    - "Any P1 incident unrelated to game day"

  post_mortem:
    due_date: "2026-06-17"
    template: "game-day-retro-template"

Common Chaos Experiments

CI/CD Integration for Chaos Tests

# .github/workflows/chaos-tests.yml
name: Chaos Engineering Pipeline
on:
  schedule:
    - cron: '0 3 * * 1-5'  # Weekdays at 3 AM UTC
  workflow_dispatch:
    inputs:
      experiment:
        description: 'Chaos experiment to run'
        required: true
        type: choice
        options:
          - pod-delete
          - network-latency
          - cpu-stress
          - all

jobs:
  chaos-experiment:
    runs-on: ubuntu-latest
    environment: production-chaos
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubectl
        uses: azure/k8s-set-context@v4
        with:
          kubeconfig: ${{ secrets.KUBE_CONFIG }}

      - name: Verify steady state (pre-chaos)
        run: |
          echo "Checking baseline metrics..."
          ERROR_RATE=$(kubectl exec -n monitoring prometheus-0 -- \
            promtool query instant \
            'rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])')
          if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
            echo "ERROR: System not in steady state (error rate: $ERROR_RATE)"
            exit 1
          fi
          echo "Steady state confirmed: error rate = $ERROR_RATE"

      - name: Run chaos experiment
        run: |
          kubectl apply -f chaos-experiments/${{ inputs.experiment }}.yaml
          echo "Experiment started. Waiting for completion..."
          sleep 120

      - name: Verify steady state (post-chaos)
        run: |
          echo "Verifying system recovered..."
          for i in {1..12}; do
            ERROR_RATE=$(kubectl exec -n monitoring prometheus-0 -- \
              promtool query instant \
              'rate(http_requests_total{status=~"5.."}[1m]) / rate(http_requests_total[1m])')
            if (( $(echo "$ERROR_RATE < 0.01" | bc -l) )); then
              echo "System recovered! Error rate: $ERROR_RATE"
              exit 0
            fi
            echo "Attempt $i/12: Error rate $ERROR_RATE, waiting..."
            sleep 10
          done
          echo "FAILURE: System did not recover within expected timeframe"
          exit 1

      - name: Cleanup experiment
        if: always()
        run: |
          kubectl delete chaosengine --all -n production
          echo "Chaos resources cleaned up"

      - name: Report results
        if: always()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "Chaos Experiment: ${{ inputs.experiment }}\nResult: ${{ job.status }}\nRun: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_CHAOS_WEBHOOK }}

Building Anti-Fragile Systems

Nassim Taleb introduced the concept of anti-fragility: systems that don't just survive stress but actually improve from it. In software, an anti-fragile system uses failures as learning opportunities to automatically become more resilient over time.

Circuit Breakers and Bulkheads

# Istio DestinationRule with circuit breaker
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payment-service-cb
  namespace: production
spec:
  host: payment-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
        maxRetries: 3
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 30

// Circuit breaker implementation with exponential backoff
class CircuitBreaker {
  constructor(options = {}) {
    this.failureThreshold = options.failureThreshold || 5;
    this.resetTimeout = options.resetTimeout || 30000;
    this.halfOpenRequests = options.halfOpenRequests || 3;

    this.state = 'CLOSED';        // CLOSED, OPEN, HALF_OPEN
    this.failureCount = 0;
    this.successCount = 0;
    this.lastFailureTime = null;
    this.halfOpenAttempts = 0;
  }

  async execute(fn) {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailureTime >= this.resetTimeout) {
        this.state = 'HALF_OPEN';
        this.halfOpenAttempts = 0;
        console.log('Circuit breaker: OPEN → HALF_OPEN');
      } else {
        throw new Error('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  onSuccess() {
    if (this.state === 'HALF_OPEN') {
      this.successCount++;
      if (this.successCount >= this.halfOpenRequests) {
        this.state = 'CLOSED';
        this.failureCount = 0;
        this.successCount = 0;
        console.log('Circuit breaker: HALF_OPEN → CLOSED');
      }
    } else {
      this.failureCount = 0;
    }
  }

  onFailure() {
    this.failureCount++;
    this.lastFailureTime = Date.now();

    if (this.state === 'HALF_OPEN' || this.failureCount >= this.failureThreshold) {
      this.state = 'OPEN';
      console.log(`Circuit breaker: → OPEN (failures: ${this.failureCount})`);
    }
  }
}

// Retry with exponential backoff and jitter
async function retryWithBackoff(fn, options = {}) {
  const maxRetries = options.maxRetries || 3;
  const baseDelay = options.baseDelay || 1000;
  const maxDelay = options.maxDelay || 30000;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      if (attempt === maxRetries) throw error;

      // Exponential backoff with full jitter
      const exponentialDelay = baseDelay * Math.pow(2, attempt);
      const jitter = Math.random() * exponentialDelay;
      const delay = Math.min(jitter, maxDelay);

      console.log(`Retry ${attempt + 1}/${maxRetries} after ${delay}ms`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

Health Checks: Liveness, Readiness, Startup Probes

# Comprehensive Kubernetes health probes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      containers:
        - name: payment-service
          image: company/payment-service:v2.4.1
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi

          # Startup probe: allows slow-starting containers
          startupProbe:
            httpGet:
              path: /health/startup
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 30  # 5s * 30 = 150s max startup

          # Liveness probe: restarts unhealthy pods
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

          # Readiness probe: removes from service
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2
            successThreshold: 2

// Health check endpoint implementation
const express = require('express');
const app = express();

let isReady = false;
let dbConnected = false;
let cacheConnected = false;

// Startup: passes once initialization is complete
app.get('/health/startup', (req, res) => {
  if (dbConnected && cacheConnected) {
    res.status(200).json({ status: 'started', uptime: process.uptime() });
  } else {
    res.status(503).json({
      status: 'starting',
      db: dbConnected,
      cache: cacheConnected
    });
  }
});

// Liveness: passes if process is healthy (not deadlocked)
app.get('/health/live', (req, res) => {
  const memUsage = process.memoryUsage();
  const heapPercent = memUsage.heapUsed / memUsage.heapTotal;

  if (heapPercent > 0.95) {
    res.status(503).json({ status: 'unhealthy', reason: 'memory pressure' });
  } else {
    res.status(200).json({ status: 'alive', memory: heapPercent.toFixed(2) });
  }
});

// Readiness: passes if ready to serve traffic
app.get('/health/ready', (req, res) => {
  if (isReady && dbConnected && cacheConnected) {
    res.status(200).json({ status: 'ready' });
  } else {
    res.status(503).json({
      status: 'not_ready',
      db: dbConnected,
      cache: cacheConnected,
      initialized: isReady
    });
  }
});

Graceful Degradation Patterns

# Feature flags for graceful degradation
# When dependencies fail, degrade gracefully instead of erroring
degradation_config:
  payment_service:
    dependency: payment-gateway
    on_failure:
      - action: queue_for_retry
        message: "Payment queued. You'll receive confirmation shortly."
      - action: disable_feature
        feature: instant-checkout
      - action: fallback
        use: cached_exchange_rates

  recommendation_engine:
    dependency: ml-service
    on_failure:
      - action: fallback
        use: popular_items_cache
      - action: reduce_personalization
        level: generic

  search_service:
    dependency: elasticsearch
    on_failure:
      - action: fallback
        use: database_fulltext_search
      - action: alert
        severity: warning
        message: "Search degraded to DB fallback"

DR Testing & Compliance

A DR plan that hasn't been tested is just a document — not a capability. Regular testing transforms theory into muscle memory, revealing gaps that would otherwise surface during an actual disaster.

DR Testing Approaches

Runbook Documentation

# runbook-database-failover.yaml
runbook:
  name: "Database Regional Failover"
  version: "3.2"
  last_tested: "2026-04-15"
  last_updated: "2026-05-01"
  owner: "Platform Team"
  estimated_time: "15-30 minutes"

  pre_conditions:
    - Aurora Global Database configured
    - DR region replica healthy and in sync
    - IAM credentials with failover permissions
    - Monitoring dashboards accessible

  steps:
    - id: 1
      action: "Confirm failover is necessary"
      details: |
        Verify primary region is genuinely failed:
        - Check AWS Health Dashboard
        - Confirm from multiple network paths
        - Rule out local network issues
      decision_maker: "Incident Commander"

    - id: 2
      action: "Notify stakeholders"
      details: |
        Post in #incidents: "Initiating DB failover to us-west-2"
        Notify: VP Eng, On-call SRE, Customer Success
      automation: "PagerDuty escalation"

    - id: 3
      action: "Promote DR replica"
      command: |
        aws rds failover-global-cluster \
          --global-cluster-identifier production-global \
          --target-db-cluster-identifier production-secondary \
          --region us-west-2
      verification: |
        aws rds describe-global-clusters \
          --global-cluster-identifier production-global \
          --query "GlobalClusters[0].GlobalClusterMembers[?IsWriter==\`true\`].DBClusterArn"

    - id: 4
      action: "Update application configuration"
      command: |
        kubectl set env deployment/api-server \
          DATABASE_URL=postgresql://production-secondary.cluster-xxx.us-west-2.rds.amazonaws.com:5432/production \
          -n production
      verification: "kubectl rollout status deployment/api-server -n production"

    - id: 5
      action: "Verify application health"
      command: |
        curl -s https://app.example.com/health | jq '.database'
      expected_output: '{"status": "connected", "region": "us-west-2"}'

  rollback:
    condition: "Failover did not resolve the issue"
    steps:
      - "Revert DATABASE_URL to primary endpoint"
      - "Investigate root cause"
      - "Schedule planned failback during maintenance window"

Compliance Requirements

# DR compliance evidence tracking
compliance_evidence:
  soc2:
    requirement: "CC7.4 - Recovery testing"
    evidence:
      - type: "DR test report"
        frequency: "quarterly"
        last_completed: "2026-04-15"
        next_due: "2026-07-15"
      - type: "Backup restoration log"
        frequency: "monthly"
        last_completed: "2026-05-01"
      - type: "RTO/RPO measurement"
        frequency: "quarterly"
        actual_rto: "4m 32s"
        target_rto: "15m"

  iso27001:
    requirement: "A.17.1 - Business continuity planning"
    controls:
      - "BCP documented and approved by management"
      - "DR plan tested at least annually"
      - "Results reviewed and plans updated"
      - "Staff trained on DR procedures"

  hipaa:
    requirement: "§164.308(a)(7) - Contingency Plan"
    elements:
      - "Data backup plan (R)"
      - "Disaster recovery plan (R)"
      - "Emergency mode operation plan (R)"
      - "Testing and revision procedures (A)"
      - "Applications and data criticality analysis (A)"

Hands-On Exercises

Conclusion & Next Steps

Disaster recovery and chaos engineering are two sides of the same coin: DR ensures you can recover from failure, while chaos engineering proves that you will. Together, they transform your infrastructure from fragile to anti-fragile — systems that not only survive disruption but improve because of it.

The key takeaways from this article:

• RTO and RPO drive every DR decision — start with Business Impact Analysis
• The 3-2-1-1-0 rule is the gold standard for backup strategy
• Infrastructure as Code is your DR plan — if you can rebuild from git, you're resilient
• Chaos engineering builds confidence through scientific experimentation, not random destruction
• A plan without testing is fiction — quarterly DR drills are the minimum
• Anti-fragility means using each failure to strengthen the system