Part 12: Resilience Engineering

Module 21: Failure as a Design Principle

Continuous Partial Failure

Here is the critical insight that separates novice from expert distributed systems engineers: distributed systems exist in a state of continuous partial failure. At any given moment in a large-scale system, something is broken — a disk is failing, a network link is dropping packets, a process is stuck in garbage collection, a deployment is rolling through.

This insight has profound implications for system design:

• You cannot prevent failure — you can only contain its impact
• Every external dependency will eventually fail, timeout, or return garbage
• Networks are unreliable — partitions happen, latency spikes, packets get corrupted
• Hardware fails gradually — disks develop bad sectors, memory develops bit flips
• Software has bugs — memory leaks, race conditions, integer overflows

Designing for Failure, Not Against It

Traditional engineering focuses on preventing failure (stronger materials, redundant components). Resilience engineering focuses on surviving failure — accepting that failures will occur and designing systems that degrade gracefully rather than catastrophically.

The spectrum of failure responses:

1. Crash (worst): System stops entirely. All users affected.
2. Cascading failure: One failure triggers others. Progressively worse until total failure.
3. Graceful degradation (goal): System continues with reduced functionality. Non-critical features disabled, core functions preserved.
4. Self-healing (ideal): System detects failure, isolates it, and recovers automatically without human intervention.

Failure Mode Analysis

Every component in your system can fail in multiple ways. Systematic failure mode analysis identifies these before they happen in production:

• Fail-stop: Component stops and is clearly dead. Easy to detect (health checks fail). Example: process crash, server power loss.
• Fail-slow: Component responds but with degraded performance. Hardest to detect — looks "alive" but poisons the system with slow responses that cause upstream timeouts. Example: GC pauses, disk degradation, noisy neighbor.
• Byzantine failure: Component behaves incorrectly — returns wrong data, corrupts state, or lies about its health. Most dangerous. Example: bit flips, software bugs, compromised nodes.
• Fail-partial: Component handles some requests correctly but fails on others. Example: database connection pool exhausted (new connections fail, existing ones work).

Module 22: Failure Domains

Failure Domain Hierarchy

A failure domain is a group of resources that share a common point of failure — when the shared component fails, everything in that domain fails together. Understanding the hierarchy of failure domains is essential for blast radius calculation.

flowchart TD
    A["🌍 Cloud Provider
(AWS, Azure, GCP)"] --> B["🗺️ Region
(us-east-1, eu-west-1)"]
    B --> C["🏢 Availability Zone
(us-east-1a, 1b, 1c)"]
    C --> D["🏗️ Rack / Network Segment
(shared switch, power)"]
    D --> E["🖥️ Physical Host
(shared hardware)"]
    E --> F["📦 VM / Container
(shared kernel)"]
    F --> G["⚙️ Process
(single binary)"]
    G --> H["🧵 Thread
(shared memory)"]

    style A fill:#132440,color:#fff
    style B fill:#16476A,color:#fff
    style C fill:#3B9797,color:#fff
    style D fill:#3B9797,color:#fff
    style E fill:#5BB5B5,color:#132440
    style F fill:#7CCBCB,color:#132440
    style G fill:#A8DEDE,color:#132440
    style H fill:#D4EDED,color:#132440
                            

Blast radius at each level:

• Thread failure: Affects one request or operation
• Process failure: Affects all connections/requests handled by that process (~100s of requests)
• VM/Container failure: Affects all processes on that instance
• Host failure: Affects all VMs/containers on that physical machine (~10s of instances)
• Rack failure: Affects all hosts sharing power/network (~40-80 hosts)
• AZ failure: Affects all racks in that facility (~1000s of hosts)
• Region failure: Affects all AZs in that region (rare but catastrophic — entire geographic area)
• Provider failure: Affects all regions (nearly unprecedented but theoretically possible — identity/control plane outage)

Blast Radius Reduction

The primary goal of resilience engineering is to minimize blast radius — ensuring that any single failure affects the smallest possible number of users and services.

Blast radius reduction strategies:

• Spread across AZs: Deploy to 3+ availability zones. An AZ failure affects only ~33% of capacity. The remaining AZs absorb the traffic.
• Multi-region active-active: Deploy across regions. A region failure affects only the users routed to that region — others are unaffected.
• Microservice isolation: Failure in the recommendations service shouldn't affect checkout. Each service has its own failure domain.
• Gradual rollouts: Deploy to 1% → 5% → 25% → 100% of traffic. A bad deployment's blast radius is capped at the current rollout percentage.
• Feature flags: Kill a misbehaving feature instantly without deploying new code. Blast radius contained to that single feature.

Cell-Based Architecture

Cell-based architecture is the most advanced blast radius reduction pattern. The system is divided into independent cells — each cell serves a subset of users/tenants and is completely isolated from other cells. A failure in one cell cannot propagate to others.

• Cell isolation: Each cell has its own compute, database, cache, and message queues. No shared state between cells.
• Cell routing: A thin routing layer maps users to cells (typically hash-based). The router itself is the only shared component — keep it stateless and simple.
• Cell sizing: Each cell serves ~5-10% of traffic. Maximum blast radius for a cell failure is bounded at 5-10%.
• Cell independence: You can deploy, scale, and troubleshoot cells independently. One cell can run a newer version while others stay on the old version.

Module 23: Fault Tolerance Patterns

Fault tolerance patterns are the building blocks of resilient systems. Each pattern addresses a specific failure mode — and combining them creates defense in depth.

Retries with Exponential Backoff + Jitter

Retries are the simplest fault tolerance pattern — if a request fails, try again. But naive retries can amplify failures into system-wide outages:

flowchart TD
    A[Request Fails] --> B{Retryable Error?}
    B -->|No: 400, 403, 404| Z[Return Error to Caller]
    B -->|Yes: 500, 503, Timeout| C{Retry Budget
Remaining?}
    C -->|No| Z
    C -->|Yes| D[Calculate Delay]
    D --> E["delay = base × 2^attempt"]
    E --> F["Add Jitter: delay × random(0.5, 1.5)"]
    F --> G["Cap at max_delay"]
    G --> H[Wait]
    H --> I[Retry Request]
    I --> J{Success?}
    J -->|Yes| K[Return Response]
    J -->|No| B
                            

Why naive retries are dangerous:

• Retry storms: If 1000 clients all retry immediately on failure, the server that's already overloaded gets 2000 requests. Then 3000. Then it dies completely.
• Thundering herd: All clients retry at the same moment (e.g., exactly 1 second after failure). The server gets a burst of retries it can't handle.
• Cascading retries: Service A retries → Service B retries → Service C retries. Each layer multiplies the retry count exponentially.

The correct retry pattern requires three safeguards:

1. Exponential backoff: Wait longer between each retry (1s, 2s, 4s, 8s...). This gives the server time to recover.
2. Jitter: Add randomness to the delay. This prevents thundering herds — clients retry at different times instead of simultaneously.
3. Retry budget: Cap the total number of retries. After N attempts, give up and return an error. Prevents infinite retry loops.

"""
Retry with Exponential Backoff + Full Jitter

This is the gold-standard retry implementation used by AWS SDKs,
gRPC, and most production-grade HTTP clients.
"""
import random
import time
from typing import Callable, Any, Optional

class RetryWithBackoff:
    """
    Production retry handler with exponential backoff and jitter.
    
    Key properties:
    - Exponential backoff: delays grow exponentially (1s, 2s, 4s, 8s...)
    - Full jitter: randomize within [0, calculated_delay] to prevent thundering herd
    - Retry budget: maximum attempts before giving up
    - Retryable detection: only retry on transient errors (5xx, timeouts)
    """
    def __init__(
        self,
        max_retries: int = 3,
        base_delay: float = 1.0,
        max_delay: float = 30.0,
        retryable_exceptions: tuple = (TimeoutError, ConnectionError),
    ):
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.retryable_exceptions = retryable_exceptions
    
    def _calculate_delay(self, attempt: int) -> float:
        """
        Full jitter: uniform random in [0, min(cap, base × 2^attempt)]
        
        Why full jitter (not equal jitter)?
        - Equal jitter: base_delay/2 + random(0, base_delay/2) — still correlated
        - Full jitter: random(0, exponential_delay) — completely decorrelated
        
        AWS research shows full jitter reduces total completion time by 40%
        compared to equal jitter in high-contention scenarios.
        """
        exponential_delay = self.base_delay * (2 ** attempt)
        capped_delay = min(exponential_delay, self.max_delay)
        # Full jitter: uniform random in [0, capped_delay]
        return random.uniform(0, capped_delay)
    
    def execute(self, func: Callable, *args, **kwargs) -> Any:
        """
        Execute function with retry logic.
        Returns the function result on success, raises on final failure.
        """
        last_exception: Optional[Exception] = None
        
        for attempt in range(self.max_retries + 1):
            try:
                result = func(*args, **kwargs)
                if attempt > 0:
                    print(f"  ✅ Succeeded on attempt {attempt + 1}")
                return result
            except self.retryable_exceptions as e:
                last_exception = e
                if attempt == self.max_retries:
                    print(f"  ❌ All {self.max_retries + 1} attempts failed")
                    raise
                
                delay = self._calculate_delay(attempt)
                print(f"  ⚠️  Attempt {attempt + 1} failed: {e}")
                print(f"     Retrying in {delay:.2f}s (attempt {attempt + 2}/{self.max_retries + 1})")
                time.sleep(delay)
        
        raise last_exception  # Should never reach here


# Demo: Simulating a flaky external service
call_count = 0

def flaky_api_call():
    """Simulates a service that fails 60% of the time"""
    global call_count
    call_count += 1
    if random.random() < 0.6:
        raise ConnectionError(f"Connection refused (call #{call_count})")
    return {"status": "ok", "data": "response_payload"}


# Execute with retry
retry_handler = RetryWithBackoff(
    max_retries=4,
    base_delay=0.1,     # Start with 100ms (demo purposes)
    max_delay=2.0,
    retryable_exceptions=(ConnectionError, TimeoutError)
)

print("Calling flaky API with retry backoff:")
try:
    result = retry_handler.execute(flaky_api_call)
    print(f"  Result: {result}")
except ConnectionError:
    print("  Service unavailable after all retries — returning cached/fallback response")

Circuit Breaker Pattern

A circuit breaker prevents a failing service from being hammered with requests that will inevitably fail. Like an electrical circuit breaker, it "trips" when failure is detected and stops sending requests — giving the downstream service time to recover.

stateDiagram-v2
    [*] --> Closed
    
    Closed --> Open: Failure threshold
exceeded (e.g., 5 failures
in 10 seconds)
    Closed --> Closed: Request succeeds
(reset failure counter)
    
    Open --> HalfOpen: Timeout expires
(e.g., after 30 seconds)
    Open --> Open: Requests immediately
rejected (fail fast)
    
    HalfOpen --> Closed: Probe request
succeeds
    HalfOpen --> Open: Probe request
fails (reset timeout)

    note right of Closed
        Normal operation.
        All requests pass through.
        Track failure count.
    end note

    note right of Open
        Circuit is tripped.
        All requests fail immediately.
        No load on downstream.
    end note

    note right of HalfOpen
        Testing recovery.
        Allow ONE probe request.
        Success → close circuit.
        Failure → reopen circuit.
    end note
                            

Circuit breaker states:

• CLOSED (normal): Requests flow through normally. The breaker tracks failures. When the failure rate exceeds a threshold (e.g., 50% of requests fail in a 10-second window), the breaker trips to OPEN.
• OPEN (tripped): All requests are immediately rejected with a fallback response — no request reaches the downstream service. This prevents wasting resources on requests that will fail, and gives the downstream time to recover. After a configured timeout (e.g., 30 seconds), transitions to HALF-OPEN.
• HALF-OPEN (testing): Allows a single "probe" request through to test if the downstream has recovered. If it succeeds → transition to CLOSED. If it fails → transition back to OPEN (with reset timeout).

"""
Circuit Breaker Implementation

A production-grade circuit breaker with:
- Configurable failure threshold and window
- Automatic state transitions
- Fallback response when open
- Thread-safe state management
"""
import time
from enum import Enum
from typing import Callable, Any, Optional

class CircuitState(Enum):
    CLOSED = "CLOSED"
    OPEN = "OPEN"
    HALF_OPEN = "HALF_OPEN"

class CircuitBreaker:
    """
    Circuit breaker that protects against cascading failures.
    
    When a downstream service fails repeatedly, the breaker opens
    and returns fallback responses instantly — preventing resource
    exhaustion and giving the service time to recover.
    """
    def __init__(
        self,
        failure_threshold: int = 5,
        recovery_timeout: float = 30.0,
        success_threshold: int = 2,
        name: str = "default"
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.success_threshold = success_threshold
        self.name = name
        
        # State
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time: Optional[float] = None
        
        # Metrics
        self.total_calls = 0
        self.total_failures = 0
        self.total_short_circuits = 0
    
    def call(self, func: Callable, fallback: Any = None, *args, **kwargs) -> Any:
        """
        Execute function through the circuit breaker.
        
        Returns function result if circuit is closed/half-open and call succeeds.
        Returns fallback if circuit is open or call fails in half-open state.
        """
        self.total_calls += 1
        
        if self.state == CircuitState.OPEN:
            # Check if recovery timeout has elapsed
            if time.time() - self.last_failure_time >= self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
                self.success_count = 0
                print(f"  [{self.name}] OPEN → HALF_OPEN (testing recovery)")
            else:
                # Fail fast — don't even try
                self.total_short_circuits += 1
                print(f"  [{self.name}] OPEN — request short-circuited")
                return fallback
        
        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            if self.state == CircuitState.OPEN:
                return fallback
            raise
    
    def _on_success(self):
        """Handle successful call"""
        if self.state == CircuitState.HALF_OPEN:
            self.success_count += 1
            if self.success_count >= self.success_threshold:
                self.state = CircuitState.CLOSED
                self.failure_count = 0
                print(f"  [{self.name}] HALF_OPEN → CLOSED (service recovered)")
        else:
            # Reset failure count on success in closed state
            self.failure_count = 0
    
    def _on_failure(self):
        """Handle failed call"""
        self.failure_count += 1
        self.total_failures += 1
        self.last_failure_time = time.time()
        
        if self.state == CircuitState.HALF_OPEN:
            # Probe failed — reopen
            self.state = CircuitState.OPEN
            print(f"  [{self.name}] HALF_OPEN → OPEN (probe failed)")
        elif self.failure_count >= self.failure_threshold:
            # Threshold exceeded — trip the breaker
            self.state = CircuitState.OPEN
            print(f"  [{self.name}] CLOSED → OPEN (threshold exceeded: {self.failure_count} failures)")
    
    def stats(self) -> dict:
        """Return circuit breaker metrics"""
        return {
            "name": self.name,
            "state": self.state.value,
            "total_calls": self.total_calls,
            "total_failures": self.total_failures,
            "total_short_circuits": self.total_short_circuits,
            "failure_rate": f"{(self.total_failures / max(self.total_calls, 1)) * 100:.1f}%"
        }


# Demo: Circuit breaker protecting a payment service
import random

breaker = CircuitBreaker(
    failure_threshold=3,
    recovery_timeout=2.0,  # Short timeout for demo
    success_threshold=2,
    name="payment-service"
)

def call_payment_service():
    """Simulates an unreliable payment service"""
    if random.random() < 0.7:  # 70% failure rate
        raise ConnectionError("Payment service timeout")
    return {"status": "charged", "amount": 99.99}

FALLBACK = {"status": "queued", "message": "Payment will be processed shortly"}

print("=== Circuit Breaker Demo ===\n")
for i in range(12):
    result = breaker.call(call_payment_service, fallback=FALLBACK)
    print(f"  Call {i+1}: {result}")
    time.sleep(0.3)

print(f"\n  Stats: {breaker.stats()}")

Bulkhead Pattern

Named after ship bulkheads (watertight compartments that prevent a hull breach from sinking the entire ship), the bulkhead pattern isolates different parts of a system so that failure in one part cannot exhaust resources shared by others.

flowchart TD
    subgraph Client ["Incoming Requests"]
        R1[Product Requests]
        R2[Payment Requests]
        R3[Notification Requests]
    end

    subgraph Bulkheads ["Isolated Resource Pools"]
        B1["Pool A: Products
Max 50 threads
Max 200 connections"]
        B2["Pool B: Payments
Max 30 threads
Max 100 connections"]
        B3["Pool C: Notifications
Max 20 threads
Max 50 connections"]
    end

    subgraph Services ["Downstream Services"]
        S1[Product Service]
        S2[Payment Gateway]
        S3[Email/SMS Service]
    end

    R1 --> B1
    R2 --> B2
    R3 --> B3
    B1 --> S1
    B2 --> S2
    B3 --> S3

    style B1 fill:#3B9797,color:#fff
    style B2 fill:#BF092F,color:#fff
    style B3 fill:#16476A,color:#fff
                            

Types of bulkhead isolation:

• Thread pool isolation: Each downstream service gets its own thread pool. If the payment service is slow and exhausts its 30 threads, the product service's 50 threads continue working normally.
• Connection pool isolation: Separate connection pools per dependency. A slow database query won't exhaust connections needed for the cache.
• Process isolation: Critical services run in separate processes/containers. A memory leak in the notification service can't crash the payment service.
• Microservice isolation: The ultimate bulkhead — each domain is its own service with independent scaling, deployment, and failure characteristics.

Timeout Strategies

Timeouts are the simplest and most important resilience pattern — yet they're frequently misconfigured. An aggressive timeout prevents a slow dependency from tying up your resources. A timeout that's too generous is barely better than no timeout at all.

Types of timeouts:

• Connection timeout: How long to wait for a TCP connection to be established. Should be short (1-5 seconds) — if a server can't accept a connection quickly, it's likely overloaded.
• Request timeout: How long to wait for a response after the connection is established. Depends on the operation (read: 1-5s, write: 5-15s, batch: 30-60s).
• Idle timeout: How long a connection can be idle before being closed. Prevents resource leaks from abandoned connections.

# Istio Retry and Timeout Policy
# Applied to all outbound traffic from the checkout service
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-service-resilience
  namespace: checkout
spec:
  hosts:
    - payment-service.payments.svc.cluster.local
  http:
    - route:
        - destination:
            host: payment-service.payments.svc.cluster.local
            port:
              number: 8080
      # Timeout: aggressive — fail fast rather than hang
      timeout: 5s
      retries:
        # Retry configuration
        attempts: 3
        perTryTimeout: 2s           # Each individual attempt gets 2s max
        retryOn: "5xx,reset,connect-failure,retriable-4xx"
        retryRemoteLocalities: true  # Retry on different AZ if available
---
# Circuit breaker via DestinationRule
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payment-service-circuit-breaker
  namespace: checkout
spec:
  host: payment-service.payments.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100          # Bulkhead: max 100 connections
        connectTimeout: 2s           # Connection timeout: 2s
      http:
        h2UpgradePolicy: DEFAULT
        maxRequestsPerConnection: 50
        maxRetries: 3                # Max concurrent retries
    outlierDetection:
      # Circuit breaker triggers
      consecutive5xxErrors: 5        # Trip after 5 consecutive 5xx
      interval: 10s                  # Check window
      baseEjectionTime: 30s          # Eject host for 30s minimum
      maxEjectionPercent: 50         # Never eject more than 50% of hosts

Timeout budgets across call chains:

In a microservice architecture, a user request often traverses multiple services: Gateway → Service A → Service B → Database. Each service adds its own timeout. If each service has a 5-second timeout, the user could wait 15+ seconds before getting an error.

#!/bin/bash
# Timeout Configuration Audit — Check all service timeouts are set correctly
# Run this against your Kubernetes cluster to audit timeout hygiene

echo "=== Timeout Configuration Audit ==="
echo ""

# Check Istio VirtualService timeouts
echo "--- Istio VirtualService Timeouts ---"
kubectl get virtualservices -A -o json | \
  jq -r '.items[] | 
    .metadata.namespace + "/" + .metadata.name + ": " + 
    (if .spec.http[0].timeout then .spec.http[0].timeout else "⚠️  NO TIMEOUT SET" end)'

echo ""
echo "--- Service Connection Pool Limits ---"
kubectl get destinationrules -A -o json | \
  jq -r '.items[] |
    .metadata.namespace + "/" + .metadata.name + ": " +
    "maxConn=" + (.spec.trafficPolicy.connectionPool.tcp.maxConnections // "UNLIMITED" | tostring) +
    " timeout=" + (.spec.trafficPolicy.connectionPool.tcp.connectTimeout // "DEFAULT" | tostring)'

echo ""
echo "--- Pods Without Resource Limits (bulkhead risk) ---"
kubectl get pods -A -o json | \
  jq -r '.items[] | select(.spec.containers[].resources.limits == null) |
    .metadata.namespace + "/" + .metadata.name + " ⚠️  No resource limits"' | head -20

echo ""
echo "=== Recommendations ==="
echo "1. Every VirtualService MUST have a timeout (default: 5s)"
echo "2. Every DestinationRule MUST have maxConnections (bulkhead)"
echo "3. Every container MUST have CPU/memory limits (process isolation)"
echo "4. perTryTimeout should be LESS than total timeout"
echo "5. Connection timeout should be SHORT (1-3s)"

Composing Patterns: Defense in Depth

No single pattern is sufficient. Production systems compose multiple patterns into a layered defense:

1. Timeout (innermost): Prevents individual requests from hanging forever
2. Retry with backoff: Handles transient failures (network blips, brief overloads)
3. Circuit breaker: Detects sustained failures and stops retrying entirely
4. Bulkhead (outermost): Isolates the blast radius — even if one dependency's circuit is open, other dependencies continue functioning

The composition order matters: Bulkhead → Circuit Breaker → Retry → Timeout → Actual Call

Case Studies

Netflix Hystrix: The Origin of Circuit Breakers in Software

AWS Cell-Based Architecture

Conclusion & Next Steps

Modules 21–23 covered the engineering discipline of building systems that survive failure — not by preventing it, but by containing it.

The key takeaways:

• Failure is normal, not exceptional. At scale, partial failure is the constant state. Design every component with explicit failure modes and recovery paths.
• Blast radius is the #1 metric. When (not if) a failure occurs, how many users are affected? Cell architecture, AZ distribution, and microservice isolation all reduce blast radius.
• Fail fast, don't fail slow. A slow response is worse than a fast error. Aggressive timeouts + fallback responses preserve user experience better than waiting and hoping.
• Retries need three safeguards: Exponential backoff (give time to recover), jitter (prevent thundering herd), and retry budget (stop eventually). Plus idempotency on the server.
• Circuit breakers prevent cascading failure. When a dependency is down, stop calling it. Return fallbacks. Test recovery with probe requests. This pattern alone prevents most cascading outages.
• Bulkheads isolate blast radius. Dedicated thread/connection pools per dependency ensure that one slow service can't exhaust resources needed by others.
• Compose patterns in layers: Bulkhead → Circuit Breaker → Retry → Timeout → Call. Each layer catches what the inner layer missed.
• Timeouts budgets propagate deadlines. Pass remaining time through the call chain. If the budget is exhausted, fail immediately — downstream calls would be wasted.