Failure Modes — Control Plane vs Data Plane Failures

Control Plane Failure Characteristics

When the control plane fails, the system becomes unmanaged but not necessarily broken. Existing workloads continue running on their last-known configuration. The system loses its ability to adapt, heal, or change — but it doesn't immediately lose its ability to function.

flowchart TD
    FAIL["Control Plane\nFAILURE"] --> LOST["Capabilities LOST"]
    FAIL --> KEEP["Capabilities RETAINED"]
    LOST --> L1["New Deployments"]
    LOST --> L2["Autoscaling"]
    LOST --> L3["Self-Healing"]
    LOST --> L4["Scheduling"]
    LOST --> L5["Policy Updates"]
    KEEP --> K1["Running Workloads"]
    KEEP --> K2["Network Routes"]
    KEEP --> K3["Load Balancing"]
    KEEP --> K4["Existing Connections"]
    KEEP --> K5["Data Persistence"]
    style FAIL fill:#BF092F,color:#fff
    style LOST fill:#BF092F,color:#fff
    style KEEP fill:#3B9797,color:#fff
                            

Data Plane Failure Characteristics

When the data plane fails, service is immediately disrupted. Users experience errors, requests fail, traffic drops. The control plane may be perfectly healthy and aware of the problem — but awareness doesn't serve traffic.

flowchart TD
    FAIL["Data Plane\nFAILURE"] --> IMM["IMMEDIATE Impact"]
    FAIL --> WORKS["Still Working"]
    IMM --> I1["Request Failures\n(5xx, timeouts)"]
    IMM --> I2["Revenue Loss\n(Every second)"]
    IMM --> I3["User Experience\n(Broken)"]
    IMM --> I4["SLA Breach\n(Clock ticking)"]
    IMM --> I5["Cascade Risk\n(Dependent services)"]
    WORKS --> W1["Control Plane\n(Healthy, aware)"]
    WORKS --> W2["Monitoring\n(Alerts firing)"]
    WORKS --> W3["Self-Healing\n(Trying to recover)"]
    WORKS --> W4["Logging\n(Recording failure)"]
    style FAIL fill:#BF092F,color:#fff
    style IMM fill:#BF092F,color:#fff
    style WORKS fill:#3B9797,color:#fff
                            

The Critical Insight

Real-World Examples

Kubernetes Control Plane Down

When the Kubernetes API server, etcd, or controller-manager goes down:

• Pods keep running — kubelet maintains containers locally
• Services keep routing — kube-proxy rules are already programmed into iptables/IPVS
• No new pods — scheduler can't assign pending pods to nodes
• No self-healing — if a pod crashes, it won't be restarted by the controller
• kubectl is broken — can't query or modify cluster state

Service Mesh Control Plane Down

When Istio's istiod (or Linkerd's control plane) fails:

• Envoy proxies continue — using last-known configuration
• mTLS continues — existing certificates valid until expiry
• Traffic policies frozen — can't add new routing rules
• New pods get no config — sidecar injection works, but no xDS config arrives
• Certificate rotation stops — time bomb (typically 24h expiry in Istio)

Cloud Provider API Outage

When AWS/Azure/GCP management APIs are unavailable:

• Running VMs/containers continue — hypervisor doesn't need the API to run workloads
• Existing load balancers route traffic — configuration is cached locally
• No new resource provisioning — can't create VMs, databases, or networks
• No autoscaling — cloud autoscaler can't call the API to add instances
• Terraform/IaC breaks — can't plan or apply changes

Failure Isolation Design

Well-architected systems explicitly design for failure isolation between control and data planes. The key principle: the data plane must be able to function autonomously when the control plane is unavailable.

flowchart TB
    subgraph STRAT["Isolation Strategies"]
                CACHE["Local Caching\nof Control Decisions"]
                GRACE["Graceful Degradation\nFallback Behaviors"]
                TIMEOUT["Timeout Independence\nDon't block on control"]
                LAST["Last-Known-Good\nConfiguration Persistence"]
    end
    subgraph EXAMPLE["Implementation Examples"]
                E1["Envoy caches xDS config\nlocally on disk"]
                E2["DNS resolvers cache\nentries past TTL in emergency"]
                E3["Kubelet continues pods\nwithout API server"]
                E4["CDN edge serves\nstale content if origin fails"]
    end
    CACHE --> E1
    GRACE --> E2
    TIMEOUT --> E3
    LAST --> E4
                            

# Kubernetes liveness probes — monitoring control plane components
# These detect control plane failures before they cascade
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
    - name: kube-apiserver
      livenessProbe:
        httpGet:
          path: /livez
          port: 6443
          scheme: HTTPS
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 15
        failureThreshold: 8    # Tolerate brief hiccups
      readinessProbe:
        httpGet:
          path: /readyz
          port: 6443
          scheme: HTTPS
        periodSeconds: 1
        timeoutSeconds: 15
      startupProbe:
        httpGet:
          path: /livez
          port: 6443
          scheme: HTTPS
        failureThreshold: 24   # 24 × 10s = 4 min startup tolerance
        periodSeconds: 10
---
# etcd health monitoring
apiVersion: v1
kind: Pod
metadata:
  name: etcd
  namespace: kube-system
spec:
  containers:
    - name: etcd
      livenessProbe:
        httpGet:
          path: /health?serializable=true
          port: 2381              # Separate health port
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 15
        failureThreshold: 8
      # etcd-specific: check if leader exists
      readinessProbe:
        exec:
          command:
            - /bin/sh
            - -c
            - |
              etcdctl endpoint health --cluster \
                --cacert=/etc/etcd/ca.crt \
                --cert=/etc/etcd/peer.crt \
                --key=/etc/etcd/peer.key
        periodSeconds: 30
        timeoutSeconds: 15

Detection & Monitoring

Monitoring control plane health separately from data plane health is essential for accurate incident classification and correct recovery prioritization.

# Comprehensive health check — classify failures correctly
echo "============================================="
echo "  CONTROL PLANE vs DATA PLANE HEALTH CHECK  "
echo "============================================="
echo ""

echo "=== CONTROL PLANE HEALTH ==="
echo "---"
# API Server responsiveness
echo -n "API Server: "
if kubectl get --raw='/healthz' 2>/dev/null | grep -q "ok"; then
    echo "HEALTHY"
else
    echo "UNHEALTHY — Cannot manage cluster"
fi

# etcd health
echo -n "etcd:       "
if kubectl get --raw='/healthz/etcd' 2>/dev/null | grep -q "ok"; then
    echo "HEALTHY"
else
    echo "UNHEALTHY — State store unavailable"
fi

# Scheduler
echo -n "Scheduler:  "
PENDING=$(kubectl get pods -A --field-selector=status.phase=Pending --no-headers 2>/dev/null | wc -l)
if [ "$PENDING" -lt 5 ]; then
    echo "HEALTHY (${PENDING} pending pods)"
else
    echo "DEGRADED (${PENDING} pending pods — possible scheduler issue)"
fi

# Controller Manager
echo -n "Controllers: "
if kubectl get lease -n kube-system kube-controller-manager -o jsonpath='{.spec.holderIdentity}' 2>/dev/null | grep -q "."; then
    echo "HEALTHY (leader elected)"
else
    echo "UNHEALTHY — No leader"
fi

echo ""
echo "=== DATA PLANE HEALTH ==="
echo "---"
# Node readiness
TOTAL_NODES=$(kubectl get nodes --no-headers 2>/dev/null | wc -l)
READY_NODES=$(kubectl get nodes --no-headers 2>/dev/null | grep " Ready" | wc -l)
echo "Nodes:    ${READY_NODES}/${TOTAL_NODES} Ready"

# Pod health
TOTAL_PODS=$(kubectl get pods -A --no-headers 2>/dev/null | wc -l)
RUNNING_PODS=$(kubectl get pods -A --no-headers 2>/dev/null | grep "Running" | wc -l)
echo "Pods:     ${RUNNING_PODS}/${TOTAL_PODS} Running"

# Service endpoints
echo -n "Endpoints: "
EMPTY_EP=$(kubectl get endpoints -A --no-headers 2>/dev/null | awk '{if($2=="") print}' | wc -l)
if [ "$EMPTY_EP" -lt 3 ]; then
    echo "HEALTHY (${EMPTY_EP} empty endpoint sets)"
else
    echo "DEGRADED (${EMPTY_EP} services with no backends)"
fi

# Network connectivity (sample pod-to-pod)
echo -n "Network:  "
if kubectl exec -n default deploy/healthcheck -- wget -qO- --timeout=5 http://kubernetes.default.svc 2>/dev/null | grep -q "."; then
    echo "HEALTHY (pod-to-service connectivity confirmed)"
else
    echo "UNKNOWN (no healthcheck pod available)"
fi

echo ""
echo "=== DIAGNOSIS ==="
if [ "$READY_NODES" -eq "$TOTAL_NODES" ] && [ "$RUNNING_PODS" -gt 0 ]; then
    echo "Data plane: HEALTHY"
else
    echo "Data plane: DEGRADED — IMMEDIATE ATTENTION REQUIRED"
fi

Recovery Priorities

The recovery priority decision depends on the failure mode:

flowchart TD
    START["Incident Detected"] --> Q1{"Data plane\nserving traffic?"}
    Q1 -->|"No"| DP["PRIORITY 1:\nRestore Data Plane"]
    Q1 -->|"Yes"| Q2{"Control plane\nhealthy?"}
    Q2 -->|"No"| Q3{"Control plane\nactively corrupting?"}
    Q3 -->|"Yes"| STOP["EMERGENCY:\nStop Control Plane\n(Prevent further damage)"]
    Q3 -->|"No (just down)"| CP["PRIORITY 2:\nRestore Control Plane\n(Self-healing disabled)"]
    Q2 -->|"Yes"| BOTH["Both healthy —\nInvestigate other causes"]
    DP --> AFTER["Then restore\ncontrol plane"]
    STOP --> ROLLBACK["Rollback to\nlast-known-good config"]
    ROLLBACK --> AFTER
                            

"""
Failure Mode Categorizer — Classify Infrastructure Failures
Determines whether a failure is control plane, data plane, or both,
and recommends recovery priority.
"""

class FailureModeCategorizer:
    """Categorize and prioritize infrastructure failures."""

    def __init__(self):
        self.control_plane_indicators = [
            "api_server_unreachable",
            "etcd_leader_lost",
            "scheduler_not_running",
            "controller_manager_down",
            "certificate_expired",
            "webhook_timeout",
            "admission_controller_failing",
        ]
        self.data_plane_indicators = [
            "pods_crashlooping",
            "nodes_not_ready",
            "network_unreachable",
            "service_5xx_errors",
            "database_connection_refused",
            "disk_full",
            "oom_killed",
        ]

    def categorize(self, symptoms):
        """Categorize failure based on observed symptoms."""
        cp_hits = [s for s in symptoms if s in self.control_plane_indicators]
        dp_hits = [s for s in symptoms if s in self.data_plane_indicators]

        if dp_hits and not cp_hits:
            return self._data_plane_failure(dp_hits)
        elif cp_hits and not dp_hits:
            return self._control_plane_failure(cp_hits)
        elif cp_hits and dp_hits:
            return self._combined_failure(cp_hits, dp_hits)
        else:
            return {"category": "unknown", "priority": "investigate"}

    def _control_plane_failure(self, indicators):
        return {
            "category": "CONTROL_PLANE",
            "severity": "HIGH",
            "priority": "P2 — System unmanaged but may be serving",
            "action": "Restore control plane; verify data plane stable",
            "time_pressure": "Hours (until certs expire or pods crash)",
            "indicators": indicators,
        }

    def _data_plane_failure(self, indicators):
        return {
            "category": "DATA_PLANE",
            "severity": "CRITICAL",
            "priority": "P1 — Immediate service impact",
            "action": "Restore data plane IMMEDIATELY",
            "time_pressure": "Minutes (active revenue loss)",
            "indicators": indicators,
        }

    def _combined_failure(self, cp_indicators, dp_indicators):
        return {
            "category": "COMBINED",
            "severity": "CRITICAL",
            "priority": "P1 — Restore data plane first, then control",
            "action": "1. Stabilize data plane, 2. Restore control plane",
            "time_pressure": "Minutes (no self-healing + no service)",
            "cp_indicators": cp_indicators,
            "dp_indicators": dp_indicators,
        }


# Example usage
categorizer = FailureModeCategorizer()

# Scenario 1: Control plane down, services still running
print("=== Scenario 1: API Server Unreachable ===")
result = categorizer.categorize(["api_server_unreachable", "etcd_leader_lost"])
for key, val in result.items():
    print(f"  {key}: {val}")

print("\n=== Scenario 2: Pods Crashing, Control Plane Fine ===")
result = categorizer.categorize(["pods_crashlooping", "service_5xx_errors"])
for key, val in result.items():
    print(f"  {key}: {val}")

print("\n=== Scenario 3: Everything Down ===")
result = categorizer.categorize([
    "api_server_unreachable", "controller_manager_down",
    "nodes_not_ready", "service_5xx_errors"
])
for key, val in result.items():
    print(f"  {key}: {val}")

The Meta-Level Understanding