Part 17: Service Mesh & Advanced Networking

Why Service Mesh

In a monolithic application, function calls between modules happen in-process — zero network latency, no serialization, no authentication between components. When you decompose that monolith into 50, 100, or 500 microservices, every inter-module call becomes a network request with all the failure modes that implies: latency, timeouts, partial failures, retries, authentication, encryption, and observability.

A service mesh is a dedicated infrastructure layer that handles service-to-service communication, making the network reliable, secure, and observable without requiring changes to application code. It extracts cross-cutting concerns from individual services into a shared infrastructure.

When You Need a Service Mesh (and When You Don't)

flowchart LR
    subgraph Monolith["Monolith (In-Process)"]
        M1[Module A] --> M2[Module B]
        M2 --> M3[Module C]
        M3 --> M1
    end

    subgraph Microservices["Microservices (Network)"]
        S1[Service A] -->|"HTTP/gRPC
Auth + TLS
Retry + Timeout"| S2[Service B]
        S2 -->|"HTTP/gRPC
Auth + TLS
Circuit Break"| S3[Service C]
        S3 -->|"HTTP/gRPC
Auth + TLS
Load Balance"| S1
        S1 -->|"Events"| S4[Service D]
        S2 -->|"gRPC"| S4
        S4 -->|"HTTP"| S3
    end

    Monolith -.->|"Decompose"| Microservices
                            

Service Mesh Architecture

Every service mesh follows a common architectural pattern: a data plane that intercepts and manages traffic, and a control plane that configures and coordinates the data plane.

Data Plane

The data plane consists of lightweight proxy servers (sidecars) deployed alongside every service instance. These proxies intercept all inbound and outbound network traffic, applying policies for routing, load balancing, authentication, and observability without the application's knowledge.

Control Plane

The control plane is the brain of the mesh. It translates high-level routing rules, security policies, and configuration into proxy-specific configuration, then distributes that configuration to all data plane proxies via APIs (typically xDS in Envoy-based meshes).

flowchart TB
    subgraph CP["Control Plane"]
        direction LR
        CP1[Config API
VirtualService, DestinationRule]
        CP2[Certificate Authority
mTLS Certificates]
        CP3[Service Discovery
Endpoints Registry]
        CP4[Policy Engine
AuthorizationPolicy]
    end

    subgraph DP["Data Plane"]
        subgraph Pod1["Pod: Service A"]
            A[App Container] <--> PA[Sidecar Proxy]
        end
        subgraph Pod2["Pod: Service B"]
            B[App Container] <--> PB[Sidecar Proxy]
        end
        subgraph Pod3["Pod: Service C"]
            C[App Container] <--> PC[Sidecar Proxy]
        end
    end

    CP1 -->|xDS Config| PA
    CP1 -->|xDS Config| PB
    CP1 -->|xDS Config| PC
    CP2 -->|Certificates| PA
    CP2 -->|Certificates| PB
    CP2 -->|Certificates| PC

    PA <-->|"mTLS"| PB
    PB <-->|"mTLS"| PC
    PA <-->|"mTLS"| PC
                            

Sidecar Injection Approaches

# Enable automatic sidecar injection for a namespace
kubectl label namespace production istio-injection=enabled

# Verify injection is enabled
kubectl get namespace production --show-labels

# Check sidecar status for a pod
kubectl get pods -n production -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'

# Disable injection for a specific pod (annotation)
kubectl patch deployment my-app -n production -p '
{
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "sidecar.istio.io/inject": "false"
        }
      }
    }
  }
}'

Envoy Proxy

Envoy is the foundational building block of most modern service meshes. Originally built at Lyft and now a CNCF graduated project, Envoy is a high-performance L4/L7 proxy designed for large-scale microservice architectures. Istio, AWS App Mesh, and Consul Connect all use Envoy as their data plane proxy.

Envoy Configuration

# Static Envoy configuration example
# envoy.yaml - standalone proxy for a service
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/api/v1"
                          route:
                            cluster: api_service
                            timeout: 30s
                            retry_policy:
                              retry_on: "5xx,reset,connect-failure"
                              num_retries: 3
                              per_try_timeout: 10s
                        - match:
                            prefix: "/api/v2"
                          route:
                            cluster: api_v2_service
                            timeout: 30s
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: api_service
      type: STRICT_DNS
      connect_timeout: 5s
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: api_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: api-service.production.svc.cluster.local
                      port_value: 8080
      circuit_breakers:
        thresholds:
          - priority: DEFAULT
            max_connections: 1024
            max_pending_requests: 1024
            max_requests: 1024
            max_retries: 3
      outlier_detection:
        consecutive_5xx: 5
        interval: 10s
        base_ejection_time: 30s
        max_ejection_percent: 50

    - name: api_v2_service
      type: STRICT_DNS
      connect_timeout: 5s
      lb_policy: LEAST_REQUEST
      load_assignment:
        cluster_name: api_v2_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: api-v2-service.production.svc.cluster.local
                      port_value: 8080

Load Balancing Algorithms

# Envoy rate limiting configuration
static_resources:
  listeners:
    - name: rate_limited_listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: api
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: backend
                      rate_limits:
                        - actions:
                            - request_headers:
                                header_name: "x-api-key"
                                descriptor_key: "api_key"
                http_filters:
                  - name: envoy.filters.http.ratelimit
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
                      domain: production
                      rate_limit_service:
                        grpc_service:
                          envoy_grpc:
                            cluster_name: rate_limit_cluster
                        transport_api_version: V3
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Istio

Istio is the most feature-rich and widely adopted service mesh. Its control plane (istiod) unifies what were formerly separate components — Pilot (traffic management), Citadel (certificate authority), and Galley (configuration validation) — into a single binary that manages the entire mesh.

Installation

# Install Istio with istioctl (recommended)
curl -L https://istio.io/downloadIstio | sh -
cd istio-1.22.0
export PATH=$PWD/bin:$PATH

# Install with the "demo" profile (all features enabled)
istioctl install --set profile=demo -y

# Verify installation
istioctl verify-install
kubectl get pods -n istio-system

# Production profile (minimal, hardened)
istioctl install --set profile=minimal \
  --set meshConfig.accessLogFile=/dev/stdout \
  --set meshConfig.enableTracing=true \
  --set values.global.proxy.resources.requests.cpu=100m \
  --set values.global.proxy.resources.requests.memory=128Mi

# Enable sidecar injection for workload namespace
kubectl label namespace production istio-injection=enabled --overwrite

Istio Custom Resource Definitions (CRDs)

# VirtualService: route traffic to different versions
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-routing
  namespace: production
spec:
  hosts:
    - reviews.production.svc.cluster.local
  http:
    # Route 90% to v1, 10% to v2 (canary)
    - match:
        - headers:
            x-canary-user:
              exact: "true"
      route:
        - destination:
            host: reviews.production.svc.cluster.local
            subset: v2
    - route:
        - destination:
            host: reviews.production.svc.cluster.local
            subset: v1
          weight: 90
        - destination:
            host: reviews.production.svc.cluster.local
            subset: v2
          weight: 10
      retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: "5xx,reset,connect-failure,retriable-4xx"
      timeout: 10s

# DestinationRule: define subsets and policies
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews-destination
  namespace: production
spec:
  host: reviews.production.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: UPGRADE
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
    loadBalancer:
      simple: LEAST_REQUEST
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2
      trafficPolicy:
        connectionPool:
          http:
            http2MaxRequests: 500

# Gateway: configure ingress traffic
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: production-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: production-tls-cert
      hosts:
        - "api.example.com"
        - "app.example.com"
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*.example.com"
      tls:
        httpsRedirect: true
---
# VirtualService binding to the Gateway
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-ingress
  namespace: production
spec:
  hosts:
    - "api.example.com"
  gateways:
    - istio-system/production-gateway
  http:
    - match:
        - uri:
            prefix: /v1/
      route:
        - destination:
            host: api-v1.production.svc.cluster.local
            port:
              number: 8080
    - match:
        - uri:
            prefix: /v2/
      route:
        - destination:
            host: api-v2.production.svc.cluster.local
            port:
              number: 8080

# AuthorizationPolicy: access control
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: api-access-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-server
  action: ALLOW
  rules:
    # Allow frontend to call API
    - from:
        - source:
            principals: ["cluster.local/ns/production/sa/frontend"]
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/v1/*"]
    # Allow monitoring to call health endpoints
    - from:
        - source:
            namespaces: ["monitoring"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/health", "/metrics"]
    # Deny everything else (implicit deny when ALLOW rules exist)

# PeerAuthentication: enforce mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: production
spec:
  mtls:
    mode: STRICT
---
# Permissive mode for migration (allows both plaintext and mTLS)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: permissive-mtls
  namespace: legacy-services
spec:
  mtls:
    mode: PERMISSIVE

sequenceDiagram
    participant Client
    participant IngressGW as Istio Ingress Gateway
    participant ProxyA as Sidecar Proxy (Service A)
    participant AppA as Service A
    participant ProxyB as Sidecar Proxy (Service B)
    participant AppB as Service B

    Client->>IngressGW: HTTPS Request
    Note over IngressGW: TLS termination
Gateway rules applied
    IngressGW->>ProxyA: Route via VirtualService
    Note over ProxyA: mTLS established
AuthorizationPolicy checked
    ProxyA->>AppA: Plain HTTP (localhost)
    AppA->>ProxyA: Outbound call to Service B
    Note over ProxyA: DestinationRule applied
Load balancing, circuit breaking
    ProxyA->>ProxyB: mTLS encrypted
    Note over ProxyB: AuthorizationPolicy checked
Rate limiting applied
    ProxyB->>AppB: Plain HTTP (localhost)
    AppB-->>ProxyB: Response
    ProxyB-->>ProxyA: mTLS encrypted response
    ProxyA-->>AppA: Response
    AppA-->>ProxyA: Final response
    ProxyA-->>IngressGW: Response
    IngressGW-->>Client: HTTPS Response
                            

Linkerd

Linkerd takes a fundamentally different philosophy from Istio: simplicity over features. Where Istio provides a comprehensive (and complex) feature set, Linkerd focuses on doing the 80% use case extremely well with minimal operational overhead. Its data plane proxy (linkerd2-proxy) is written in Rust for memory safety and performance.

# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh
export PATH=$HOME/.linkerd2/bin:$PATH

# Pre-flight checks
linkerd check --pre

# Install Linkerd control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Verify installation
linkerd check

# Inject sidecars into a namespace
kubectl get deploy -n production -o yaml | linkerd inject - | kubectl apply -f -

# Or annotate namespace for auto-injection
kubectl annotate namespace production linkerd.io/inject=enabled

# Linkerd TrafficSplit (SMI spec) for canary deployment
apiVersion: split.smi-spec.io/v1alpha4
kind: TrafficSplit
metadata:
  name: reviews-canary
  namespace: production
spec:
  service: reviews
  backends:
    - service: reviews-v1
      weight: 900
    - service: reviews-v2
      weight: 100
---
# Linkerd ServiceProfile for per-route metrics and retries
apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: reviews.production.svc.cluster.local
  namespace: production
spec:
  routes:
    - name: GET /api/reviews
      condition:
        method: GET
        pathRegex: /api/reviews(/.*)?
      responseClasses:
        - condition:
            status:
              min: 500
              max: 599
          isFailure: true
      timeout: 5s
      isRetryable: true
    - name: POST /api/reviews
      condition:
        method: POST
        pathRegex: /api/reviews
      timeout: 10s

Service Mesh Comparison

mTLS & Zero-Trust Networking

In traditional networking, services inside a network perimeter are implicitly trusted. Zero-trust networking eliminates this assumption: every request must be authenticated and authorized, regardless of network location. Mutual TLS (mTLS) is the foundation — both client and server present certificates to prove identity.

SPIFFE/SPIRE Identity Framework

# SPIFFE ID format: spiffe://trust-domain/path
# Examples:
# spiffe://cluster.local/ns/production/sa/api-server
# spiffe://cluster.local/ns/production/sa/frontend

# Istio automatically assigns SPIFFE IDs based on Kubernetes ServiceAccount
# Check the identity of a pod's sidecar:
# istioctl proxy-config secret  -n production

# SPIRE server configuration (alternative to Istio CA)
apiVersion: v1
kind: ConfigMap
metadata:
  name: spire-server
  namespace: spire
data:
  server.conf: |
    server {
      bind_address = "0.0.0.0"
      bind_port = "8081"
      trust_domain = "example.org"
      data_dir = "/run/spire/data"
      log_level = "INFO"
      ca_ttl = "168h"
      default_x509_svid_ttl = "1h"

      ca_subject {
        country = ["US"]
        organization = ["Example Corp"]
      }
    }

    plugins {
      DataStore "sql" {
        plugin_data {
          database_type = "sqlite3"
          connection_string = "/run/spire/data/datastore.sqlite3"
        }
      }
      NodeAttestor "k8s_psat" {
        plugin_data {
          clusters = {
            "production" = {
              service_account_allow_list = ["spire:spire-agent"]
            }
          }
        }
      }
      KeyManager "disk" {
        plugin_data {
          keys_path = "/run/spire/data/keys.json"
        }
      }
    }

Kubernetes Network Policies

# Defense in depth: Network Policies + Service Mesh mTLS
# Layer 1: Kubernetes NetworkPolicy (L3/L4)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Allow traffic from frontend pods
    - from:
        - podSelector:
            matchLabels:
              app: frontend
        - namespaceSelector:
            matchLabels:
              name: production
      ports:
        - protocol: TCP
          port: 8080
    # Allow Prometheus scraping
    - from:
        - namespaceSelector:
            matchLabels:
              name: monitoring
      ports:
        - protocol: TCP
          port: 9090
  egress:
    # Allow DNS
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
    # Allow database access
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
---
# Layer 2: Istio AuthorizationPolicy (L7 - identity-based)
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: api-server-authz
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-server
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/production/sa/frontend"]
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/*"]
      when:
        - key: request.headers[x-request-id]
          notValues: [""]

Traffic Management

Traffic management is arguably the killer feature of service meshes. Without a mesh, implementing canary deployments, A/B testing, or fault injection requires custom load balancer configuration, application-level routing code, or complex Kubernetes configurations. With a mesh, these become declarative YAML configurations.

Canary Deployments with Traffic Splitting

flowchart LR
    subgraph Phase1["Phase 1: Initial"]
        LB1[Mesh Router] -->|"100%"| V1A[v1 Pods]
    end

    subgraph Phase2["Phase 2: Canary Start"]
        LB2[Mesh Router] -->|"90%"| V1B[v1 Pods]
        LB2 -->|"10%"| V2A[v2 Pods
1 replica]
    end

    subgraph Phase3["Phase 3: Expand"]
        LB3[Mesh Router] -->|"50%"| V1C[v1 Pods]
        LB3 -->|"50%"| V2B[v2 Pods
3 replicas]
    end

    subgraph Phase4["Phase 4: Complete"]
        LB4[Mesh Router] -->|"100%"| V2C[v2 Pods]
    end

    Phase1 -->|"Deploy v2"| Phase2
    Phase2 -->|"Metrics OK"| Phase3
    Phase3 -->|"Promote"| Phase4
                            

# Progressive canary deployment with Istio
# Step 1: Deploy v2 alongside v1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v2
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v2
  template:
    metadata:
      labels:
        app: reviews
        version: v2
    spec:
      containers:
        - name: reviews
          image: reviews:2.0.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
---
# Step 2: Route 10% to canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-canary
  namespace: production
spec:
  hosts:
    - reviews
  http:
    - route:
        - destination:
            host: reviews
            subset: v1
          weight: 90
        - destination:
            host: reviews
            subset: v2
          weight: 10
---
# Step 3: Increase to 50% after validation
# kubectl apply with weight: 50/50
---
# Step 4: Full rollout - 100% to v2
# kubectl apply with weight: 0/100, then delete v1 deployment

Fault Injection & Chaos Testing

# Fault injection for resilience testing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-fault-injection
  namespace: production
spec:
  hosts:
    - reviews
  http:
    # Inject 5-second delay for 10% of requests
    - fault:
        delay:
          percentage:
            value: 10.0
          fixedDelay: 5s
      route:
        - destination:
            host: reviews
            subset: v1
---
# Inject HTTP 503 errors for 5% of requests
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments-chaos
  namespace: production
spec:
  hosts:
    - payments
  http:
    - fault:
        abort:
          percentage:
            value: 5.0
          httpStatus: 503
      route:
        - destination:
            host: payments
            subset: v1
---
# Header-based A/B testing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: frontend-ab-test
  namespace: production
spec:
  hosts:
    - frontend
  http:
    # Users with experiment header get v2
    - match:
        - headers:
            x-experiment:
              exact: "new-checkout"
      route:
        - destination:
            host: frontend
            subset: v2
    # Everyone else gets v1
    - route:
        - destination:
            host: frontend
            subset: v1

# Circuit breaking with Istio DestinationRule
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payments-circuit-breaker
  namespace: production
spec:
  host: payments
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
        connectTimeout: 5s
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
        maxRetries: 3
    outlierDetection:
      # Eject host after 5 consecutive 5xx errors
      consecutive5xxErrors: 5
      # Check every 10 seconds
      interval: 10s
      # Eject for at least 30 seconds
      baseEjectionTime: 30s
      # Never eject more than 50% of hosts
      maxEjectionPercent: 50
      # Also eject on gateway errors (502, 503, 504)
      consecutiveGatewayErrors: 3

Observability with Service Mesh

One of the most immediate benefits of a service mesh is automatic observability. Without writing a single line of instrumentation code, the mesh provides: request rate, error rate, and latency (the RED metrics) for every service-to-service call, distributed traces that show the full request path, and access logs for forensic analysis.

flowchart TB
    subgraph DataPlane["Data Plane (Sidecar Proxies)"]
        P1[Proxy A] -->|"Metrics"| M[Prometheus]
        P2[Proxy B] -->|"Metrics"| M
        P3[Proxy C] -->|"Metrics"| M
        P1 -->|"Traces"| T[Jaeger/Zipkin]
        P2 -->|"Traces"| T
        P3 -->|"Traces"| T
        P1 -->|"Access Logs"| L[Loki/EFK]
        P2 -->|"Access Logs"| L
        P3 -->|"Access Logs"| L
    end

    subgraph Visualization["Visualization Layer"]
        M --> G[Grafana Dashboards]
        T --> G
        L --> G
        M --> K[Kiali Service Graph]
        T --> K
    end

    subgraph Alerting["Alerting"]
        M --> A[Alertmanager]
        A --> S[Slack/PagerDuty]
    end
                            

Kiali & Grafana Integration

# Install Kiali for Istio visualization
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/addons/jaeger.yaml

# Access Kiali dashboard
istioctl dashboard kiali

# Access Grafana dashboards
istioctl dashboard grafana

# Check mesh-wide metrics via Prometheus
kubectl port-forward svc/prometheus -n istio-system 9090:9090

# Query: Request rate per service
# rate(istio_requests_total{reporter="destination"}[5m])

# Query: P99 latency per service
# histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket{reporter="destination"}[5m])) by (le, destination_service_name))

# Query: Error rate per service
# sum(rate(istio_requests_total{reporter="destination",response_code=~"5.."}[5m])) by (destination_service_name) / sum(rate(istio_requests_total{reporter="destination"}[5m])) by (destination_service_name)

# Istio telemetry configuration for enhanced observability
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-observability
  namespace: istio-system
spec:
  # Enable access logging for all services
  accessLogging:
    - providers:
        - name: envoy
      filter:
        expression: "response.code >= 400"
  # Configure tracing sampling rate
  tracing:
    - randomSamplingPercentage: 10.0
      providers:
        - name: zipkin
      customTags:
        environment:
          literal:
            value: "production"
        cluster:
          literal:
            value: "us-east-1"
  # Metrics customization
  metrics:
    - providers:
        - name: prometheus
      overrides:
        - match:
            metric: REQUEST_COUNT
            mode: CLIENT_AND_SERVER
          tagOverrides:
            request_host:
              operation: UPSERT
              value: "request.host"

API Gateways & Ingress

The boundary between API gateways, ingress controllers, and service meshes is increasingly blurry. Understanding where each technology fits is essential for designing a coherent networking architecture.

Kubernetes Gateway API

# Kubernetes Gateway API resources
# GatewayClass: provided by infrastructure team
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: istio
spec:
  controllerName: istio.io/gateway-controller
---
# Gateway: managed by cluster operators
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: istio-system
spec:
  gatewayClassName: istio
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: production-tls
            namespace: istio-system
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: "true"
---
# HTTPRoute: managed by application developers
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-routes
  namespace: production
spec:
  parentRefs:
    - name: production-gateway
      namespace: istio-system
  hostnames:
    - "api.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/users
      backendRefs:
        - name: users-service
          port: 8080
          weight: 90
        - name: users-service-v2
          port: 8080
          weight: 10
    - matches:
        - path:
            type: PathPrefix
            value: /v1/orders
      backendRefs:
        - name: orders-service
          port: 8080
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: x-request-source
                value: gateway

# NGINX Ingress Controller with annotations
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/limit-rps: "100"
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls-cert
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 8080

Advanced Patterns

Multi-Cluster Mesh

Running a service mesh across multiple Kubernetes clusters enables cross-cluster service discovery, failover, and load balancing. Istio supports two primary multi-cluster topologies:

# Istio multi-cluster setup (primary-remote model)
# Cluster 1 (Primary): runs istiod control plane
# Cluster 2 (Remote): uses Cluster 1's control plane

# On primary cluster: expose istiod externally
cat 
<< EOF | kubectl apply -f - --context=cluster1
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istiod-gateway
  namespace: istio-system
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        number: 15012
        name: tls-istiod
        protocol: TLS
      tls:
        mode: PASSTHROUGH
      hosts:
        - "*.global"
EOF

# Create remote secret for cluster2
istioctl create-remote-secret \
  --context=cluster2 \
  --name=cluster2 | kubectl apply -f - --context=cluster1

# Install Istio on remote cluster pointing to primary
istioctl install --context=cluster2 \
  --set profile=remote \
  --set values.istiod.enabled=false \
  --set values.global.remotePilotAddress=istiod.cluster1.example.com

# Verify cross-cluster connectivity
istioctl remote-clusters --context=cluster1

Ambient Mesh & eBPF

# Install Istio Ambient Mesh
istioctl install --set profile=ambient -y

# Add namespace to ambient mesh (no sidecars!)
kubectl label namespace production istio.io/dataplane-mode=ambient

# Verify ztunnel pods are running (one per node)
kubectl get pods -n istio-system -l app=ztunnel

# Deploy a waypoint proxy for L7 features on specific service
istioctl waypoint apply --namespace production --name reviews-waypoint

# Bind the waypoint to a service
kubectl label service reviews -n production \
  istio.io/use-waypoint=reviews-waypoint

# Cilium Service Mesh (eBPF-based, no sidecar)
# Install Cilium with service mesh features
# helm install cilium cilium/cilium --namespace kube-system \
#   --set kubeProxyReplacement=true \
#   --set envoy.enabled=true \
#   --set ingressController.enabled=true

# CiliumEnvoyConfig for L7 policies
apiVersion: cilium.io/v2
kind: CiliumEnvoyConfig
metadata:
  name: reviews-l7-policy
  namespace: production
spec:
  services:
    - name: reviews
      namespace: production
  backendServices:
    - name: reviews
      namespace: production
  resources:
    - "@type": type.googleapis.com/envoy.config.listener.v3.Listener
      name: reviews-listener
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: reviews
                route_config:
                  virtual_hosts:
                    - name: reviews
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: "production/reviews"
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Hands-On Exercises

# Prerequisites: kind or minikube cluster running
# Step 1: Install Istio
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.22.0 sh -
cd istio-1.22.0
export PATH=$PWD/bin:$PATH
istioctl install --set profile=demo -y

# Step 2: Enable sidecar injection
kubectl create namespace bookinfo
kubectl label namespace bookinfo istio-injection=enabled

# Step 3: Deploy Bookinfo sample application
kubectl apply -n bookinfo -f samples/bookinfo/platform/kube/bookinfo.yaml

# Step 4: Wait for pods to be ready (should have 2/2 containers)
kubectl get pods -n bookinfo -w

# Step 5: Expose via Istio ingress gateway
kubectl apply -n bookinfo -f samples/bookinfo/networking/bookinfo-gateway.yaml

# Step 6: Get the ingress gateway URL
export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway \
  -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
echo "http://$INGRESS_HOST:$INGRESS_PORT/productpage"

# Step 7: Verify mesh connectivity
istioctl analyze -n bookinfo
istioctl proxy-status

# Step 1: Create DestinationRule with subsets
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews
  namespace: bookinfo
spec:
  host: reviews
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2
    - name: v3
      labels:
        version: v3
---
# Step 2: Start with 100% to v1
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
  namespace: bookinfo
spec:
  hosts:
    - reviews
  http:
    - route:
        - destination:
            host: reviews
            subset: v1
          weight: 100

# Step 3: Shift 10% to v2
kubectl apply -f - << EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
  namespace: bookinfo
spec:
  hosts:
    - reviews
  http:
    - route:
        - destination:
            host: reviews
            subset: v1
          weight: 90
        - destination:
            host: reviews
            subset: v2
          weight: 10
EOF

# Step 4: Generate traffic and observe in Kiali
for i in $(seq 1 100); do
  curl -s -o /dev/null "http://$INGRESS_HOST:$INGRESS_PORT/productpage"
done

# Step 5: Check metrics - verify ~10% going to v2
istioctl dashboard kiali

# Step 6: Progressive increase (50/50, then 100% v2)
# Monitor error rate before each increase

# Step 1: Enable strict mTLS mesh-wide
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
---
# Step 2: Create deny-all policy for bookinfo namespace
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: bookinfo
spec:
  {}
---
# Step 3: Allow productpage to call reviews and details
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-productpage-to-reviews
  namespace: bookinfo
spec:
  selector:
    matchLabels:
      app: reviews
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/bookinfo/sa/bookinfo-productpage"]
      to:
        - operation:
            methods: ["GET"]
---
# Step 4: Allow reviews to call ratings
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-reviews-to-ratings
  namespace: bookinfo
spec:
  selector:
    matchLabels:
      app: ratings
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/bookinfo/sa/bookinfo-reviews"]
      to:
        - operation:
            methods: ["GET"]

# Step 5: Verify mTLS is active
istioctl authn tls-check reviews.bookinfo.svc.cluster.local

# Step 6: Test unauthorized access (should fail with RBAC denied)
kubectl exec -n bookinfo deploy/ratings -c ratings -- \
  curl -s http://reviews:9080/reviews/1
# Expected: RBAC: access denied

# Step 7: Test authorized path (should succeed)
kubectl exec -n bookinfo deploy/productpage -c productpage -- \
  curl -s http://reviews:9080/reviews/1
# Expected: 200 OK with review data

# Step 1: Deploy observability addons
kubectl apply -f samples/addons/prometheus.yaml
kubectl apply -f samples/addons/grafana.yaml
kubectl apply -f samples/addons/kiali.yaml
kubectl apply -f samples/addons/jaeger.yaml

# Wait for all pods to be ready
kubectl wait --for=condition=Ready pods --all -n istio-system --timeout=300s

# Step 2: Generate sustained traffic for metrics
while true; do
  curl -s -o /dev/null "http://$INGRESS_HOST:$INGRESS_PORT/productpage"
  sleep 0.5
done &

# Step 3: Open Kiali - observe service graph
istioctl dashboard kiali
# Navigate: Graph > bookinfo namespace > Versioned app graph
# Verify: traffic flow, response times, success rates visible

# Step 4: Open Grafana - explore Istio dashboards
istioctl dashboard grafana
# Navigate: Dashboards > Istio > Istio Mesh Dashboard
# Verify: global request volume, success rate, 4xx/5xx errors

# Step 5: Open Jaeger - distributed traces
istioctl dashboard jaeger
# Navigate: Search > Service: productpage > Find Traces
# Verify: full trace from productpage -> reviews -> ratings

# Step 6: Custom Prometheus queries
kubectl port-forward svc/prometheus -n istio-system 9090:9090
# Query 1: Request rate per service
# rate(istio_requests_total{reporter="destination",namespace="bookinfo"}[1m])
# Query 2: P95 latency
# histogram_quantile(0.95, sum(rate(istio_request_duration_milliseconds_bucket{namespace="bookinfo"}[5m])) by (le, destination_service_name))

Conclusion & Next Steps

A service mesh transforms microservices networking from a per-service responsibility into a platform capability. By extracting traffic management, security, and observability into infrastructure, teams can focus on business logic while platform engineers ensure consistent, secure communication across the entire fleet.

Key takeaways:

• Evaluate need carefully — Service meshes add complexity; ensure you have enough services and operational maturity to justify the investment
• Envoy is the foundation — Understanding Envoy's listener-route-cluster-endpoint model helps debug any Envoy-based mesh
• Istio for features, Linkerd for simplicity — Choose based on your team's operational capacity and feature requirements
• mTLS should be non-negotiable — In any multi-service architecture, encrypt and authenticate all internal traffic
• Traffic management enables safe deployments — Canary releases, fault injection, and circuit breaking reduce deployment risk dramatically
• Observability is immediate value — Automatic RED metrics and distributed tracing justify mesh adoption even before traffic management
• Gateway API is the future — Invest in Gateway API over legacy Ingress for new deployments
• Watch ambient mesh and eBPF — The sidecar model is being challenged by lighter-weight approaches that reduce overhead