Cilium Track Part 2: Hubble & L7 Observability

Hubble Architecture

Hubble is Cilium's observability layer that provides deep visibility into network flows at L3/L4 and L7. It consists of three components that work together to collect, aggregate, and visualize network data from eBPF programs.

• Hubble Server — Runs on each node as part of the Cilium agent, collecting flow events from eBPF maps
• Hubble Relay — Aggregates flows from all Hubble servers across the cluster into a single gRPC stream
• Hubble UI — Web-based service map visualization showing real-time traffic flows between services
• Hubble CLI — Command-line tool for querying flows with powerful filters

Enable Hubble

If Cilium is already installed, enable Hubble with the CLI. This deploys Hubble Relay and optionally the Hubble UI:

# Enable Hubble with the Cilium CLI
cilium hubble enable

# Enable Hubble with the UI component
cilium hubble enable --ui

# Verify Hubble is running
cilium hubble status
# Expected output:
# Healthcheck (via localhost:4245): Ok
# Current/Max Flows: 4095/4095 (100.0%)
# Flows/s: 42.3

# Port-forward Hubble Relay for CLI access
cilium hubble port-forward &

# Port-forward Hubble UI for web access
kubectl port-forward -n kube-system svc/hubble-ui 12000:80 &
# Access at http://localhost:12000

Flow Observation

The hubble observe command provides real-time network flow visibility with powerful filtering options. Flows include source/destination identity, verdict (forwarded/dropped), and protocol details.

# Observe all flows in real-time
hubble observe

# Filter by namespace
hubble observe --namespace production

# Filter by specific pod
hubble observe --to-pod production/backend-7d4f8b6c9-abc12

# Filter dropped traffic only (policy violations)
hubble observe --verdict DROPPED

# Filter by protocol and port
hubble observe --protocol TCP --port 8080

# Combine filters: dropped traffic to backend in production
hubble observe --namespace production \
  --to-label app=backend \
  --verdict DROPPED

# Output as JSON for processing
hubble observe --namespace production -o json | jq '.flow.source.labels'

# Follow mode with timestamps
hubble observe -f --since 5m --namespace production

L7 Protocol Visibility

Cilium can inspect L7 protocols when configured via CiliumNetworkPolicy with L7 rules. This enables HTTP path filtering, gRPC method filtering, DNS query logging, and Kafka topic access control.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: l7-http-visibility
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api-gateway
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/products.*"
              - method: "POST"
                path: "/api/v1/orders"
              - method: "GET"
                path: "/healthz"

DNS-based visibility and filtering:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: dns-visibility
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*.production.svc.cluster.local"
              - matchPattern: "api.external-service.com"

After applying L7 policies, observe the enriched flow data:

# Observe L7 HTTP flows with request details
hubble observe --protocol http --namespace production

# Example output includes HTTP method, path, and response code:
# TIMESTAMP    SOURCE         DESTINATION    TYPE    VERDICT   SUMMARY
# 12:34:56     frontend-xyz   api-gw-abc     l7      FORWARDED HTTP/1.1 GET /api/v1/products 200
# 12:34:57     frontend-xyz   api-gw-abc     l7      DROPPED   HTTP/1.1 DELETE /api/v1/users 403

# Filter by HTTP status code
hubble observe --http-status 500 --namespace production

# Filter by HTTP method
hubble observe --http-method POST --namespace production

Prometheus Metrics & Grafana

Hubble exports rich networking metrics to Prometheus via the hubble-metrics configuration. These metrics are derived directly from eBPF flow data and cover L3/L4/L7 dimensions.

# values.yaml for Cilium Helm chart with Hubble metrics
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
  metrics:
    enabled:
      - dns
      - drop
      - tcp
      - flow
      - icmp
      - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
    serviceMonitor:
      enabled: true
      labels:
        release: prometheus-stack
    dashboards:
      enabled: true
      namespace: monitoring

# Apply Hubble metrics configuration via Helm upgrade
helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  -f values.yaml

# Verify metrics are being exposed
kubectl port-forward -n kube-system ds/cilium 9962:9962
curl -s http://localhost:9962/metrics | grep hubble

# Key metrics available:
# hubble_flows_processed_total — total flows by verdict
# hubble_drop_total — dropped packets by reason
# hubble_dns_queries_total — DNS queries by rcode
# hubble_http_requests_total — HTTP requests by method, status
# hubble_tcp_connecttime_seconds — TCP connection latency

Exercises

Key Takeaways

• Hubble provides kernel-level network observability without sidecars or application changes
• Hubble Relay aggregates flows cluster-wide; Hubble UI shows real-time service maps
• L7 visibility requires CiliumNetworkPolicy with protocol-specific rules (HTTP paths, DNS patterns)
• hubble observe supports powerful filters: namespace, pod, verdict, protocol, HTTP method/status
• Hubble metrics integrate with Prometheus/Grafana for dashboards and alerting on network behavior