Part 15: Container Security Fundamentals

Why Container Security Matters

Containers revolutionised deployment by sharing the host kernel instead of virtualising hardware. This shared-kernel architecture delivers sub-second startup and near-native performance, but it fundamentally changes the security equation: every container on a host trusts the same kernel. A kernel exploit in one container compromises them all.

Unlike virtual machines where a hypervisor provides hardware-enforced isolation, container isolation relies entirely on kernel features — namespaces, cgroups, seccomp filters, and mandatory access controls. These are software boundaries, not hardware ones.

The Numbers Don't Lie

Industry data paints a concerning picture of container security in production:

The Container Threat Model

Understanding where attacks originate is the foundation of effective defence. Container threats span the entire lifecycle — from build time (malicious base images) through distribution (registry compromise) to runtime (privilege escalation).

Attack Surface Map

flowchart TB
    subgraph Build["Build Time Threats"]
        B1["Malicious base images"]
        B2["Vulnerable dependencies"]
        B3["Secrets in layers"]
        B4["Trojanised build tools"]
    end
    subgraph Distribution["Distribution Threats"]
        D1["Registry compromise"]
        D2["Man-in-the-middle"]
        D3["Tag mutation"]
        D4["Unsigned images"]
    end
    subgraph Runtime["Runtime Threats"]
        R1["Privilege escalation"]
        R2["Container breakout"]
        R3["Kernel exploits"]
        R4["Resource abuse (cryptomining)"]
    end
    subgraph Orchestration["Orchestration Threats"]
        O1["Misconfigured RBAC"]
        O2["Exposed API servers"]
        O3["Lateral movement"]
        O4["etcd compromise"]
    end

    Build --> Distribution --> Runtime --> Orchestration

    style Build fill:#fff5f5,stroke:#BF092F
    style Distribution fill:#f0f5ff,stroke:#16476A
    style Runtime fill:#fff5f5,stroke:#BF092F
    style Orchestration fill:#f0f9f9,stroke:#3B9797
                            

The most common attack patterns in production environments:

• Supply chain poisoning — Attacker compromises a popular base image or dependency; thousands of downstream images inherit the backdoor
• Privilege escalation — Container running as root uses kernel vulnerability to escape namespace isolation
• Cryptojacking — Attacker gains access to container, deploys cryptocurrency miners consuming compute resources
• Data exfiltration — Exploiting misconfigured volume mounts or network policies to access sensitive data
• Lateral movement — Using one compromised container as a pivot point to attack other services on the same network

The Shared Kernel Risk

This is the fundamental security trade-off of containers versus virtual machines:

flowchart TB
    subgraph VM["Virtual Machine Model"]
        VM1["App A"] --> VMK1["Guest Kernel A"]
        VM2["App B"] --> VMK2["Guest Kernel B"]
        VMK1 --> HV["Hypervisor (Hardware Boundary)"]
        VMK2 --> HV
        HV --> HW["Host Hardware"]
    end
    subgraph Container["Container Model"]
        C1["App A"] --> NS1["Namespace + cgroup"]
        C2["App B"] --> NS2["Namespace + cgroup"]
        NS1 --> SK["Shared Kernel (Software Boundary)"]
        NS2 --> SK
        SK --> HW2["Host Hardware"]
    end

    style VM fill:#f0f9f9,stroke:#3B9797
    style Container fill:#fff5f5,stroke:#BF092F
    style HV fill:#f0f9f9,stroke:#3B9797
    style SK fill:#fff5f5,stroke:#BF092F
                            

Notable container escape vulnerabilities in recent history:

Image Vulnerabilities

Container images inherit every vulnerability from their base image, installed packages, and application dependencies. The problem compounds through layers of inheritance:

# This innocent-looking Dockerfile inherits hundreds of CVEs
FROM ubuntu:22.04

# ubuntu:22.04 ships with ~120 packages, many with known CVEs
# Adding more packages adds more vulnerabilities
RUN apt-get update && apt-get install -y \
    curl \
    wget \
    python3 \
    nodejs \
    npm

# By this point, your image likely has 200+ known vulnerabilities
COPY app/ /app/
CMD ["python3", "/app/main.py"]

Why do base images have so many vulnerabilities?

• General-purpose images (ubuntu, debian) include hundreds of packages for compatibility — most of which your application never uses
• Tag mutability — ubuntu:latest changes over time; a scan result from last week may not reflect today's image
• Transitive dependencies — Installing one package pulls in dozens of dependencies, each with its own CVE history
• Stale images — Teams build once and deploy for months without rebuilding, accumulating newly-discovered CVEs

Let's see the actual vulnerability count of popular base images:

# Scan popular base images with Trivy
trivy image ubuntu:22.04
# Output: Total: 127 (UNKNOWN: 0, LOW: 82, MEDIUM: 33, HIGH: 10, CRITICAL: 2)

trivy image node:18
# Output: Total: 634 (UNKNOWN: 1, LOW: 398, MEDIUM: 156, HIGH: 67, CRITICAL: 12)

trivy image python:3.11
# Output: Total: 428 (UNKNOWN: 0, LOW: 267, MEDIUM: 108, HIGH: 43, CRITICAL: 10)

trivy image alpine:3.19
# Output: Total: 0 (no known vulnerabilities)

trivy image gcr.io/distroless/static-debian12
# Output: Total: 0 (no known vulnerabilities)

Vulnerability Scanning

Vulnerability scanners compare the packages in your image against databases of known CVEs (NVD, vendor advisories). They identify what's vulnerable, what the severity is, and whether a fix exists.

Scanning with Trivy

Trivy (by Aqua Security) is the most popular open-source container scanner — fast, accurate, and easy to integrate:

# Install Trivy
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin

# Scan a local image
trivy image myapp:latest

# Scan with severity filter (only HIGH and CRITICAL)
trivy image --severity HIGH,CRITICAL myapp:latest

# Scan and fail if vulnerabilities found (for CI/CD)
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan a Dockerfile (configuration scanning)
trivy config Dockerfile

# Output as JSON for programmatic processing
trivy image --format json --output results.json myapp:latest

# Scan with ignore file (known false positives)
trivy image --ignorefile .trivyignore myapp:latest

Understanding Trivy output:

$ trivy image nginx:1.25
2024-01-15T10:30:00.000Z  INFO  Vulnerability scanning...

nginx:1.25 (debian 12.4)
=========================
Total: 83 (UNKNOWN: 0, LOW: 52, MEDIUM: 21, HIGH: 8, CRITICAL: 2)

┌──────────────────┬────────────────┬──────────┬────────────────────────┬───────────────┬──────────────────────────────────────┐
│     Library      │ Vulnerability  │ Severity │   Installed Version    │ Fixed Version │                Title                 │
├──────────────────┼────────────────┼──────────┼────────────────────────┼───────────────┼──────────────────────────────────────┤
│ libssl3          │ CVE-2024-0727  │ CRITICAL │ 3.0.11-1~deb12u2       │ 3.0.13-1      │ openssl: denial of service via null  │
│ libcurl4         │ CVE-2023-46218 │ HIGH     │ 7.88.1-10+deb12u4      │ 7.88.1-10+5   │ curl: cookie injection with none ... │
│ zlib1g           │ CVE-2023-45853 │ MEDIUM   │ 1:1.2.13.dfsg-1        │               │ zlib: integer overflow in zipOpen2   │
└──────────────────┴────────────────┴──────────┴────────────────────────┴───────────────┴──────────────────────────────────────┘

Scanner Comparison

Minimal Base Images

The most effective way to reduce vulnerabilities is to reduce the attack surface. Fewer packages means fewer CVEs, less code for attackers to exploit, and smaller images (faster pulls, less storage).

# Example: Go application with scratch base (0 CVEs possible)
FROM golang:1.22 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /server .

# Final image: zero packages, zero shell, zero CVEs
FROM scratch
COPY --from=builder /server /server
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
EXPOSE 8080
ENTRYPOINT ["/server"]

# Example: Python application with distroless
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/install -r requirements.txt
COPY . .

FROM gcr.io/distroless/python3-debian12
COPY --from=builder /install /usr/local/lib/python3.11/site-packages
COPY --from=builder /app /app
WORKDIR /app
CMD ["main.py"]

Image Signing & Verification

How do you know the image you're pulling is the image that was built? Without cryptographic verification, you're trusting that the registry hasn't been compromised and that no one has tampered with the image in transit.

Image signing provides integrity (image hasn't been modified) and authenticity (image was built by a trusted party).

cosign (Sigstore)

cosign is the modern standard for container image signing, part of the Linux Foundation's Sigstore project:

# Install cosign
go install github.com/sigstore/cosign/v2/cmd/cosign@latest

# Generate a keypair (traditional key-based signing)
cosign generate-key-pair
# Creates: cosign.key (private), cosign.pub (public)

# Sign an image
cosign sign --key cosign.key myregistry.io/myapp:v1.0.0

# Verify an image
cosign verify --key cosign.pub myregistry.io/myapp:v1.0.0

# Keyless signing (uses OIDC identity — no key management!)
cosign sign myregistry.io/myapp:v1.0.0
# Opens browser for OIDC authentication (GitHub, Google, Microsoft)
# Signature includes your verified identity

# Verify keyless signature
cosign verify \
  --certificate-identity=developer@company.com \
  --certificate-oidc-issuer=https://accounts.google.com \
  myregistry.io/myapp:v1.0.0

Docker Content Trust (DCT)

# Enable Docker Content Trust globally
export DOCKER_CONTENT_TRUST=1

# Now all push/pull operations require signatures
docker push myregistry.io/myapp:v1.0.0
# Automatically signs on push

docker pull myregistry.io/myapp:v1.0.0
# Fails if image is unsigned

# Disable temporarily for specific operations
DOCKER_CONTENT_TRUST=0 docker pull unsigned-image:latest

# Verify an official Kubernetes image
cosign verify \
  --certificate-identity-regexp='.*@kubernetes.io' \
  --certificate-oidc-issuer='https://accounts.google.com' \
  registry.k8s.io/kube-apiserver:v1.29.0

Dockerfile Security Best Practices

A secure Dockerfile follows the principle of least privilege at every layer. Here's a production-grade secure Dockerfile template:

# ============================================================
# SECURE DOCKERFILE TEMPLATE
# ============================================================

# 1. Pin specific versions (never use :latest)
FROM node:20.11.0-alpine3.19 AS builder

# 2. Set working directory
WORKDIR /app

# 3. Copy dependency files first (layer caching)
COPY package.json package-lock.json ./

# 4. Install production dependencies only
RUN npm ci --only=production && npm cache clean --force

# 5. Copy application code
COPY --chown=node:node src/ ./src/

# ============================================================
# Production stage — minimal runtime image
# ============================================================
FROM node:20.11.0-alpine3.19

# 6. Install security updates
RUN apk update && apk upgrade --no-cache && rm -rf /var/cache/apk/*

# 7. Create non-root user (never run as root)
RUN addgroup -g 1001 appgroup && \
    adduser -u 1001 -G appgroup -D appuser

# 8. Set working directory
WORKDIR /app

# 9. Copy from builder with correct ownership
COPY --from=builder --chown=appuser:appgroup /app .

# 10. Switch to non-root user
USER appuser

# 11. Expose only needed ports
EXPOSE 3000

# 12. Use exec form for proper signal handling
ENTRYPOINT ["node"]
CMD ["src/server.js"]

# 13. Add health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1

Key rules to follow:

Registry Security

The registry is the distribution point for your images — compromising it means every deployment pulls malicious code. Registry security covers authentication, transport encryption, content trust, and access policies.

# Pin images by digest (immutable reference)
# Instead of: FROM nginx:1.25
FROM nginx@sha256:6a59a71c54a40d74f5e9a3f7b2e3d3c4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0

# Set up a private registry with TLS
docker run -d \
  --name registry \
  -p 5000:5000 \
  -v /certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  -e REGISTRY_AUTH=htpasswd \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
  registry:2

# Enable content trust for registry pulls
export DOCKER_CONTENT_TRUST=1
export DOCKER_CONTENT_TRUST_SERVER=https://notary.example.com

Container Security Scanning in CI/CD

Security scanning must happen before images reach production. Integrating scanners into CI/CD pipelines creates automated gates that reject vulnerable images.

GitHub Actions with Trivy

# .github/workflows/container-security.yml
name: Container Security Scan

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  security-scan:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'

      - name: Upload Trivy scan results to GitHub Security
        uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: 'trivy-results.sarif'

      - name: Run Trivy config scanner (Dockerfile)
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'config'
          scan-ref: '.'
          exit-code: '1'

GitLab CI Integration

# .gitlab-ci.yml
container_scanning:
  stage: test
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  variables:
    FULL_IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  script:
    - trivy image --exit-code 0 --severity LOW,MEDIUM $FULL_IMAGE_NAME
    - trivy image --exit-code 1 --severity HIGH,CRITICAL $FULL_IMAGE_NAME
  artifacts:
    reports:
      container_scanning: gl-container-scanning-report.json
  allow_failure: false

Exercises

Conclusion & Next Steps

Container security begins with understanding the threat model: shared kernels, vulnerable images, and supply chain risks. The defences layer upon each other:

• Minimal images reduce attack surface (fewer packages = fewer CVEs)
• Vulnerability scanning identifies known issues before deployment
• Image signing ensures integrity and authenticity through the supply chain
• Secure Dockerfiles follow least-privilege principles (non-root, pinned versions, no secrets)
• CI/CD gates automate enforcement so security doesn't depend on human diligence

But image security is only half the story. Once containers are running, they need runtime protection — kernel hardening, capability dropping, and anomaly detection.