Part 17: Supply Chain Security & Secrets

The Supply Chain Problem

Modern containers don't run code you wrote — they run code your dependencies wrote. A typical production container image contains thousands of components from hundreds of authors. Each is a potential attack vector.

The Dependency Iceberg

flowchart TB
    A["Your Application Code
(~5% of image)"] --> B["Direct Dependencies
(npm, pip, go modules)"]
    B --> C["Transitive Dependencies
(deps of your deps)"]
    C --> D["System Libraries
(glibc, openssl, zlib)"]
    D --> E["Base Image OS Packages
(apt/apk packages)"]
    E --> F["Linux Kernel Interfaces
(shared with host)"]

    style A fill:#f0f9f9,stroke:#3B9797
    style B fill:#f8f9fa,stroke:#132440
    style C fill:#f8f9fa,stroke:#132440
    style D fill:#fff5f5,stroke:#BF092F
    style E fill:#fff5f5,stroke:#BF092F
    style F fill:#fff5f5,stroke:#BF092F
                            

Supply chain attacks that shook the industry:

Software Bill of Materials (SBOM)

An SBOM is a machine-readable inventory of every component in your software — package names, versions, licenses, and relationships. It's the "ingredient list" for your container image.

Two standard formats dominate:

Generating SBOMs

# Generate SBOM with Syft (by Anchore) — the most popular SBOM generator
# Install Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

# Generate SPDX SBOM from a container image
syft myapp:latest -o spdx-json > sbom-spdx.json

# Generate CycloneDX SBOM
syft myapp:latest -o cyclonedx-json > sbom-cdx.json

# Generate SBOM from a directory (source code)
syft dir:./my-project -o spdx-json > source-sbom.json

# Generate SBOM with Trivy
trivy image --format spdx-json --output sbom.json myapp:latest

# Attach SBOM to an OCI image (using cosign)
cosign attach sbom --sbom sbom-cdx.json myregistry.io/myapp:v1.0.0

# Verify and download attached SBOM
cosign verify-attestation --type spdxjson myregistry.io/myapp:v1.0.0

Example SBOM output (CycloneDX, abbreviated):

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.5",
  "serialNumber": "urn:uuid:3e671687-395b-41f5-a30f-a58921a69b79",
  "version": 1,
  "metadata": {
    "component": {
      "type": "container",
      "name": "myapp",
      "version": "v1.0.0"
    }
  },
  "components": [
    {
      "type": "library",
      "name": "express",
      "version": "4.18.2",
      "purl": "pkg:npm/express@4.18.2",
      "licenses": [{ "license": { "id": "MIT" } }]
    },
    {
      "type": "library",
      "name": "openssl",
      "version": "3.0.13",
      "purl": "pkg:deb/debian/openssl@3.0.13-1~deb12u1",
      "licenses": [{ "license": { "id": "Apache-2.0" } }]
    }
  ]
}

Image Provenance & Attestations

Provenance answers: "Where was this image built, from what source code, by which pipeline?" It provides verifiable evidence of an image's origin, preventing scenarios where an attacker builds a malicious image outside your CI/CD and pushes it to your registry.

The SLSA framework (Supply-chain Levels for Software Artifacts) defines progressive security levels:

# Generate SLSA provenance with GitHub Actions
# The slsa-github-generator creates non-falsifiable provenance
# See: github.com/slsa-framework/slsa-github-generator

# Verify image provenance with cosign
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity-regexp='https://github.com/myorg/.*' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  myregistry.io/myapp:v1.0.0

# View provenance attestation content
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity-regexp='.*' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  myregistry.io/myapp:v1.0.0 | jq '.payload' | base64 -d | jq .

Signing Images with cosign

cosign (part of Sigstore) provides two signing modes: traditional key-based signing and modern keyless signing using OIDC identity providers. Keyless signing is preferred because it eliminates key management complexity.

sequenceDiagram
    participant Dev as Developer/CI
    participant OIDC as Identity Provider
(GitHub, Google)
    participant Fulcio as Fulcio CA
    participant Rekor as Rekor Transparency Log
    participant Reg as Container Registry

    Dev->>OIDC: Authenticate (OAuth2)
    OIDC-->>Dev: OIDC Token (identity proof)
    Dev->>Fulcio: Request signing certificate
(short-lived, OIDC token)
    Fulcio-->>Dev: X.509 Certificate (10 min validity)
    Dev->>Dev: Sign image digest with certificate
    Dev->>Rekor: Record signature in transparency log
    Dev->>Reg: Attach signature to image
    Note over Rekor: Immutable public record
of who signed what, when
                            

# Keyless signing in CI/CD (GitHub Actions)
# No keys to manage — identity comes from the CI platform
cosign sign myregistry.io/myapp@sha256:abc123...
# Fulcio issues short-lived certificate based on GitHub OIDC token
# Signature recorded in Rekor transparency log

# Verify with identity constraints
cosign verify \
  --certificate-identity='https://github.com/myorg/myrepo/.github/workflows/build.yml@refs/heads/main' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  myregistry.io/myapp@sha256:abc123...

# Sign with attestations (SBOM + provenance in one step)
cosign attest --predicate sbom.json --type cyclonedx myregistry.io/myapp:v1.0.0
cosign attest --predicate provenance.json --type slsaprovenance myregistry.io/myapp:v1.0.0

# Complete CI/CD signing workflow (GitHub Actions)
# .github/workflows/sign.yml
# steps:
#   - uses: sigstore/cosign-installer@v3
#   - run: cosign sign ${IMAGE}@${DIGEST}
#     env:
#       COSIGN_EXPERIMENTAL: "true"  # Enable keyless

Notary v2 & Docker Content Trust

Notary v2 (now part of the CNCF Notation project) provides a different trust model than cosign, using traditional PKI with delegation roles suitable for enterprise environments:

# Notation (Notary v2) — enterprise image signing
# Install notation CLI
curl -Lo notation.tar.gz https://github.com/notaryproject/notation/releases/download/v1.1.0/notation_1.1.0_linux_amd64.tar.gz
tar xzf notation.tar.gz -C /usr/local/bin notation

# Add a signing key
notation key add --name mykey --plugin azure-kv --id https://myvault.vault.azure.net/keys/signing-key/abc123

# Sign an image
notation sign myregistry.io/myapp@sha256:abc123...

# Verify an image
notation verify myregistry.io/myapp@sha256:abc123...

# List signatures on an image
notation list myregistry.io/myapp@sha256:abc123...

Trusted Registries & Policies

Signing images is only half the equation — you must also enforce that only signed images can run. Admission controllers in Kubernetes reject unsigned or untrusted images before they're scheduled.

# Kyverno policy: Only allow signed images from trusted registry
# kyverno-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-cosign-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "myregistry.io/*"
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/myorg/*"
                    rekor:
                      url: "https://rekor.sigstore.dev"

# OPA/Gatekeeper constraint: Deny unsigned images
# constraint.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireImageSignature
metadata:
  name: require-signed-images
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["production"]
  parameters:
    registries:
      - "myregistry.io"
    cosignVerification:
      required: true
      keyRef: "cosign-public-key"

The Secret Management Problem

Secrets (API keys, database passwords, certificates) are the most common security mistake in container deployments. They leak through multiple channels:

# BAD: Secret is permanently stored in image layer
FROM node:20-alpine
ENV DATABASE_URL=postgres://admin:s3cr3t@db.example.com/prod
COPY . /app

# Even if you try to "delete" it later:
RUN unset DATABASE_URL
# The ENV is still visible in the image metadata!

# Anyone can see it:
docker inspect myapp:latest | jq '.[0].Config.Env'
# ["DATABASE_URL=postgres://admin:s3cr3t@db.example.com/prod"]

# Or extract from image history:
docker history --no-trunc myapp:latest
# Shows the ENV instruction with full secret value

Docker Secrets (Swarm Mode)

Docker Secrets is a built-in secret management system for Docker Swarm. Secrets are encrypted at rest, transmitted over TLS, and mounted as in-memory files at /run/secrets/ — never touching disk or image layers.

# Create a secret from a file
echo "my-super-secret-password" | docker secret create db_password -

# Create from a file
docker secret create tls_cert ./server.crt

# List secrets
docker secret ls
# ID                          NAME          CREATED
# qdpvj3k4x3t0z7k2hf9lm1n8a  db_password   2 minutes ago
# r8k3j5l2m1n4p6q9s0t7v8w3x  tls_cert      1 minute ago

# Use secrets in a service
docker service create \
  --name mydb \
  --secret db_password \
  --secret tls_cert \
  -e DB_PASSWORD_FILE=/run/secrets/db_password \
  postgres:16

# Inside the container, the secret is a file:
# /run/secrets/db_password (contains "my-super-secret-password")
# Mounted as tmpfs — never written to disk

# docker-compose.yml with secrets
version: "3.8"
services:
  webapp:
    image: myapp:latest
    secrets:
      - db_password
      - api_key
    environment:
      - DB_PASSWORD_FILE=/run/secrets/db_password
      - API_KEY_FILE=/run/secrets/api_key

secrets:
  db_password:
    external: true    # Pre-created with docker secret create
  api_key:
    file: ./secrets/api_key.txt    # From local file (dev only)

Build-Time Secrets

BuildKit's --mount=type=secret provides build-time secrets that are never committed to image layers. The secret is available during the RUN instruction but leaves no trace in the final image.

# syntax=docker/dockerfile:1.4

FROM node:20-alpine

WORKDIR /app
COPY package.json package-lock.json ./

# Mount secret during npm install (for private registry auth)
# The secret is available at /run/secrets/npmrc during this RUN only
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci --only=production

COPY . .

USER node
CMD ["node", "server.js"]

# Build with the secret (passed at build time, never stored in image)
docker build --secret id=npmrc,src=$HOME/.npmrc -t myapp:latest .

# Verify the secret is NOT in the image
docker history --no-trunc myapp:latest
# No trace of .npmrc content in any layer

# Multiple secrets example
docker build \
  --secret id=npmrc,src=$HOME/.npmrc \
  --secret id=github_token,src=./github-token.txt \
  -t myapp:latest .

# In Dockerfile:
# RUN --mount=type=secret,id=github_token \
#     GITHUB_TOKEN=$(cat /run/secrets/github_token) \
#     go install private-repo/package@latest

# SSH agent forwarding for private Git repos (BuildKit)
# syntax=docker/dockerfile:1.4

FROM golang:1.22 AS builder

WORKDIR /app

# Mount SSH agent socket — clone private repos without embedding keys
RUN --mount=type=ssh \
    git clone git@github.com:myorg/private-lib.git /deps/private-lib

COPY . .
RUN go build -o /server .

FROM gcr.io/distroless/static
COPY --from=builder /server /server
CMD ["/server"]

# Build with SSH agent forwarding
docker build --ssh default -t myapp:latest .

External Secret Managers

For production deployments, external secret managers provide centralized secret storage with access control, rotation, audit logging, and dynamic credential generation.

# HashiCorp Vault integration example
# 1. Enable database secrets engine (dynamic credentials)
vault secrets enable database
vault write database/config/mydb \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db:5432/mydb" \
  allowed_roles="app-role"

# 2. Create a role that generates short-lived credentials
vault write database/roles/app-role \
  db_name=mydb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';" \
  default_ttl="1h" \
  max_ttl="24h"

# 3. Application requests dynamic credentials (auto-expires in 1 hour)
vault read database/creds/app-role
# Key                Value
# lease_id           database/creds/app-role/abc123
# username           v-app-role-xyz789
# password           A1B2C3D4-temp-password

Environment Variable Security

Environment variables are the most common (and most dangerous) way to pass secrets to containers. They're visible through multiple channels:

# Environment variables are VISIBLE to anyone with Docker access
docker inspect mycontainer | jq '.[0].Config.Env'
# ["DATABASE_URL=postgres://admin:password@db:5432/prod"]

# Also visible in /proc inside the container
docker exec mycontainer cat /proc/1/environ | tr '\0' '\n'
# DATABASE_URL=postgres://admin:password@db:5432/prod

# And in docker compose config output
docker compose config
# Shows all interpolated environment variables

# Leaked in crash dumps, debug logs, and error reports
# Application error: "connection failed to postgres://admin:password@..."

Safer alternatives to plain environment variables:

Exercises

Conclusion & Next Steps

Supply chain security closes the loop on container security: Part 15 secured the image, Part 16 hardened the runtime, and this article secured the pipeline that produces and distributes those images.

• SBOMs provide visibility into every component, enabling rapid incident response when new CVEs emerge
• Provenance attestations prove where images were built, preventing unauthorized builds from reaching production
• Image signing (cosign/Notation) guarantees integrity and authenticity with cryptographic verification
• Admission policies (Kyverno, OPA) enforce that only signed, verified images can deploy
• Secret management keeps credentials out of image layers, environment variables, and version control
• BuildKit secrets provide build-time credentials that never persist in the final image

Together, Parts 15-17 form a complete container security posture: secure images, hardened runtime, and verified supply chain. Next, we shift from single containers to multi-container applications with Docker Compose.