Part 8: Continuous Deployment & Advanced GitOps

Introduction — From CI to CD

In Part 7, we built robust continuous integration pipelines that validate, test, and package our code on every push. But CI alone only gets us halfway — code that is tested but never deployed provides no value to users. Continuous Deployment (CD) closes the loop by automatically delivering validated changes to production environments.

GitOps takes this further by making Git the single source of truth for both application code and infrastructure state. With ArgoCD watching our manifest repositories, any change to desired state triggers automatic reconciliation — the cluster converges to match what's declared in Git.

In this article, we'll build a complete CD pipeline that:

• Builds and pushes container images on self-hosted runners
• Updates Kubernetes manifests programmatically
• Commits changes to a GitOps repository
• Triggers ArgoCD sync and waits for healthy deployment
• Promotes changes through dev → staging → production

flowchart LR
    A[Developer Push] --> B[CI Pipeline]
    B --> C[Build & Test]
    C --> D[Push Image]
    D --> E[Update Manifests]
    E --> F[Commit to GitOps Repo]
    F --> G[ArgoCD Detects Change]
    G --> H[Sync to Cluster]
    H --> I[Health Check]
    I -->|Healthy| J[Deployed ✓]
    I -->|Degraded| K[Auto-Rollback]
                            

Self-Hosted GitHub Runners

GitHub-hosted runners are convenient but come with limitations: restricted network access, limited compute, and no persistent tooling. For CD pipelines that need to push to private registries or access internal clusters, self-hosted runners are essential.

Why Self-Hosted?

• Network access — Connect to private container registries, internal ArgoCD instances, and VPN-protected clusters
• Performance — Larger machines with pre-cached Docker layers for faster builds
• Cost control — Unlimited minutes on your own infrastructure
• Custom tooling — Pre-installed CLIs, credentials, and configurations

Installation & Registration

#!/bin/bash
# Download and configure a self-hosted GitHub Actions runner

# Create a dedicated directory
mkdir -p /opt/actions-runner && cd /opt/actions-runner

# Download the latest runner package (Linux x64)
RUNNER_VERSION="2.314.1"
curl -o actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz -L \
  "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz"

# Extract
tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz

# Configure the runner (interactive — provide org URL and token)
./config.sh \
  --url https://github.com/YOUR_ORG \
  --token YOUR_REGISTRATION_TOKEN \
  --name "cd-runner-01" \
  --labels "self-hosted,linux,cd-pipeline,docker" \
  --work "_work" \
  --runasservice

# Install and start as a systemd service
sudo ./svc.sh install
sudo ./svc.sh start

# Verify status
sudo ./svc.sh status

Using Labels in Workflows

# .github/workflows/cd-pipeline.yml
name: CD Pipeline

on:
  push:
    branches: [main]

jobs:
  build-and-push:
    # Target self-hosted runner with specific labels
    runs-on: [self-hosted, linux, cd-pipeline, docker]
    steps:
      - uses: actions/checkout@v4

      - name: Build container image
        run: |
          docker build -t myregistry.io/myapp:${{ github.sha }} .
          docker push myregistry.io/myapp:${{ github.sha }}

ArgoCD CLI

The ArgoCD CLI provides full control over applications, projects, and synchronization from the command line — essential for both interactive management and pipeline automation.

Installation

#!/bin/bash
# Install ArgoCD CLI on Linux

# Download the latest stable release
ARGOCD_VERSION=$(curl -s https://api.github.com/repos/argoproj/argo-cd/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -sSL -o /usr/local/bin/argocd \
  "https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_VERSION}/argocd-linux-amd64"

chmod +x /usr/local/bin/argocd

# Verify installation
argocd version --client

# Login to ArgoCD server
argocd login argocd.internal.company.com \
  --username admin \
  --password "$ARGOCD_PASSWORD" \
  --grpc-web

# List all applications
argocd app list

# Get detailed app status
argocd app get myapp --refresh

# Sync an application
argocd app sync myapp --prune --force

# Wait for app to become healthy
argocd app wait myapp --health --timeout 300

ArgoCD API Integration

For pipeline automation, the ArgoCD REST API provides programmatic access without requiring the CLI binary. This is particularly useful in containerized CI environments where you want minimal dependencies.

Authentication Tokens

#!/bin/bash
# Generate an ArgoCD API token for automation

# Get a session token via username/password
ARGOCD_TOKEN=$(curl -s -k "https://argocd.internal.company.com/api/v1/session" \
  -d '{"username":"admin","password":"'"${ARGOCD_PASSWORD}"'"}' \
  -H "Content-Type: application/json" | jq -r '.token')

echo "Token: ${ARGOCD_TOKEN}"

# Create a long-lived API token for a project
# (requires ArgoCD 2.4+)
argocd proj role create-token my-project ci-role \
  --expires-in 720h

# Use the token in API calls
curl -s -k "https://argocd.internal.company.com/api/v1/applications/myapp" \
  -H "Authorization: Bearer ${ARGOCD_TOKEN}" | jq '.status.health'

Programmatic Sync via API

#!/bin/bash
# Trigger ArgoCD sync via REST API

ARGOCD_SERVER="https://argocd.internal.company.com"
APP_NAME="myapp"

# Trigger sync
curl -s -k -X POST "${ARGOCD_SERVER}/api/v1/applications/${APP_NAME}/sync" \
  -H "Authorization: Bearer ${ARGOCD_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "revision": "HEAD",
    "prune": true,
    "strategy": {
      "apply": {
        "force": false
      }
    }
  }'

# Poll for sync completion
while true; do
  STATUS=$(curl -s -k "${ARGOCD_SERVER}/api/v1/applications/${APP_NAME}" \
    -H "Authorization: Bearer ${ARGOCD_TOKEN}" | jq -r '.status.sync.status')
  HEALTH=$(curl -s -k "${ARGOCD_SERVER}/api/v1/applications/${APP_NAME}" \
    -H "Authorization: Bearer ${ARGOCD_TOKEN}" | jq -r '.status.health.status')

  echo "Sync: ${STATUS} | Health: ${HEALTH}"

  if [[ "$STATUS" == "Synced" && "$HEALTH" == "Healthy" ]]; then
    echo "✓ Application is synced and healthy"
    break
  elif [[ "$HEALTH" == "Degraded" ]]; then
    echo "✗ Application is degraded — check logs"
    exit 1
  fi

  sleep 10
done

Building a CD Pipeline

A production CD pipeline is composed of multiple jobs with clear responsibilities and dependencies. Each job handles one concern: build, push, update manifests, sync. This separation enables parallel execution, granular retry, and clear audit trails.

flowchart TD
    A[Push to main] --> B[Job 1: Build & Test]
    B --> C[Job 2: Push Image]
    C --> D[Job 3: Update Manifests]
    D --> E[Job 4: ArgoCD Sync]
    E --> F[Job 5: Smoke Tests]
    F -->|Pass| G[Job 6: Promote to Staging]
    F -->|Fail| H[Rollback & Alert]
    G --> I[Job 7: Integration Tests]
    I -->|Pass| J[Manual Approval Gate]
    J --> K[Job 8: Promote to Production]
                            

# .github/workflows/cd-full-pipeline.yml
name: Full CD Pipeline

on:
  push:
    branches: [main]
    paths:
      - 'src/**'
      - 'Dockerfile'

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}
  GITOPS_REPO: my-org/k8s-manifests

jobs:
  # ─── Job 1: Build and Test ────────────────────────────────
  build-test:
    runs-on: [self-hosted, linux, cd-pipeline]
    outputs:
      image-tag: ${{ steps.meta.outputs.tags }}
      image-digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Run unit tests
        run: |
          npm ci
          npm test -- --coverage

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=
            type=ref,event=branch

      - name: Build and push
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  # ─── Job 2: Update Manifests ──────────────────────────────
  update-manifests:
    needs: build-test
    runs-on: [self-hosted, linux, cd-pipeline]
    outputs:
      commit-sha: ${{ steps.commit.outputs.sha }}
    steps:
      - name: Checkout GitOps repo
        uses: actions/checkout@v4
        with:
          repository: ${{ env.GITOPS_REPO }}
          token: ${{ secrets.GITOPS_PAT }}
          path: manifests

      - name: Update image tag in dev overlay
        run: |
          cd manifests/overlays/dev
          yq eval '.images[0].newTag = "${{ github.sha }}"' -i kustomization.yaml
          cat kustomization.yaml

      - name: Commit and push
        id: commit
        run: |
          cd manifests
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add .
          git commit -m "chore(dev): update image to ${{ github.sha }}"
          git push
          echo "sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT

  # ─── Job 3: Sync ArgoCD ───────────────────────────────────
  sync-argocd:
    needs: update-manifests
    runs-on: [self-hosted, linux, cd-pipeline]
    steps:
      - name: Sync and wait for healthy
        env:
          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_TOKEN }}
        run: |
          argocd app sync myapp-dev \
            --server "$ARGOCD_SERVER" \
            --auth-token "$ARGOCD_AUTH_TOKEN" \
            --grpc-web \
            --revision "${{ needs.update-manifests.outputs.commit-sha }}" \
            --prune

          argocd app wait myapp-dev \
            --server "$ARGOCD_SERVER" \
            --auth-token "$ARGOCD_AUTH_TOKEN" \
            --grpc-web \
            --health \
            --timeout 300

Accessing Values Across Jobs

In multi-job workflows, sharing data between jobs requires explicit mechanisms. GitHub Actions provides three approaches: job outputs, artifacts, and environment files.

Job Outputs

# Job outputs — best for small strings (image tags, SHAs, flags)
jobs:
  producer:
    runs-on: ubuntu-latest
    outputs:
      # Declare outputs that other jobs can reference
      image-tag: ${{ steps.tag.outputs.value }}
      deploy-env: ${{ steps.env.outputs.name }}
    steps:
      - name: Generate image tag
        id: tag
        run: echo "value=sha-$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Determine environment
        id: env
        run: |
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            echo "name=production" >> $GITHUB_OUTPUT
          else
            echo "name=staging" >> $GITHUB_OUTPUT
          fi

  consumer:
    needs: producer
    runs-on: ubuntu-latest
    steps:
      - name: Use outputs from producer
        run: |
          echo "Image tag: ${{ needs.producer.outputs.image-tag }}"
          echo "Deploy to: ${{ needs.producer.outputs.deploy-env }}"

Modifying YAML Programmatically

The critical step in a GitOps CD pipeline is updating the image tag in Kubernetes manifests. The yq tool (a YAML-aware processor, like jq for JSON) is the recommended approach — it preserves formatting, comments, and structure.

#!/bin/bash
# Using yq to update image tags in Kubernetes manifests

# Install yq (Mike Farah's version — not the Python one)
YQ_VERSION="v4.40.5"
wget -qO /usr/local/bin/yq \
  "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_amd64"
chmod +x /usr/local/bin/yq

# ─── Kustomization overlay approach ────────────────────────
# Update newTag in kustomization.yaml
yq eval '.images[0].newTag = "abc123def"' -i overlays/dev/kustomization.yaml

# Update multiple images
yq eval '
  (.images[] | select(.name == "myapp")).newTag = "v1.2.3" |
  (.images[] | select(.name == "sidecar")).newTag = "v0.5.1"
' -i overlays/dev/kustomization.yaml

# ─── Direct deployment manifest approach ───────────────────
# Update the image in a deployment YAML
yq eval '
  .spec.template.spec.containers[0].image = "ghcr.io/myorg/myapp:abc123def"
' -i deployments/myapp.yaml

# ─── Helm values approach ──────────────────────────────────
# Update image.tag in values.yaml
yq eval '.image.tag = "abc123def"' -i charts/myapp/values-dev.yaml

# Verify the change
echo "Updated manifest:"
yq eval '.spec.template.spec.containers[0].image' deployments/myapp.yaml

# ✗ BAD — fragile sed approach (breaks on multi-line, special chars)
sed -i "s|image: myapp:.*|image: myapp:${NEW_TAG}|" deployment.yaml

# ✓ GOOD — yq understands YAML structure
yq eval ".spec.template.spec.containers[0].image = \"myapp:${NEW_TAG}\"" -i deployment.yaml

Updating Remote Repositories

In GitOps, the CD pipeline doesn't deploy directly — it updates a separate manifest repository and lets ArgoCD handle the actual deployment. This separation provides clear ownership, independent versioning, and auditable change history.

# Git operations in a GitHub Actions job
- name: Checkout manifest repository
  uses: actions/checkout@v4
  with:
    repository: my-org/k8s-manifests
    token: ${{ secrets.GITOPS_PAT }}  # PAT with repo write access
    path: gitops-repo
    ref: main

- name: Update image tag
  run: |
    cd gitops-repo
    yq eval '.images[0].newTag = "${{ env.IMAGE_TAG }}"' \
      -i overlays/dev/kustomization.yaml

- name: Commit and push manifest change
  run: |
    cd gitops-repo
    git config user.name "cd-pipeline[bot]"
    git config user.email "cd-pipeline[bot]@users.noreply.github.com"

    # Check if there are actual changes
    if git diff --quiet; then
      echo "No changes to commit"
      exit 0
    fi

    git add overlays/dev/kustomization.yaml
    git commit -m "deploy(dev): update myapp to ${{ env.IMAGE_TAG }}

    Triggered by: ${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }}
    Workflow: ${{ github.workflow }} #${{ github.run_number }}"

    # Push with retry logic for concurrent updates
    for i in 1 2 3; do
      git pull --rebase origin main
      git push origin main && break
      echo "Push failed, retrying ($i/3)..."
      sleep 5
    done

ArgoCD Sync from Pipelines

While ArgoCD can detect changes automatically via polling or webhooks, explicitly triggering sync from the pipeline gives you precise control over timing and the ability to wait for health status before proceeding.

#!/bin/bash
# argocd-sync.sh — Reusable sync script for CD pipelines

set -euo pipefail

APP_NAME="${1:?Usage: argocd-sync.sh  [timeout]}"
TIMEOUT="${2:-300}"

echo "► Triggering sync for ${APP_NAME}..."

# Hard refresh to pick up latest manifests
argocd app get "${APP_NAME}" --hard-refresh --grpc-web

# Sync with pruning (removes orphaned resources)
argocd app sync "${APP_NAME}" \
  --grpc-web \
  --prune \
  --timeout "${TIMEOUT}" \
  --retry-limit 3 \
  --retry-backoff-duration 10s

echo "► Waiting for ${APP_NAME} to become healthy..."

# Wait for health with timeout
if argocd app wait "${APP_NAME}" \
  --grpc-web \
  --health \
  --timeout "${TIMEOUT}"; then
  echo "✓ ${APP_NAME} is healthy!"
  argocd app get "${APP_NAME}" --grpc-web | grep -E "Health|Sync|Images"
else
  echo "✗ ${APP_NAME} did not become healthy within ${TIMEOUT}s"
  echo "Current status:"
  argocd app get "${APP_NAME}" --grpc-web
  echo "Recent events:"
  argocd app resources "${APP_NAME}" --grpc-web
  exit 1
fi

Environment Promotion

Production-grade CD requires promoting changes through environments: dev → staging → production. Each stage validates the deployment before advancing, catching issues early where they're cheapest to fix.

flowchart LR
    subgraph DEV["Development"]
        D1[Auto-deploy on merge]
        D2[Smoke tests]
    end
    subgraph STAGING["Staging"]
        S1[Promote after dev success]
        S2[Integration tests]
        S3[Performance tests]
    end
    subgraph PROD["Production"]
        P1[Manual approval gate]
        P2[Canary deployment]
        P3[Full rollout]
    end
    DEV -->|Automated| STAGING
    STAGING -->|Manual Gate| PROD
                            

Promotion Strategies

# Promotion job — copies image tag from dev to staging
promote-to-staging:
  needs: [sync-argocd, smoke-tests]
  runs-on: [self-hosted, linux, cd-pipeline]
  environment:
    name: staging
    url: https://staging.myapp.com
  steps:
    - name: Checkout manifests
      uses: actions/checkout@v4
      with:
        repository: ${{ env.GITOPS_REPO }}
        token: ${{ secrets.GITOPS_PAT }}

    - name: Promote image tag to staging
      run: |
        # Read current dev tag
        DEV_TAG=$(yq eval '.images[0].newTag' overlays/dev/kustomization.yaml)
        echo "Promoting tag: ${DEV_TAG}"

        # Apply to staging overlay
        yq eval ".images[0].newTag = \"${DEV_TAG}\"" \
          -i overlays/staging/kustomization.yaml

    - name: Commit promotion
      run: |
        git config user.name "cd-pipeline[bot]"
        git config user.email "cd-pipeline[bot]@users.noreply.github.com"
        git add overlays/staging/
        git commit -m "promote(staging): update to ${DEV_TAG}

        Source: dev deployment validated
        Tests: smoke tests passed ✓"
        git push origin main

The Complete CI/CD Workflow

Putting it all together, here's the end-to-end flow from developer commit to production deployment. Each step has clear inputs, outputs, and failure modes.

flowchart TD
    A[Developer pushes to main] --> B{CI Pipeline}
    B --> B1[Lint & Static Analysis]
    B --> B2[Unit Tests]
    B --> B3[Security Scan]
    B1 --> C{All CI Passed?}
    B2 --> C
    B3 --> C
    C -->|No| FAIL[Notify & Block]
    C -->|Yes| D[Build Container Image]
    D --> E[Push to Registry]
    E --> F[Update Dev Manifests]
    F --> G[Commit to GitOps Repo]
    G --> H[ArgoCD Sync Dev]
    H --> I{Dev Healthy?}
    I -->|No| ROLLBACK1[Rollback Dev]
    I -->|Yes| J[Run Smoke Tests]
    J -->|Pass| K[Promote to Staging]
    K --> L[ArgoCD Sync Staging]
    L --> M[Integration Tests]
    M -->|Pass| N[Manual Approval]
    N --> O[Promote to Production]
    O --> P[Canary Rollout]
    P --> Q{Metrics OK?}
    Q -->|Yes| R[Full Rollout ✓]
    Q -->|No| ROLLBACK2[Rollback Production]
                            

Troubleshooting

CD pipelines have more failure modes than CI because they interact with external systems (registries, clusters, GitOps controllers). Here are the most common issues and their resolutions.

Image Pull Errors

#!/bin/bash
# Diagnose ImagePullBackOff errors

# Check pod events
kubectl describe pod myapp-xyz -n dev | grep -A 5 "Events"

# Common causes:
# 1. Image doesn't exist — verify the tag was pushed
docker manifest inspect ghcr.io/myorg/myapp:abc123

# 2. Registry auth — ensure imagePullSecrets exist
kubectl get secret regcred -n dev -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

# 3. Network — test registry connectivity from node
kubectl run test --image=busybox --rm -it -- wget -qO- https://ghcr.io/v2/

Sync Failures

#!/bin/bash
# Diagnose ArgoCD sync failures

# Check application sync status and errors
argocd app get myapp --grpc-web --show-operation

# View sync result details
argocd app manifests myapp --grpc-web --source live | head -50

# Common fixes:
# 1. Out-of-sync due to manual changes — enable self-heal
argocd app set myapp --self-heal --grpc-web

# 2. Resource already exists (from another app)
argocd app sync myapp --grpc-web --force

# 3. RBAC — ArgoCD service account lacks permissions
kubectl auth can-i create deployments \
  --as=system:serviceaccount:argocd:argocd-application-controller \
  -n dev

Common Issues Reference

Conclusion & What's Next

We've built a complete continuous deployment pipeline that embodies GitOps principles — Git as the source of truth, declarative desired state, automated reconciliation, and clear promotion paths through environments. The key components we assembled:

• Self-hosted runners for network access and performance
• ArgoCD CLI and API for programmatic deployment control
• Multi-job pipelines with clear job outputs and dependencies
• yq for YAML manipulation — structured, safe, and predictable
• Environment promotion with gated progression and automated testing