CI/CD Platform Deep Dive: Argo CD & GitOps

Introduction

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. Unlike traditional CI/CD platforms that push deployments to clusters, Argo CD pulls — it continuously monitors Git repositories and automatically synchronizes the live cluster state with the desired state defined in Git. When someone commits a change to the manifests repository, Argo CD detects the drift and reconciles it.

This fundamental inversion — from "CI pushes to cluster" to "cluster pulls from Git" — eliminates an entire class of problems: credential sprawl (no cluster credentials in CI systems), configuration drift (continuous reconciliation corrects manual changes), and audit gaps (Git history becomes the definitive deployment record).

Pull vs Push Deployment

Traditional CI/CD (Jenkins, GitHub Actions, GitLab CI) uses a push model: the CI system authenticates to the cluster and applies changes. This requires storing cluster credentials in the CI system — a security risk. Pull-based GitOps inverts this: the agent running inside the cluster watches Git and applies changes locally, eliminating external credential exposure.

GitOps Principles

GitOps is a set of practices that use Git as the single source of truth for declarative infrastructure and applications. The four core principles (formalized by the OpenGitOps project under CNCF):

1. Declarative — The entire system is described declaratively (Kubernetes manifests, Helm charts, Kustomize overlays).
2. Versioned and Immutable — The desired state is stored in Git, providing full history, audit trail, and easy rollback via git revert.
3. Pulled Automatically — Software agents (Argo CD) automatically pull the desired state from Git — no manual kubectl apply.
4. Continuously Reconciled — Agents compare desired state (Git) with actual state (cluster) and correct any drift automatically.

flowchart LR
    DEV[Developer] -->|Push code| CI[CI Pipeline]
    CI -->|Build & Test| REG[Container Registry]
    CI -->|Update manifests| GIT[Git Repo - Manifests]
    
    GIT -->|Watches| ARGO[Argo CD]
    ARGO -->|Syncs| K8S[Kubernetes Cluster]
    K8S -->|Pulls image| REG
    
    K8S -.->|Status| ARGO
    ARGO -.->|Drift detection| GIT
    
    style DEV fill:#132440,color:#fff
    style CI fill:#16476A,color:#fff
    style GIT fill:#3B9797,color:#fff
    style ARGO fill:#BF092F,color:#fff
    style K8S fill:#132440,color:#fff
    style REG fill:#16476A,color:#fff
                            

The separation of concerns is critical: CI handles building and testing (producing artifacts), while CD (Argo CD) handles deploying those artifacts to clusters. The manifests repository is the contract between them.

Argo CD Architecture

Argo CD runs as a set of Kubernetes controllers in your cluster. Understanding its components helps with debugging, scaling, and security hardening.

flowchart TD
    UI[Web UI / CLI] --> API[API Server]
    API --> REPO[Repo Server]
    API --> CTRL[Application Controller]
    API --> REDIS[Redis Cache]
    
    REPO -->|Clone & render| GIT[Git Repositories]
    CTRL -->|Watch & sync| K8S[Kubernetes API]
    CTRL --> REDIS
    REPO --> REDIS
    
    DEX[Dex - SSO] --> API
    NOTIFY[Notifications Controller] --> CTRL
    
    style API fill:#3B9797,color:#fff
    style CTRL fill:#132440,color:#fff
    style REPO fill:#16476A,color:#fff
    style REDIS fill:#BF092F,color:#fff
                            

• API Server — Exposes the gRPC/REST API consumed by the Web UI, CLI, and CI systems. Handles authentication and authorization.
• Repository Server — Clones Git repos, renders Helm charts, processes Kustomize overlays, and caches the results. Stateless and horizontally scalable.
• Application Controller — The core reconciliation loop. Watches Application CRDs, compares desired vs actual state, and performs syncs. Runs leader-elected for HA.
• Redis — In-memory cache for repo manifests and app state. Reduces Git polling and API server load.
• Dex — OpenID Connect provider for SSO integration (LDAP, SAML, GitHub, GitLab, etc.).

Installation & Setup

Argo CD can be installed via plain manifests, Helm chart, or as part of a managed offering (OpenShift GitOps, Akuity). Here are the primary methods:

# Method 1: Plain manifests (non-HA, good for dev/test)
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Method 2: HA installation (production)
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

# Method 3: Helm chart (most configurable)
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --set server.service.type=LoadBalancer \
  --set configs.params."server\.insecure"=true

# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

# Install Argo CD CLI
curl -sSL -o argocd-linux-amd64 \
  https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
chmod +x argocd-linux-amd64
sudo mv argocd-linux-amd64 /usr/local/bin/argocd

# Login via CLI
argocd login argocd.example.com --username admin --password $INITIAL_PASSWORD

# Change admin password
argocd account update-password

Application CRD

The Application is Argo CD's core custom resource. It defines what to deploy (source), where to deploy (destination), and how to sync (policy). Every managed workload is represented by an Application.

# Basic Application manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default

  source:
    repoURL: https://github.com/myorg/my-app-manifests.git
    targetRevision: main
    path: overlays/production

  destination:
    server: https://kubernetes.default.svc
    namespace: production

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

# Application with Helm source
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus
  namespace: argocd
spec:
  project: monitoring
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: kube-prometheus-stack
    targetRevision: 55.5.0
    helm:
      releaseName: prometheus
      valuesObject:
        grafana:
          enabled: true
          adminPassword: ${grafana_password}
        alertmanager:
          enabled: true
        prometheus:
          prometheusSpec:
            retention: 30d
            storageSpec:
              volumeClaimTemplate:
                spec:
                  resources:
                    requests:
                      storage: 50Gi
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated:
      selfHeal: true

Sync Policies

Sync policies control how and when Argo CD reconciles the desired state with the actual state. The key settings:

# Sync windows — restrict when syncs can occur
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  syncWindows:
    # Allow syncs only during business hours on weekdays
    - kind: allow
      schedule: '0 9 * * 1-5'
      duration: 8h
      applications:
        - '*'
    # Deny all syncs during weekends
    - kind: deny
      schedule: '0 0 * * 0,6'
      duration: 48h
      applications:
        - '*'
    # Allow emergency hotfixes anytime for critical apps
    - kind: allow
      schedule: '* * * * *'
      duration: 24h
      applications:
        - 'critical-*'
      manualSync: true

ApplicationSets

ApplicationSets dynamically generate multiple Applications from a single template. They're essential for multi-cluster deployments, multi-tenant platforms, and monorepo management. Instead of manually creating 50 Application manifests for 50 microservices, an ApplicationSet generates them from a directory structure or cluster list.

# Git Directory Generator — one Application per subdirectory
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: microservices
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/k8s-manifests.git
        revision: main
        directories:
          - path: apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

# Cluster Generator — deploy to all registered clusters
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-services
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production
  template:
    metadata:
      name: 'platform-{{name}}'
    spec:
      project: platform
      source:
        repoURL: https://github.com/myorg/platform-manifests.git
        targetRevision: main
        path: base
        kustomize:
          patches:
            - target:
                kind: Namespace
              patch: |
                - op: replace
                  path: /metadata/labels/cluster
                  value: '{{name}}'
      destination:
        server: '{{server}}'
        namespace: platform
      syncPolicy:
        automated:
          selfHeal: true

# Matrix Generator — combine clusters x environments
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: multi-env-deploy
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  tier: production
          - list:
              elements:
                - app: frontend
                  path: apps/frontend
                - app: backend
                  path: apps/backend
                - app: worker
                  path: apps/worker
  template:
    metadata:
      name: '{{app}}-{{name}}'
    spec:
      source:
        repoURL: https://github.com/myorg/manifests.git
        path: '{{path}}/overlays/{{metadata.labels.region}}'
      destination:
        server: '{{server}}'
        namespace: '{{app}}'

Multi-Cluster Management

Argo CD excels at managing deployments across multiple Kubernetes clusters from a single control plane. The hub-and-spoke model has one Argo CD instance (the hub) managing applications across all target clusters (spokes).

flowchart TD
    GIT[Git Repository] --> HUB[Argo CD Hub Cluster]
    
    HUB -->|Syncs| PROD1[Production US-East]
    HUB -->|Syncs| PROD2[Production EU-West]
    HUB -->|Syncs| STAGING[Staging]
    HUB -->|Syncs| DEV[Development]
    
    HUB -->|Manages itself| HUB
    
    style GIT fill:#3B9797,color:#fff
    style HUB fill:#BF092F,color:#fff
    style PROD1 fill:#132440,color:#fff
    style PROD2 fill:#132440,color:#fff
    style STAGING fill:#16476A,color:#fff
    style DEV fill:#16476A,color:#fff
                            

# Register a remote cluster with Argo CD
argocd cluster add my-production-cluster \
  --name production-us-east \
  --kubeconfig ~/.kube/config \
  --context production-us-east

# List registered clusters
argocd cluster list

# Add cluster labels for ApplicationSet generators
argocd cluster set production-us-east \
  --label environment=production \
  --label region=us-east

Argo Rollouts

Argo Rollouts extends Kubernetes with advanced deployment strategies — canary releases with traffic management, blue-green deployments, and automated analysis-driven promotions. It replaces the standard Kubernetes Deployment with a Rollout CRD that provides fine-grained control over how new versions are introduced.

# Canary Rollout with traffic management and analysis
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
  namespace: production
spec:
  replicas: 10
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: app
          image: myregistry/my-app:latest
          ports:
            - containerPort: 8080
  strategy:
    canary:
      canaryService: my-app-canary
      stableService: my-app-stable
      trafficRouting:
        istio:
          virtualServices:
            - name: my-app-vsvc
              routes:
                - primary
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: my-app-canary
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

# Analysis Template — automated promotion/rollback decisions
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 30s
      count: 5
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))

# Blue-Green Rollout
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app-bg
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: app
          image: myregistry/my-app:latest
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: false
      prePromotionAnalysis:
        templates:
          - templateName: smoke-tests
      postPromotionAnalysis:
        templates:
          - templateName: success-rate
      scaleDownDelaySeconds: 300

Integration Patterns

Argo CD integrates with the broader Argo ecosystem and CI tools to form complete delivery pipelines:

• Argo CD + CI Pipeline — CI builds the image, updates the manifests repo (image tag), Argo CD detects the change and syncs.
• Argo CD Image Updater — Automatically detects new container images in registries and updates the Application's image tag without CI involvement.
• Argo CD + Argo Events — Event-driven triggers (webhooks, message queues) that create Argo CD sync operations or ApplicationSets.
• Argo CD + Argo Workflows — Complex pre/post-sync operations (database migrations, smoke tests) orchestrated as Argo Workflows triggered by sync hooks.

# CI pipeline step: Update manifests repo after image build
# (GitHub Actions example — triggers Argo CD sync)
- name: Update manifests
  run: |
    git clone https://github.com/myorg/k8s-manifests.git
    cd k8s-manifests
    
    # Update image tag in kustomization
    cd overlays/production
    kustomize edit set image myregistry/my-app=myregistry/my-app:${{ github.sha }}
    
    git add .
    git commit -m "deploy: my-app ${{ github.sha }}"
    git push

Repository Structure

How you organize your GitOps repositories significantly impacts maintainability and security. Common patterns:

# Recommended monorepo structure with Kustomize
k8s-manifests/
├── apps/
│   ├── frontend/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── overlays/
│   │       ├── staging/
│   │       │   ├── kustomization.yaml
│   │       │   └── replica-patch.yaml
│   │       └── production/
│   │           ├── kustomization.yaml
│   │           ├── replica-patch.yaml
│   │           └── hpa.yaml
│   ├── backend/
│   │   ├── base/
│   │   └── overlays/
│   └── worker/
│       ├── base/
│       └── overlays/
├── platform/
│   ├── cert-manager/
│   ├── ingress-nginx/
│   └── monitoring/
└── argocd-apps/
    ├── frontend.yaml
    ├── backend.yaml
    └── platform.yaml

Security

Argo CD security spans RBAC, SSO, project isolation, and secrets management:

• RBAC — Fine-grained policies controlling who can sync, override, delete, or view Applications. Integrated with SSO groups.
• Projects — Logical groupings that restrict which repositories, clusters, and namespaces an Application can target. Prevent teams from deploying to each other's namespaces.
• SSO — Integrate with OIDC providers (Okta, Azure AD, GitHub) via the bundled Dex server.
• Secrets — Argo CD cannot natively manage secrets (they'd be in Git). Use Sealed Secrets, External Secrets Operator, or Vault for secret injection.

# Argo CD RBAC policy
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.csv: |
    # Team frontend can manage their apps
    p, role:frontend-team, applications, *, frontend-project/*, allow
    p, role:frontend-team, logs, get, frontend-project/*, allow
    
    # Platform team can manage all apps and clusters
    p, role:platform-admin, applications, *, */*, allow
    p, role:platform-admin, clusters, *, *, allow
    p, role:platform-admin, projects, *, *, allow
    
    # Map SSO groups to roles
    g, frontend-devs, role:frontend-team
    g, platform-eng, role:platform-admin
  policy.default: role:readonly

Troubleshooting

Common Argo CD issues and their resolutions:

• OutOfSync but healthy — Resource has fields set by Kubernetes (defaulted fields, status subresource) that don't match the manifest. Fix with ignoreDifferences in the Application spec.
• Sync failed: "resource already exists" — Resource was created outside Argo CD. Adopt it by adding the argocd.argoproj.io/managed-by annotation.
• Slow sync with many resources — Increase Application Controller replicas, enable sharding, or split large Applications into smaller ones.
• Repository not accessible — Check repository credentials in argocd-repo-creds secret. Verify network connectivity from the repo-server pod.

# Ignore differences for fields managed by Kubernetes
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Managed by HPA
    - group: ""
      kind: Service
      jqPathExpressions:
        - .spec.clusterIP  # Auto-assigned by k8s

Exercises