Part 5: GitOps Foundations & Architecture

Introduction — What is GitOps?

GitOps is an operational framework that takes DevOps best practices used for application development — such as version control, collaboration, compliance, and CI/CD — and applies them to infrastructure automation. At its core, GitOps uses Git repositories as the single source of truth for declarative infrastructure and applications.

The term was coined by Weaveworks in 2017, but the principles draw from decades of configuration management experience. GitOps answers a fundamental question: "What should my system look like, and how do I ensure it always looks that way?"

Traditional deployment pipelines push changes to environments through scripts and manual approvals. GitOps flips this model: a software agent running inside the target environment continuously pulls the desired state from Git and reconciles any differences. This shift from push to pull is what makes GitOps uniquely powerful for cloud-native operations.

In this article, we'll build a comprehensive understanding of GitOps — from core principles and architecture patterns to repository strategies, tooling, security considerations, and real-world case studies that demonstrate its transformative impact.

GitOps Core Principles

The OpenGitOps project (a CNCF Sandbox project) defines four key principles that any GitOps implementation must follow:

Principle 1: Declarative

A system managed by GitOps must have its desired state expressed declaratively. This means describing what the system should look like rather than how to get there. Kubernetes manifests, Terraform configurations, and Helm charts are all declarative — they define end states without specifying imperative steps.

# Declarative: "I want 3 replicas of this app"
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      containers:
        - name: payment-service
          image: registry.example.com/payment:v2.4.1
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"

Principle 2: Versioned and Immutable

The desired state is stored in a way that enforces immutability, versioning, and retains a complete version history. Git naturally provides all three: every commit is immutable (content-addressable via SHA), every change is versioned, and the full history is preserved indefinitely.

Principle 3: Pulled Automatically

Software agents automatically pull the desired state declarations from the source. Rather than waiting for external triggers, controllers continuously monitor the Git repository and initiate reconciliation when differences are detected.

Principle 4: Continuously Reconciled

Software agents continuously observe the actual system state and attempt to apply the desired state. If the actual state diverges (due to manual changes, failures, or drift), the agent corrects it — ensuring the system always converges toward the declared state.

Push vs Pull Deployment Models

Understanding the difference between push-based and pull-based deployment is essential to grasping why GitOps represents a paradigm shift.

Traditional Push-Based CI/CD

In a push model, an external CI/CD system (Jenkins, GitHub Actions, Azure Pipelines) builds artifacts and then pushes them to the target environment. The pipeline needs credentials to access production systems, and deployment happens as a one-time event.

flowchart LR
    A[Developer] -->|git push| B[Git Repository]
    B -->|webhook| C[CI/CD Pipeline]
    C -->|build| D[Container Registry]
    C -->|push credentials| E[Production Cluster]
    D -->|pull image| E
    style C fill:#f9f,stroke:#333,stroke-width:2px
    style E fill:#bbf,stroke:#333,stroke-width:2px
                            

Push model challenges:

• CI/CD system requires production credentials (security risk)
• No visibility into actual cluster state after deployment
• Manual changes in production cause undetected drift
• Rollback requires re-running pipelines
• Deployment state is transient — lives only in pipeline logs

GitOps Pull-Based Model

In the pull model, a controller running inside the target environment monitors the Git repository and pulls changes when detected. No external system needs credentials to the production cluster.

flowchart LR
    A[Developer] -->|git push| B[Git Repository]
    B -->|CI builds image| C[Container Registry]
    D[GitOps Controller] -->|polls/watches| B
    D -->|reconciles| E[Production Cluster]
    E -->|pulls image| C
    D -.->|lives inside| E
    style D fill:#3B9797,stroke:#333,stroke-width:2px,color:#fff
    style E fill:#132440,stroke:#333,stroke-width:2px,color:#fff
                            

Pull model advantages:

• No external access to production needed (improved security)
• Continuous reconciliation detects and corrects drift
• Git history is the deployment audit trail
• Rollback is a simple git revert
• Desired state is always visible and versioned

GitOps Architecture

A complete GitOps architecture involves several components working together in a continuous reconciliation loop. Let's trace the full workflow from developer commit to production deployment.

flowchart TD
    A[Developer commits code] --> B[App Repo - source code]
    B -->|CI Pipeline triggers| C[Build & Test]
    C -->|Push image| D[Container Registry]
    C -->|Update manifest| E[Config Repo - desired state]
    E -->|Controller detects change| F[GitOps Controller]
    F -->|Compare desired vs actual| G{Drift Detected?}
    G -->|Yes| H[Apply Changes to Cluster]
    G -->|No| I[System in Sync ✓]
    H --> J[Cluster Reconciled]
    J --> K[Health Check & Validation]
    K -->|Healthy| I
    K -->|Unhealthy| L[Alert & Auto-Rollback]
    L --> E
    style F fill:#3B9797,stroke:#333,stroke-width:2px,color:#fff
    style I fill:#28a745,stroke:#333,stroke-width:2px,color:#fff
    style L fill:#BF092F,stroke:#333,stroke-width:2px,color:#fff
                            

Component Breakdown

Application Repository — Contains source code, Dockerfiles, and CI pipeline definitions. Developers work here day-to-day. When code merges to main, CI builds and publishes container images.

Configuration Repository — Contains Kubernetes manifests, Helm values, Kustomize overlays, or Terraform configs representing the desired state. Image tags are updated here (manually or by automation) to trigger deployments.

GitOps Controller — Runs inside the cluster (e.g., Argo CD, Flux). Continuously monitors the config repo for changes, compares declared state against live state, and reconciles differences.

Container Registry — Stores built images. The cluster pulls images referenced in manifests during reconciliation.

# Example: Kustomization that the GitOps controller watches
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production

resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml
  - hpa.yaml

images:
  - name: registry.example.com/payment-service
    newTag: v2.4.1  # Updated by CI automation

configMapGenerator:
  - name: payment-config
    literals:
      - LOG_LEVEL=info
      - MAX_RETRIES=3
      - TIMEOUT_MS=5000

Drift Detection & Self-Healing

Configuration drift occurs when the actual state of a system diverges from its declared desired state. This can happen through manual kubectl commands, operator bugs, resource evictions, or external modifications. GitOps controllers solve this through continuous drift detection.

How Drift Detection Works

The controller maintains a three-way comparison:

1. Desired state — What's declared in the Git repository
2. Last applied state — What the controller last successfully applied
3. Live state — What currently exists in the cluster

When live state ≠ desired state, the controller takes action. Depending on configuration, it may:

• Auto-sync — Immediately correct the drift (self-healing mode)
• Alert only — Notify operators and wait for manual approval
• Selective sync — Auto-heal certain resources, alert on others

# Argo CD Application with auto-sync and self-healing
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-service
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/org/platform-config.git
    targetRevision: main
    path: apps/payment-service/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true        # Remove resources not in Git
      selfHeal: true     # Correct drift automatically
      allowEmpty: false  # Prevent accidental deletion of all resources
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Why Self-Healing Matters

Self-healing provides three critical guarantees:

• Reliability — Accidental deletions or misconfigurations are automatically corrected
• Security — Unauthorized modifications are detected and reverted
• Compliance — Systems always match their audited, approved configuration

GitOps Repository Strategies

One of the most debated topics in GitOps is how to structure repositories. The right strategy depends on team size, number of services, compliance requirements, and operational complexity.

Monorepo vs Polyrepo

App Repo vs Config Repo Separation

The most widely recommended GitOps pattern separates application source code from deployment configuration:

# App Repository (source code + CI)
app-repo/
├── src/                    # Application source code
├── Dockerfile              # Container build definition
├── tests/                  # Unit and integration tests
└── .github/workflows/      # CI pipeline (build, test, publish)

# Config Repository (desired state for GitOps)
config-repo/
├── base/                   # Shared base manifests
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/               # Development environment overrides
│   │   ├── kustomization.yaml
│   │   └── replicas-patch.yaml
│   ├── staging/           # Staging environment
│   │   ├── kustomization.yaml
│   │   └── resources-patch.yaml
│   └── production/        # Production environment
│       ├── kustomization.yaml
│       ├── replicas-patch.yaml
│       ├── hpa.yaml
│       └── pdb.yaml
└── infrastructure/         # Cluster-level resources
    ├── namespaces.yaml
    ├── network-policies.yaml
    └── rbac.yaml

Environment Promotion Strategies

Two primary approaches exist for promoting changes across environments:

Branch-per-environment: Separate Git branches represent environments. Changes promote via branch merges (dev → staging → production). Simple but can lead to merge conflicts and divergent branches.

Folder-per-environment (recommended): A single branch (main) contains folders for each environment. Kustomize overlays customize base manifests per environment. Promotion happens through commits that update higher-environment folders.

The GitOps Toolkit

Several mature tools implement GitOps principles. The two dominant options in the Kubernetes ecosystem are Argo CD and Flux CD, each with distinct philosophies and strengths.

Argo CD

Argo CD is the most popular GitOps tool, developed by Intuit and now a CNCF graduated project. It provides a rich Web UI, application-centric views, and supports Kubernetes manifests, Helm, Kustomize, and Jsonnet.

Key features: Web UI dashboard, RBAC with SSO, multi-cluster management, ApplicationSets for templating, health assessment, sync waves and hooks, and progressive delivery support via Argo Rollouts.

Flux CD

Flux is the other major GitOps controller, also a CNCF graduated project. It follows a more modular, composable architecture using the GitOps Toolkit (a set of specialized controllers). Flux is CLI-first and integrates tightly with Kubernetes custom resources.

Key features: Modular architecture, Helm Controller, Kustomize Controller, image automation (auto-update image tags), notification controller, multi-tenancy, and OCI artifact support.

Comparison Table

Other Notable Tools

• Jenkins X — Opinionated CI/CD + GitOps for Kubernetes (uses Flux under the hood)
• Rancher Fleet — GitOps at scale for multi-cluster management (100,000+ clusters)
• Codefresh (GitOps) — Commercial Argo CD platform with added analytics and security
• Weave GitOps Enterprise — Commercial Flux platform with UI, policy, and progressive delivery

GitOps for Non-Kubernetes

While GitOps originated in the Kubernetes ecosystem, its principles apply broadly to any infrastructure managed declaratively. The key requirement is a reconciliation agent that can compare desired vs actual state.

Terraform + GitOps

Tools like Terraform Cloud, Atlantis, and Spacelift implement GitOps patterns for cloud infrastructure. The Git repository contains Terraform configurations, and PRs trigger plan/apply workflows.

# Flux Terraform Controller - GitOps for Terraform
apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: Terraform
metadata:
  name: vpc-production
  namespace: flux-system
spec:
  interval: 1h
  approvePlan: auto
  path: ./terraform/vpc/production
  sourceRef:
    kind: GitRepository
    name: infrastructure
    namespace: flux-system
  writeOutputsToSecret:
    name: vpc-outputs
  varsFrom:
    - kind: Secret
      name: aws-credentials

Ansible + GitOps

For configuration management of VMs and bare metal, Ansible playbooks stored in Git can be executed on a schedule by tools like AWX/Ansible Tower or custom controllers. The reconciliation interval ensures hosts converge to desired state.

Cloud Resources

Crossplane extends Kubernetes with custom resources that represent cloud infrastructure (AWS, Azure, GCP). Combined with a GitOps controller, this enables managing cloud resources through the same Git-based workflow as applications:

• Crossplane + Argo CD — Declare RDS instances, S3 buckets, and VPCs as Kubernetes resources, managed via GitOps
• AWS Controllers for Kubernetes (ACK) — Native AWS resource management via K8s custom resources
• Azure Service Operator — Manage Azure resources declaratively within Kubernetes

GitOps Security Considerations

GitOps introduces unique security challenges. Storing all configuration in Git means secrets management becomes critical, and Git repository access controls become your deployment permissions.

Secrets Management

The fundamental problem: secrets cannot be stored in plaintext in Git (even private repos are insufficient for sensitive credentials). Three dominant approaches solve this:

1. Sealed Secrets (Bitnami)

Encrypts secrets using a cluster-side controller's public key. Only the cluster can decrypt them, making encrypted values safe to store in Git.

# Create a sealed secret safe for Git storage
kubeseal --controller-namespace kube-system \
  --controller-name sealed-secrets \
  --format yaml \
  < secret.yaml > sealed-secret.yaml

# The sealed-secret.yaml is safe to commit to Git
# Only the cluster's sealed-secrets controller can decrypt it
cat sealed-secret.yaml

2. SOPS (Mozilla)

Encrypts secret values (not keys) in YAML/JSON files using AWS KMS, GCP KMS, Azure Key Vault, or PGP. Flux has native SOPS integration.

3. External Secrets Operator

References secrets stored in external vaults (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault). The operator syncs external secrets into Kubernetes Secret objects at runtime.

# External Secrets Operator - reference secrets from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: payment-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: production/payment-service/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/payment-service/database
        property: password

RBAC & Access Control

In GitOps, Git repository permissions become deployment permissions. This means:

• Branch protection rules enforce deployment approval workflows
• CODEOWNERS files require specific team review for production changes
• Signed commits ensure only verified authors can trigger deployments
• PR policies enforce testing, security scanning, and review requirements

Audit & Compliance

GitOps naturally produces comprehensive audit trails. Every deployment is a Git commit with:

• Who made the change (commit author)
• What was changed (diff)
• When it happened (timestamp)
• Why it was made (commit message, PR description)
• Who approved it (PR reviewers)

This makes compliance frameworks (SOC 2, ISO 27001, PCI DSS) significantly easier to satisfy — auditors can review the Git log directly.

Case Studies

Intuit — Pioneer of Argo CD

Intuit (TurboTax, QuickBooks) created Argo CD to manage their migration to Kubernetes. With 4,000+ engineers and hundreds of microservices, they needed a GitOps tool that provided visibility, RBAC, and multi-cluster support.

Results: Reduced deployment time from 2 weeks to minutes. Achieved 99.99% availability. Developers deploy independently without ops bottlenecks. Argo CD was donated to the CNCF and became a graduated project.

Weaveworks — GitOps Creators

Weaveworks coined the term "GitOps" and built Flux to solve their own deployment challenges. Their experience managing production Kubernetes clusters for customers led to the core GitOps principles.

Key insight: Weaveworks found that 90% of production incidents were caused by configuration drift — changes made directly in clusters that weren't reflected in version control. GitOps eliminated this category of incidents entirely.

BMW Group

BMW adopted GitOps to manage their connected vehicle platform across multiple regions. Using Flux CD with multi-tenancy, they enable independent teams to deploy services to shared clusters while maintaining security boundaries.

Scale: 100+ microservices across 5 regions, deployed by 50+ teams. GitOps reduced their mean time to recovery (MTTR) by 70% through instant rollback via Git revert.

Conclusion & What's Next

GitOps represents a fundamental shift in how we think about deployment and infrastructure management. By making Git the single source of truth and employing continuous reconciliation, GitOps delivers:

• Auditability — Every change is tracked, reviewed, and reversible
• Security — No external credential exposure, drift is automatically corrected
• Reliability — Self-healing ensures systems always converge to desired state
• Developer experience — Deploy through familiar Git workflows (PRs, branches, merges)
• Compliance — Git history satisfies audit requirements automatically

The key architectural decisions — pull vs push, monorepo vs polyrepo, secrets management strategy — depend on your organization's scale, compliance needs, and team structure. Start simple (single config repo, folder-per-environment, Sealed Secrets) and evolve as requirements grow.