Flux Kustomization & Reconciliation — Part 2

Kustomization Deep Dive

                            
                            Name collision warning: Flux's Kustomization (apiVersion: kustomize.toolkit.fluxcd.io/v1) is a different resource from Kustomize's kustomization.yaml file. The Flux Kustomization is a CRD that controls reconciliation. A Kustomize kustomization.yaml is a file that tells kubectl kustomize how to build manifests. Flux uses both: the Flux Kustomization CRD calls kustomize-controller which runs kustomize build internally.
                        

Key Spec Fields

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: grade-api
  namespace: flux-system
spec:
  # ── SOURCE ─────────────────────────────────────────────────────
  sourceRef:
    kind: GitRepository       # Or OCIRepository
    name: grade-api
    namespace: flux-system    # Can reference cross-namespace (see below)

  # ── PATH ───────────────────────────────────────────────────────
  path: ./deploy/k8s/overlays/prod
  # If there is no kustomization.yaml at this path, Flux creates one
  # automatically from any YAML files it finds there.

  # ── RECONCILIATION INTERVAL ────────────────────────────────────
  interval: 5m       # How often to reconcile even if no new commit
  retryInterval: 1m  # How often to retry after a failure
  timeout: 3m        # Max time allowed for a single reconcile

  # ── TARGETING ──────────────────────────────────────────────────
  targetNamespace: grade-api      # Override namespace for all resources
  serviceAccountName: flux-reconciler  # Use specific SA for apply

  # ── PRUNING ────────────────────────────────────────────────────
  prune: true   # Delete K8s resources removed from Git (STRONGLY recommended)

  # ── FORCE ──────────────────────────────────────────────────────
  force: false  # If true, re-create resources that can't be updated (e.g. immutable fields)

  # ── PATCHES (inline Kustomize patches without a kustomization.yaml) ──
  patches:
    - patch: |
        - op: replace
          path: /spec/replicas
          value: 3
      target:
        kind: Deployment
        name: grade-api

  # ── CONFIG MAPS / SECRETS FROM GENERATORS ─────────────────────
  configMapGenerator:
    - name: app-config
      literals:
        - environment=production

  # ── POST-BUILD VARIABLE SUBSTITUTION ─────────────────────────
  postBuild:
    substitute:
      cluster_env: production
      cluster_region: us-east-1
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
      - kind: Secret
        name: cluster-secrets
        optional: true

Pruning & Garbage Collection

# prune: true is the GitOps guarantee — resources removed from Git
# are deleted from the cluster.

# Without prune, deleted manifests leave orphan resources:
# git rm deploy/k8s/service.yaml
# git push
# → Service still exists in cluster (stale resource)

# With prune: true:
# git rm deploy/k8s/service.yaml  
# git push
# → Flux deletes Service from cluster

# Check what Flux is tracking (its inventory)
kubectl get kustomization grade-api -n flux-system -o yaml | grep -A 20 inventory

# Resources Flux manages have a label added automatically:
# kustomize.toolkit.fluxcd.io/name: grade-api
# kustomize.toolkit.fluxcd.io/namespace: flux-system

Health Checks

Built-in Health Assessment

Flux automatically assesses health for well-known resource types without explicit configuration:

Deployment — Ready when spec.replicas == status.readyReplicas
StatefulSet — Ready when spec.replicas == status.readyReplicas
DaemonSet — Ready when status.numberReady == status.desiredNumberScheduled
Job — Ready when status.succeeded >= 1
HelmRelease — Ready when release is deployed and healthy

Custom Health Checks

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: grade-api
  namespace: flux-system
spec:
  sourceRef:
    kind: GitRepository
    name: grade-api
  path: ./deploy/k8s
  prune: true
  interval: 5m
  timeout: 5m     # Must be longer than healthCheck wait time

  # Explicit health checks — Flux waits for these to be Ready
  # before marking the Kustomization as Ready itself.
  # This is what lets depends-on work correctly.
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: grade-api
      namespace: grade-api
    - apiVersion: apps/v1
      kind: Deployment
      name: postgres
      namespace: grade-api
    - apiVersion: batch/v1
      kind: Job
      name: grade-api-migrate
      namespace: grade-api
    # For CRDs (e.g., Argo CD Application):
    - apiVersion: argoproj.io/v1alpha1
      kind: Application
      name: grade-api
      namespace: argocd

# Watch health check progression
flux get kustomization grade-api --watch
# NAME       REVISION       SUSPENDED  READY   MESSAGE
# grade-api  main/abc1234   False      False   Health check failed after 30s: Deployment/grade-api is not ready
# grade-api  main/abc1234   False      True    Applied revision: main/abc1234

# Events show health check details
kubectl describe kustomization grade-api -n flux-system
# Events:
#   Type    Reason       Message
#   Normal  ReconcileOk  Kustomization reconciled successfully
#   Warning HealthCheck  Deployment grade-api not ready: 0/1 replicas available

depends-on Ordering

Flux's dependsOn is the equivalent of Argo CD sync waves — it ensures one Kustomization is fully Ready before another starts reconciling:

# 1. Infrastructure first (CRDs, namespaces, operators)
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-crds
  namespace: flux-system
spec:
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/crds
  prune: false      # Don't prune CRDs — they're shared
  interval: 1h
  timeout: 5m

---
# 2. Infrastructure controllers (depends on CRDs being installed)
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-controllers
  namespace: flux-system
spec:
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers
  prune: true
  interval: 1h
  dependsOn:
    - name: infra-crds        # Wait for CRDs Kustomization to be Ready

---
# 3. Applications (depend on controllers being ready)
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  interval: 30m
  dependsOn:
    - name: infra-controllers  # Wait for controller Kustomization

# Cross-namespace dependsOn (Flux 2.x)
spec:
  dependsOn:
    - name: cert-manager
      namespace: cert-manager-flux   # Kustomization in another namespace

# Visualize dependency graph
flux tree kustomization flux-system
# flux-system
# └── Kustomization/flux-system
#     ├── Kustomization/infra-crds
#     ├── Kustomization/infra-controllers
#     │   └── [depends on: infra-crds]
#     └── Kustomization/apps
#         └── [depends on: infra-controllers]

Post-Build Variable Substitution

Flux's postBuild.substitute lets you write environment-agnostic YAML and inject values at apply time — without templating your YAML files:

# In your manifest (use ${VAR} syntax):
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grade-api
  namespace: ${namespace}
spec:
  replicas: ${replicas}
  template:
    spec:
      containers:
        - name: grade-api
          image: ghcr.io/your-org/grade-api:${image_tag}
          env:
            - name: ENVIRONMENT
              value: ${cluster_env}
            - name: DB_HOST
              value: ${db_host}

# ConfigMap with cluster variables (committed to Git)
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-vars
  namespace: flux-system
data:
  cluster_env: "production"
  cluster_region: "us-east-1"
  replicas: "3"
  namespace: "grade-api"
  db_host: "postgres.grade-api.svc.cluster.local"

---
# Secret with sensitive values (managed by External Secrets or Sealed Secrets)
apiVersion: v1
kind: Secret
metadata:
  name: cluster-secrets
  namespace: flux-system
stringData:
  image_tag: "v1.5.2"

# Kustomization using postBuild
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: grade-api-prod
  namespace: flux-system
spec:
  sourceRef:
    kind: GitRepository
    name: grade-api
  path: ./deploy/k8s/base    # Single base, no environment-specific overlays needed
  prune: true
  interval: 5m
  postBuild:
    # Inline substitutions (lower precedence than substituteFrom)
    substitute:
      cluster_env: production
    # Load from ConfigMap/Secret (higher precedence)
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars      # From above
      - kind: Secret
        name: cluster-secrets
        optional: true          # Don't fail if Secret doesn't exist

Multi-Environment Structure

# Recommended repo structure for multi-environment Flux
grade-api-gitops/
├── clusters/
│   ├── staging/
│   │   ├── flux-system/          # Flux bootstrap files
│   │   └── apps.yaml             # One Kustomization pointing at apps/staging
│   └── production/
│       ├── flux-system/
│       └── apps.yaml
├── infrastructure/
│   ├── crds/                     # CRD manifests
│   └── controllers/              # cert-manager, ingress-nginx, etc.
└── apps/
    ├── base/
    │   └── grade-api/
    │       ├── deployment.yaml
    │       ├── service.yaml
    │       └── kustomization.yaml   # Kustomize base
    ├── staging/
    │   └── grade-api/
    │       ├── kustomization.yaml   # Kustomize overlay
    │       └── cluster-vars.yaml    # ConfigMap with staging values
    └── production/
        └── grade-api/
            ├── kustomization.yaml
            └── cluster-vars.yaml    # ConfigMap with prod values

# clusters/production/apps.yaml — single entry point for production
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  dependsOn:
    - name: infra-controllers

# apps/production/grade-api/kustomization.yaml (Kustomize file)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base/grade-api
patchesStrategicMerge:
  - replicas-patch.yaml
images:
  - name: ghcr.io/your-org/grade-api
    newTag: v1.5.2

Exercises

                            
                            Exercise 1 — Health Check Gating: Deploy a Kustomization with a health check on a Deployment. Intentionally break the Deployment (set an invalid image). Watch the Kustomization stay NotReady. Fix the image and watch it recover. Note how the reconciliation status changes.
                        

                            
                            Exercise 2 — depends-on Chain: Create three Kustomizations: infra, middleware, and app. Set middleware to depend on infra, and app to depend on middleware. Break infra — observe that middleware and app stop reconciling. Fix infra — watch the chain recover in order.
                        

                            
                            Exercise 3 — Variable Substitution: Rewrite the grade-api Deployment manifest to use ${replicas} and ${cluster_env} placeholders. Create a ConfigMap with the values. Add postBuild.substituteFrom to the Kustomization. Verify the rendered output has the correct values substituted.
                        

Key Takeaways & Next Steps

                            
                            Key Takeaways:
                            Always enable prune: true — it's what makes GitOps actually mean the cluster matches Git
Health checks turn Kustomization readiness into a meaningful signal that downstream dependsOn can rely on
dependsOn creates ordered rollouts without sync waves — infra before apps, controllers before workloads
Post-build substitution lets you maintain a single base manifest and inject environment-specific values via ConfigMaps and Secrets
The recommended structure: clusters/ for bootstrap config, infrastructure/ for CRDs and operators, apps/ for workloads

                        

Next in This Track

In Part 3: HelmRelease & OCI Sources, we deploy and manage Helm charts declaratively with Flux's HelmRelease resource, pull charts from OCI registries, manage chart values with ConfigMaps and Secrets, and handle upgrades and rollbacks.

Cookie Consent

Flux Track Part 2: Kustomization & Reconciliation

Table of Contents

Kustomization Deep Dive

Key Spec Fields

Pruning & Garbage Collection

Health Checks

Built-in Health Assessment

Custom Health Checks

depends-on Ordering

Post-Build Variable Substitution

Multi-Environment Structure

Exercises

Key Takeaways & Next Steps

Next in This Track

Cookie Consent

Flux Track Part 2: Kustomization & Reconciliation

Table of Contents

Kustomization Deep Dive

Key Spec Fields

Pruning & Garbage Collection

Health Checks

Built-in Health Assessment

Custom Health Checks

depends-on Ordering

Post-Build Variable Substitution

Multi-Environment Structure

Exercises

Key Takeaways & Next Steps

Next in This Track

Continue the Track

Part 1: Bootstrap & Sources

Part 3: HelmRelease & OCI