App-of-Apps Pattern
The Root Application
The App-of-Apps pattern uses Argo CD to manage Argo CD Applications. A "root" Application points to a Git directory containing other Application YAML files. When Argo CD syncs the root Application, it creates all child Applications, which then sync their respective workloads:
flowchart TD
ROOT[Root Application\n apps/root-app.yaml] --> INFRA[infra-apps\n cert-manager, ingress-nginx]
ROOT --> PLATFORM[platform-apps\n prometheus, vault, eso]
ROOT --> APP1[grade-api-dev]
ROOT --> APP2[grade-api-prod]
ROOT --> APP3[notification-service-prod]
INFRA --> CM[cert-manager]
INFRA --> NGINX[ingress-nginx]
PLATFORM --> PROM[prometheus-stack]
PLATFORM --> VAULT[vault]
APP1 --> K8S1[Kubernetes: dev ns]
APP2 --> K8S2[Kubernetes: production ns]
APP3 --> K8S3[Kubernetes: production ns]
style ROOT fill:#e8f4f4,stroke:#3B9797
style INFRA fill:#f0f0ff,stroke:#16476A
style PLATFORM fill:#f0f0ff,stroke:#16476A
# GitOps repo structure for App-of-Apps
cluster-gitops/
├── apps/ ← Root Application points here
│ ├── infra/
│ │ ├── cert-manager.yaml
│ │ └── ingress-nginx.yaml
│ ├── platform/
│ │ ├── prometheus.yaml
│ │ └── vault.yaml
│ └── workloads/
│ ├── grade-api-dev.yaml
│ └── grade-api-prod.yaml
├── bootstrap/
│ └── root-app.yaml ← Apply once to bootstrap
└── README.md
# bootstrap/root-app.yaml — apply once to start everything
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
source:
repoURL: https://github.com/your-org/cluster-gitops
targetRevision: HEAD
path: apps # Points to the apps/ directory
destination:
server: https://kubernetes.default.svc
namespace: argocd # Child Applications go in argocd namespace
syncPolicy:
automated:
prune: true
selfHeal: true
# apps/workloads/grade-api-prod.yaml
# One of the child Applications managed by the root
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: grade-api-prod
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
sources:
- repoURL: oci://ghcr.io/your-org/charts
chart: grade-api
targetRevision: 0.2.0
helm:
releaseName: grade-api
valueFiles:
- $values/environments/production/values.yaml
- repoURL: https://github.com/your-org/cluster-gitops
targetRevision: HEAD
ref: values
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Cluster Bootstrap
# Bootstrap a brand-new cluster — just 3 commands:
# 1. Install Argo CD
kubectl apply -n argocd \
-f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# 2. Wait for Argo CD to be ready
kubectl wait --for=condition=available \
deployment/argocd-server \
-n argocd \
--timeout=300s
# 3. Apply the root Application — Argo CD takes over from here
kubectl apply -f bootstrap/root-app.yaml
# Everything else is automated:
# - Argo CD syncs the root app
# - Root app creates all child Applications
# - Each child Application deploys its workloads
# - Total time to full cluster: depends on workloads, typically 5-15 min
ApplicationSets
ApplicationSets are a more powerful evolution of App-of-Apps. Instead of manually writing Application YAML files, you define a template and a generator. The generator produces parameter sets; the template is rendered once per parameter set:
Git Directory Generator
# Automatically create one Application per directory in the repo
# When you add a new directory, Argo CD auto-creates the Application
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: cluster-addons
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/your-org/cluster-gitops
revision: HEAD
directories:
- path: "addons/*" # Match all dirs under addons/
- path: "addons/excluded" # Exclude specific dirs
exclude: true
template:
metadata:
name: '{{path.basename}}' # App name = directory name
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
source:
repoURL: https://github.com/your-org/cluster-gitops
targetRevision: HEAD
path: '{{path}}' # Use the matched directory path
destination:
server: https://kubernetes.default.svc
namespace: '{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
List Generator
# Generate Applications from a static list of parameter sets
# Great for: deploying the same chart to multiple environments
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: grade-api-environments
namespace: argocd
spec:
generators:
- list:
elements:
- environment: dev
namespace: grade-api-dev
cluster: https://kubernetes.default.svc
imageTag: latest
replicas: "1"
- environment: staging
namespace: grade-api-staging
cluster: https://staging-cluster.example.com
imageTag: "1.2.0-rc1"
replicas: "2"
- environment: production
namespace: production
cluster: https://prod-cluster.example.com
imageTag: "1.2.0"
replicas: "4"
template:
metadata:
name: 'grade-api-{{environment}}'
namespace: argocd
spec:
source:
repoURL: oci://ghcr.io/your-org/charts
chart: grade-api
targetRevision: 0.2.0
helm:
releaseName: grade-api
parameters:
- name: replicaCount
value: '{{replicas}}'
- name: image.tag
value: '{{imageTag}}'
valueFiles:
- $values/environments/{{environment}}/values.yaml
sources: [] # Note: multi-source requires different syntax
destination:
server: '{{cluster}}'
namespace: '{{namespace}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Matrix Generator
# Matrix generator: cross-product of two generators
# Example: deploy every application to every cluster
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: all-apps-all-clusters
namespace: argocd
spec:
generators:
- matrix:
generators:
# Generator 1: clusters
- clusters:
selector:
matchLabels:
env: production # Only production clusters
# Generator 2: Git directories (one per application)
- git:
repoURL: https://github.com/your-org/cluster-gitops
revision: HEAD
directories:
- path: "workloads/*"
template:
metadata:
name: '{{name}}-{{path.basename}}' # cluster-app naming
namespace: argocd
spec:
source:
repoURL: https://github.com/your-org/cluster-gitops
targetRevision: HEAD
path: '{{path}}'
destination:
server: '{{server}}'
namespace: '{{path.basename}}'
Pull Request Generator
# Deploy every open PR to its own preview environment
# Enables "environment per PR" for testing before merge
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: grade-api-pr-preview
namespace: argocd
spec:
generators:
- pullRequest:
github:
owner: your-org
repo: grade-api
# Only PRs with this label get a preview environment
labels:
- preview
tokenRef:
secretName: github-token
key: token
requeueAfterSeconds: 30
template:
metadata:
name: 'grade-api-pr-{{number}}'
namespace: argocd
spec:
source:
repoURL: https://github.com/your-org/grade-api-gitops
targetRevision: '{{branch}}' # Deploy the PR's branch
path: helm/grade-api
helm:
parameters:
- name: image.tag
value: '{{head_sha}}' # Image built from PR commit
- name: ingress.host
value: 'pr-{{number}}.preview.example.com'
destination:
server: https://kubernetes.default.svc
namespace: 'preview-pr-{{number}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Sync Waves
Wave Annotations
Sync waves solve ordering problems within a single sync. Argo CD applies resources in wave order (lowest to highest number), waiting for each wave to be Healthy before proceeding:
# Resources with no annotation get wave 0 by default
# Wave -5: Namespaces must exist before anything else
apiVersion: v1
kind: Namespace
metadata:
name: grade-api
annotations:
argocd.argoproj.io/sync-wave: "-5"
# Wave -3: CRDs must be installed before their instances
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: grades.gradeapi.example.com
annotations:
argocd.argoproj.io/sync-wave: "-3"
# Wave -1: Secrets and ConfigMaps before Deployments
apiVersion: v1
kind: Secret
metadata:
name: grade-api-db-secret
annotations:
argocd.argoproj.io/sync-wave: "-1"
# Wave 0 (default): Core application resources
apiVersion: apps/v1
kind: Deployment
metadata:
name: grade-api
# No annotation = wave 0
# Wave 2: Services after Pods are ready
apiVersion: v1
kind: Service
metadata:
name: grade-api
annotations:
argocd.argoproj.io/sync-wave: "2"
# Wave 5: Ingress after Service is ready
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grade-api
annotations:
argocd.argoproj.io/sync-wave: "5"
Dependency Ordering in App-of-Apps
# Use sync waves in child Application manifests
# to order which applications sync first
# Wave -10: Infrastructure apps first (cert-manager, CRDs)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cert-manager
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "-10"
spec:
# ...
# Wave -5: Platform apps after infra (ingress-nginx needs cert-manager)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ingress-nginx
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "-5"
spec:
# ...
# Wave 0: Application workloads last
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: grade-api-prod
namespace: argocd
# No annotation = wave 0
spec:
# ...
Sync Phases
Sync phases and sync waves work together. Phases are the outer structure (PreSync, Sync, PostSync), and waves are the ordering within each phase:
# Complete sync ordering — from first to last:
#
# Phase: PreSync (all resources with hook: PreSync)
# → Wave -10: Pre-sync data backup job
# → Wave -5: Pre-sync migration validation job
# → Wave 0: Main pre-sync job (migration)
#
# Phase: Sync (all non-hook resources, ordered by wave)
# → Wave -5: Namespaces, CRDs
# → Wave -1: Secrets, ConfigMaps
# → Wave 0: Deployments, StatefulSets
# → Wave 5: Services, HPA
# → Wave 10: Ingress, NetworkPolicies
#
# Phase: PostSync (all resources with hook: PostSync)
# → Wave 0: Health verification job
# → Wave 5: Notification job
# Annotate a resource for a specific phase AND wave:
metadata:
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/sync-wave: "-5"
argocd.argoproj.io/hook-delete-policy: HookSucceeded
Exercises
apps/ directory containing 3 Application YAML files (nginx, redis, grade-api). Create a root Application pointing to the apps/ path. Apply only the root Application and watch Argo CD automatically create and sync all 3 child Applications.
argocd app get --watch.
Key Takeaways & Next Steps
- App-of-Apps: Applications are code in Git, enabling cluster bootstrap from a single kubectl apply
- ApplicationSets eliminate manual Application maintenance — generators produce Applications automatically
- Git directory generator: one Application per directory — adding a directory = new Application
- Pull request generator enables "environment per PR" preview deployments
- Sync waves order resource application within a sync — lower waves apply before higher ones
- Combine wave annotations on child Application manifests to order infrastructure before workloads
Next in This Track
In Part 4: RBAC & Projects, we add multi-tenancy to Argo CD — AppProjects for access boundaries, RBAC for team permissions, SSO integration with GitHub/Okta, and the declarative user management patterns used by platform teams.