OPA/Gatekeeper Track Part 3: Audit, Mutation & Testing

Audit Mode

Gatekeeper's audit controller periodically scans all existing resources in the cluster and evaluates them against every active Constraint. Violations are recorded in the Constraint's status.violations field — even for resources that were created before the policy existed.

Audit Controller

The audit controller runs as a separate Pod (gatekeeper-audit) and scans at a configurable interval (default: 60 seconds). It does NOT block existing resources — it only reports violations.

# Check audit controller is running
kubectl get pods -n gatekeeper-system -l control-plane=audit-controller
# NAME                                READY   STATUS    RESTARTS   AGE
# gatekeeper-audit-6b9f8d7c5b-xk2lm  1/1     Running   0          2h

# Configure audit interval via Helm values
# helm upgrade gatekeeper gatekeeper/gatekeeper \
#   --namespace gatekeeper-system \
#   --set auditInterval=30 \
#   --set auditMatchKindOnly=true

Querying Violations

Every Constraint stores its audit violations in status.violations. Use kubectl or jq to inspect them:

# List all violations across ALL constraints
kubectl get constraints -o json | jq '.items[] | {
  name: .metadata.name,
  kind: .kind,
  totalViolations: .status.totalViolations,
  violations: [.status.violations[]? | {
    resource: .name,
    namespace: .namespace,
    message: .message
  }]
}'

# Query violations for a specific constraint
kubectl get k8srequiredlabels require-team-label -o json | jq '.status.violations'
# [
#   {
#     "enforcementAction": "dryrun",
#     "group": "apps",
#     "kind": "Deployment",
#     "message": "Resource nginx-legacy is missing required labels: {\"team\"}",
#     "name": "nginx-legacy",
#     "namespace": "default",
#     "version": "v1"
#   },
#   {
#     "enforcementAction": "dryrun",
#     "group": "apps",
#     "kind": "Deployment",
#     "message": "Resource redis-cache is missing required labels: {\"cost-center\", \"team\"}",
#     "name": "redis-cache",
#     "namespace": "backend",
#     "version": "v1"
#   }
# ]

# Count total violations per constraint
kubectl get constraints -o json | jq '.items[] | "\(.metadata.name): \(.status.totalViolations // 0) violations"'
# "require-team-label: 12 violations"
# "prod-allowed-registries: 3 violations"
# "deny-privileged: 0 violations"

# Export violations to CSV for reporting
kubectl get constraints -o json | jq -r '
  .items[] | .metadata.name as $constraint |
  .status.violations[]? |
  [$constraint, .namespace, .kind, .name, .message] | @csv
' > violations-report.csv

Mutation Policies

Gatekeeper supports mutation webhooks that automatically modify resources during admission. Instead of rejecting non-compliant resources, mutation policies fix them. Gatekeeper provides two mutation CRDs:

• AssignMetadata — Adds or overwrites labels and annotations on resource metadata
• Assign — Sets or overrides any field in the resource spec (e.g., imagePullPolicy, securityContext)

AssignMetadata

AssignMetadata automatically adds labels or annotations to matching resources. This is commonly used to enforce organizational tagging without relying on developers to remember every required label.

# assign-metadata-team-label.yaml
# Automatically add a "managed-by: gatekeeper" label to all Deployments
apiVersion: mutations.gatekeeper.sh/v1
kind: AssignMetadata
metadata:
  name: add-managed-by-label
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment", "StatefulSet"]
    # Only apply in namespaces with this label
    namespaceSelector:
      matchLabels:
        policy-managed: "true"
  location: "metadata.labels.managed-by"
  parameters:
    assign:
      value: "gatekeeper"

# assign-metadata-cost-center.yaml
# Add a default cost-center annotation if not already present
apiVersion: mutations.gatekeeper.sh/v1
kind: AssignMetadata
metadata:
  name: default-cost-center
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
    excludedNamespaces:
      - "kube-system"
      - "gatekeeper-system"
  location: "metadata.annotations.cost-center"
  parameters:
    assign:
      value: "shared-platform"

# Apply the mutation policies
kubectl apply -f assign-metadata-team-label.yaml
kubectl apply -f assign-metadata-cost-center.yaml

# Verify: create a Deployment and inspect its labels
kubectl create deployment test-mutation --image=nginx -n default
kubectl get deployment test-mutation -o jsonpath='{.metadata.labels}' | jq .
# {
#   "app": "test-mutation",
#   "managed-by": "gatekeeper"
# }

kubectl get deployment test-mutation -o jsonpath='{.metadata.annotations}' | jq .
# {
#   "cost-center": "shared-platform"
# }

Assign

Assign modifies any field in the resource spec. Common use cases include forcing imagePullPolicy: Always, injecting security contexts, or setting resource limits.

# assign-image-pull-policy.yaml
# Force imagePullPolicy: Always on all containers in production
apiVersion: mutations.gatekeeper.sh/v1
kind: Assign
metadata:
  name: force-always-pull
spec:
  applyTo:
    - groups: ["apps"]
      kinds: ["Deployment", "StatefulSet"]
      versions: ["v1"]
  match:
    scope: Namespaced
    namespaces:
      - "production"
      - "staging"
    excludedNamespaces:
      - "kube-system"
  location: "spec.template.spec.containers[name:*].imagePullPolicy"
  parameters:
    assign:
      value: "Always"

# assign-security-context.yaml
# Inject runAsNonRoot: true into all Pods
apiVersion: mutations.gatekeeper.sh/v1
kind: Assign
metadata:
  name: force-non-root
spec:
  applyTo:
    - groups: [""]
      kinds: ["Pod"]
      versions: ["v1"]
    - groups: ["apps"]
      kinds: ["Deployment", "StatefulSet"]
      versions: ["v1"]
  match:
    scope: Namespaced
    excludedNamespaces:
      - "kube-system"
      - "gatekeeper-system"
  location: "spec.template.spec.securityContext.runAsNonRoot"
  parameters:
    assign:
      value: true

# Apply the Assign policies
kubectl apply -f assign-image-pull-policy.yaml
kubectl apply -f assign-security-context.yaml

# Test: create a Deployment in production namespace without specifying imagePullPolicy
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mutated-app
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mutated-app
  template:
    metadata:
      labels:
        app: mutated-app
    spec:
      containers:
        - name: app
          image: nginx:1.25
EOF

# Verify mutation was applied
kubectl get deployment mutated-app -n production -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}'
# Always

kubectl get deployment mutated-app -n production -o jsonpath='{.spec.template.spec.securityContext.runAsNonRoot}'
# true

conftest CI Testing

conftest is an open-source tool that runs OPA/Rego policies against structured data files (YAML, JSON, HCL, Dockerfile, etc.) without needing a Kubernetes cluster. This lets you shift policy testing left — catching violations in CI before resources ever reach the cluster.

Install & Usage

# Install conftest (macOS/Linux)
brew install conftest

# Or download binary directly
curl -L https://github.com/open-policy-agent/conftest/releases/download/v0.46.0/conftest_0.46.0_Linux_x86_64.tar.gz | tar xz
sudo mv conftest /usr/local/bin/

# Verify installation
conftest --version
# Version: 0.46.0

Write Rego policies in a policy/ directory. conftest looks for the deny or violation rule in the main package by default:

# Create policy directory
mkdir -p policy/

# Write a policy file
cat > policy/deployment.rego <<'EOF'
package main

# Deny deployments without resource limits
deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.resources.limits
  msg := sprintf("Container '%s' must have resource limits defined", [container.name])
}

# Deny images using :latest tag
deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  endswith(container.image, ":latest")
  msg := sprintf("Container '%s' uses :latest tag — pin to a specific version", [container.name])
}

# Deny containers without image tag (implies :latest)
deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not contains(container.image, ":")
  msg := sprintf("Container '%s' image has no tag — pin to a specific version", [container.name])
}

# Require 'team' label on Deployments
deny[msg] {
  input.kind == "Deployment"
  not input.metadata.labels.team
  msg := "Deployment must have a 'team' label"
}
EOF

# deployment.yaml — a non-compliant manifest to test against
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: web
          image: nginx:latest
        - name: sidecar
          image: busybox

# Run conftest against the deployment manifest
conftest test deployment.yaml --policy policy/

# FAIL - deployment.yaml - main - Container 'web' uses :latest tag — pin to a specific version
# FAIL - deployment.yaml - main - Container 'sidecar' image has no tag — pin to a specific version
# FAIL - deployment.yaml - main - Container 'web' must have resource limits defined
# FAIL - deployment.yaml - main - Container 'sidecar' must have resource limits defined
# FAIL - deployment.yaml - main - Deployment must have a 'team' label
#
# 5 tests, 0 passed, 0 warnings, 5 failures

# Test multiple files at once
conftest test manifests/*.yaml --policy policy/

# Output in structured format for CI tooling
conftest test deployment.yaml --policy policy/ --output json

# Test only specific policies using --namespace
conftest test deployment.yaml --policy policy/ --namespace main

# Use --fail-on-warn to also fail on warn[] rules
conftest test deployment.yaml --policy policy/ --fail-on-warn

CI Pipeline Integration

Integrate conftest into GitHub Actions, GitLab CI, or any CI system to block non-compliant manifests before they reach the cluster:

# .github/workflows/policy-check.yaml
name: Policy Check
on:
  pull_request:
    paths:
      - 'k8s/**'
      - 'policy/**'

jobs:
  conftest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install conftest
        run: |
          curl -L https://github.com/open-policy-agent/conftest/releases/download/v0.46.0/conftest_0.46.0_Linux_x86_64.tar.gz | tar xz
          sudo mv conftest /usr/local/bin/

      - name: Run policy tests
        run: |
          conftest test k8s/*.yaml --policy policy/ --output github
          # --output github formats results as GitHub annotations

      - name: Verify Rego unit tests
        run: |
          # Run OPA's built-in test framework on policy files
          opa test policy/ -v

# Write Rego unit tests alongside policies
cat > policy/deployment_test.rego <<'EOF'
package main

# Test: Deployment with :latest tag should be denied
test_deny_latest_tag {
  input := {
    "kind": "Deployment",
    "metadata": {"name": "test", "labels": {"team": "eng"}},
    "spec": {"template": {"spec": {"containers": [
      {"name": "web", "image": "nginx:latest", "resources": {"limits": {"cpu": "100m"}}}
    ]}}}
  }
  count(deny) > 0
}

# Test: Deployment with pinned tag should pass
test_allow_pinned_tag {
  input := {
    "kind": "Deployment",
    "metadata": {"name": "test", "labels": {"team": "eng"}},
    "spec": {"template": {"spec": {"containers": [
      {"name": "web", "image": "nginx:1.25.3", "resources": {"limits": {"cpu": "100m"}}}
    ]}}}
  }
  count(deny) == 0
}
EOF

# Run Rego unit tests
opa test policy/ -v
# policy/deployment_test.rego:
# data.main.test_deny_latest_tag: PASS (1.2ms)
# data.main.test_allow_pinned_tag: PASS (0.8ms)
# ----------------
# PASS: 2/2

Enforcement Actions

dryrun vs warn vs deny

Gatekeeper supports three enforcement modes per Constraint. Choose the right one based on your rollout stage:

# enforcementAction: dryrun
# Resource is ALLOWED. Violations appear ONLY in constraint status.violations[]
# and in audit logs. User sees NO feedback at admission time.
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-labels-dryrun
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
  parameters:
    labels: ["team", "cost-center"]

# enforcementAction: warn
# Resource is ALLOWED. User sees a WARNING message in kubectl output.
# Violations also appear in status.violations[].
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-labels-warn
spec:
  enforcementAction: warn
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
  parameters:
    labels: ["team", "cost-center"]

# kubectl output when creating a non-compliant Deployment:
# Warning: [require-labels-warn] Resource my-app is missing required labels: {"team"}
# deployment.apps/my-app created    <-- resource IS created despite warning

# enforcementAction: deny (default)
# Resource is REJECTED. User sees an error. Resource is NOT created.
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-labels-deny
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
  parameters:
    labels: ["team", "cost-center"]

# kubectl output when creating a non-compliant Deployment:
# Error from server (Forbidden): admission webhook "validation.gatekeeper.sh"
# denied the request: [require-labels-deny] Resource my-app is missing
# required labels: {"cost-center", "team"}

Safe Rollout Strategy

The recommended approach to deploying new policies safely follows a three-phase progression:

# Phase 1: DRYRUN — Deploy constraint in audit-only mode
# Duration: 1-2 weeks
# Goal: Discover all existing violations without impacting anyone
kubectl apply -f - <<EOF
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-team-label
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
  parameters:
    labels: ["team"]
EOF

# Monitor violations
watch kubectl get k8srequiredlabels require-team-label \
  -o jsonpath='{.status.totalViolations}'

# Phase 2: WARN — Notify developers without blocking
# Duration: 1-2 weeks
# Goal: Give teams time to fix violations with clear warnings
kubectl patch k8srequiredlabels require-team-label \
  --type='merge' -p '{"spec":{"enforcementAction":"warn"}}'

# Phase 3: DENY — Enforce the policy
# Timing: After violations reach 0 (or acceptable threshold)
kubectl patch k8srequiredlabels require-team-label \
  --type='merge' -p '{"spec":{"enforcementAction":"deny"}}'

# Verify enforcement is active
kubectl get k8srequiredlabels require-team-label \
  -o jsonpath='{.spec.enforcementAction}'
# deny

Exercises

Key Takeaways