Part 20: Secrets Management — Vault & Key Rotation

The Secrets Problem

Every production system needs secrets — database passwords, API keys, TLS certificates, SSH keys, encryption keys, OAuth tokens. The challenge isn't having secrets; it's managing them securely across hundreds of services, multiple environments, and dozens of engineers without creating a single point of compromise.

A proper secrets management strategy addresses five concerns: storage (where secrets live at rest), distribution (how they reach applications), rotation (changing secrets regularly), access control (who/what can read which secrets), and auditing (logging every access for forensics).

Anti-Patterns

Hardcoded Secrets

The worst anti-pattern: secrets embedded directly in source code. They end up in version control, container images, CI logs, and every developer's laptop. A single git clone gives an attacker everything.

# Detect hardcoded secrets in your codebase using truffleHog
# Scans entire git history for high-entropy strings and known patterns
pip install trufflehog
trufflehog git file://. --only-verified

# Alternative: use gitleaks (faster, regex-based)
# Install: https://github.com/gitleaks/gitleaks
gitleaks detect --source . --verbose

# GitHub also scans for secrets automatically (secret scanning)
# Enable in repo Settings → Code security and analysis

Env Vars

Environment variables are better than hardcoding but still problematic. They're visible in /proc/<pid>/environ, leaked in crash dumps, logged by debugging tools, inherited by child processes, and often dumped in CI output. They provide no rotation, no auditing, and no access control.

# Environment variables are NOT secret on a Linux system
# Any process running as the same user can read them

# View env vars of any running process (same user)
cat /proc/$(pgrep -f "myapp")/environ | tr '\0' '\n' | grep -i password

# Docker containers: env vars visible via inspect
docker inspect mycontainer --format '{{json .Config.Env}}' | python3 -m json.tool

# Kubernetes: env vars in pod spec are stored in etcd (unencrypted by default)
kubectl get pod mypod -o jsonpath='{.spec.containers[0].env}' | python3 -m json.tool

Git Commits

Even if you delete a secret from the current branch, it lives forever in git history. A force-push doesn't help if anyone has fetched the commit. You must rotate the secret immediately — consider it compromised.

# Search entire git history for secrets
git log -p --all -S "AKIA"          # AWS access key pattern
git log -p --all -S "password"      # Generic password strings
git log -p --all --diff-filter=D -- "*.env"   # Deleted .env files

# Remove a secret from git history (STILL ROTATE THE SECRET)
# Using BFG Repo-Cleaner (faster than filter-branch)
java -jar bfg.jar --replace-text passwords.txt my-repo.git
cd my-repo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push --force

# Pre-commit hook to prevent secrets from being committed
# .pre-commit-config.yaml
# repos:
#   - repo: https://github.com/gitleaks/gitleaks
#     rev: v8.18.0
#     hooks:
#       - id: gitleaks

HashiCorp Vault

Architecture

HashiCorp Vault is the industry-standard secrets management tool. It provides a unified interface to any secret while providing tight access control and a detailed audit log. Vault encrypts secrets at rest, requires authentication for every operation, and can generate dynamic short-lived credentials on demand.

flowchart TD
    C["Client\n(App / CI / Human)"] --> AM["Auth Method\n(Token, K8s SA, OIDC, AppRole)"]
    AM --> VS["Vault Server\n(Policy Engine + Audit Log)"]
    VS --> SE["Secret Engine\n(KV, Database, PKI, Transit)"]
    SE --> SB["Storage Backend\n(Consul, Raft, S3, DynamoDB)"]

    VS --> AL["Audit Log\n(File, Syslog, Socket)"]
    VS --> P["Policies\n(path-based ACLs)"]
            

# Start Vault in dev mode (in-memory, unsealed, root token = "root")
vault server -dev -dev-root-token-id="root"

# In another terminal, configure client
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'

# Check Vault status
vault status
# Key             Value
# Seal Type       shamir
# Initialized     true
# Sealed          false   ← dev mode auto-unseals
# Total Shares    1
# Version         1.15.x

# Write a secret to the KV (Key-Value) engine
vault kv put secret/myapp/database \
    username="app_user" \
    password="s3cr3t-p@ssw0rd" \
    host="db.internal:5432"

# Read the secret back
vault kv get secret/myapp/database
vault kv get -field=password secret/myapp/database

Secret Engines

Vault's power comes from secret engines — pluggable backends that generate, store, or encrypt data. The KV engine stores static secrets, but the database and PKI engines generate dynamic credentials with automatic revocation.

# Enable the database secret engine
vault secrets enable database

# Configure a PostgreSQL connection
vault write database/config/mydb \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/mydb?sslmode=require" \
    allowed_roles="readonly" \
    username="vault_admin" \
    password="vault-admin-password"

# Create a role that generates short-lived credentials
vault write database/roles/readonly \
    db_name=mydb \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# Generate dynamic credentials (unique per request, auto-revoked after TTL)
vault read database/creds/readonly
# Key                Value
# lease_id           database/creds/readonly/abc123
# lease_duration     1h
# username           v-token-readonly-xyz789
# password           A1B2-c3D4-e5F6-g7H8

Auth Methods

Vault supports multiple authentication methods — each suited to different environments. The goal: prove your identity without a shared secret (chicken-and-egg problem).

# Enable Kubernetes auth (pods authenticate via service account JWT)
vault auth enable kubernetes

vault write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Create a role binding a K8s service account to a Vault policy
vault write auth/kubernetes/role/myapp \
    bound_service_account_names=myapp-sa \
    bound_service_account_namespaces=production \
    policies=myapp-secrets \
    ttl=1h

# Enable AppRole auth (for CI/CD pipelines)
vault auth enable approle

vault write auth/approle/role/ci-pipeline \
    secret_id_ttl=10m \
    token_ttl=20m \
    token_max_ttl=30m \
    policies=ci-secrets

# Write a policy (path-based ACL)
vault policy write myapp-secrets - <<EOF
path "secret/data/myapp/*" {
    capabilities = ["read", "list"]
}
path "database/creds/readonly" {
    capabilities = ["read"]
}
EOF

Cloud KMS & Secret Stores

Every major cloud provider offers managed secrets services. They integrate with IAM, provide automatic rotation for some secret types, and eliminate the operational burden of running Vault infrastructure.

# Kubernetes secrets are just base64 — NOT encryption
echo -n "my-password" | base64
# bXktcGFzc3dvcmQ=

# Create a K8s secret
kubectl create secret generic db-creds \
    --from-literal=username=app_user \
    --from-literal=password=s3cr3t-p@ss

# Decode a K8s secret (trivial)
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
# s3cr3t-p@ss  ← anyone with RBAC access can read this

# Enable etcd encryption at rest (kube-apiserver config)
# /etc/kubernetes/encryption-config.yaml:
# apiVersion: apiserver.config.k8s.io/v1
# kind: EncryptionConfiguration
# resources:
#   - resources: [secrets]
#     providers:
#       - aescbc:
#           keys:
#             - name: key1
#               secret: 
#       - identity: {}

Secret Rotation

Secret rotation limits the blast radius of a compromise. If credentials are rotated every hour, a leaked secret is usable for at most 60 minutes. Vault's dynamic secrets solve this elegantly — each request generates a unique credential with a TTL, automatically revoked on expiry.

flowchart LR
    App["Application"] --> V["Vault"]
    V --> DS["Dynamic Secret Engine"]
    DS --> DB["Database"]
    DB --> Cred["Unique Credentials\n(username + password)\nTTL: 1 hour"]
    Cred --> App
    Cred -.->|"TTL expires"| Rev["Auto-Revocation\n(DROP ROLE)"]
    Rev --> DB
            

# Vault lease management — check and revoke dynamic secrets
vault list sys/leases/lookup/database/creds/readonly

# Revoke a specific lease immediately
vault lease revoke database/creds/readonly/abc123

# Revoke ALL leases for a path (emergency rotation)
vault lease revoke -prefix database/creds/readonly

# Force-rotate a root credential (Vault generates new password)
vault write -force database/rotate-root/mydb

# AWS Secrets Manager: configure automatic rotation
aws secretsmanager rotate-secret \
    --secret-id prod/myapp/db-password \
    --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:rotate-db \
    --rotation-rules '{"AutomaticallyAfterDays": 30}'

# View rotation status
aws secretsmanager describe-secret --secret-id prod/myapp/db-password \
    --query '{LastRotated: LastRotatedDate, NextRotation: NextRotationDate}'

Sealed Secrets & GitOps

In GitOps workflows, all configuration lives in git — but you can't commit plaintext secrets. Sealed Secrets (Bitnami) and SOPS (Mozilla) solve this by encrypting secrets that can only be decrypted by the target cluster or specific KMS keys.

# === Sealed Secrets (Bitnami) ===
# Install the controller in your cluster
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Fetch the public key (used to encrypt — safe to distribute)
kubeseal --fetch-cert > pub-cert.pem

# Create a sealed secret from a regular secret
kubectl create secret generic db-creds \
    --from-literal=password=s3cr3t --dry-run=client -o yaml | \
    kubeseal --cert pub-cert.pem -o yaml > sealed-db-creds.yaml

# sealed-db-creds.yaml is safe to commit to git
# Only the controller in the cluster can decrypt it
kubectl apply -f sealed-db-creds.yaml
# Controller decrypts → creates regular Secret in the namespace

# === SOPS (Secrets OPerationS) — encrypt values in YAML/JSON/ENV files ===
# Install: brew install sops (or download from GitHub)

# Configure SOPS to use AWS KMS for encryption
# .sops.yaml (in repo root):
# creation_rules:
#   - path_regex: .*secrets.*\.yaml$
#     kms: 'arn:aws:kms:us-east-1:123456789:key/abc-def-123'

# Encrypt a file (only values are encrypted, keys remain readable)
sops --encrypt secrets.yaml > secrets.enc.yaml

# Edit encrypted file in place (decrypts → editor → re-encrypts)
sops secrets.enc.yaml

# Decrypt at deploy time (requires KMS access)
sops --decrypt secrets.enc.yaml | kubectl apply -f -

# SOPS supports: AWS KMS, GCP KMS, Azure Key Vault, age, PGP
# Multiple keys = multiple decryptors (team + CI + production)

# External Secrets Operator — example SecretStore + ExternalSecret

# 1. Install ESO via Helm
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

# 2. Create a SecretStore pointing to AWS Secrets Manager
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
EOF

# 3. Create an ExternalSecret that syncs a remote secret into K8s
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
  namespace: production
spec:
  refreshInterval: 5m     # Re-sync every 5 minutes (catches rotations)
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-creds        # K8s Secret name created by ESO
  data:
    - secretKey: username
      remoteRef:
        key: prod/myapp/db
        property: username
    - secretKey: password
      remoteRef:
        key: prod/myapp/db
        property: password
EOF

# 4. Verify the synced secret
kubectl get externalsecret db-creds -n production
# STATUS: SecretSynced

Exercises

# Exercise 1: Run Vault in dev mode and store/retrieve a secret
vault server -dev -dev-root-token-id="root" &
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'
vault kv put secret/exercise api_key="test-key-12345"
vault kv get -field=api_key secret/exercise

# Exercise 2: Scan your repo for leaked secrets
# Install gitleaks: https://github.com/gitleaks/gitleaks/releases
gitleaks detect --source . --verbose --no-git

# Exercise 3: Create and decode a Kubernetes secret
kubectl create secret generic test-secret \
    --from-literal=token="super-secret" --dry-run=client -o yaml
# Notice the base64 value — decode it:
echo "c3VwZXItc2VjcmV0" | base64 -d

# Exercise 4: Encrypt a file with SOPS and age
# Install age: https://github.com/FiloSottile/age
age-keygen -o key.txt
export SOPS_AGE_RECIPIENTS=$(grep "public key" key.txt | awk '{print $NF}')
echo "db_password: s3cr3t" > secrets.yaml
sops --encrypt secrets.yaml > secrets.enc.yaml
cat secrets.enc.yaml   # Values encrypted, keys readable
SOPS_AGE_KEY_FILE=key.txt sops --decrypt secrets.enc.yaml

Conclusion & Next Steps

Secrets management is a foundational security practice: hardcoded secrets and plain environment variables are insufficient for production. HashiCorp Vault provides dynamic, short-lived credentials with audit logging and fine-grained policies. Cloud KMS services offer managed alternatives with native IAM integration. Secret rotation — especially Vault's dynamic secrets — limits breach impact to the TTL window. Sealed Secrets and SOPS enable GitOps without committing plaintext. The External Secrets Operator bridges cloud secret stores with Kubernetes, and auto-refresh ensures rotated credentials propagate without manual intervention.