Back to Distributed Systems & Kubernetes Series

Vault Track Part 4: Dynamic Secrets & PKI

June 6, 2026 Wasil Zafar 40 min read

Dynamic secrets are Vault's superpower. Instead of storing a password, Vault creates a fresh database user with a short TTL every time your service requests credentials. No long-lived passwords to rotate. The PKI engine does the same for certificates — acting as your internal CA for mutual TLS between microservices.

Table of Contents

  1. Database Secrets Engine
  2. PKI Secrets Engine
  3. cert-manager + Vault Integration
  4. Exercises
  5. Key Takeaways

Database Secrets Engine

Configure PostgreSQL

# Enable the database secrets engine
vault secrets enable database

# Configure the PostgreSQL connection
# Vault needs a "root" DB user to create/revoke users
vault write database/config/grade-api-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="grade-api-role,analytics-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.grade-api.svc.cluster.local:5432/gradedb?sslmode=require" \
  username="vault_root" \
  password="vault-root-password"

# Rotate the root credentials immediately
# Vault takes ownership of the root password; no human will know it
vault write -force database/rotate-root/grade-api-db

# Verify configuration
vault read database/config/grade-api-db

Database Roles

# Create a dynamic role — Vault runs creation_statements when credentials are requested
vault write database/roles/grade-api-role \
  db_name=grade-api-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Read-only role for analytics
vault write database/roles/analytics-role \
  db_name=grade-api-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="8h"

Generating Credentials

# Generate credentials (Vault creates a real PostgreSQL user)
vault read database/creds/grade-api-role

# Key            Value
# ---            -----
# lease_id       database/creds/grade-api-role/abc123
# lease_duration 1h0m0s
# lease_renewable true
# password       A1a-xyzabcdef12345
# username       v-root-grade-ap-abcdef123-1234567890
#
# This is a real PostgreSQL user, created right now.
# It will be automatically revoked at expiry.
# Renew a lease (extend TTL before it expires)
vault lease renew database/creds/grade-api-role/abc123

# Revoke a lease immediately (e.g., on security incident)
vault lease revoke database/creds/grade-api-role/abc123

# Revoke ALL leases from a role
vault lease revoke -prefix database/creds/grade-api-role/

# List active leases
vault list sys/leases/lookup/database/creds/grade-api-role/
# Use with Vault Agent Injector — agent auto-renews the dynamic credentials
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "grade-api"

  # Dynamic credentials endpoint (not a KV path)
  vault.hashicorp.com/agent-inject-secret-db: "database/creds/grade-api-role"
  vault.hashicorp.com/agent-inject-template-db: |
    {{- with secret "database/creds/grade-api-role" -}}
    export DB_USERNAME="{{ .Data.username }}"
    export DB_PASSWORD="{{ .Data.password }}"
    {{- end }}

  # Vault Agent will renew before expiry and send HUP to the process
  vault.hashicorp.com/agent-inject-command-db: "kill -HUP $(pidof grade-api-server)"

PKI Secrets Engine

Setup Root & Intermediate CA

# Enable PKI engines for root and intermediate CAs
vault secrets enable -path=pki pki
vault secrets enable -path=pki_int pki

# Set max TTL for the root CA
vault secrets tune -max-lease-ttl=87600h pki   # 10 years

# Generate the root CA certificate (self-signed)
vault write -field=certificate pki/root/generate/internal \
  common_name="grade-api-root-ca" \
  issuer_name="root-2026" \
  ttl=87600h > root-ca.crt

# Configure CRL and issuer URLs
vault write pki/config/urls \
  issuing_certificates="https://vault.vault.svc.cluster.local:8200/v1/pki/ca" \
  crl_distribution_points="https://vault.vault.svc.cluster.local:8200/v1/pki/crl"

# Generate the intermediate CA CSR
vault write -format=json pki_int/intermediate/generate/internal \
  common_name="grade-api-intermediate-ca" \
  issuer_name="grade-api-intermediate" \
  | jq -r '.data.csr' > intermediate.csr

# Sign the intermediate CA with root CA
vault write -format=json pki/root/sign-intermediate \
  issuer_ref="root-2026" \
  csr=@intermediate.csr \
  format=pem_bundle \
  ttl=43800h \
  | jq -r '.data.certificate' > intermediate.cert.pem

# Import the signed certificate back to the intermediate CA
vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

PKI Roles

# Create a role that defines what cert parameters are allowed
vault write pki_int/roles/grade-api-tls \
  issuer_ref="grade-api-intermediate" \
  allowed_domains="grade-api.svc.cluster.local,grade-api.grade-api.svc.cluster.local" \
  allow_subdomains=true \
  allow_localhost=false \
  max_ttl="720h" \     # 30 days max
  key_type="rsa" \
  key_bits=2048 \
  require_cn=false \   # Allow requests without CN (use SANs instead)
  server_flag=true \
  client_flag=true    # Mutual TLS — cert can authenticate as client too

Issuing Certificates

# Issue a certificate for a specific service
vault write pki_int/issue/grade-api-tls \
  common_name="grade-api.grade-api.svc.cluster.local" \
  alt_names="grade-api.grade-api.svc.cluster.local,grade-api.svc.cluster.local" \
  ttl="24h"

# Response:
# certificate: -----BEGIN CERTIFICATE-----\n...
# issuing_ca:  -----BEGIN CERTIFICATE-----\n...
# private_key: -----BEGIN RSA PRIVATE KEY-----\n...
# serial_number: 7d:f4:a2:...
# expiration: 1234567890

cert-manager + Vault Integration

# Install cert-manager (if not already installed)
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true
# Create a cert-manager Issuer that uses Vault PKI
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: grade-api
spec:
  vault:
    server: http://vault.vault.svc.cluster.local:8200
    path: pki_int/sign/grade-api-tls    # Vault PKI sign endpoint
    auth:
      kubernetes:
        role: grade-api
        mountPath: /v1/auth/kubernetes
        serviceAccountRef:
          name: grade-api-sa

---
# Request a certificate via cert-manager
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: grade-api-tls
  namespace: grade-api
spec:
  secretName: grade-api-tls-secret     # cert-manager creates this K8s Secret
  issuerRef:
    name: vault-issuer
    kind: Issuer
  dnsNames:
    - grade-api.grade-api.svc.cluster.local
  duration: 24h
  renewBefore: 8h                      # Renew 8h before expiry

Exercises

Exercise 1 — Dynamic DB Credentials: Stand up a PostgreSQL Pod in your cluster. Configure Vault's database engine with a vault_root user. Create a role with 5-minute TTL. Request credentials, connect to PostgreSQL, and list the users. Wait 5 minutes — verify the user was deleted automatically.
Exercise 2 — PKI CA Chain: Set up a root CA and intermediate CA in Vault. Issue a 1-hour certificate for test.svc.cluster.local. Use openssl to verify the cert chain. Let it expire and re-issue.
Exercise 3 — cert-manager Integration: Create a cert-manager Issuer backed by your Vault PKI. Request a Certificate. Verify the K8s Secret is created with the TLS cert. Mount it in an nginx Pod and confirm HTTPS works.

Key Takeaways

Key Takeaways:
  • Dynamic database secrets eliminate long-lived passwords — Vault creates a real DB user on demand and deletes it at TTL expiry
  • Rotate the Vault root DB password immediately after setup — no human should know it
  • Vault Agent with dynamic secrets sends renewal signals (SIGHUP) to your process — the app reloads config without restart
  • PKI engine makes Vault your internal Certificate Authority — issue short-lived certs for mTLS at scale
  • cert-manager + Vault Issuer is the production pattern — cert-manager handles cert lifecycle, renewal, and k8s Secret creation
  • Short cert TTLs (24h) and auto-renewal provide defense-in-depth — a leaked cert expires quickly