Back to Distributed Systems & Kubernetes Series

Vault Track Part 1: Concepts & Install on Kubernetes

June 6, 2026 Wasil Zafar 40 min read

HashiCorp Vault is the industry-standard secret management platform. It stores, manages, and controls access to tokens, passwords, certificates, and encryption keys — with full audit logging. Running it on Kubernetes with Helm gives you HA Vault integrated with your cluster's identity system.

Table of Contents

  1. Vault Core Concepts
  2. Install on Kubernetes
  3. Initialize & Unseal
  4. Writing & Reading Secrets
  5. Vault CLI Essentials
  6. Exercises
  7. Key Takeaways & Next Steps

Vault Core Concepts

Secrets Engines

Vault's functionality is organized into pluggable backends called secrets engines. Each engine handles a different type of secret:

  • KV v2 (Key-Value) — Store arbitrary key-value secrets with versioning. Most commonly used. Secrets are static — you put them in, Vault stores them.
  • Database — Dynamically generate short-lived database credentials. Vault creates a DB user on-demand when requested, with a configurable TTL. No static passwords to rotate.
  • PKI — Issue X.509 certificates on-demand. Acts as a Certificate Authority. Used for mTLS between services.
  • AWS/GCP/Azure — Dynamically generate cloud provider credentials (IAM roles, service account keys) with limited TTL.
  • Transit — Encryption-as-a-service. Encrypt/decrypt data without exposing the key. Data never leaves Vault encrypted.
  • SSH — Sign SSH public keys or generate OTPs for SSH access. Enables zero-standing-access to servers.

Auth Methods

Before accessing any secret, a client must authenticate. Vault supports many auth methods:

  • Kubernetes — Pods authenticate using their service account JWT. Vault verifies with the Kubernetes API. No credentials needed in the Pod spec.
  • Token — Root tokens and short-lived tokens. Token auth is always enabled; don't use root tokens in automation.
  • AppRole — Machine-to-machine auth: a RoleID (non-secret) + SecretID (secret, short-lived). Good for CI/CD pipelines not on Kubernetes.
  • AWS/GCP/Azure IAM — Authenticate using cloud provider identity. Good for VMs, Lambda functions, or GKE Workload Identity.
  • OIDC/JWT — Authenticate using OIDC tokens from GitHub Actions, GitLab CI, or any OIDC provider.

Policies

# Vault policies use HCL (HashiCorp Configuration Language)
# They grant or deny capabilities on paths

# grade-api-policy.hcl
path "secret/data/grade-api/*" {
  capabilities = ["read", "list"]
}

path "secret/data/grade-api/db" {
  capabilities = ["read"]
}

path "database/creds/grade-api-role" {
  capabilities = ["read"]
}

# Deny access to other teams' secrets
path "secret/data/payment-api/*" {
  capabilities = ["deny"]
}

# Capabilities: create, read, update, delete, list, sudo, deny
# 'deny' overrides any allow in other policies

# Write the policy to Vault
vault policy write grade-api grade-api-policy.hcl

# List policies
vault policy list

# Read a policy
vault policy read grade-api

Install on Kubernetes

Helm Installation

# Add HashiCorp Helm repo
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Install Vault in dev mode first (no HA, unsealed, for learning)
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set "server.dev.enabled=true"

# Wait for Vault to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=vault -n vault --timeout=60s

# Exec into the vault pod
kubectl exec -it vault-0 -n vault -- vault status
# Key             Value
# ---             -----
# Seal Type       shamir
# Initialized     true
# Sealed          false      (dev mode starts unsealed)

HA Configuration

# values.yaml for production HA Vault install
server:
  replicas: 3

  # Integrated storage (Raft) — no external Consul needed
  ha:
    enabled: true
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          telemetry {
            unauthenticated_metrics_access = "true"
          }
        }

        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }

        service_registration "kubernetes" {}

  dataStorage:
    enabled: true
    size: 10Gi
    storageClass: standard     # Use your cluster's storage class

  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 1Gi
      cpu: 1000m

ui:
  enabled: true
  serviceType: ClusterIP   # Use Ingress, not LoadBalancer

injector:
  enabled: true            # Vault Agent Injector (Part 3)

# Install production Vault
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  -f values.yaml

# Check all 3 pods come up
kubectl get pods -n vault
# NAME      READY   STATUS    RESTARTS   AGE
# vault-0   0/1     Running   0          30s
# vault-1   0/1     Running   0          28s
# vault-2   0/1     Running   0          26s
# All show 0/1 because Vault is not initialized/unsealed yet

Initialize & Unseal

# Initialize Vault (only done once per cluster)
kubectl exec vault-0 -n vault -- vault operator init \
  -key-shares=5 \       # Split the master key into 5 shares
  -key-threshold=3 \    # Need 3 of 5 to unseal
  -format=json > vault-keys.json

# CRITICAL: Save vault-keys.json securely (AWS KMS, encrypted backup, etc.)
# It contains the unseal keys and the root token
# NEVER commit to Git

# Extract keys
VAULT_TOKEN=$(cat vault-keys.json | jq -r ".root_token")
UNSEAL_KEY_1=$(cat vault-keys.json | jq -r ".unseal_keys_b64[0]")
UNSEAL_KEY_2=$(cat vault-keys.json | jq -r ".unseal_keys_b64[1]")
UNSEAL_KEY_3=$(cat vault-keys.json | jq -r ".unseal_keys_b64[2]")

# Unseal vault-0 (must provide 3 of 5 keys)
kubectl exec vault-0 -n vault -- vault operator unseal $UNSEAL_KEY_1
kubectl exec vault-0 -n vault -- vault operator unseal $UNSEAL_KEY_2
kubectl exec vault-0 -n vault -- vault operator unseal $UNSEAL_KEY_3

# Join and unseal vault-1 and vault-2
kubectl exec vault-1 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
kubectl exec vault-1 -n vault -- vault operator unseal $UNSEAL_KEY_1
kubectl exec vault-1 -n vault -- vault operator unseal $UNSEAL_KEY_2
kubectl exec vault-1 -n vault -- vault operator unseal $UNSEAL_KEY_3

kubectl exec vault-2 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
kubectl exec vault-2 -n vault -- vault operator unseal $UNSEAL_KEY_1
kubectl exec vault-2 -n vault -- vault operator unseal $UNSEAL_KEY_2
kubectl exec vault-2 -n vault -- vault operator unseal $UNSEAL_KEY_3

# Verify all pods are 1/1
kubectl get pods -n vault
# NAME      READY   STATUS    RESTARTS   AGE
# vault-0   1/1     Running   0          5m
# vault-1   1/1     Running   0          5m
# vault-2   1/1     Running   0          5m
Auto-unseal for production: Manual unseal is operationally painful — every restart requires human intervention. For production, use Vault's auto-unseal with AWS KMS, GCP Cloud KMS, or Azure Key Vault. Set seal "awskms" in the Vault config with your KMS key ARN, and Vault decrypts its unseal key automatically on startup.

Writing & Reading Secrets

# Port-forward to Vault UI/API
kubectl port-forward vault-0 8200:8200 -n vault &
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=$VAULT_TOKEN   # From init step

# Enable the KV v2 secrets engine
vault secrets enable -path=secret kv-v2

# Write a secret
vault kv put secret/grade-api/db \
  username="grade_api_user" \
  password="super-secret-db-password" \
  host="postgres.grade-api.svc.cluster.local"

# Read the secret
vault kv get secret/grade-api/db
# == Secret Path ==
# secret/data/grade-api/db
# == Data ==
# Key       Value
# ---       -----
# host      postgres.grade-api.svc.cluster.local
# password  super-secret-db-password
# username  grade_api_user

# List secrets at a path
vault kv list secret/grade-api/

# Update (creates a new version)
vault kv patch secret/grade-api/db password="new-password-v2"

# Get a specific version
vault kv get -version=1 secret/grade-api/db

# Delete (marks as deleted; version still exists)
vault kv delete secret/grade-api/db

# Destroy a specific version (permanent)
vault kv destroy -versions=1 secret/grade-api/db

Vault CLI Essentials

# Status
vault status
vault operator raft list-peers    # HA cluster members

# Auth
vault login $ROOT_TOKEN            # Login with root token (avoid in prod)
vault token lookup                 # Info about current token
vault token create -ttl=1h -policy=grade-api   # Create short-lived token

# Secrets
vault kv put secret/path key=value
vault kv get secret/path
vault kv list secret/
vault kv delete secret/path
vault kv metadata get secret/path  # Version history

# Policies
vault policy list
vault policy write myapp myapp.hcl
vault policy read myapp

# Auth methods
vault auth list
vault auth enable kubernetes
vault auth enable approle

# Secrets engines
vault secrets list
vault secrets enable -path=database database
vault secrets enable -path=pki pki

# Audit
vault audit enable file file_path=/vault/logs/audit.log
vault audit list

Exercises

Exercise 1 — Dev Mode: Install Vault in dev mode on a Minikube or Kind cluster. Port-forward the UI. Write 3 secrets at different paths. Practice reading, updating, and versioning secrets. Explore the Vault UI at http://localhost:8200.
Exercise 2 — Policy: Write a policy that allows read-only access to secret/data/grade-api/* but denies access to secret/data/grade-api/admin. Create a token with that policy. Verify: reading grade-api/db succeeds, reading grade-api/admin is denied.
Exercise 3 — HA Install: Install Vault with Raft HA (3 replicas). Initialize and unseal all 3 nodes. Check cluster membership with vault operator raft list-peers. Kill vault-0 and verify vault-1 becomes the new leader.

Key Takeaways & Next Steps

Key Takeaways:
  • Vault organizes functionality into secrets engines (KV, Database, PKI, Transit) and auth methods (Kubernetes, AppRole, OIDC)
  • Policies use HCL and control which paths a token/entity can access with which capabilities
  • KV v2 provides versioning — deleted secrets can be recovered; destroyed secrets cannot
  • HA with Raft integrated storage requires no external Consul — 3-node Raft cluster is production-grade
  • Use auto-unseal (AWS KMS, GCP KMS) in production — manual unseal doesn't scale
  • The root token should be revoked after initial setup — use AppRole or Kubernetes auth for all automation

Next in This Track

In Part 2: Kubernetes Auth, we configure Vault's Kubernetes auth method so Pods authenticate using their service account JWTs — no static credentials, no Secrets to manage. Vault issues short-lived tokens scoped to policies based on namespace and service account.

Continue the Track

Part 2: Kubernetes Auth

Authenticate Pods to Vault using service account JWTs.

Read Part 2
External Secrets Track

Use ESO to sync Vault secrets into Kubernetes Secrets automatically.

Read External Secrets Part 1