Part 17: TLS/SSL Deep Dive — PKI & Certificates

Public Key Infrastructure

PKI (Public Key Infrastructure) is the system of roles, policies, hardware, software, and procedures needed to create, manage, distribute, use, store, and revoke digital certificates. It's the trust foundation that makes HTTPS, code signing, email encryption (S/MIME), and VPNs work at internet scale.

The core problem PKI solves: how does a client know that a public key genuinely belongs to example.com and not an attacker? The answer: a trusted third party (a Certificate Authority) cryptographically signs a statement binding the public key to the domain name.

Certificate Authorities

A Certificate Authority (CA) is an entity trusted to issue digital certificates. CAs verify the identity of certificate requestors before signing. The CA hierarchy has three levels:

• Root CA — Self-signed, stored offline in HSMs (Hardware Security Modules). Pre-installed in OS/browser trust stores. Never signs end-entity certs directly.
• Intermediate CA — Signed by the Root CA. Does the day-to-day signing of server certificates. If compromised, only the intermediate is revoked, not the root.
• End-Entity Certificate — The server/client certificate signed by an Intermediate CA. This is what your web server presents during the TLS handshake.

Chain of Trust

During TLS, the server sends its certificate plus the intermediate CA certificate(s). The client walks up the chain: verifies the server cert was signed by the intermediate, then verifies the intermediate was signed by a root CA in its trust store.

flowchart TD
    ROOT["Root CA Certificate\n(Self-signed, in trust store)\nValidity: 20-30 years"]
    INT["Intermediate CA Certificate\n(Signed by Root CA)\nValidity: 5-10 years"]
    SERVER["Server Certificate\n(Signed by Intermediate CA)\nValidity: 90 days - 1 year"]
    CLIENT["Client Browser / OS\nTrust Store"]

    ROOT -->|"Signs"| INT
    INT -->|"Signs"| SERVER
    CLIENT -->|"Trusts Root CA\n(pre-installed)"| ROOT
    CLIENT -->|"Verifies chain:\nServer → Intermediate → Root"| SERVER
            

# View the certificate chain for a live website
openssl s_client -connect www.google.com:443 -showcerts < /dev/null 2>/dev/null | \
  grep -E "s:|i:" | head -10
# s: = subject (who the cert is for)
# i: = issuer (who signed it)

# View your system's trusted root CAs (Debian/Ubuntu)
ls /etc/ssl/certs/ | head -20
cat /etc/ssl/certs/ca-certificates.crt | grep "BEGIN CERTIFICATE" | wc -l
# Typically 130-150 root CAs in a default trust store

# View root CAs on RHEL/CentOS
trust list | grep -c "type: certificate"

X.509 Certificates

X.509 is the ITU-T standard defining the format of public key certificates. Every TLS certificate on the internet is an X.509v3 certificate, encoded in either DER (binary) or PEM (Base64-encoded DER wrapped in -----BEGIN CERTIFICATE----- headers).

Certificate Fields

Extensions

X.509v3 extensions add critical metadata. The most important ones:

# View full X.509 certificate details for a website
openssl s_client -connect example.com:443 < /dev/null 2>/dev/null | \
  openssl x509 -noout -text | head -60

# View just the SAN entries
openssl s_client -connect github.com:443 < /dev/null 2>/dev/null | \
  openssl x509 -noout -ext subjectAltName

# View expiry date
openssl s_client -connect example.com:443 < /dev/null 2>/dev/null | \
  openssl x509 -noout -dates
# notBefore=...
# notAfter=...

# View the public key algorithm and size
openssl s_client -connect example.com:443 < /dev/null 2>/dev/null | \
  openssl x509 -noout -pubkey | openssl pkey -pubin -text -noout | head -5

Certificate Lifecycle

Every certificate goes through: generation → signing → deployment → monitoring → renewal/revocation. Automating this lifecycle is critical — expired certificates cause outages.

CSR Generation

A Certificate Signing Request (CSR) contains the public key and subject information. The private key never leaves the server — only the CSR is sent to the CA.

# Generate a private key + CSR in one command
openssl req -new -newkey rsa:2048 -nodes \
  -keyout server.key \
  -out server.csr \
  -subj "/C=US/ST=California/L=San Francisco/O=MyCompany/CN=api.example.com" \
  -addext "subjectAltName=DNS:api.example.com,DNS:www.api.example.com"

# View the CSR contents
openssl req -in server.csr -noout -text

# Generate an EC (Elliptic Curve) key + CSR (faster, smaller keys)
openssl ecparam -genkey -name prime256v1 -out ec-server.key
openssl req -new -key ec-server.key \
  -out ec-server.csr \
  -subj "/CN=api.example.com" \
  -addext "subjectAltName=DNS:api.example.com"

Signing

The CA validates the CSR (domain ownership, organisation identity) and signs it with their private key, producing the certificate. For internal PKI, you can be your own CA.

# Create a self-signed Root CA (for internal/dev use)
openssl req -x509 -new -nodes \
  -keyout ca.key -out ca.crt \
  -days 3650 \
  -subj "/C=US/O=Internal Corp/CN=Internal Root CA" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign"

# Sign the server CSR with your CA
openssl x509 -req -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt \
  -days 365 \
  -extfile <(printf "subjectAltName=DNS:api.example.com,DNS:www.api.example.com\nextendedKeyUsage=serverAuth")

# Verify the certificate was signed by your CA
openssl verify -CAfile ca.crt server.crt
# server.crt: OK

Renewal

Certificates expire. Let's Encrypt certs last 90 days; commercial certs last up to 398 days (since 2020). Renewal generates a new CSR (optionally with the same key) and obtains a fresh signature from the CA.

Mutual TLS (mTLS)

Standard TLS is one-way: the client verifies the server's certificate. Mutual TLS (mTLS) adds client certificate authentication — the server also requests and verifies a certificate from the client. This provides strong mutual authentication without passwords.

mTLS is widely used for:

• Service-to-service communication (microservices, service meshes)
• Zero-trust networks (BeyondCorp, Cloudflare Access)
• IoT device authentication
• Financial APIs (PSD2 Open Banking requires mTLS)

# Generate a client certificate (signed by the same internal CA)
openssl req -new -newkey rsa:2048 -nodes \
  -keyout client.key -out client.csr \
  -subj "/CN=service-a/O=Internal Corp"

openssl x509 -req -in client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt -days 365 \
  -extfile <(printf "extendedKeyUsage=clientAuth")

# Test mTLS connection with curl
# Server requires client cert → pass --cert and --key
curl --cacert ca.crt \
     --cert client.crt \
     --key client.key \
     https://api.internal.example.com/health

# Configure nginx to require client certificates
# In nginx.conf server block:
#   ssl_client_certificate /etc/nginx/ca.crt;
#   ssl_verify_client on;
#   ssl_verify_depth 2;

Let's Encrypt & ACME Protocol

Let's Encrypt is a free, automated, open Certificate Authority that has issued billions of certificates. It uses the ACME (Automatic Certificate Management Environment) protocol (RFC 8555) to automate domain validation and certificate issuance — no human intervention required.

sequenceDiagram
    participant CB as Certbot (Client)
    participant ACME as ACME Server (Let's Encrypt)
    participant WEB as Web Server (port 80)

    CB->>ACME: 1. Request certificate for example.com
    ACME->>CB: 2. Challenge: place token at /.well-known/acme-challenge/{token}
    CB->>WEB: 3. Write token file to web root
    CB->>ACME: 4. Ready — please validate
    ACME->>WEB: 5. HTTP GET /.well-known/acme-challenge/{token}
    WEB->>ACME: 6. Returns token + key authorization
    ACME->>ACME: 7. Validates domain control
    ACME->>CB: 8. Validation passed — here's your signed certificate
    CB->>CB: 9. Install cert + reload web server
            

Challenge types:

• HTTP-01 — Place a file at http://domain/.well-known/acme-challenge/ (requires port 80 open)
• DNS-01 — Create a _acme-challenge.domain TXT record (supports wildcards, no port 80 needed)
• TLS-ALPN-01 — Serve a special self-signed cert on port 443 with ALPN extension

# Install certbot (the official ACME client)
sudo apt install certbot python3-certbot-nginx   # Debian/Ubuntu

# Obtain a certificate with HTTP-01 challenge (nginx plugin)
sudo certbot --nginx -d example.com -d www.example.com

# Obtain a wildcard cert with DNS-01 challenge (manual DNS)
sudo certbot certonly --manual --preferred-challenges dns \
  -d "*.example.com" -d "example.com"

# Automatic renewal (certbot installs a systemd timer/cron)
sudo certbot renew --dry-run   # Test renewal without making changes

# View installed certificates
sudo certbot certificates

# Renewal with post-hook to reload nginx
sudo certbot renew --deploy-hook "systemctl reload nginx"

OpenSSL Certificate Operations

OpenSSL is the Swiss Army knife for TLS debugging and certificate management. These are the commands you'll use daily when working with certificates.

# === Inspect a remote server's TLS configuration ===
# Connect and show full handshake + certificate
openssl s_client -connect example.com:443 -servername example.com < /dev/null

# Show only the certificate chain (subject + issuer)
openssl s_client -connect example.com:443 -showcerts < /dev/null 2>/dev/null | \
  grep -E "s:|i:"

# Check which TLS version and cipher was negotiated
openssl s_client -connect example.com:443 < /dev/null 2>/dev/null | \
  grep -E "Protocol|Cipher"

# === Verify a certificate chain locally ===
# Verify server.crt against system trust store
openssl verify server.crt

# Verify against a specific CA bundle
openssl verify -CAfile ca-bundle.crt server.crt

# Verify a full chain (intermediate + server)
openssl verify -CAfile root-ca.crt -untrusted intermediate.crt server.crt

# === Convert between certificate formats ===
# PEM to DER (binary)
openssl x509 -in cert.pem -outform DER -out cert.der

# DER to PEM
openssl x509 -in cert.der -inform DER -outform PEM -out cert.pem

# PEM to PKCS#12 (bundle cert + key for import into browsers/Java)
openssl pkcs12 -export -out bundle.p12 \
  -inkey server.key -in server.crt -certfile ca.crt

# PKCS#12 to PEM (extract cert and key)
openssl pkcs12 -in bundle.p12 -out extracted.pem -nodes

# === Check certificate expiry (useful in monitoring scripts) ===
# Check when a remote cert expires
echo | openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -enddate

# Check if cert expires within 30 days (exit code 0 = still valid)
openssl x509 -in server.crt -checkend 2592000 -noout
echo $?   # 0 = valid for 30+ more days, 1 = expires within 30 days

# One-liner for monitoring: check multiple domains
for domain in example.com api.example.com mail.example.com; do
  expiry=$(echo | openssl s_client -connect "$domain:443" 2>/dev/null | \
    openssl x509 -noout -enddate | cut -d= -f2)
  echo "$domain expires: $expiry"
done

Exercises

# Exercise 1: View the certificate chain for your favourite website
openssl s_client -connect github.com:443 < /dev/null 2>/dev/null | \
  grep -E "s:|i:"

# Exercise 2: Create a self-signed CA and sign a server cert
openssl req -x509 -new -nodes -keyout myca.key -out myca.crt \
  -days 3650 -subj "/CN=My Test CA"
openssl req -new -newkey rsa:2048 -nodes \
  -keyout myserver.key -out myserver.csr \
  -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"
openssl x509 -req -in myserver.csr -CA myca.crt -CAkey myca.key \
  -CAcreateserial -out myserver.crt -days 365 \
  -extfile <(printf "subjectAltName=DNS:localhost,IP:127.0.0.1")
openssl verify -CAfile myca.crt myserver.crt

# Exercise 3: Check how many days until a cert expires
echo | openssl s_client -connect google.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# Exercise 4: View the SAN entries on a multi-domain cert
echo | openssl s_client -connect google.com:443 2>/dev/null | \
  openssl x509 -noout -ext subjectAltName

# Exercise 5: Check which TLS version is negotiated
openssl s_client -connect example.com:443 -tls1_3 < /dev/null 2>/dev/null | \
  grep "Protocol"

Conclusion & Next Steps

PKI is the trust infrastructure that makes encrypted communication possible at internet scale. The certificate chain (Root CA → Intermediate → End-Entity) provides hierarchical trust, X.509 defines the universal certificate format, and ACME/Let's Encrypt automates the entire lifecycle. mTLS extends trust to both sides of a connection — essential for zero-trust architectures and service meshes. Master OpenSSL's inspection commands and you can debug any TLS issue you encounter.