Part 18: SSH & Secure Remote Access

SSH Protocol Overview

SSH (Secure Shell) is a cryptographic network protocol for secure remote login, command execution, and tunneling over an untrusted network. It replaced insecure predecessors (Telnet, rlogin, rsh) by encrypting all traffic — including passwords — between client and server. SSH operates on TCP port 22 by default and uses a client-server architecture.

The SSH protocol has three layers: the Transport Layer (key exchange, encryption, integrity), the User Authentication Layer (password, public key, keyboard-interactive), and the Connection Layer (multiplexed channels for shell, exec, forwarding).

sequenceDiagram
    participant C as Client
    participant S as Server

    C->>S: TCP connect (port 22)
    S->>C: Server version string (SSH-2.0-OpenSSH_9.x)
    C->>S: Client version string (SSH-2.0-OpenSSH_9.x)
    Note over C,S: Key Exchange (Diffie-Hellman / Curve25519)
    S->>C: Server host key (verify fingerprint)
    Note over C,S: Encrypted channel established
    C->>S: Auth request: public key (Ed25519)
    S->>C: Challenge (sign this nonce)
    C->>S: Signature (signed with private key)
    S->>C: Auth success
    Note over C,S: Session established — shell/exec/tunnel
            

Key-Based Authentication

Public key authentication is more secure than passwords — no secret is transmitted over the network. The client proves it holds the private key by signing a challenge from the server. The server verifies the signature against the public key listed in ~/.ssh/authorized_keys.

ssh-keygen

# Generate an Ed25519 key pair (RECOMMENDED — modern, fast, small, secure)
ssh-keygen -t ed25519 -C "wasil@workstation" -f ~/.ssh/id_ed25519
# Prompts for passphrase (protect your private key at rest)

# Generate RSA 4096-bit key (wider compatibility, but larger and slower)
ssh-keygen -t rsa -b 4096 -C "wasil@workstation" -f ~/.ssh/id_rsa_4096

# View your public key (this goes on the server)
cat ~/.ssh/id_ed25519.pub
# ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... wasil@workstation

# View the key fingerprint (for verification)
ssh-keygen -lf ~/.ssh/id_ed25519.pub
# 256 SHA256:abc123... wasil@workstation (ED25519)

# Change passphrase on existing key
ssh-keygen -p -f ~/.ssh/id_ed25519

authorized_keys

# Copy your public key to the server (easiest method)
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server.example.com
# This appends your public key to ~/.ssh/authorized_keys on the server

# Manual method (if ssh-copy-id unavailable)
cat ~/.ssh/id_ed25519.pub | ssh user@server "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

# Verify permissions (SSH is strict about permissions)
# On the server:
ls -la ~/.ssh/
# drwx------  .ssh/             (700 — owner only)
# -rw-------  authorized_keys   (600 — owner read/write only)
# -rw-------  id_ed25519        (600 — private key)
# -rw-r--r--  id_ed25519.pub    (644 — public key, readable)

# Restrict a key to specific commands (in authorized_keys)
# command="/usr/bin/backup.sh",no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAA...

SSH Config File

The ~/.ssh/config file saves you from typing long SSH commands repeatedly. It defines per-host settings: hostname, user, key, port, proxy, and more. SSH reads this file on every connection.

# ~/.ssh/config — Example with multiple hosts

# Default settings for all hosts
Host *
    AddKeysToAgent yes
    IdentitiesOnly yes
    ServerAliveInterval 60
    ServerAliveCountMax 3

# Production server (direct)
Host prod
    HostName 10.0.1.50
    User deploy
    IdentityFile ~/.ssh/id_ed25519
    Port 2222

# Jump through bastion to reach internal database server
Host db-internal
    HostName 10.0.2.100
    User dbadmin
    IdentityFile ~/.ssh/id_ed25519
    ProxyJump bastion

# Bastion / jump host
Host bastion
    HostName bastion.example.com
    User wasil
    IdentityFile ~/.ssh/id_ed25519
    ForwardAgent no

# GitHub (use specific deploy key)
Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/github_ed25519

# Wildcard for all staging servers
Host staging-*
    User deploy
    IdentityFile ~/.ssh/id_ed25519_staging
    ProxyJump bastion

# Usage with SSH config (short aliases instead of long commands)
ssh prod                    # Connects to 10.0.1.50:2222 as deploy
ssh db-internal             # Jumps through bastion automatically
ssh staging-web             # Matches wildcard, uses ProxyJump

# ProxyJump (-J) without config file
ssh -J wasil@bastion.example.com dbadmin@10.0.2.100

# Test SSH config parsing
ssh -G prod                 # Show resolved config for "prod" host

SSH Tunneling

SSH tunneling (port forwarding) creates encrypted channels that carry other protocols' traffic through the SSH connection. There are three types: Local (-L), Remote (-R), and Dynamic (-D).

flowchart TD
    subgraph Local["-L Local Forwarding"]
        LC[Client :8080] -->|encrypted| LS[SSH Server]
        LS --> LT[Target db:5432]
    end

    subgraph Remote["-R Remote Forwarding"]
        RT[Remote users :80] --> RS[SSH Server :80]
        RS -->|encrypted| RC[Client localhost:3000]
    end

    subgraph Dynamic["-D Dynamic SOCKS"]
        DA[App → SOCKS :1080] -->|encrypted| DS[SSH Server]
        DS --> DT1[Any host:port]
        DS --> DT2[Any host:port]
    end
            

Local Forwarding (-L)

Local forwarding binds a port on your local machine and forwards traffic through the SSH tunnel to a destination reachable from the server. Use case: access a remote database that only listens on localhost.

# Local port forwarding: access remote PostgreSQL via localhost:5433
# Syntax: ssh -L [local_addr:]local_port:remote_host:remote_port server
ssh -L 5433:localhost:5432 user@db-server.example.com -N
# -N = no remote command (tunnel only)

# Now connect to PostgreSQL locally:
psql -h 127.0.0.1 -p 5433 -U myuser mydb

# Access internal web app through bastion
ssh -L 8080:internal-app.corp:80 user@bastion.example.com -N
# Browse http://localhost:8080 → reaches internal-app.corp:80

# Multiple forwards in one connection
ssh -L 5433:db:5432 -L 6380:redis:6379 -L 8080:web:80 user@bastion -N

Remote Forwarding (-R)

Remote forwarding binds a port on the SSH server and forwards traffic back to your local machine (or another host reachable from you). Use case: expose a local dev server to the internet through a VPS.

# Remote port forwarding: expose local dev server to VPS
# Syntax: ssh -R [remote_addr:]remote_port:local_host:local_port server
ssh -R 8080:localhost:3000 user@vps.example.com -N
# Anyone connecting to vps.example.com:8080 reaches your localhost:3000

# Allow external connections (requires GatewayPorts yes in sshd_config)
ssh -R 0.0.0.0:8080:localhost:3000 user@vps.example.com -N

# Expose local SSH to a remote host (reverse SSH tunnel)
ssh -R 2222:localhost:22 user@relay.example.com -N
# From relay: ssh -p 2222 localhost  → reaches your machine

Dynamic/SOCKS Proxy (-D)

Dynamic forwarding creates a local SOCKS proxy. Any application configured to use it routes all TCP traffic through the SSH tunnel — like a lightweight VPN.

# Dynamic SOCKS proxy on localhost:1080
ssh -D 1080 user@server.example.com -N

# Configure applications to use SOCKS5 proxy at 127.0.0.1:1080
# Browser: Settings → Proxy → SOCKS5, 127.0.0.1, port 1080
# curl with SOCKS proxy:
curl --socks5 127.0.0.1:1080 http://internal-service.corp/api/status

# Use with proxychains (route any command through SOCKS)
# /etc/proxychains.conf: socks5 127.0.0.1 1080
proxychains nmap -sT internal-network.corp/24

SSH Agent & Agent Forwarding

The SSH agent holds your decrypted private keys in memory so you don't re-enter your passphrase for every connection. Agent forwarding lets a remote server use your local agent's keys for onward connections (e.g., pulling from GitHub on a server without storing keys there).

# Start the SSH agent (usually auto-started by desktop environment)
eval "$(ssh-agent -s)"
# Agent pid 12345

# Add your key to the agent
ssh-add ~/.ssh/id_ed25519
# Enter passphrase once — agent remembers it

# List keys loaded in agent
ssh-add -l
# 256 SHA256:abc123... wasil@workstation (ED25519)

# Add key with lifetime (auto-removed after 4 hours)
ssh-add -t 4h ~/.ssh/id_ed25519

# Agent forwarding: use local keys on remote server
ssh -A user@bastion.example.com
# On bastion, you can now: git clone git@github.com:org/repo.git
# (uses your local key via forwarded agent)

# In SSH config:
# Host bastion
#     ForwardAgent yes

File Transfer — scp & rsync

SSH provides secure file transfer via scp (simple copy) and rsync (incremental sync). Both use SSH as the transport layer and respect your SSH config.

# === scp — Secure Copy ===

# Copy local file to remote server
scp ./app.tar.gz user@server:/opt/releases/

# Copy remote file to local machine
scp user@server:/var/log/app.log ./app.log

# Copy entire directory recursively
scp -r ./build/ user@server:/var/www/html/

# Use SSH config alias
scp ./deploy.sh prod:/opt/scripts/

# Specify port and key
scp -P 2222 -i ~/.ssh/id_ed25519 file.txt user@host:/tmp/

# === rsync — Incremental Sync (preferred over scp for large transfers) ===

# Sync local directory to remote (only transfers changed files)
rsync -avz --progress ./project/ user@server:/opt/project/
# -a = archive (preserves permissions, timestamps, symlinks)
# -v = verbose
# -z = compress during transfer
# --progress = show transfer progress

# Sync remote to local (pull)
rsync -avz user@server:/var/log/app/ ./logs/

# Dry run (show what would change without doing it)
rsync -avzn ./project/ user@server:/opt/project/

# Exclude patterns
rsync -avz --exclude='node_modules' --exclude='.git' ./app/ user@server:/opt/app/

# Delete files on destination that don't exist on source (mirror)
rsync -avz --delete ./build/ user@server:/var/www/html/

# Use specific SSH key and port
rsync -avz -e "ssh -p 2222 -i ~/.ssh/id_ed25519" ./data/ user@host:/data/

# Bandwidth limit (KB/s) — don't saturate the link
rsync -avz --bwlimit=5000 ./large-backup/ user@server:/backups/

SSH Hardening

A default SSH configuration is functional but not production-ready. Hardening reduces the attack surface by disabling weak authentication methods, restricting access, and limiting exposure.

# /etc/ssh/sshd_config — Production hardening

# Disable password authentication (keys only)
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

# Disable root login
PermitRootLogin no

# Allow only specific users/groups
AllowUsers deploy wasil
# AllowGroups ssh-users

# Use only SSH protocol 2 (protocol 1 is insecure and deprecated)
Protocol 2

# Restrict key exchange and ciphers to strong algorithms
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com
HostKeyAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256

# Limit authentication attempts and timeout
MaxAuthTries 3
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2

# Disable unused features
X11Forwarding no
AllowTcpForwarding no          # Enable only if tunneling is needed
AllowAgentForwarding no        # Enable only on jump hosts
PermitTunnel no

# Change default port (security through obscurity — reduces log noise)
Port 2222

# Restrict to specific listen address (if multi-homed)
ListenAddress 10.0.1.50

# Apply sshd_config changes
sudo sshd -t                   # Test config syntax BEFORE restarting
sudo systemctl restart sshd    # Restart SSH daemon

# Verify hardening with ssh-audit (open-source auditing tool)
# Install: pip install ssh-audit
ssh-audit localhost
ssh-audit server.example.com

# Fail2ban — auto-block IPs after failed login attempts
sudo apt install fail2ban
# /etc/fail2ban/jail.local:
# [sshd]
# enabled = true
# port = 2222
# maxretry = 3
# bantime = 3600

Exercises

# Exercise 1: Generate an Ed25519 key and inspect it
ssh-keygen -t ed25519 -C "exercise-key" -f /tmp/exercise_key -N ""
ssh-keygen -lf /tmp/exercise_key.pub
cat /tmp/exercise_key.pub

# Exercise 2: Create an SSH config with aliases
cat <<'EOF' > /tmp/ssh_config_test
Host myserver
    HostName 192.168.1.100
    User admin
    IdentityFile ~/.ssh/id_ed25519
    Port 22

Host internal
    HostName 10.0.0.50
    User deploy
    ProxyJump myserver
EOF
ssh -F /tmp/ssh_config_test -G myserver   # Verify config parsing

# Exercise 3: Local port forwarding (test with nc)
# Terminal 1 (simulated remote service):
nc -l 9999 &
# Terminal 2 (SSH tunnel — requires sshd running locally):
# ssh -L 8888:localhost:9999 localhost -N &
# Terminal 3: echo "tunneled" | nc 127.0.0.1 8888

# Exercise 4: Inspect your SSH agent
ssh-add -l 2>/dev/null || echo "No agent or no keys loaded"

# Exercise 5: Audit SSH server configuration
# ssh-audit localhost  (install: pip install ssh-audit)

Conclusion & Next Steps

SSH is the backbone of secure remote administration. Key-based authentication eliminates password vulnerabilities. The SSH config file turns complex multi-hop connections into simple aliases. Tunneling (-L, -R, -D) provides encrypted channels for any protocol. Agent forwarding enables seamless multi-hop workflows but requires caution. And hardening (disabling passwords, restricting ciphers, limiting access) transforms SSH from functional to production-secure. Combined with bastion patterns and short-lived certificates, SSH remains the primary secure access mechanism even in cloud-native environments.