Part 9: Scalability Fundamentals

Module 14: Scalability Fundamentals

Scalability is a system's ability to handle increased load by adding resources. It sounds simple, but the engineering decisions around how to scale determine whether your system gracefully handles 10x growth or collapses under 2x pressure.

Why Scale? Growth Curves & Breaking Points

Systems face different growth patterns, each demanding different scaling strategies:

• Linear growth: Steady user acquisition (B2B SaaS). Predictable — you can plan capacity months ahead.
• Exponential growth: Viral products, social networks. Traffic doubles every week. Planning is impossible — you must architect for elastic scaling from day one.
• Step functions: Marketing campaigns, product launches, seasonal events. Flat baseline with sudden spikes. Auto-scaling with pre-warming is critical.
• Unpredictable bursts: News events, viral moments. Zero warning. Systems must absorb 100x normal traffic or gracefully degrade.

Vertical Scaling: Scale Up (Bigger Machine)

Vertical scaling means upgrading a single machine — more CPU cores, more RAM, faster disks, better network cards. It's the first instinct and often the right first move.

Advantages:

• Simplicity: No distributed systems complexity. Single process, single database, simple deployment.
• Strong consistency: One machine = no network partitions, no eventual consistency headaches.
• Lower latency: Everything in-memory on one box. No network hops between components.
• Less operational overhead: One server to monitor, patch, and secure.

The Wall — Why Vertical Scaling Fails:

flowchart LR
    subgraph COST["Cost vs. Capacity"]
        direction TB
        A["$100/mo
4 CPU, 16GB RAM
~1,000 req/s"]
        B["$400/mo
16 CPU, 64GB RAM
~3,500 req/s"]
        C["$1,600/mo
64 CPU, 256GB RAM
~10,000 req/s"]
        D["$12,000/mo
128 CPU, 1TB RAM
~18,000 req/s"]
        E["$$$$$
Physical Limit
Can't buy bigger"]
    end

    A -->|"4x cost"| B
    B -->|"4x cost"| C
    C -->|"7.5x cost"| D
    D -->|"No option"| E

    style A fill:#e8f4f4,stroke:#3B9797,color:#132440
    style B fill:#e8f4f4,stroke:#3B9797,color:#132440
    style C fill:#f0f4f8,stroke:#16476A,color:#132440
    style D fill:#fdf0f0,stroke:#BF092F,color:#132440
    style E fill:#fdf0f0,stroke:#BF092F,color:#132440
                            

• Hardware ceiling: The biggest machine money can buy has limits. As of 2026, you can get ~448 CPU cores and 24TB RAM — but no matter how much you spend, physics caps single-machine performance.
• Cost curve: Doubling capacity costs more than double the money. A 128-core server costs 10x a 16-core server, not 8x. Diminishing returns accelerate.
• Single point of failure: One machine means one failure domain. When it goes down (hardware failure, kernel panic, power loss), everything goes down.
• Downtime for upgrades: Replacing hardware or upgrading RAM requires taking the server offline. Zero-downtime vertical scaling is nearly impossible.

Horizontal Scaling: Scale Out (More Machines)

Horizontal scaling means running multiple instances of your service behind a load balancer. Instead of one big machine, you use many smaller ones working in parallel.

Advantages:

• Near-unlimited capacity: Add more machines as needed. No hardware ceiling.
• Fault tolerance: If one machine dies, others continue serving. Redundancy is built in.
• Cost efficiency at scale: Commodity hardware is cheap. 10 × $100/month machines often outperform 1 × $5,000/month machine.
• Geographic distribution: Machines can be in different data centers, closer to users.

The Price — Horizontal Scaling Challenges:

• State management: Where does session data live? In-memory state on one server isn't visible to others. Solution: externalize state to Redis, database, or use stateless design.
• Data distribution: A single database becomes the bottleneck. You need read replicas, sharding, or distributed databases — each with consistency tradeoffs.
• Coordination overhead: Distributed locks, leader election, consensus protocols (Raft, Paxos). Complexity grows non-linearly with nodes.
• Network unreliability: Machines communicate over networks that can partition, delay, or lose messages. The CAP theorem becomes your daily reality.
• Deployment complexity: Rolling updates across 50 servers is harder than updating one. You need orchestration (Kubernetes), service discovery, and health checking.

flowchart TD
    START["Need More Capacity"] --> Q1{"Current load
< 50% of max
single machine?"}
    Q1 -->|"Yes"| VERT["Scale Vertically
Upgrade instance size"]
    Q1 -->|"No"| Q2{"Need >99.9%
availability?"}
    Q2 -->|"Yes"| HORIZ["Scale Horizontally
Multiple instances + LB"]
    Q2 -->|"No"| Q3{"Can afford
downtime for
upgrades?"}
    Q3 -->|"Yes"| VERT
    Q3 -->|"No"| HORIZ
    VERT --> CHECK{"Hit vertical
ceiling?"}
    CHECK -->|"Yes"| HORIZ
    CHECK -->|"No"| DONE["Monitor & Reassess"]
    HORIZ --> STATELESS{"Service is
stateless?"}
    STATELESS -->|"Yes"| EASY["Add instances
behind LB"]
    STATELESS -->|"No"| HARD["Externalize state
then scale out"]

    style START fill:#e8f4f4,stroke:#3B9797,color:#132440
    style VERT fill:#f0f4f8,stroke:#16476A,color:#132440
    style HORIZ fill:#f0f4f8,stroke:#16476A,color:#132440
    style EASY fill:#e8f4f4,stroke:#3B9797,color:#132440
    style HARD fill:#fdf0f0,stroke:#BF092F,color:#132440
                            

Amdahl's Law Applied to System Scaling

Amdahl's Law states that the maximum speedup from parallelization is limited by the sequential (non-parallelizable) portion of the work. Applied to scaling:

Practical implications:

• If 5% of processing is sequential → max theoretical speedup = 20×
• If 20% is sequential → max speedup = 5×
• If 50% is sequential → max speedup = 2×

This is why identifying and eliminating sequential bottlenecks (global locks, single-writer databases, shared mutable state) is more impactful than adding more servers. You don't scale by adding hardware — you scale by removing serialization points.

"""
Amdahl's Law Calculator for System Scaling

Demonstrates the theoretical maximum speedup achievable when
scaling horizontally, given a sequential (non-parallelizable) fraction.
"""

def amdahls_law(sequential_fraction, num_processors):
    """
    Calculate theoretical maximum speedup.
    
    Args:
        sequential_fraction: Fraction of work that must be sequential (0 to 1)
        num_processors: Number of parallel processors (servers)
    
    Returns:
        Maximum theoretical speedup factor
    """
    parallel_fraction = 1 - sequential_fraction
    speedup = 1 / (sequential_fraction + (parallel_fraction / num_processors))
    return speedup

# Simulate scaling with different sequential fractions
print("Amdahl's Law: Impact of Sequential Bottlenecks")
print("=" * 60)

sequential_fractions = [0.01, 0.05, 0.10, 0.20, 0.50]
server_counts = [1, 2, 4, 8, 16, 32, 64, 128, 256]

print(f"\n{'Servers':<10}", end="")
for frac in sequential_fractions:
    print(f"{'seq=' + str(int(frac*100)) + '%':<12}", end="")
print()
print("-" * 70)

for servers in server_counts:
    print(f"{servers:<10}", end="")
    for frac in sequential_fractions:
        speedup = amdahls_law(frac, servers)
        print(f"{speedup:<12.1f}", end="")
    print()

print("\n" + "=" * 60)
print("\nKey Insight: With 10% sequential work, 128 servers")
print(f"only give {amdahls_law(0.10, 128):.1f}x speedup (not 128x).")
print(f"With 1% sequential: {amdahls_law(0.01, 128):.1f}x — much better!")
print(f"\nTheoretical max (infinite servers):")
for frac in sequential_fractions:
    max_speedup = 1 / frac
    print(f"  {int(frac*100)}% sequential → max {max_speedup:.0f}x speedup")

Module 15: Load Distribution

Once you have multiple servers, you need to decide which server handles each request. This is the job of a load balancer — and the algorithm it uses determines whether your fleet is utilized efficiently or if some servers are overloaded while others sit idle.

Load Balancing: L4 vs. L7

Load balancers operate at different layers of the network stack, with fundamentally different capabilities:

flowchart TD
    CLIENT["Client Request"] --> LB{"Load Balancer"}

    subgraph L4["Layer 4 (Transport)"]
        direction TB
        L4_DESC["Sees: IP + Port
Decisions: TCP/UDP level
Speed: Very fast (hardware)"]
        L4_USE["Use for:
• Raw TCP throughput
• Database connections
• Non-HTTP protocols"]
    end

    subgraph L7["Layer 7 (Application)"]
        direction TB
        L7_DESC["Sees: Full HTTP request
URL, headers, cookies, body
Speed: Slower (software)"]
        L7_USE["Use for:
• Path-based routing
• A/B testing
• Session affinity
• SSL termination
• Request manipulation"]
    end

    LB -->|"TCP level"| L4
    LB -->|"HTTP level"| L7

    style CLIENT fill:#e8f4f4,stroke:#3B9797,color:#132440
    style L4_DESC fill:#f0f4f8,stroke:#16476A,color:#132440
    style L4_USE fill:#f0f4f8,stroke:#16476A,color:#132440
    style L7_DESC fill:#e8f4f4,stroke:#3B9797,color:#132440
    style L7_USE fill:#e8f4f4,stroke:#3B9797,color:#132440
                            

Health Checks — the load balancer's safety net:

• Active health checks: LB periodically pings each server (e.g., GET /healthz every 10s). If 3 consecutive checks fail, remove from rotation.
• Passive health checks: LB monitors actual traffic responses. If a server returns 5xx errors above threshold, mark unhealthy.
• Graceful drain: Before shutting down a server, stop sending new requests but allow in-flight requests to complete (connection draining).

Session Affinity (Sticky Sessions) — when stateless isn't practical:

• Route all requests from the same user to the same backend server
• Methods: cookie-based (LB injects server ID cookie), IP-based (hash client IP), header-based
• Warning: Sticky sessions fight against horizontal scaling. If the sticky server dies, the user loses session state. Prefer externalized state (Redis) instead.

# NGINX Load Balancer Configuration — Layer 7 with health checks
# Production-ready example with multiple algorithms

# Upstream definition — Round Robin (default)
upstream api_round_robin {
    server 10.0.1.10:8080 weight=5;    # 5x traffic (beefier machine)
    server 10.0.1.11:8080 weight=3;    # 3x traffic
    server 10.0.1.12:8080 weight=2;    # 2x traffic (smaller instance)
    server 10.0.1.13:8080 backup;      # Only used when all others fail

    # Health check: mark server down after 3 failures, retry after 30s
    # max_fails and fail_timeout control passive health checking
}

# Upstream definition — Least Connections
upstream api_least_conn {
    least_conn;
    server 10.0.2.10:8080;
    server 10.0.2.11:8080;
    server 10.0.2.12:8080;
    server 10.0.2.13:8080;
}

# Upstream definition — IP Hash (session affinity)
upstream api_sticky {
    ip_hash;
    server 10.0.3.10:8080;
    server 10.0.3.11:8080;
    server 10.0.3.12:8080;
    # If a server goes down, its traffic redistributes to remaining servers
    # When it recovers, clients gradually return (consistent hashing)
}

server {
    listen 443 ssl http2;
    server_name api.example.com;

    ssl_certificate     /etc/ssl/certs/api.example.com.pem;
    ssl_certificate_key /etc/ssl/private/api.example.com.key;

    # Route based on URL path to different upstreams
    location /api/v1/ {
        proxy_pass http://api_least_conn;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout and retry configuration
        proxy_connect_timeout 5s;
        proxy_read_timeout 30s;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
    }

    # WebSocket connections — sticky sessions required
    location /ws/ {
        proxy_pass http://api_sticky;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600s;  # Keep WebSocket alive for 1 hour
    }

    # Health check endpoint (for upstream load balancers)
    location /healthz {
        access_log off;
        return 200 "healthy\n";
    }
}

Load Balancing Algorithms Deep-Dive

The choice of algorithm determines how evenly work is distributed — and each has tradeoffs that matter at scale:

Consistent Hashing: The Algorithm That Enables Distributed Caches

Standard modular hashing (server = hash(key) % N) breaks catastrophically when you add or remove servers — every key remaps. With 100 servers and you add one, ~99% of keys move to different servers. For a cache, that means 99% cache miss rate — effectively a cold start.

Consistent hashing solves this: when you add or remove a server, only ~1/N of keys are remapped. The algorithm places both servers and keys on a virtual ring (0 to 2³²), and each key routes to the nearest server clockwise.

flowchart TD
    subgraph RING["Hash Ring (0 to 2^32)"]
        direction LR
        S1["Server A
position: 1000"]
        S2["Server B
position: 4000"]
        S3["Server C
position: 7000"]
        NEW["NEW Server D
position: 5500"]
    end

    subgraph KEYS["Key Assignment"]
        K1["key='user:42'
hash=2500
→ Server B"]
        K2["key='session:99'
hash=5000
→ Server C"]
        K3["key='cache:hello'
hash=8500
→ Server A"]
        K4["key='data:xyz'
hash=5200
→ was C, now D"]
    end

    S1 --> S2
    S2 --> S3
    S3 --> S1
    S2 -.->|"Add D between B and C"| NEW
    NEW -.-> S3

    style S1 fill:#e8f4f4,stroke:#3B9797,color:#132440
    style S2 fill:#e8f4f4,stroke:#3B9797,color:#132440
    style S3 fill:#e8f4f4,stroke:#3B9797,color:#132440
    style NEW fill:#fdf0f0,stroke:#BF092F,color:#132440
    style K4 fill:#fdf0f0,stroke:#BF092F,color:#132440
                            

"""
Consistent Hashing Implementation

Demonstrates the ring-based approach used by distributed caches
(Memcached, Redis Cluster), databases (DynamoDB, Cassandra),
and CDNs for minimal key redistribution when nodes change.
"""

import hashlib
from bisect import bisect_right

class ConsistentHashRing:
    """
    Consistent hash ring with virtual nodes for even distribution.
    
    Virtual nodes (vnodes) solve the uneven distribution problem:
    instead of one point per server, each server gets multiple
    points spread around the ring.
    """
    
    def __init__(self, nodes=None, virtual_nodes=150):
        self.virtual_nodes = virtual_nodes  # vnodes per physical node
        self.ring = {}          # hash_position -> node_name
        self.sorted_keys = []   # sorted list of hash positions
        self.nodes = set()      # set of physical node names
        
        if nodes:
            for node in nodes:
                self.add_node(node)
    
    def _hash(self, key):
        """Generate consistent hash position (0 to 2^32)."""
        md5 = hashlib.md5(key.encode('utf-8')).hexdigest()
        return int(md5, 16) % (2**32)
    
    def add_node(self, node):
        """Add a node with virtual nodes spread across the ring."""
        self.nodes.add(node)
        for i in range(self.virtual_nodes):
            virtual_key = f"{node}#vnode{i}"
            hash_pos = self._hash(virtual_key)
            self.ring[hash_pos] = node
            self.sorted_keys.append(hash_pos)
        self.sorted_keys.sort()
    
    def remove_node(self, node):
        """Remove a node — only its keys redistribute."""
        self.nodes.discard(node)
        for i in range(self.virtual_nodes):
            virtual_key = f"{node}#vnode{i}"
            hash_pos = self._hash(virtual_key)
            del self.ring[hash_pos]
            self.sorted_keys.remove(hash_pos)
    
    def get_node(self, key):
        """Find which node owns a given key."""
        if not self.sorted_keys:
            return None
        hash_pos = self._hash(key)
        # Find the first ring position >= key's hash (clockwise)
        idx = bisect_right(self.sorted_keys, hash_pos)
        # Wrap around if past the last position
        if idx == len(self.sorted_keys):
            idx = 0
        return self.ring[self.sorted_keys[idx]]


# Demonstration: Adding/removing nodes
ring = ConsistentHashRing(nodes=["cache-1", "cache-2", "cache-3"])

# Distribute 1000 keys and count per server
keys = [f"user:{i}" for i in range(1000)]
distribution = {}
for key in keys:
    node = ring.get_node(key)
    distribution[node] = distribution.get(node, 0) + 1

print("=== Initial Distribution (3 nodes) ===")
for node, count in sorted(distribution.items()):
    print(f"  {node}: {count} keys ({count/10:.1f}%)")

# Add a 4th node — only ~25% of keys should move
ring.add_node("cache-4")
new_distribution = {}
moved = 0
for key in keys:
    new_node = ring.get_node(key)
    new_distribution[new_node] = new_distribution.get(new_node, 0) + 1
    old_node = None
    # Check if key moved
    temp_ring = ConsistentHashRing(nodes=["cache-1", "cache-2", "cache-3"])
    if temp_ring.get_node(key) != new_node:
        moved += 1

print(f"\n=== After Adding cache-4 ===")
for node, count in sorted(new_distribution.items()):
    print(f"  {node}: {count} keys ({count/10:.1f}%)")
print(f"\n  Keys redistributed: {moved}/1000 ({moved/10:.1f}%)")
print(f"  Theoretical minimum: ~{1000//4} (25%)")

Global Distribution: Bringing Compute Closer to Users

Physics imposes a hard limit: light travels ~200km per millisecond in fiber. A round trip from London to Sydney is ~130ms minimum — no optimization can beat the speed of light. Global distribution reduces latency by placing servers closer to users.

DNS-Based Routing:

• DNS resolves to different IPs based on client location (GeoDNS)
• Simple to implement (Route 53, Cloudflare DNS)
• Weakness: DNS caching means slow failover (TTL-dependent), and client DNS resolver location may not match actual user location

Anycast Routing:

• Same IP address advertised from multiple locations via BGP
• Network routes to the nearest (topologically) server automatically
• Used by CDNs (Cloudflare), DNS providers (1.1.1.1), and DDoS protection
• Failover is instant — if one site goes down, BGP routes automatically shift

Edge Computing:

• Run application logic at CDN edge nodes (Cloudflare Workers, Lambda@Edge, Deno Deploy)
• Sub-50ms response times globally for personalization, A/B testing, auth checks
• Limitation: limited compute resources, no persistent connections to databases (use edge-compatible databases like Turso, Neon, PlanetScale)

flowchart TD
    USER["User Request"] --> DNS["GeoDNS Resolution"]
    DNS -->|"Europe"| EU_POP["EU Edge PoP
(London, Frankfurt)"]
    DNS -->|"Americas"| US_POP["US Edge PoP
(Virginia, Oregon)"]
    DNS -->|"Asia-Pacific"| AP_POP["APAC Edge PoP
(Singapore, Tokyo)"]

    EU_POP --> EU_EDGE["Edge Logic
(Auth, Cache, A/B)"]
    US_POP --> US_EDGE["Edge Logic
(Auth, Cache, A/B)"]
    AP_POP --> AP_EDGE["Edge Logic
(Auth, Cache, A/B)"]

    EU_EDGE -->|"Cache MISS"| EU_ORIGIN["EU Origin
(Ireland)"]
    US_EDGE -->|"Cache MISS"| US_ORIGIN["US Origin
(us-east-1)"]
    AP_EDGE -->|"Cache MISS"| AP_ORIGIN["APAC Origin
(ap-southeast-1)"]

    EU_ORIGIN --> DB_PRIMARY["Primary DB
(us-east-1)"]
    US_ORIGIN --> DB_PRIMARY
    AP_ORIGIN --> DB_PRIMARY

    style USER fill:#e8f4f4,stroke:#3B9797,color:#132440
    style EU_EDGE fill:#f0f4f8,stroke:#16476A,color:#132440
    style US_EDGE fill:#f0f4f8,stroke:#16476A,color:#132440
    style AP_EDGE fill:#f0f4f8,stroke:#16476A,color:#132440
    style DB_PRIMARY fill:#fdf0f0,stroke:#BF092F,color:#132440
                            

# Load Testing — Verify scaling behavior under pressure
# Using 'hey' (Go-based HTTP load generator)

# Install hey (if not already available)
# go install github.com/rakyll/hey@latest

# Baseline: 100 concurrent users, 10,000 total requests
echo "=== Baseline Load Test (single server) ==="
hey -n 10000 -c 100 -m GET \
    -H "Authorization: Bearer $API_TOKEN" \
    https://api.example.com/v1/health

# Ramp up: 500 concurrent users
echo ""
echo "=== Stress Test (500 concurrent) ==="
hey -n 50000 -c 500 -m GET \
    -H "Authorization: Bearer $API_TOKEN" \
    https://api.example.com/v1/health

# Sustained load: 200 concurrent for 60 seconds
echo ""
echo "=== Sustained Load (200 concurrent, 60s) ==="
hey -z 60s -c 200 -m GET \
    -H "Authorization: Bearer $API_TOKEN" \
    https://api.example.com/v1/health

# Key metrics to watch:
# - p99 latency (should stay under 500ms)
# - Error rate (should stay under 0.1%)
# - Requests/sec throughput
# - Status code distribution (any 502/503 = backend overload)

Case Studies

Twitter's Fail Whale: Early Scaling Failure

Discord: Achieving 140M+ Users with Smart Scaling

Conclusion & Next Steps

Modules 14 and 15 covered the fundamental mechanics of scaling — why we scale, the two primary strategies (vertical and horizontal), and the algorithms that distribute load across a fleet.

The key takeaways:

• Vertical scaling is your first move — simple, cheap, and sufficient longer than most engineers expect. Don't distribute prematurely.
• Horizontal scaling is inevitable — once you hit the vertical ceiling or need high availability, you must distribute. Accept the complexity cost.
• Amdahl's Law is your ceiling. The sequential fraction of your workload limits maximum speedup. Eliminate serialization points before adding more servers.
• Load balancing algorithm choice matters. Round robin for uniform workloads, least connections for varying request durations, consistent hashing for caches and stateful routing.
• Consistent hashing enables distributed caches. Without it, adding/removing a cache server causes near-total cache invalidation — a stampede that can take down your database.
• Global distribution reduces latency but adds complexity. GeoDNS, anycast, and edge computing bring compute closer to users — but data replication and consistency become harder.