System Design Series Part 2: Scalability Fundamentals

1. User Experience

Amazon found that every 100ms of latency costs them 1% in sales. Google discovered that a 500ms delay reduced search traffic by 20%. Users don't wait—they leave.

2. Cost Efficiency

A well-scaled system uses resources efficiently. Over-provisioning wastes money; under-provisioning causes outages. The goal is to scale proportionally with demand.

3. Business Growth

Systems that can't scale become bottlenecks to business growth. When a viral marketing campaign succeeds, your infrastructure must handle the surge.

No Single Point of Failure

When one server fails, others continue serving requests. With proper load balancing, users may not even notice the failure.

Linear Cost Scaling

Commodity hardware is cheaper per unit of compute than high-end servers. Ten $1,000 servers often provide more capacity than one $10,000 server.

Theoretically Unlimited

You can keep adding servers until you hit network or coordination limits. Companies like Google run millions of servers.

The Twelve-Factor App

The Twelve-Factor methodology provides guidelines for building scalable applications:

• Factor VI: Execute the app as one or more stateless processes
• Factor IX: Maximize robustness with fast startup and graceful shutdown
• Factor XI: Treat logs as event streams

Simplicity

No code changes required. No data partitioning. No distributed coordination. Just bigger hardware.

Data Consistency

Single server means no distributed transactions, no eventual consistency, no split-brain scenarios.

Lower Latency

No network calls between servers. All operations happen in-memory on a single machine.

Hardware Failures

• Disk failures: ~2-4% annual failure rate for HDDs
• Server failures: Power supplies, memory, network cards
• Network failures: Router issues, cable cuts, switch failures
• Data center failures: Power outages, cooling failures, natural disasters

Software Failures

• Bugs causing crashes or incorrect behavior
• Memory leaks leading to gradual degradation
• Resource exhaustion (connections, file handles)
• Cascading failures from dependency issues

Human Errors

• Configuration mistakes (wrong environment, typos)
• Deployment issues (bad code, incomplete rollout)
• Operational errors (accidentally deleting data)

Studies show human error causes 70-80% of outages.

N+1 Redundancy

Run one more instance than required to handle load. If one fails, remaining capacity is sufficient.

# N+1 Example
required_capacity = 10000  # requests/second
single_server_capacity = 2500

# Minimum servers needed: 10000/2500 = 4
# With N+1: 5 servers (4 for capacity + 1 for failure)
servers_deployed = 5
capacity_per_server = required_capacity / (servers_deployed - 1)
# Each server handles 2500 RPS, one can fail without impact

N+2 or 2N Redundancy

For critical systems, maintain higher redundancy levels. 2N (double capacity) allows maintenance and failure simultaneously.

Geographic Redundancy

Deploy across multiple data centers or regions to survive facility-level failures.

• Same Region: Multiple availability zones (protects against data center failure)
• Multi-Region: Multiple geographic regions (protects against regional disasters)

Failover

Automatically switch to a backup component when the primary fails.

# Failover pseudocode
class DatabaseClient:
    def __init__(self):
        self.primary = "db-primary.example.com"
        self.secondary = "db-secondary.example.com"
        self.current = self.primary
    
    def execute(self, query):
        try:
            return self.connect(self.current).execute(query)
        except ConnectionError:
            # Failover to secondary
            self.current = self.secondary
            return self.connect(self.current).execute(query)

Replication

Keep multiple copies of data synchronized across nodes.

• Synchronous: All replicas confirm before write completes (consistent but slower)
• Asynchronous: Write completes immediately, replicas sync later (faster but potential data loss)

Circuit Breaker

Prevent cascade failures by stopping requests to failing services.

# Circuit Breaker Pattern
class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_timeout=60):
        self.failures = 0
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.state = "CLOSED"  # CLOSED, OPEN, HALF_OPEN
        self.last_failure_time = None
    
    def call(self, func):
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.reset_timeout:
                self.state = "HALF_OPEN"
            else:
                raise CircuitOpenError("Circuit breaker is open")
        
        try:
            result = func()
            self.on_success()
            return result
        except Exception as e:
            self.on_failure()
            raise e
    
    def on_success(self):
        self.failures = 0
        self.state = "CLOSED"
    
    def on_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()
        if self.failures >= self.failure_threshold:
            self.state = "OPEN"

Graceful Degradation

Continue providing partial service when components fail.

• Recommendation service down? Show popular items instead
• Search slow? Return cached results
• Payment processing delayed? Queue orders for retry

Retries with Exponential Backoff

Retry failed operations with increasing delays to prevent overwhelming recovering services.

# Exponential Backoff with Jitter
import random
import time

def retry_with_backoff(func, max_retries=5, base_delay=1):
    for attempt in range(max_retries):
        try:
            return func()
        except Exception as e:
            if attempt == max_retries - 1:
                raise e
            
            # Exponential backoff: 1s, 2s, 4s, 8s, 16s...
            delay = base_delay * (2 ** attempt)
            
            # Add jitter to prevent thundering herd
            jitter = random.uniform(0, delay * 0.1)
            
            print(f"Attempt {attempt + 1} failed, retrying in {delay + jitter:.2f}s")
            time.sleep(delay + jitter)

Netflix's Active-Active Architecture

Netflix runs active-active across three AWS regions (US-East, US-West, EU-West). Benefits:

• Any region can handle full production traffic
• Regional failures don't affect users (traffic shifts automatically)
• Can deploy updates region-by-region for safer rollouts

Cost: Running 3x infrastructure plus complex data replication (Cassandra with eventual consistency).

1. Backup & Restore

Regular backups with restoration procedures. Cheapest but slowest recovery.

• RTO: Hours to days
• RPO: Depends on backup frequency (hourly, daily)
• Cost: Storage costs only

2. Pilot Light

Core services (database) running at minimal capacity in DR region. Scale up when needed.

• RTO: Tens of minutes to hours
• RPO: Minutes (continuous replication)
• Cost: Minimal running infrastructure

3. Warm Standby

Scaled-down but fully functional version running continuously.

• RTO: Minutes
• RPO: Seconds to minutes
• Cost: ~20-30% of production

4. Hot Standby (Multi-Site Active-Active)

Full production replica ready for immediate failover.

• RTO: Seconds (automatic)
• RPO: Near-zero (synchronous replication)
• Cost: 100%+ of production

Auto-Scaling Groups

Automatically add/remove instances based on metrics (CPU, memory, request count).

# AWS Auto Scaling Configuration (pseudo-YAML)
AutoScalingGroup:
  MinSize: 2
  MaxSize: 20
  DesiredCapacity: 4
  
  ScalingPolicies:
    - Name: ScaleOutOnHighCPU
      MetricType: CPUUtilization
      Threshold: 70%
      ScalingAdjustment: +2 instances
      Cooldown: 300 seconds
    
    - Name: ScaleInOnLowCPU  
      MetricType: CPUUtilization
      Threshold: 30%
      ScalingAdjustment: -1 instance
      Cooldown: 600 seconds

Read Replicas

For read-heavy workloads, create read replicas to distribute query load.

• Primary handles writes and time-sensitive reads
• Replicas handle read queries (search, reports, analytics)
• Typical read/write ratio: 90/10 to 99/1

Database Sharding

Partition data across multiple databases for write scaling. We'll cover this in depth in Part 4.

Multi-Level Caching

Cache at multiple levels to maximize efficiency:

1. Browser Cache: Static assets (hours to days TTL)
2. CDN Cache: Static and semi-dynamic content (minutes to hours)
3. Application Cache: Computed results, session data (seconds to minutes)
4. Database Cache: Query results, hot data (seconds to minutes)