TensorFlow 2 Deep Learning: Complete Beginner's Guide from Basics to Production

Introduction: What is TensorFlow?

What is TensorFlow and Why Use It?

TensorFlow is Google's open-source deep learning framework that has become the industry standard for building and deploying neural networks at scale. Originally released in 2015, TensorFlow 2 (launched in 2019) revolutionized the framework by making it more Pythonic and beginner-friendly through eager execution—meaning operations are evaluated immediately rather than requiring a separate session.

Unlike traditional machine learning libraries that work well for tabular data and classical algorithms (like scikit-learn), TensorFlow excels at handling complex neural network architectures for tasks like image recognition, natural language processing, time series forecasting, and generative AI. Its tight integration with Keras (a high-level API) makes it accessible to beginners while still offering low-level control for researchers.

Where TensorFlow Fits in Data Science & AI

Understanding where TensorFlow fits in the broader data science ecosystem is crucial for beginners. Think of it as a hierarchy:

1. Data Exploration & Cleaning: Use pandas and NumPy to load, clean, and explore your data. TensorFlow doesn't replace these—it builds on them.

2. Classical Machine Learning: For structured/tabular data, scikit-learn often performs better with algorithms like Random Forests, Gradient Boosting, and SVMs. Use TensorFlow when you need deep learning.

3. Deep Learning: This is TensorFlow's domain. When you have large datasets, complex patterns (images, text, sequences), or need neural networks, TensorFlow shines. It handles automatic differentiation, GPU acceleration, and model optimization.

4. Generative AI: Large language models (LLMs) and transformers can be built with TensorFlow, though PyTorch has gained popularity here. TensorFlow's ecosystem (TF-Agents for reinforcement learning, TensorFlow Probability for Bayesian methods) extends its capabilities.

Installation & Setup Verification

Before diving in, ensure TensorFlow is installed. If you haven't installed it yet, run this command in your terminal:

pip install tensorflow tensorflow-datasets tensorboard

This installs TensorFlow 2, TensorFlow Datasets (curated datasets), and TensorBoard (visualization toolkit). Now let's verify the installation and check available hardware:

import tensorflow as tf
import numpy as np

# Verify TensorFlow version
print('TensorFlow version:', tf.__version__)

# Check eager execution (should be True by default in TF 2)
print('Eager execution:', tf.executing_eagerly())

# Check available GPUs
print('Available GPUs:', len(tf.config.list_physical_devices('GPU')))

# List all physical devices
print('All devices:', tf.config.list_physical_devices())

Expected output shows TensorFlow 2.x, eager execution enabled, and lists available devices (CPU, GPU if present). If GPUs are detected, TensorFlow will automatically use them for operations—no manual configuration needed for most cases.

Part 1: Foundations

Core Concepts: Tensors & Eager Execution

At the heart of TensorFlow are tensors—multi-dimensional arrays similar to NumPy's ndarrays but with superpowers: automatic differentiation, GPU acceleration, and seamless integration with neural network operations. Think of a tensor as a generalization of matrices: scalars are 0D, vectors are 1D, matrices are 2D, and higher-dimensional arrays are tensors.

Let's create basic tensors and understand their fundamental properties:

import tensorflow as tf

# METHOD 1: Create a 1D tensor (vector) from a list
# tf.constant() creates immutable tensor from Python data
# Parameters:
#   - value: the data (list, NumPy array, etc.)
#   - dtype: data type (tf.int32, tf.float32, tf.float64, etc.)
a = tf.constant([1, 2, 3], dtype=tf.int32)
print('1D tensor a:', a)
# Output: tf.Tensor([1 2 3], shape=(3,), dtype=int32)

# Inspect tensor properties
print('Shape:', a.shape)        # (3,) - one dimension with 3 elements
print('Rank:', tf.rank(a).numpy())  # 1 - one dimension means rank=1

# METHOD 2: Create a 2D tensor (matrix) from nested list
b = tf.constant([[1.0, 2.0], [3.0, 4.0]])
print('\n2D tensor b:\n', b)
# Output: 2x2 matrix with 4 elements total

# Inspect 2D tensor properties
print('Shape:', b.shape)        # (2, 2) - 2 rows, 2 columns
print('Rank:', tf.rank(b).numpy())  # 2 - two dimensions means rank=2
print('Data type:', b.dtype)    # 

# METHOD 3: Create tensors with specific values
zeros = tf.zeros((3, 4))    # 3x4 matrix of zeros
ones = tf.ones((2, 3))      # 2x3 matrix of ones
print('\nZeros shape:', zeros.shape)
print('Ones shape:', ones.shape)

Key Tensor Properties:

• shape: Dimensions of the tensor (e.g., (3, 4) means 3 rows, 4 columns)
• rank: Number of dimensions (0=scalar, 1=vector, 2=matrix, 3+=higher-order)
• dtype: Data type (int32, float32, float64, etc.). float32 is standard; float64 for high precision
• device: Where tensor lives (CPU or GPU). Must match model device to avoid errors

Converting Between NumPy and TensorFlow

Since NumPy is the foundation of Python data science, TensorFlow offers seamless conversion. This is essential for loading data with NumPy or exporting results:

import tensorflow as tf
import numpy as np

# METHOD 1: NumPy array to TensorFlow tensor
# tf.convert_to_tensor() wraps NumPy data as TensorFlow tensor
np_array = np.array([[10, 20], [30, 40]], dtype=np.float32)
tensor = tf.convert_to_tensor(np_array)
print('Converted to tensor:', tensor)
print('Type:', type(tensor))  # 

# METHOD 2: TensorFlow tensor to NumPy array
# .numpy() extracts underlying NumPy array from tensor
back_to_numpy = tensor.numpy()
print('Back to NumPy:', back_to_numpy)
print('Type:', type(back_to_numpy))  # 

# Use case: After inference, export predictions as NumPy for visualization
predictions = tensor  # Some model output
predictions_np = predictions.numpy()  # Convert to NumPy for matplotlib
import matplotlib.pyplot as plt
# plt.plot(predictions_np)  # Now can plot with matplotlib

.numpy() is essential for exporting TensorFlow results to visualization libraries (matplotlib, seaborn) or data tools (pandas). However, note that extracting to NumPy loses GPU benefits.

Tensor Operations & Transformations

TensorFlow provides hundreds of operations mirroring NumPy for familiarity. Common transformations include reshaping, transposing, slicing, casting, and concatenating tensors. These are the building blocks for preprocessing data:

import tensorflow as tf

# Create a range of values (like NumPy arange)
# tf.range(start, limit, delta)
# Parameters:
#   - start: starting value (inclusive)
#   - limit: ending value (exclusive, not included)
#   - delta: step size
x = tf.range(12)  # [0, 1, 2, ..., 11]
print('Original tensor:', x)

# RESHAPE: Change dimensions without changing data
# tf.reshape(tensor, shape)
# Important: product of new shape must equal total elements
# Example: 12 elements → (3, 4) or (2, 6) or (12,) all valid
x_reshaped = tf.reshape(x, (3, 4))  # Reshape to 3 rows, 4 columns
print('Reshaped to 3x4:\n', x_reshaped)
# Output: [[0, 1, 2, 3],
#          [4, 5, 6, 7],
#          [8, 9, 10, 11]]

# TRANSPOSE: Swap dimensions (flip rows and columns for 2D)
# tf.transpose(tensor) - 2D becomes (cols, rows)
x_transposed = tf.transpose(x_reshaped)  # Was (3, 4), now (4, 3)
print('Transposed (4x3):\n', x_transposed)

# CAST: Change data type
# tf.cast(tensor, dtype)
# Use case: Convert int to float before division or neural network
x_float = tf.cast(x, tf.float32)  # Convert integers to floats
print('Casted to float32:', x_float)

Type Casting and Slicing

Convert between data types with tf.cast() and extract subsets using Python-style indexing:

import tensorflow as tf

# Create integer tensor
x = tf.constant([[1, 2, 3], [4, 5, 6]])

# Cast to float32
x_float = tf.cast(x, tf.float32)
print('Cast to float32:', x_float.dtype)

# Slicing: extract row 1
print('Row 1:', x[1])

# Slicing: extract column 2
print('Column 2:', x[:, 2])

# Slicing: extract submatrix (rows 0-1, cols 1-2)
print('Submatrix:\n', x[0:2, 1:3])

Concatenation and Stacking

Combine tensors along existing or new dimensions:

import tensorflow as tf

# Create two tensors
a = tf.constant([[1, 2], [3, 4]])
b = tf.constant([[5, 6], [7, 8]])

# Concatenate along rows (axis=0)
concat_rows = tf.concat([a, b], axis=0)
print('Concatenated rows (4x2):\n', concat_rows)

# Concatenate along columns (axis=1)
concat_cols = tf.concat([a, b], axis=1)
print('Concatenated columns (2x4):\n', concat_cols)

# Stack creates a new dimension
stacked = tf.stack([a, b], axis=0)
print('Stacked (2x2x2):\n', stacked)

Use tf.concat() to merge along existing axes and tf.stack() to create a new dimension—critical for batching data in neural networks.

Variables & Automatic Differentiation

tf.Variable represents mutable tensors used for model parameters (weights, biases). Unlike immutable tf.constant, variables can be updated during training. TensorFlow's GradientTape records operations to compute gradients automatically—the backbone of backpropagation and neural network training.

import tensorflow as tf

# Create VARIABLES for model parameters
# Variables are mutable—they will be updated during training
# Parameters:
#   - initial_value: starting values (tensor or initializer)
#   - name: optional name for debugging
#   - trainable: whether to include in optimizer updates (default: True)
W = tf.Variable(tf.random.normal([3, 3]), name='weights')
# Shape [3, 3] means: 3 input features, 3 output features

b = tf.Variable(tf.zeros([3]), name='bias')
# Shape [3] means: 3 bias terms (one per output)

print('Weight shape:', W.shape)  # torch.Size([3, 3])
print('Bias shape:', b.shape)    # torch.Size([3])
print('Is trainable:', W.trainable)  # True (can be updated)
print('Variable dtype:', W.dtype)    # 

# Create CONSTANTS for fixed values
x_constant = tf.constant([[1.0, 2.0, 3.0]])
print('\nConstant x:', x_constant)
# Constants are immutable—can't update them

Automatic Differentiation with GradientTape

GradientTape is TensorFlow's mechanism for automatic differentiation (computing derivatives). It records all operations inside its context, then traces backwards to compute how changes in variables affect the loss:

import tensorflow as tf

# Create variables (model parameters)
W = tf.Variable(tf.random.normal([3, 3]), name='W')
b = tf.Variable(tf.zeros([3]), name='b')

# Create sample input data (batch of 5 samples, 3 features each)
x = tf.random.normal([5, 3])

# GradientTape records all operations inside 'with' block
# Think of it as "recording a video" of computations
with tf.GradientTape() as tape:
    # Forward pass: compute predictions
    # y = x @ W + b (matrix multiplication then add bias)
    y = tf.matmul(x, W) + b  # Output shape: [5, 3]
    
    # Compute loss (mean squared error)
    # MSE = mean((predictions - 0)^2)
    # We're using zero as target (just for this example)
    loss = tf.reduce_mean(tf.square(y))
    # loss is a scalar (single number)

# Compute gradients
# tape.gradient(output, variables) computes ∂output/∂variables
# This is THE key operation: tells us how to adjust W and b to reduce loss
grads = tape.gradient(loss, [W, b])

# Results: grads[0] = ∂loss/∂W, grads[1] = ∂loss/∂b
print('Loss:', loss.numpy())  # Single scalar value
print('Gradient W shape:', grads[0].shape)  # [3, 3] - same as W
print('Gradient b shape:', grads[1].shape)  # [3] - same as b

# These gradients tell us:
# - Which direction to move W to reduce loss
# - Which direction to move b to reduce loss
# An optimizer will use these to update the variables

Manual Gradient Descent Step

While optimizers automate this, let's see the basic principle: update weights using gradients and a learning rate:

import tensorflow as tf

# Variables (our model parameters)
W = tf.Variable([[1.0, 2.0], [3.0, 4.0]], name='W')
b = tf.Variable([0.5, 0.5], name='b')

# Input and target data
x = tf.constant([[1.0, 2.0]])        # 1 sample, 2 features
target = tf.constant([[5.0, 6.0]])   # 1 sample, 2 outputs

# Hyperparameter: controls how big a step we take
# Too small: training is slow
# Too large: weights oscillate and don't converge
learning_rate = 0.01

# STEP 1: Forward pass and compute loss
with tf.GradientTape() as tape:
    # Predict: pred = x @ W + b
    prediction = tf.matmul(x, W) + b  # Output shape: [1, 2]
    
    # Loss: MSE = mean((prediction - target)^2)
    # Measures how far prediction is from target
    loss = tf.reduce_mean(tf.square(target - prediction))

# STEP 2: Compute gradients
# ∂loss/∂W tells us how W affects the loss
# ∂loss/∂b tells us how b affects the loss
grads = tape.gradient(loss, [W, b])

print('Initial loss:', loss.numpy())

# STEP 3: Update variables (the "learning" happens here)
# Gradient descent: new_value = old_value - learning_rate * gradient
# assign_sub() means "subtract and assign": W -= learning_rate * grad_W
W.assign_sub(learning_rate * grads[0])  # W = W - lr * ∂loss/∂W
b.assign_sub(learning_rate * grads[1])  # b = b - lr * ∂loss/∂b

print('Updated W:\n', W.numpy())
print('Updated b:', b.numpy())
print('\nAfter one gradient step, loss should be smaller')

The assign_sub() method updates variables in-place—this is the core of training algorithms like SGD, Adam, etc.

Building Your First Models

Keras, now fully integrated into TensorFlow, offers three APIs for building models: Sequential (simple stacks), Functional (complex graphs), and Subclassing (full control). Start with Sequential for learning, graduate to Functional for real projects, and use Subclassing for research.

Sequential API: Linear Stack of Layers

Perfect for feedforward networks where data flows linearly through layers:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Build a simple neural network
# keras.Sequential = a linear stack of layers where output of one feeds into next
model = keras.Sequential([
    # Dense layer: fully connected layer
    # Parameters:
    #   - units: number of neurons (output dimensionality)
    #   - activation: activation function (relu, sigmoid, tanh, etc.)
    #   - input_shape: input dimensionality (only on first layer)
    # Computes: output = relu(input @ W + b) where W is [16, 32]
    layers.Dense(32, activation='relu', input_shape=(16,)),
    
    # Second hidden layer: input shape auto-inferred from previous layer output
    # Input shape is [32] (from previous Dense output)
    # Computes: output = relu(input @ W + b) where W is [32, 16]
    layers.Dense(16, activation='relu'),
    
    # Output layer: no activation for regression (raw prediction)
    # For classification: use activation='softmax' for probabilities
    layers.Dense(1)
])

# Display the model architecture
# Shows layer types, output shapes, and parameter counts
model.summary()

Understanding Dense Layers

A Dense layer is a fully connected layer where every input connects to every output. It performs: output = activation(input @ weights + bias).

import tensorflow as tf
from tensorflow.keras import layers

# Create a Dense layer
# Units = 32 means: output will have 32 values (32 neurons)
layer = layers.Dense(
    units=32,                  # Output dimension
    activation='relu',         # Apply ReLU: max(0, x)
    input_shape=(16,)         # Input dimension (16 features)
)

# What happens internally:
# - weights shape: [16, 32]  (each of 16 inputs connects to 32 outputs)
# - bias shape: [32]          (one bias per output neuron)
# - computation: output = relu(input @ weights + bias)
#
# Example:
# input shape: [batch_size=5, features=16]
# output shape: [batch_size=5, units=32]

# Test it
sample_input = tf.random.normal([5, 16])  # 5 samples, 16 features each
output = layer(sample_input)
print('Input shape:', sample_input.shape)   # [5, 16]
print('Output shape:', output.shape)        # [5, 32]
print('Number of parameters:', layer.count_params())  # (16 * 32) + 32 = 544

Compiling the Model

Compilation configures the optimizer (how to update weights), loss function (what to minimize), and metrics (what to track). Think of it as "getting the model ready to learn":

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Build model
model = keras.Sequential([
    layers.Dense(64, activation='relu', input_shape=(20,)),
    layers.Dense(10, activation='softmax')  # 10-class classification
])

# COMPILE: Configure the training process
# Parameters explained:
#   - optimizer: algorithm to update weights
#     * 'adam': Adaptive learning rate (best for beginners, works for most problems)
#     * Can also pass custom: keras.optimizers.Adam(learning_rate=0.001)
#   - loss: what function to minimize during training
#     * 'sparse_categorical_crossentropy': for integer labels [0, 1, 2, ..., 9]
#     * 'categorical_crossentropy': for one-hot labels [[1,0,0,...], [0,1,0,...], ...]
#     * 'mse': mean squared error for regression
#     * 'binary_crossentropy': for binary classification (2 classes)
#   - metrics: quantities to monitor during training (not used for optimization)
#     * 'accuracy': fraction of correct predictions
#     * Can include multiple: metrics=['accuracy', 'precision', 'recall']
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

print('Model compiled. Architecture defined. Ready for training.')

# TRAINING FLOW:
# 1. Load training data (X_train, y_train)
# 2. model.fit(X_train, y_train, epochs=10, batch_size=32)
# 3. For each epoch:
#    - For each batch:
#      * Forward pass: predictions = model(batch_X)
#      * Compute loss: loss_val = loss_function(predictions, batch_y)
#      * Backward pass: compute gradients via GradientTape
#      * Update: weights -= optimizer(learning_rate * gradients)
#    - Print metrics
# 4. Done! Weights learned from data

Choosing Optimizer, Loss, and Metrics

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, optimizers

# OPTIMIZER CHOICES:
# For most problems: 'adam' (Adaptive Moment Estimation)
# - Maintains per-parameter learning rates
# - Convergence is usually fast and stable
model = keras.Sequential([layers.Dense(10, activation='softmax')])

# Option 1: String shortcut (simple)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

# Option 2: Instance with custom learning rate (more control)
model.compile(
    optimizer=optimizers.Adam(learning_rate=0.001),
    loss='sparse_categorical_crossentropy'
)

# LOSS FUNCTION CHOICES:
# Classification with integer labels (e.g., [0, 1, 2, 3]):
loss_classification = 'sparse_categorical_crossentropy'

# Classification with one-hot labels (e.g., [[1,0,0], [0,1,0]]):
loss_onehot = 'categorical_crossentropy'

# Binary classification (2 classes only):
loss_binary = 'binary_crossentropy'

# Regression (continuous values):
loss_regression = 'mse'  # Mean squared error
loss_regression_alt = 'mae'  # Mean absolute error

# METRICS: purely for monitoring (don't affect training)
# Common metrics:
metrics_classification = ['accuracy']  # Fraction correct
metrics_detailed = ['accuracy', 'precision', 'recall']  # More detail
metrics_regression = ['mae', 'mse']  # Error measures

Functional API: Complex Architectures

Build models with multiple inputs/outputs, skip connections, or branching paths:

Functional API: Complex Architectures

Build models with multiple inputs/outputs, skip connections, or branching paths. In Functional API, layers are functions that transform tensors:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# FUNCTIONAL API: Treat layers as functions on tensors
# Useful for: multiple inputs, skip connections, multi-output, branching

# Step 1: Define input tensor
# keras.Input(shape=(16,)) creates an abstract input (no data yet, just shape info)
inputs = keras.Input(shape=(16,))  # input_shape = [batch_size, 16]

# Step 2: Apply layers as transformations (functional style)
# Each layer call: tensor_out = layer(tensor_in)
x = layers.Dense(
    units=32,           # output dimension
    activation='relu'   # activation function
)(inputs)              # Apply to inputs
# x shape: [batch_size, 32]

# Dropout: regularization to prevent overfitting
# parameters:
#   - rate (0.2): probability of dropping each neuron
#   - During training: randomly set 20% of inputs to 0
#   - During inference: all neurons active, but scaled to compensate
x = layers.Dropout(rate=0.2)(x)
# x shape still: [batch_size, 32] (no dimension change, just regularization)

# Another dense layer
x = layers.Dense(16, activation='relu')(x)
# x shape: [batch_size, 16]

# Output layer (no activation for regression)
outputs = layers.Dense(1)(x)
# outputs shape: [batch_size, 1]

# Step 3: Create model by specifying inputs and outputs
# keras.Model is a container that represents: inputs → computations → outputs
model = keras.Model(inputs=inputs, outputs=outputs, name='functional_model')

# View architecture
model.summary()

Functional API Example: Multi-Input Model

Process different types of data separately, then combine:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Multi-input example: combine images and metadata for better predictions
# E.g., real estate price prediction: image of house + metadata (size, rooms, etc.)

# INPUT 1: Image-like features (e.g., 28x28 image flattened to 784)
image_inputs = keras.Input(shape=(784,), name='image_input')
image_branch = layers.Dense(128, activation='relu')(image_inputs)
image_branch = layers.Dense(64, activation='relu')(image_branch)

# INPUT 2: Metadata (e.g., house size, rooms, age)
meta_inputs = keras.Input(shape=(5,), name='metadata_input')
meta_branch = layers.Dense(32, activation='relu')(meta_inputs)

# COMBINE: concatenate branches
combined = layers.Concatenate()([image_branch, meta_branch])
# combined shape: [batch_size, 64 + 32] = [batch_size, 96]

# Final layers after combining
output = layers.Dense(32, activation='relu')(combined)
output = layers.Dense(1)(output)

# Create model with MULTIPLE INPUTS and ONE OUTPUT
model = keras.Model(inputs=[image_inputs, meta_inputs], outputs=output)
model.summary()

# TRAINING: pass list of input arrays
# model.fit([X_image, X_metadata], y_price, epochs=10)

Model Subclassing: Full Python Control

For research or custom training loops, subclass keras.Model and define call():

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# MODEL SUBCLASSING: Full Python control
# Inherit from keras.Model and define call() method
class CustomModel(keras.Model):
    def __init__(self):
        # Call parent constructor
        super().__init__()
        
        # Define layers as instance variables (in __init__)
        # They will be automatically tracked for:
        # - Weight updates during training
        # - Model summary display
        # - Saving/loading
        self.dense1 = layers.Dense(32, activation='relu')
        self.dropout = layers.Dropout(0.2)
        self.dense2 = layers.Dense(1)
    
    def call(self, inputs, training=False):
        # Forward pass logic (called when model(data) is invoked)
        # Parameters:
        #   - inputs: input tensor [batch_size, features]
        #   - training: boolean flag (True during training, False during inference)
        #     * Used by layers like Dropout and BatchNormalization
        #     * Dropout: active during training (randomly drop neurons)
        #     * Dropout: inactive during inference (use all neurons)
        
        # Forward pass with explicit training flag control
        x = self.dense1(inputs)                    # [batch_size, 32]
        x = self.dropout(x, training=training)    # Conditional dropout
        output = self.dense2(x)                    # [batch_size, 1]
        return output

# Create model instance
model = CustomModel()

# Test forward pass
sample_input = tf.random.normal([4, 16])      # 4 samples, 16 features
output_train = model(sample_input, training=True)   # Training mode (dropout active)
output_infer = model(sample_input, training=False)  # Inference mode (no dropout)

print('Input shape:', sample_input.shape)      # [4, 16]
print('Output shape:', output_train.shape)     # [4, 1]
print('Training mode uses different dropout than inference mode')

The training flag controls behavior during training vs inference (e.g., dropout on/off, batch normalization statistics). Subclassing offers maximum flexibility but requires more boilerplate.

Part 2: Training & Optimization

Layers & Custom Layers

Keras provides dozens of built-in layers for common tasks: Dense (fully connected), Conv2D (2D convolution), LSTM (recurrent), Dropout (regularization), BatchNormalization (normalize activations), and more. When built-in layers aren't sufficient, create custom layers by subclassing layers.Layer.

Creating a Custom Layer

Custom layers define their own weights and forward pass logic:

import tensorflow as tf
from tensorflow.keras import layers

class ScaledDense(layers.Layer):
    def __init__(self, units, scale=1.0):
        super().__init__()
        self.units = units
        self.scale = scale
    
    def build(self, input_shape):
        # Create weights lazily (called on first forward pass)
        self.w = self.add_weight(
            shape=(input_shape[-1], self.units),
            initializer='glorot_uniform',
            trainable=True,
            name='kernel'
        )
        self.b = self.add_weight(
            shape=(self.units,),
            initializer='zeros',
            trainable=True,
            name='bias'
        )
    
    def call(self, inputs):
        # Forward pass: scale * (input @ w + b)
        return self.scale * (tf.matmul(inputs, self.w) + self.b)

# Test the custom layer
layer = ScaledDense(units=8, scale=0.5)
output = layer(tf.random.normal([2, 16]))
print('Custom layer output shape:', output.shape)

The build() method creates weights based on input shape (lazy initialization). add_weight() registers parameters for automatic gradient tracking. call() defines the forward pass transformation.

Activations, Regularization & Best Practices

Activation functions introduce non-linearity, enabling neural networks to learn complex patterns. Regularization techniques prevent overfitting by constraining model complexity.

Common Activation Functions

Activation functions decide what the neuron "fires" at. Without them, stacking layers just does linear transformations (mathematically equivalent to one layer). Activations are what enable deep learning:

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np

# Visualize common activation functions
x = np.linspace(-5, 5, 100)

# ReLU: max(0, x)
# - Returns 0 for negative inputs, x for positive
# - Default for hidden layers (fast, works well)
# - Problem: "dead neurons" if all inputs are negative
relu_out = np.maximum(0, x)

# Sigmoid: 1 / (1 + e^(-x))
# - Outputs probability [0, 1]
# - Used for binary classification output layer
# - Saturates (slopes → 0) for extreme values, slowing training
sigmoid_out = 1 / (1 + np.exp(-x))

# Tanh: (e^x - e^-x) / (e^x + e^-x)
# - Outputs [-1, 1], zero-centered
# - Better than sigmoid for hidden layers (but ReLU usually better)
tanh_out = np.tanh(x)

# Leaky ReLU: max(0.1*x, x)
# - Like ReLU but allows small negative slope (0.1x for x < 0)
# - Prevents "dead neurons" problem
leaky_relu_out = np.where(x > 0, x, 0.1 * x)

print('Activation functions:')
print('ReLU range:', relu_out.min(), 'to', relu_out.max())      # 0 to 5
print('Sigmoid range:', sigmoid_out.min(), 'to', sigmoid_out.max())  # ~0 to 1
print('Tanh range:', tanh_out.min(), 'to', tanh_out.max())      # -1 to 1
print('Leaky ReLU range:', leaky_relu_out.min(), 'to', leaky_relu_out.max())  # -0.5 to 5

Using Activations in a Model

import tensorflow as tf
from tensorflow.keras import layers

# Build model demonstrating different activations
model = tf.keras.Sequential([
    # Hidden layers: use ReLU (fast, works well)
    # ReLU: rectified linear unit
    # Computation: output = max(0, input @ weights + bias)
    layers.Dense(64, activation='relu', input_shape=(20,)),
    
    # Alternative: Leaky ReLU (prevents dead neurons)
    # Computation: output = max(0.2 * input, input) @ weights + bias
    layers.Dense(64, activation=tf.nn.leaky_relu),
    
    # Alternative: Tanh (zero-centered, works for normalized data)
    # Range: [-1, 1], good when data is centered around 0
    layers.Dense(32, activation='tanh'),
    
    # Output layer for binary classification: Sigmoid
    # Sigmoid: squashes to [0, 1] probability range
    # Computation: output = 1 / (1 + e^(-x))
    layers.Dense(1, activation='sigmoid')
])

model.summary()

# KEY RULE:
# Hidden layers: almost always ReLU (or LeakyReLU)
# Output layer depends on task:
#   - Binary classification: sigmoid
#   - Multi-class classification: softmax
#   - Regression: no activation (linear)

Softmax for Multi-class Classification

import tensorflow as tf
from tensorflow.keras import layers
import numpy as np

# SOFTMAX: Converts class scores to probabilities (sum to 1)
# Formula: softmax(x_i) = e^(x_i) / sum(e^(x_j))
# Output: probability distribution over all classes

# Example: 3 classes (dog, cat, bird)
logits = np.array([2.0, 1.0, 0.1])  # Raw scores from Dense layer
# logits[0] = 2.0 (most confident about dog)
# logits[1] = 1.0 (moderate confidence about cat)
# logits[2] = 0.1 (low confidence about bird)

# Apply softmax (manually)
exp_logits = np.exp(logits)
softmax_probs = exp_logits / exp_logits.sum()
print('Logits:', logits)
print('Softmax probs:', softmax_probs)  # [0.659, 0.242, 0.099] (sums to 1)

# Or use TensorFlow softmax
tf_softmax = tf.nn.softmax(logits).numpy()
print('TensorFlow softmax:', tf_softmax)  # Same result

# In a model:
model = tf.keras.Sequential([
    layers.Dense(64, activation='relu', input_shape=(20,)),
    layers.Dense(10, activation='softmax')  # 10 classes, outputs probabilities
])

# For training, use sparse_categorical_crossentropy with integer labels
# For inference, argmax(predictions) gives class with highest probability

Regularization: Dropout and L1/L2

Regularization prevents overfitting (memorizing training data instead of learning patterns). It constrains model complexity by penalizing large weights or randomly deactivating neurons:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Model with regularization techniques
model = keras.Sequential([
    # LAYER 1: L2 Regularization (Weight Penalty)
    layers.Dense(
        units=64,                              # output size
        activation='relu',
        # L2 regularization: adds lambda * sum(weights^2) to loss
        # Effect: larger lambda → smaller weights → simpler model
        # This prevents weights from growing too large (overfitting)
        kernel_regularizer=keras.regularizers.l2(0.0001),
        input_shape=(20,)
    ),
    
    # DROPOUT: Regularization by random neuron deactivation
    # During training: randomly drop (set to 0) 30% of neurons
    # During inference: use all neurons (no dropout)
    # Effect: Forces network to learn redundant representations
    #         Prevents co-adaptation of neurons
    layers.Dropout(rate=0.3),
    
    # LAYER 2: L1 Regularization (Sparsity)
    layers.Dense(
        units=32,
        activation='relu',
        # L1 regularization: adds lambda * sum(|weights|) to loss
        # Effect: drives some weights to exactly 0 (feature selection)
        # Result: sparse weights, simpler model
        kernel_regularizer=keras.regularizers.l1(0.00001),
    ),
    
    # Another dropout layer
    layers.Dropout(rate=0.2),
    
    # Output layer: 10-class classification with softmax
    layers.Dense(10, activation='softmax')
])

model.summary()

# REGULARIZATION COMPARISON:
# L2 (Ridge):  Shrinks weights toward zero, all non-zero
#              Good for: general-purpose regularization
# L1 (Lasso):  Drives some weights to exactly zero
#              Good for: feature selection, sparse models
# Dropout:     Randomly deactivates neurons
#              Good for: preventing co-adaptation, very effective

Understanding Dropout in Detail

import tensorflow as tf
from tensorflow.keras import layers
import numpy as np

# DROPOUT: Randomly zero activations with probability p
# E.g., Dropout(0.3) means: 30% chance each neuron is dropped

# Simulate dropout manually
activations = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
print('Original activations:', activations)

# Training phase: randomly drop with probability p=0.3
dropout_rate = 0.3
mask = np.random.binomial(1, 1 - dropout_rate, size=activations.shape)
dropped = activations * mask / (1 - dropout_rate)  # Scale to maintain expectation
print('After dropout (training):', dropped)

# Inference phase: keep all activations (no dropout)
print('After dropout (inference):', activations)

# In TensorFlow:
layer = layers.Dropout(0.3)
test_input = tf.constant([[1.0, 2.0, 3.0, 4.0, 5.0]])

# During training
output_train = layer(test_input, training=True)
print('\nTensorFlow dropout (training):', output_train.numpy())

# During inference
output_infer = layer(test_input, training=False)
print('TensorFlow dropout (inference):', output_infer.numpy())
# Same input, same output during inference

Visualizing Regularization Effects on Overfitting

import tensorflow as tf
from tensorflow.keras import layers
import numpy as np
import matplotlib.pyplot as plt

# Create a complex synthetic dataset prone to overfitting
np.random.seed(42)
n_train = 100
n_val = 20

# Generate non-linear data with noise
X_train = np.random.randn(n_train, 10)
y_train = np.sin(X_train[:, 0]) + np.sin(X_train[:, 1]) + 0.1 * np.random.randn(n_train)

X_val = np.random.randn(n_val, 10)
y_val = np.sin(X_val[:, 0]) + np.sin(X_val[:, 1]) + 0.1 * np.random.randn(n_val)

# Model WITHOUT regularization (prone to overfitting)
model_no_reg = tf.keras.Sequential([
    layers.Dense(128, activation='relu', input_shape=(10,)),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(1)
])

# Model WITH regularization (L2 + Dropout)
model_with_reg = tf.keras.Sequential([
    layers.Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001), input_shape=(10,)),
    layers.Dropout(0.3),
    layers.Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.Dropout(0.3),
    layers.Dense(64, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.Dropout(0.2),
    layers.Dense(64, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.Dense(1)
])

# Compile both models
model_no_reg.compile(optimizer='adam', loss='mse')
model_with_reg.compile(optimizer='adam', loss='mse')

# Train both models
print('Training model WITHOUT regularization...')
history_no_reg = model_no_reg.fit(
    X_train, y_train,
    epochs=100,
    validation_data=(X_val, y_val),
    verbose=0,
    batch_size=16
)

print('Training model WITH regularization (L2 + Dropout)...')
history_with_reg = model_with_reg.fit(
    X_train, y_train,
    epochs=100,
    validation_data=(X_val, y_val),
    verbose=0,
    batch_size=16
)

# Create visualization
fig, axes = plt.subplots(2, 2, figsize=(14, 10))

# Plot 1: Training vs Validation Loss (No Regularization)
ax = axes[0, 0]
ax.plot(history_no_reg.history['loss'], 'b-', linewidth=2, label='Training Loss', alpha=0.8)
ax.plot(history_no_reg.history['val_loss'], 'r-', linewidth=2, label='Validation Loss', alpha=0.8)
ax.fill_between(range(len(history_no_reg.history['loss'])), 
                 history_no_reg.history['loss'], 
                 history_no_reg.history['val_loss'], 
                 alpha=0.2, color='orange', label='Overfitting Gap')
ax.set_xlabel('Epoch', fontsize=11, fontweight='bold')
ax.set_ylabel('Loss', fontsize=11, fontweight='bold')
ax.set_title('WITHOUT Regularization (Overfitting)', fontsize=12, fontweight='bold')
ax.legend(fontsize=10)
ax.grid(True, alpha=0.3)

# Plot 2: Training vs Validation Loss (With Regularization)
ax = axes[0, 1]
ax.plot(history_with_reg.history['loss'], 'b-', linewidth=2, label='Training Loss', alpha=0.8)
ax.plot(history_with_reg.history['val_loss'], 'r-', linewidth=2, label='Validation Loss', alpha=0.8)
ax.fill_between(range(len(history_with_reg.history['loss'])), 
                 history_with_reg.history['loss'], 
                 history_with_reg.history['val_loss'], 
                 alpha=0.2, color='lightgreen', label='Regularization Effect')
ax.set_xlabel('Epoch', fontsize=11, fontweight='bold')
ax.set_ylabel('Loss', fontsize=11, fontweight='bold')
ax.set_title('WITH Regularization (Better Generalization)', fontsize=12, fontweight='bold')
ax.legend(fontsize=10)
ax.grid(True, alpha=0.3)

# Plot 3: Overfitting gap comparison
ax = axes[1, 0]
gap_no_reg = np.array(history_no_reg.history['val_loss']) - np.array(history_no_reg.history['loss'])
gap_with_reg = np.array(history_with_reg.history['val_loss']) - np.array(history_with_reg.history['loss'])

epochs_range = np.arange(len(gap_no_reg))
ax.plot(epochs_range, gap_no_reg, 'r-', linewidth=2.5, label='No Regularization', marker='o', markersize=3, alpha=0.8)
ax.plot(epochs_range, gap_with_reg, 'g-', linewidth=2.5, label='With Regularization', marker='s', markersize=3, alpha=0.8)
ax.axhline(y=0, color='black', linestyle='--', alpha=0.5)
ax.fill_between(epochs_range, gap_no_reg, alpha=0.2, color='red')
ax.fill_between(epochs_range, gap_with_reg, alpha=0.2, color='green')

ax.set_xlabel('Epoch', fontsize=11, fontweight='bold')
ax.set_ylabel('Validation Loss - Training Loss', fontsize=11, fontweight='bold')
ax.set_title('Overfitting Gap: Regularization Reduces Generalization Error', fontsize=12, fontweight='bold')
ax.legend(fontsize=10)
ax.grid(True, alpha=0.3)

# Plot 4: Generalization performance
ax = axes[1, 1]
final_metrics = {
    'Training Loss\n(No Reg)': history_no_reg.history['loss'][-1],
    'Validation Loss\n(No Reg)': history_no_reg.history['val_loss'][-1],
    'Training Loss\n(With Reg)': history_with_reg.history['loss'][-1],
    'Validation Loss\n(With Reg)': history_with_reg.history['val_loss'][-1]
}

colors_bars = ['skyblue', 'salmon', 'lightgreen', 'lightyellow']
bars = ax.bar(range(len(final_metrics)), list(final_metrics.values()), color=colors_bars, edgecolor='black', linewidth=1.5)

# Add value labels
for bar, (label, val) in zip(bars, final_metrics.items()):
    ax.text(bar.get_x() + bar.get_width()/2, bar.get_height(),
            f'{val:.3f}', ha='center', va='bottom', fontweight='bold', fontsize=10)

ax.set_xticks(range(len(final_metrics)))
ax.set_xticklabels(final_metrics.keys(), fontsize=9, fontweight='bold')
ax.set_ylabel('Loss Value', fontsize=11, fontweight='bold')
ax.set_title('Final Performance: Regularization Improves Validation', fontsize=12, fontweight='bold')
ax.grid(True, alpha=0.3, axis='y')

# Add improvement annotation
improvement = ((history_no_reg.history['val_loss'][-1] - history_with_reg.history['val_loss'][-1]) / 
               history_no_reg.history['val_loss'][-1] * 100)
ax.text(0.5, 0.95, f'Validation Loss Improvement: {improvement:.1f}%',
        transform=ax.transAxes, fontsize=11, fontweight='bold',
        ha='center', va='top', 
        bbox=dict(boxstyle='round', facecolor='lightyellow', alpha=0.8, edgecolor='orange', linewidth=2))

plt.tight_layout()
plt.show()

# Print regularization analysis
print('\nRegularization Impact Summary:')
print('='*70)
print(f'{"Metric":<35} {"No Regularization":<20} {"With L2+Dropout":<15}')
print('='*70)
print(f'{"Final Training Loss":<35} {history_no_reg.history["loss"][-1]:>18.4f}   {history_with_reg.history["loss"][-1]:>13.4f}')
print(f'{"Final Validation Loss":<35} {history_no_reg.history["val_loss"][-1]:>18.4f}   {history_with_reg.history["val_loss"][-1]:>13.4f}')

train_val_gap_no_reg = history_no_reg.history['val_loss'][-1] - history_no_reg.history['loss'][-1]
train_val_gap_with_reg = history_with_reg.history['val_loss'][-1] - history_with_reg.history['loss'][-1]
print(f'{"Train-Val Gap (overfitting)":<35} {train_val_gap_no_reg:>18.4f}   {train_val_gap_with_reg:>13.4f}')
print('='*70)

print(f'\nKey Insights:')
print(f'✓ Regularization reduced overfitting gap by {abs(train_val_gap_no_reg - train_val_gap_with_reg):.4f}')
print(f'✓ Validation loss improved by {improvement:.1f}% with regularization')
print(f'✓ Model generalizes better to unseen data with L2 + Dropout')

Loss Functions & Custom Losses

Loss functions measure how wrong the model's predictions are. The optimizer minimizes this loss by updating weights. Choosing the correct loss function is critical—it directly affects what the model learns:

Common Built-in Losses

import tensorflow as tf
from tensorflow import keras
import numpy as np

# LOSS FUNCTION SELECTION GUIDE:
# Each task requires a different loss function

# ===== BINARY CLASSIFICATION (2 classes) =====
# Example: Spam detection (Spam vs Not Spam)
binary_model = keras.Sequential([
    keras.layers.Dense(16, activation='relu', input_shape=(10,)),
    keras.layers.Dense(1, activation='sigmoid')  # sigmoid → [0, 1]
])
binary_model.compile(
    optimizer='adam',
    loss='binary_crossentropy',  # Use for 2-class problems
    metrics=['accuracy']
)

# ===== MULTI-CLASS CLASSIFICATION (3+ classes) =====
# Example: Image classification (10 digit classes 0-9)
# Two variants based on label format:

# VARIANT 1: Integer labels [0, 1, 2, ..., 9]
multiclass_integer = keras.Sequential([
    keras.layers.Dense(32, activation='relu', input_shape=(20,)),
    keras.layers.Dense(10, activation='softmax')  # 10 output neurons
])
multiclass_integer.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',  # Use for integer labels
    metrics=['accuracy']
)

# VARIANT 2: One-hot encoded labels [[1,0,0,...], [0,1,0,...], ...]
multiclass_onehot = keras.Sequential([
    keras.layers.Dense(32, activation='relu', input_shape=(20,)),
    keras.layers.Dense(10, activation='softmax')
])
multiclass_onehot.compile(
    optimizer='adam',
    loss='categorical_crossentropy',  # Use for one-hot labels
    metrics=['accuracy']
)

# ===== REGRESSION (continuous values) =====
# Example: House price prediction (output: $100k-$500k)
regression_model = keras.Sequential([
    keras.layers.Dense(64, activation='relu', input_shape=(15,)),
    keras.layers.Dense(1)  # No activation: can output any value
])
regression_model.compile(
    optimizer='adam',
    loss='mse',  # Mean Squared Error: emphasis on large errors
    metrics=['mae']  # Monitor MAE: Mean Absolute Error
)

print('Models compiled with appropriate loss functions.')

# LOSS FUNCTION COMPARISON:
print('\nLoss functions explained:')
print('binary_crossentropy: For binary classification')
print('sparse_categorical_crossentropy: Multi-class, integer labels')
print('categorical_crossentropy: Multi-class, one-hot labels')
print('mse: Regression, penalizes large errors heavily')
print('mae: Regression, treats all errors equally')

Understanding Loss Functions Mathematically

import tensorflow as tf
import numpy as np

# BINARY CROSS-ENTROPY: -[y*log(p) + (1-y)*log(1-p)]
# Where: y = true label (0 or 1), p = predicted probability [0, 1]

y_true = np.array([1, 0, 1])        # Ground truth labels
y_pred = np.array([0.9, 0.2, 0.8]) # Predicted probabilities

# Compute manually
loss = -(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
print('Binary cross-entropy (per sample):', loss)
print('Average loss:', loss.mean())

# TensorFlow version
tf_loss = tf.keras.losses.binary_crossentropy(y_true, y_pred)
print('TensorFlow binary_crossentropy:', tf_loss.numpy())

# CATEGORICAL CROSS-ENTROPY: -sum(y_true * log(y_pred))
# Where: y_true = one-hot [1,0,0], y_pred = softmax probabilities

y_true_onehot = np.array([[1, 0, 0], [0, 1, 0]])  # True labels
y_pred_softmax = np.array([[0.7, 0.2, 0.1], [0.1, 0.8, 0.1]])  # Predictions

# Loss = -log(correct_class_prob)
loss_class1 = -np.log(y_pred_softmax[0, 0])  # Sample 1: -log(0.7)
loss_class2 = -np.log(y_pred_softmax[1, 1])  # Sample 2: -log(0.8)
print('\nCategorical cross-entropy:', [loss_class1, loss_class2])

# Mean Squared Error (MSE) for regression:
y_true_regression = np.array([100, 200, 150])  # House prices
y_pred_regression = np.array([105, 195, 160])  # Predictions

mse = ((y_true_regression - y_pred_regression) ** 2).mean()
print('\nMSE (regression):', mse)

Visualizing Loss Functions

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# Create prediction range [0, 1] and true label values [0, 1]
predictions = np.linspace(0.01, 0.99, 100)

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

# Binary Cross-Entropy Loss
# When true label = 1 (correct prediction should be close to 1)
bce_loss_true = -np.log(predictions)
# When true label = 0 (correct prediction should be close to 0)
bce_loss_false = -np.log(1 - predictions)

ax = axes[0]
ax.plot(predictions, bce_loss_true, 'b-', linewidth=2.5, label='True label = 1')
ax.plot(predictions, bce_loss_false, 'r-', linewidth=2.5, label='True label = 0')
ax.set_xlabel('Predicted Probability', fontsize=11)
ax.set_ylabel('Loss Value', fontsize=11)
ax.set_title('Binary Cross-Entropy Loss', fontsize=12, fontweight='bold')
ax.legend(fontsize=10)
ax.grid(True, alpha=0.3)
ax.set_ylim([0, 5])

# Mean Squared Error (MSE) Loss
# Predictions vs true value (assuming true value = 0 for simplicity)
mse_loss = (predictions - 0) ** 2
mae_loss = np.abs(predictions - 0)

ax = axes[1]
ax.plot(predictions, mse_loss, 'g-', linewidth=2.5, label='MSE = (y - ŷ)²')
ax.plot(predictions, mae_loss, 'orange', linewidth=2.5, label='MAE = |y - ŷ|')
ax.set_xlabel('Prediction Error (distance from 0)', fontsize=11)
ax.set_ylabel('Loss Value', fontsize=11)
ax.set_title('Regression Losses: MSE vs MAE', fontsize=12, fontweight='bold')
ax.legend(fontsize=10)
ax.grid(True, alpha=0.3)

# Hinge Loss (for SVM-like problems)
# hinge = max(1 - y*ŷ, 0), where y ∈ {-1, 1} and ŷ ∈ [-1, 1]
scores = np.linspace(-2, 2, 100)
y_true_pos = 1  # Positive class
y_true_neg = -1  # Negative class
hinge_pos = np.maximum(1 - y_true_pos * scores, 0)
hinge_neg = np.maximum(1 - y_true_neg * scores, 0)

ax = axes[2]
ax.plot(scores, hinge_pos, 'purple', linewidth=2.5, label='True class = +1')
ax.plot(scores, hinge_neg, 'brown', linewidth=2.5, label='True class = -1')
ax.axvline(x=1, color='gray', linestyle='--', alpha=0.5, label='Margin boundary')
ax.axvline(x=-1, color='gray', linestyle='--', alpha=0.5)
ax.set_xlabel('Model Score (prediction)', fontsize=11)
ax.set_ylabel('Loss Value', fontsize=11)
ax.set_title('Hinge Loss (SVM)', fontsize=12, fontweight='bold')
ax.legend(fontsize=10)
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# Compare loss behavior at different prediction accuracies
print('\nLoss Function Behavior (True label = 1):')
print('┌──────────────┬────────────────┬──────────┬─────────┐')
print('│ Prediction   │ Binary CE Loss │ MSE Loss │ Notes   │')
print('├──────────────┼────────────────┼──────────┼─────────┤')
predictions_test = [0.1, 0.3, 0.5, 0.7, 0.9, 0.99]
for pred in predictions_test:
    bce = -np.log(pred)
    mse = (1 - pred) ** 2
    notes = "✓ Good" if pred > 0.7 else ("⚠ OK" if pred > 0.5 else "✗ Bad")
    print(f'│ {pred:12.2f} │ {bce:14.3f} │ {mse:8.3f} │ {notes:7s} │')
print('└──────────────┴────────────────┴──────────┴─────────┘')

# Key insights
print('\nKey Insights:')
print('• Binary CE penalizes wrong predictions exponentially')
print('• MSE penalizes quadratically (smoother gradient)')
print('• For probability outputs: use Cross-Entropy')
print('• For regression (continuous values): use MSE or MAE')
print('• Categorical CE generalizes Binary CE to multi-class')

Creating a Custom Loss Function

Implement custom loss as a function accepting y_true and y_pred:

import tensorflow as tf

def custom_mse(y_true, y_pred):
    """Custom mean squared error with optional weighting."""
    squared_diff = tf.square(y_true - y_pred)
    return tf.reduce_mean(squared_diff)

# Test the custom loss
y_true = tf.constant([[1.0], [2.0], [3.0]])
y_pred = tf.constant([[1.5], [1.8], [3.2]])

loss_value = custom_mse(y_true, y_pred)
print('Custom MSE:', loss_value.numpy())

# Use in model compilation
model = tf.keras.Sequential([
    tf.keras.layers.Dense(16, activation='relu', input_shape=(10,)),
    tf.keras.layers.Dense(1)
])
model.compile(optimizer='adam', loss=custom_mse)

Custom losses are useful for domain-specific objectives like weighted errors, focal loss for imbalanced data, or contrastive loss for metric learning.

Optimizers & Learning Rate Schedules

Optimizers update weights based on gradients to minimize loss. Different optimizers have different strategies: Adam adapts learning rates per parameter, SGD uses a fixed rate with momentum, AdamW decouples weight decay from gradients. Choosing the right optimizer and learning rate affects convergence speed and final accuracy:

Common Optimizers and Their Strategies

import tensorflow as tf
from tensorflow import keras

# ===== ADAM: Default, Adaptive Learning Rate =====
# Combines: momentum (moving average of gradients)
#          + RMSprop (per-parameter learning rates)
# Parameters:
#   - learning_rate: initial step size (typical: 0.001)
#   - beta_1: momentum decay (default 0.9)
#   - beta_2: RMSprop decay (default 0.999)
adam_optimizer = keras.optimizers.Adam(
    learning_rate=0.001,  # Initial step size
    beta_1=0.9,           # Momentum: smooth out oscillations
    beta_2=0.999          # RMSprop: adapt per-parameter
)

# ===== SGD: Stochastic Gradient Descent with Momentum =====
# Simpler than Adam, sometimes more stable
# Parameters:
#   - learning_rate: step size (typically larger than Adam, ~0.01)
#   - momentum: fraction of previous gradient to keep (default 0.0)
#   - nesterov: use Nesterov momentum (look-ahead) (default False)
sgd_optimizer = keras.optimizers.SGD(
    learning_rate=0.01,   # Step size for SGD
    momentum=0.9,         # Include 90% of previous gradient
    nesterov=True         # Look-ahead version
)

# ===== ADAMW: Adam with Decoupled Weight Decay =====
# Better than Adam for modern architectures (Transformers, Vision Transformers)
# Parameters:
#   - weight_decay: L2 regularization strength
adamw_optimizer = keras.optimizers.AdamW(
    learning_rate=0.001,
    weight_decay=0.01     # Regularization: penalties large weights
)

# ===== RMSprop: Root Mean Square Propagation =====
# Good for recurrent neural networks (RNNs, LSTMs)
# Adapts learning rate based on magnitude of recent gradients
rmsprop_optimizer = keras.optimizers.RMSprop(
    learning_rate=0.001,
    rho=0.9               # Decay rate for moving average
)

print('Optimizers created.')

# QUICK DECISION GUIDE:
print('\nOptimizer selection:')
print('- Default/Safe choice: Adam (learning_rate=0.001)')
print('- Fine-tuning (transfer learning): Adam with lower learning_rate')
print('- Modern architectures: AdamW with weight decay')
print('- RNNs/LSTMs: RMSprop')
print('- When Adam fails: Try SGD with momentum')

Visualizing Optimizer Convergence

import numpy as np
import matplotlib.pyplot as plt

# Simulate loss curves for different optimizers
# Training a simple model on a quadratic loss function
epochs = 100
np.random.seed(42)

# SGD with fixed learning rate (slow, steady)
sgd_loss = 10 * np.exp(-0.02 * np.arange(epochs)) + 0.3 + np.random.normal(0, 0.1, epochs)

# SGD with momentum (faster convergence)
sgd_momentum_loss = 10 * np.exp(-0.035 * np.arange(epochs)) + 0.25 + np.random.normal(0, 0.08, epochs)

# Adam (adaptive, converges quickly)
adam_loss = 10 * np.exp(-0.05 * np.arange(epochs)) + 0.2 + np.random.normal(0, 0.06, epochs)

# AdamW (like Adam but with weight decay)
adamw_loss = 10 * np.exp(-0.05 * np.arange(epochs)) + 0.15 + np.random.normal(0, 0.06, epochs)

# Plot convergence curves
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Plot 1: Loss curves comparison
ax = axes[0]
ax.plot(sgd_loss, 'o-', linewidth=2, markersize=2, label='SGD (lr=0.01)', alpha=0.8)
ax.plot(sgd_momentum_loss, 's-', linewidth=2, markersize=2, label='SGD + Momentum', alpha=0.8)
ax.plot(adam_loss, '^-', linewidth=2, markersize=2, label='Adam (lr=0.001)', alpha=0.8)
ax.plot(adamw_loss, 'd-', linewidth=2, markersize=2, label='AdamW (weight decay)', alpha=0.8)

ax.set_xlabel('Epoch', fontsize=12, fontweight='bold')
ax.set_ylabel('Loss Value', fontsize=12, fontweight='bold')
ax.set_title('Optimizer Convergence Comparison', fontsize=13, fontweight='bold')
ax.legend(fontsize=11, loc='upper right')
ax.grid(True, alpha=0.3)
ax.set_yscale('log')  # Log scale to see differences clearly

# Plot 2: Learning rate schedules
ax = axes[1]
steps = np.arange(200)

# Constant learning rate
lr_constant = np.ones_like(steps) * 0.001

# Exponential decay: lr = initial_lr * decay_rate^(step / decay_steps)
initial_lr = 0.001
decay_rate = 0.96
decay_steps = 10
lr_exponential = initial_lr * (decay_rate ** (steps / decay_steps))

# Polynomial decay: lr = (initial_lr - final_lr) * (1 - step/steps)^power + final_lr
final_lr = 0.00001
power = 1
lr_polynomial = (initial_lr - final_lr) * ((1 - steps / 200) ** power) + final_lr

# Cosine annealing: lr = final_lr + 0.5 * (initial_lr - final_lr) * (1 + cos(π * step / steps))
lr_cosine = final_lr + 0.5 * (initial_lr - final_lr) * (1 + np.cos(np.pi * steps / 200))

ax.plot(steps, lr_constant * 1000, 'b-', linewidth=2.5, label='Constant (1e-3)')
ax.plot(steps, lr_exponential * 1000, 'g-', linewidth=2.5, label='Exponential Decay')
ax.plot(steps, lr_polynomial * 1000, 'r-', linewidth=2.5, label='Polynomial Decay')
ax.plot(steps, lr_cosine * 1000, 'm-', linewidth=2.5, label='Cosine Annealing')

ax.set_xlabel('Training Step', fontsize=12, fontweight='bold')
ax.set_ylabel('Learning Rate (×1e-3)', fontsize=12, fontweight='bold')
ax.set_title('Learning Rate Schedules', fontsize=13, fontweight='bold')
ax.legend(fontsize=11, loc='upper right')
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# Print optimizer comparison table
print('\nOptimizer Characteristics:')
print('┌─────────────┬──────────┬──────────────┬─────────────────────┐')
print('│ Optimizer   │ Speed    │ Final Loss   │ Best For            │')
print('├─────────────┼──────────┼──────────────┼─────────────────────┤')
print('│ SGD         │ Slow     │ 0.3 (noisy)  │ Simple, interpretable│')
print('│ SGD+Mom     │ Fast     │ 0.25 (noisy) │ Standard choice     │')
print('│ Adam        │ V.Fast   │ 0.2 (smooth) │ Deep networks       │')
print('│ AdamW       │ V.Fast   │ 0.15 (best)  │ Weight decay needed │')
print('│ RMSprop     │ Fast     │ 0.22 (good)  │ RNN/LSTM            │')
print('└─────────────┴──────────┴──────────────┴─────────────────────┘')

print('\nLearning Rate Schedule Effects:')
print('• Constant: Simple, but may overshoot minimum or get stuck')
print('• Exponential: Smooth decay, good for most problems')
print('• Polynomial: Gradual decrease, allows fine-tuning at end')
print('• Cosine: Smooth + warm restart variant for ensemble training')

Learning Rate Schedules

Learning rate often needs adjustment during training—start high for rapid progress, decrease later for fine-tuning. Learning rate schedules automate this:

import tensorflow as tf
from tensorflow import keras

# ===== EXPONENTIAL DECAY =====
# Multiply learning rate by a factor every N steps
# Formula: lr(t) = lr_0 * decay_rate ^ (t / decay_steps)
# Effect: gradual decrease in learning rate

initial_learning_rate = 0.1
decay_steps = 1000          # Decay every 1000 steps
decay_rate = 0.96          # Multiply by 0.96 each time
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate,
    decay_steps,
    decay_rate
)
optimizer = keras.optimizers.Adam(learning_rate=lr_schedule)

# ===== STEP DECAY =====
# Multiply learning rate at specific steps
# Example: halve learning rate at epochs [10, 20, 30]

# Using callback (see Callbacks section for more)
from tensorflow.keras.callbacks import ReduceLROnPlateau
reduce_lr = ReduceLROnPlateau(
    monitor='val_loss',     # Metric to monitor
    factor=0.5,             # Multiply LR by 0.5
    patience=5,             # Wait 5 epochs of no improvement
    min_lr=0.00001          # Lower bound
)

# ===== POLYNOMIAL DECAY =====
# Learning rate decreases polynomially: lr = lr_0 * (1 - t/total)^p

total_steps = 10000
power = 1.0  # Linear decay (power=1); quadratic (power=2)
lr_schedule_poly = keras.optimizers.schedules.PolynomialDecay(
    0.1,           # Initial LR
    total_steps,
    end_learning_rate=0.00001,
    power=power
)
optimizer = keras.optimizers.Adam(learning_rate=lr_schedule_poly)

print('Learning rate schedules created.')

# RULE OF THUMB:
# - Start with Adam(learning_rate=0.001)
# - If loss plateaus: decrease LR or use schedule
# - If loss oscillates: decrease LR
# - If training too slow: increase LR (carefully!)

Learning Rate Schedules

Decay learning rate over time for better convergence:

import tensorflow as tf
from tensorflow import keras

# Exponential decay: lr = initial_lr * decay_rate^(step/decay_steps)
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.01,
    decay_steps=1000,
    decay_rate=0.96,
    staircase=True  # Discretize decay steps
)

# Create optimizer with schedule
optimizer = keras.optimizers.Adam(learning_rate=lr_schedule)

# Check learning rate at different steps
print('LR at step 0:', lr_schedule(0).numpy())
print('LR at step 1000:', lr_schedule(1000).numpy())
print('LR at step 2000:', lr_schedule(2000).numpy())

Other schedules: PiecewiseConstantDecay (step-wise), PolynomialDecay, CosineDecay (warm restarts). Use ReduceLROnPlateau callback for adaptive decay based on validation metrics.

Data Pipelines with tf.data

The tf.data API builds efficient input pipelines that overlap data loading with model training (prefetching), shuffle data for better generalization, and batch samples for GPU efficiency. Always use tf.data for datasets that don't fit in memory or require complex preprocessing.

Basic Pipeline: Batch and Prefetch

import tensorflow as tf
import numpy as np

# CREATE DATASET from in-memory tensor
# tf.data.Dataset is the standard way to feed data to keras.fit()
data = tf.random.uniform([1000, 16])  # 1000 samples, 16 features

# Convert to Dataset object
dataset = tf.data.Dataset.from_tensor_slices(data)

# BATCH: Group samples into mini-batches
# Parameters:
#   - batch_size: samples per gradient update (32, 64, 128 typical)
#   - Larger batch → faster computation but less frequent updates
#   - Smaller batch → slower but noisier gradients (can be good for regularization)
dataset = dataset.batch(32)

# PREFETCH: Load next batch while training current batch
# Parameters:
#   - buffer_size: how many batches to prefetch
#   - tf.data.AUTOTUNE: Let TensorFlow pick optimal value automatically
# Effect: Overlaps I/O (loading) and computation (training) → huge speedup
dataset = dataset.prefetch(buffer_size=tf.data.AUTOTUNE)

# ITERATE through batches
# dataset.take(3) grabs first 3 batches without loading entire dataset
for batch in dataset.take(3):
    print('Batch shape:', batch.shape)  # [32, 16] for first 2, [8, 16] for last
    # Total samples: 1000
    # 1000 / 32 = 31.25 batches
    # Batch 1: 32 samples
    # Batch 2: 32 samples
    # ...
    # Batch 31: 8 samples (remainder)

Complete Pipeline: Shuffle → Batch → Prefetch

import tensorflow as tf
import numpy as np

# Create sample dataset with features and labels
X = np.random.randn(500, 10).astype('float32')  # 500 samples, 10 features
y = np.random.randint(0, 2, (500,)).astype('int32')  # Binary labels

# BUILD PIPELINE in correct order
dataset = tf.data.Dataset.from_tensor_slices((X, y))

# STEP 1: SHUFFLE - randomize order for better generalization
# Parameters:
#   - buffer_size: how many samples to shuffle together
#   - Use buffer_size >= dataset size for perfect shuffling
#   - Larger buffer = better randomization but more memory
#   - Must come BEFORE batching to avoid sorting by class
# Why? Prevents bias where model learns ordering instead of features
dataset = dataset.shuffle(buffer_size=500)

# STEP 2: BATCH - group into mini-batches
# Group 32 consecutive samples together
# After shuffling, batch will have mixed samples (not sorted)
dataset = dataset.batch(32)

# STEP 3: PREFETCH - load next batch during training
# Overlaps I/O and computation for speed
dataset = dataset.prefetch(tf.data.AUTOTUNE)

# VERIFY pipeline
for features, labels in dataset.take(1):
    print('Features shape:', features.shape)  # (32, 10)
    print('Labels shape:', labels.shape)      # (32,)
    print('First batch labels:', labels.numpy())  # Mixed 0s and 1s (correct!)

# PIPELINE SUMMARY:
# Dataset has 500 samples
# After shuffle + batch(32) + prefetch: 16 batches
# - Batches 1-15: 32 samples each
# - Batch 16: 4 samples (remainder)

Advanced: Preprocessing with map()

Apply preprocessing functions with map() for data augmentation and normalization on-the-fly:

import tensorflow as tf
import numpy as np

# Create image-like dataset (100 images, 28x28 pixels, grayscale)
X = np.random.randint(0, 256, [100, 28, 28], dtype='uint8')
y = np.random.randint(0, 10, [100], dtype='int32')

dataset = tf.data.Dataset.from_tensor_slices((X, y))

# PREPROCESSING FUNCTION
# This function is applied to each element in the dataset
def preprocess_image(x, y):
    # Convert to float and normalize to [0, 1]
    x = tf.cast(x, tf.float32)     # Convert uint8 → float32
    x = x / 255.0                   # Normalize: [0, 256] → [0, 1]
    
    # Data augmentation: add random rotation-like transform
    # In practice, use tf.image.rot90, tf.image.flip_left_right, etc.
    noise = tf.random.normal([28, 28], stddev=0.05)
    x = x + noise
    
    return x, y

# APPLY TRANSFORMATION with map()
# Parameters:
#   - function: preprocessing function to apply to each element
#   - num_parallel_calls: how many samples to process in parallel
#     tf.data.AUTOTUNE: Let TensorFlow decide
# Effect: Parallelizes preprocessing across CPU cores
dataset = dataset.map(preprocess_image, num_parallel_calls=tf.data.AUTOTUNE)
dataset = dataset.batch(32)
dataset = dataset.prefetch(tf.data.AUTOTUNE)

# COMPLETE PIPELINE for training:
# Original data (100, 28, 28) → preprocess → batch(32) → prefetch
# For model.fit(dataset, epochs=10)
for batch_x, batch_y in dataset.take(1):
    print('Batch shape after preprocessing:', batch_x.shape)  # (32, 28, 28)
    print('Pixel range [0, 1]:', batch_x.min().numpy(), '-', batch_x.max().numpy())

Pipeline Performance Tips

import tensorflow as tf

# GOOD: Correct order for best performance
dataset = tf.data.Dataset.from_tensor_slices(range(1000))
dataset = dataset.shuffle(1000)          # Randomize
dataset = dataset.batch(32)              # Group
dataset = dataset.prefetch(tf.data.AUTOTUNE)  # Prefetch

# BAD: Prefetch before batch (batches aren't prefetched!)
dataset_bad = tf.data.Dataset.from_tensor_slices(range(1000))
dataset_bad = dataset_bad.prefetch(tf.data.AUTOTUNE)  # Wrong place!
dataset_bad = dataset_bad.batch(32)      # Should be before prefetch

# OPTIMIZATION: Cache preprocessed data if it fits in memory
# .cache() stores all data in memory after preprocessing
dataset_cached = tf.data.Dataset.from_tensor_slices(range(100))
dataset_cached = dataset_cached.map(lambda x: x ** 2)  # Expensive operation
dataset_cached = dataset_cached.cache()  # Store in memory after this point
dataset_cached = dataset_cached.batch(32)
dataset_cached = dataset_cached.prefetch(tf.data.AUTOTUNE)
# Second epoch will use cached data (much faster!)

print('Dataset pipeline optimized for speed')

Visualizing Data Pipeline Performance Impact

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import time

# Create large synthetic dataset
n_samples = 5000
X = np.random.randn(n_samples, 100).astype('float32')
y = np.random.randint(0, 10, (n_samples,)).astype('int32')

# Define expensive preprocessing function (simulates slow I/O or augmentation)
def expensive_preprocess(x, y):
    # Simulate expensive computation (e.g., image augmentation)
    x = x + tf.random.normal(tf.shape(x), stddev=0.1)
    x = x * tf.random.uniform(tf.shape(x), 0.9, 1.1)
    return x, y

# Pipeline 1: NAIVE (no optimization)
dataset_naive = tf.data.Dataset.from_tensor_slices((X, y))
dataset_naive = dataset_naive.map(expensive_preprocess)
dataset_naive = dataset_naive.batch(32)
# MISSING: prefetch!

# Pipeline 2: WITH PREFETCH (overlaps I/O and computation)
dataset_prefetch = tf.data.Dataset.from_tensor_slices((X, y))
dataset_prefetch = dataset_prefetch.map(expensive_preprocess, num_parallel_calls=tf.data.AUTOTUNE)
dataset_prefetch = dataset_prefetch.batch(32)
dataset_prefetch = dataset_prefetch.prefetch(tf.data.AUTOTUNE)  # KEY: prefetch here!

# Pipeline 3: WITH CACHE + PREFETCH (best for repeated epochs)
dataset_cached = tf.data.Dataset.from_tensor_slices((X, y))
dataset_cached = dataset_cached.map(expensive_preprocess, num_parallel_calls=tf.data.AUTOTUNE)
dataset_cached = dataset_cached.cache()  # Cache after preprocessing
dataset_cached = dataset_cached.batch(32)
dataset_cached = dataset_cached.prefetch(tf.data.AUTOTUNE)

# Measure performance across epochs
def measure_epoch_time(dataset, name, num_batches=None):
    """Measure time to iterate through dataset once"""
    start = time.time()
    batch_count = 0
    for _ in dataset:
        batch_count += 1
        if num_batches and batch_count >= num_batches:
            break
    elapsed = time.time() - start
    return elapsed

# Run measurements
num_epochs = 3
epoch_times_naive = []
epoch_times_prefetch = []
epoch_times_cached = []

print('Measuring data pipeline performance across epochs...')
print('(This measures time to iterate through dataset, not training time)\n')

for epoch in range(num_epochs):
    print(f'Epoch {epoch + 1}/{num_epochs}')
    t_naive = measure_epoch_time(dataset_naive, 'Naive')
    t_prefetch = measure_epoch_time(dataset_prefetch, 'Prefetch')
    t_cached = measure_epoch_time(dataset_cached, 'Cached')
    
    epoch_times_naive.append(t_naive)
    epoch_times_prefetch.append(t_prefetch)
    epoch_times_cached.append(t_cached)
    
    print(f'  Naive:            {t_naive:.3f}s')
    print(f'  With Prefetch:    {t_prefetch:.3f}s ({(1-t_prefetch/t_naive)*100:.1f}% faster)')
    print(f'  With Cache+Prefetch: {t_cached:.3f}s ({(1-t_cached/t_naive)*100:.1f}% faster)')

# Create visualization
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Plot 1: Per-epoch timing comparison
ax = axes[0]
epochs_range = np.arange(1, num_epochs + 1)
width = 0.25

bars1 = ax.bar(epochs_range - width, epoch_times_naive, width, label='Naive (No Optimization)', color='salmon', alpha=0.8, edgecolor='black')
bars2 = ax.bar(epochs_range, epoch_times_prefetch, width, label='With Prefetch', color='skyblue', alpha=0.8, edgecolor='black')
bars3 = ax.bar(epochs_range + width, epoch_times_cached, width, label='With Cache + Prefetch', color='lightgreen', alpha=0.8, edgecolor='black')

# Add value labels
for bars in [bars1, bars2, bars3]:
    for bar in bars:
        height = bar.get_height()
        ax.text(bar.get_x() + bar.get_width()/2., height,
                f'{height:.2f}s', ha='center', va='bottom', fontweight='bold', fontsize=9)

ax.set_xlabel('Epoch', fontsize=12, fontweight='bold')
ax.set_ylabel('Time (seconds)', fontsize=12, fontweight='bold')
ax.set_title('Data Pipeline Performance: Naive vs Optimized', fontsize=13, fontweight='bold')
ax.set_xticks(epochs_range)
ax.legend(fontsize=10, loc='upper right')
ax.grid(True, alpha=0.3, axis='y')

# Plot 2: Speedup comparison
ax = axes[1]
prefetch_speedup = (1 - np.array(epoch_times_prefetch) / np.array(epoch_times_naive)) * 100
cached_speedup = (1 - np.array(epoch_times_cached) / np.array(epoch_times_naive)) * 100

x_pos = np.arange(len(epochs_range))
ax.plot(x_pos, prefetch_speedup, 'b-o', linewidth=2.5, markersize=8, label='Prefetch vs Naive', alpha=0.8)
ax.plot(x_pos, cached_speedup, 'g-s', linewidth=2.5, markersize=8, label='Cache+Prefetch vs Naive', alpha=0.8)
ax.axhline(y=0, color='black', linestyle='--', alpha=0.5)

ax.fill_between(x_pos, prefetch_speedup, alpha=0.2, color='blue')
ax.fill_between(x_pos, cached_speedup, alpha=0.2, color='green')

ax.set_xlabel('Epoch', fontsize=12, fontweight='bold')
ax.set_ylabel('Speedup (%)', fontsize=12, fontweight='bold')
ax.set_title('Performance Improvement from Optimization', fontsize=13, fontweight='bold')
ax.set_xticks(x_pos)
ax.set_xticklabels(epochs_range)
ax.legend(fontsize=10)
ax.grid(True, alpha=0.3)

# Add trend annotation
avg_prefetch_speedup = prefetch_speedup.mean()
avg_cached_speedup = cached_speedup.mean()
ax.text(0.5, 0.95, 
        f'Avg Prefetch Speedup: {avg_prefetch_speedup:.1f}%\nAvg Cache+Prefetch Speedup: {avg_cached_speedup:.1f}%',
        transform=ax.transAxes, fontsize=11, fontweight='bold',
        ha='center', va='top',
        bbox=dict(boxstyle='round', facecolor='lightyellow', alpha=0.9, edgecolor='orange', linewidth=2))

plt.tight_layout()
plt.show()

# Print summary table
print('\nData Pipeline Performance Summary:')
print('='*80)
print(f'{"Epoch":<10} {"Naive":<15} {"Prefetch":<15} {"Cache+Prefetch":<15} {"Cache Benefit":<20}')
print('='*80)

for i, (t_naive, t_pref, t_cache) in enumerate(zip(epoch_times_naive, epoch_times_prefetch, epoch_times_cached), 1):
    prefetch_benefit = (1 - t_pref / t_naive) * 100
    cache_benefit = (1 - t_cache / t_pref) * 100 if t_pref > 0 else 0
    print(f'{i:<10} {t_naive:>12.3f}s   {t_pref:>12.3f}s   {t_cache:>12.3f}s   {cache_benefit:>15.1f}% extra')

print('='*80)
print(f'\nKey Insights:')
print(f'✓ Prefetch overlaps I/O with computation: {np.mean(prefetch_speedup):.1f}% average speedup')
print(f'✓ Cache stores preprocessed data: epoch 2+ benefit from cached data')
print(f'✓ Combination is most powerful: {np.mean(cached_speedup):.1f}% average speedup')
print(f'✓ For multi-epoch training, caching provides cumulative benefit')

Part 3: Training Workflows

Training: model.fit() vs Custom Loops

Keras provides model.fit() for high-level training—perfect for most use cases. It handles epochs, batches, metrics tracking, and callbacks automatically. For research or custom training logic (e.g., GANs, reinforcement learning), use custom loops with GradientTape.

High-Level Training with model.fit()

The simplest and most common way to train models. Keras handles all the complexity (batching, gradient computation, weight updates, metric tracking):

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Create synthetic dataset (200 samples, 16 features)
X_train = np.random.randn(200, 16).astype('float32')  # Training data
y_train = np.random.randn(200, 1).astype('float32')   # Training labels

# Build neural network
model = keras.Sequential([
    layers.Dense(32, activation='relu', input_shape=(16,)),
    layers.Dense(16, activation='relu'),
    layers.Dense(1)  # Output layer (regression)
])

# Compile: configure optimizer, loss, metrics
model.compile(optimizer='adam', loss='mse', metrics=['mae'])

# TRAIN with model.fit()
# Parameters:
#   - X_train, y_train: training data and labels
#   - epochs: number of passes through entire dataset
#   - batch_size: samples per gradient update
#   - validation_split: fraction of data for validation (e.g., 0.2 = 20%)
#   - verbose: 0 (silent), 1 (progress bar), 2 (one line per epoch)
history = model.fit(
    X_train, y_train,
    epochs=5,                    # 5 complete passes through training data
    batch_size=32,               # Update weights every 32 samples
    validation_split=0.2,        # Reserve 20% for validation
    verbose=1                    # Show progress bar
)

# HISTORY: contains loss/metrics at each epoch
print('Training losses:', history.history['loss'])        # Per-epoch training loss
print('Validation losses:', history.history['val_loss'])  # Per-epoch validation loss
print('Training MAE:', history.history['mae'])
print('Validation MAE:', history.history['val_mae'])

# What happens under the hood in model.fit():
# For each epoch:
#   1. Shuffle training data
#   2. Split into batches
#   3. For each batch:
#      - Forward pass: predictions = model(batch_X)
#      - Compute loss: loss = loss_function(predictions, batch_y)
#      - Backward pass: compute gradients via GradientTape
#      - Update: weights -= optimizer(learning_rate * gradients)
#   4. Evaluate on validation set
#   5. Print metrics

Custom Training Loop with Fine-Grained Control

When you need custom training logic (GANs, reinforcement learning, multi-task learning), write the loop manually using GradientTape:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Data
X = np.random.randn(200, 16).astype('float32')
y = np.random.randn(200, 1).astype('float32')

# Model
model = keras.Sequential([
    layers.Dense(32, activation='relu', input_shape=(16,)),
    layers.Dense(1)
])

# Optimizer
optimizer = keras.optimizers.Adam(learning_rate=0.001)

# Loss function
loss_fn = keras.losses.MeanSquaredError()

# Hyperparameters
epochs = 5
batch_size = 32

# CUSTOM TRAINING LOOP: Full control over each step
for epoch in range(epochs):
    print(f'\nEpoch {epoch+1}/{epochs}')
    epoch_loss = 0
    num_batches = 0
    
    # Iterate through batches manually
    for i in range(0, len(X), batch_size):
        x_batch = X[i:i+batch_size]
        y_batch = y[i:i+batch_size]
        
        # STEP 1: Forward pass with GradientTape recording
        # Everything inside 'with' block is recorded for gradient computation
        with tf.GradientTape() as tape:
            # Forward: make predictions
            predictions = model(x_batch, training=True)
            # Compute loss
            loss_value = loss_fn(y_batch, predictions)
            # Can add custom losses here, weighted combinations, etc.
        
        # STEP 2: Backward pass: compute gradients
        # Tells us how each weight affects the loss
        grads = tape.gradient(loss_value, model.trainable_variables)
        
        # STEP 3: Update weights
        # apply_gradients: W = W - learning_rate * gradient
        optimizer.apply_gradients(zip(grads, model.trainable_variables))
        
        epoch_loss += loss_value.numpy()
        num_batches += 1
    
    print(f'Average loss: {epoch_loss/num_batches:.4f}')

print('Training complete!')

# ADVANTAGES of custom loops:
# 1. Full control over gradients (e.g., gradient clipping, manipulation)
# 2. Support custom loss combinations
# 3. Implement GAN training (separate generator/discriminator updates)
# 4. Multi-task learning with weighted losses
# 5. Research-specific training procedures

# DISADVANTAGES:
# 1. More boilerplate code
# 2. Harder to debug
# 3. Easy to make mistakes (forgot training flag, gradient accumulation, etc.)
# 4. Slower than optimized model.fit()

Comparison: model.fit() vs Custom Loop

import tensorflow as tf
from tensorflow.keras import layers

# MODEL.FIT() - Recommended for most cases
# Pros:
#   - Concise, readable code
#   - Automatic metric tracking and callbacks
#   - Optimized for speed
#   - Less room for bugs
# Cons:
#   - Less flexibility for advanced scenarios

# Build model
model = tf.keras.Sequential([layers.Dense(10, activation='relu')])
model.compile(optimizer='adam', loss='mse')
# history = model.fit(X, y, epochs=10, batch_size=32)

# CUSTOM LOOP - For research/advanced use
# Pros:
#   - Full control over training
#   - Support complex scenarios (GANs, multi-task, etc.)
#   - Custom gradient manipulation
# Cons:
#   - More code to write and debug
#   - Easier to make mistakes
#   - Slower if not carefully optimized

# When to use each:
print('Use model.fit() if:')
print('  - Standard supervised learning (classification, regression)')
print('  - You want simple, clean code')
print('  - You are a beginner')
print('')
print('Use custom loop if:')
print('  - Adversarial training (GANs, adversarial examples)')
print('  - Multi-task learning (different losses per task)')
print('  - Reinforcement learning (state-value functions, policy gradients)')
print('  - Meta-learning (learn-to-learn)')
print('  - Custom gradient computation needed')

Callbacks for Training Control

Callbacks are functions executed at specific training stages (epoch end, batch end) to monitor, modify, or stop training. Keras provides powerful built-in callbacks: EarlyStopping, ModelCheckpoint, ReduceLROnPlateau, TensorBoard, and more.

EarlyStopping & ModelCheckpoint

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Data
X = np.random.randn(300, 20).astype('float32')
y = np.random.randint(0, 2, (300,)).astype('int32')

# Model
model = keras.Sequential([
    layers.Dense(64, activation='relu', input_shape=(20,)),
    layers.Dropout(0.3),
    layers.Dense(32, activation='relu'),
    layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Define callbacks
callbacks = [
    # Stop training when validation loss stops improving
    keras.callbacks.EarlyStopping(
        monitor='val_loss',
        patience=3,  # Stop after 3 epochs without improvement
        restore_best_weights=True,  # Restore weights from best epoch
        verbose=1
    ),
    
    # Save model when validation loss improves
    keras.callbacks.ModelCheckpoint(
        filepath='best_model.keras',
        monitor='val_loss',
        save_best_only=True,
        verbose=1
    ),
    
    # Reduce learning rate when metric plateaus
    keras.callbacks.ReduceLROnPlateau(
        monitor='val_loss',
        factor=0.5,  # Multiply LR by 0.5
        patience=2,  # Wait 2 epochs before reducing
        min_lr=1e-6,
        verbose=1
    )
]

# Train with callbacks
history = model.fit(
    X, y,
    epochs=20,
    batch_size=32,
    validation_split=0.2,
    callbacks=callbacks,
    verbose=0  # Suppress epoch logs (callbacks provide updates)
)

print(f'\nTraining completed. Best epoch restored.')
print(f'Total epochs run: {len(history.history["loss"])}')

EarlyStopping prevents overfitting by halting training when validation metrics degrade. ModelCheckpoint saves the best model version—crucial for long training runs. ReduceLROnPlateau adapts learning rate when progress stalls.

Custom Callback

Create custom callbacks for domain-specific logging or actions:

import tensorflow as tf
from tensorflow import keras
import numpy as np

class CustomLogger(keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        # Log custom metrics at epoch end
        print(f'\nEpoch {epoch+1} complete.')
        print(f'  Training loss: {logs["loss"]:.4f}')
        if 'val_loss' in logs:
            print(f'  Validation loss: {logs["val_loss"]:.4f}')
    
    def on_train_end(self, logs=None):
        print('\nTraining finished!')

# Sample model and data
X = np.random.randn(100, 10).astype('float32')
y = np.random.randn(100, 1).astype('float32')

model = keras.Sequential([
    keras.layers.Dense(16, activation='relu', input_shape=(10,)),
    keras.layers.Dense(1)
])
model.compile(optimizer='adam', loss='mse')

# Train with custom callback
model.fit(X, y, epochs=3, callbacks=[CustomLogger()], verbose=0)

Custom callbacks enable logging to external systems, dynamic hyperparameter adjustments, or early experiment termination based on custom criteria.

Model Persistence: Saving & Loading

Save models for deployment, transfer learning, or resuming training. TensorFlow supports multiple formats: SavedModel (recommended for production), .keras (Keras native format), and legacy .h5 (HDF5).

Saving and Loading Models

import tensorflow as tf
from tensorflow import keras
import numpy as np

# Build and train a simple model
model = keras.Sequential([
    keras.layers.Dense(32, activation='relu', input_shape=(16,)),
    keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train briefly
X = np.random.randn(100, 16).astype('float32')
y = np.random.randint(0, 2, (100,)).astype('int32')
model.fit(X, y, epochs=2, verbose=0)

# Save in .keras format (recommended)
model.save('my_model.keras')
print('Model saved to my_model.keras')

# Load the model
loaded_model = keras.models.load_model('my_model.keras')
print('Model loaded successfully.')

# Verify predictions match
original_pred = model.predict(X[:5], verbose=0)
loaded_pred = loaded_model.predict(X[:5], verbose=0)
print('Predictions match:', np.allclose(original_pred, loaded_pred))

The .keras format saves architecture, weights, optimizer state, and training config—everything needed to resume training or deploy.

Saving Weights Only

import tensorflow as tf
from tensorflow import keras
import numpy as np

# Model
model = keras.Sequential([
    keras.layers.Dense(16, activation='relu', input_shape=(10,)),
    keras.layers.Dense(1)
])

# Save weights only (smaller file, requires architecture separately)
model.save_weights('weights_only.weights.h5')
print('Weights saved.')

# Load weights into same architecture
new_model = keras.Sequential([
    keras.layers.Dense(16, activation='relu', input_shape=(10,)),
    keras.layers.Dense(1)
])
new_model.load_weights('weights_only.weights.h5')
print('Weights loaded into new model.')

Saving weights only is useful for transfer learning (reuse pretrained weights with modified architecture) or reducing file size when architecture is known.

TensorBoard Visualization

TensorBoard is TensorFlow's visualization toolkit for monitoring training metrics, visualizing model architectures, profiling performance, and debugging. It runs as a web server displaying real-time or logged data.

Logging to TensorBoard

import tensorflow as tf
from tensorflow import keras
import numpy as np
import datetime

# Create log directory with timestamp
log_dir = 'logs/fit/' + datetime.datetime.now().strftime('%Y%m%d-%H%M%S')

# TensorBoard callback
tensorboard_callback = keras.callbacks.TensorBoard(
    log_dir=log_dir,
    histogram_freq=1,  # Log weight histograms every epoch
    write_graph=True,  # Visualize model graph
    update_freq='epoch'  # Log after each epoch
)

# Sample model and data
X = np.random.randn(200, 16).astype('float32')
y = np.random.randn(200, 1).astype('float32')

model = keras.Sequential([
    keras.layers.Dense(32, activation='relu', input_shape=(16,)),
    keras.layers.Dense(16, activation='relu'),
    keras.layers.Dense(1)
])
model.compile(optimizer='adam', loss='mse', metrics=['mae'])

# Train with TensorBoard logging
model.fit(
    X, y,
    epochs=5,
    validation_split=0.2,
    callbacks=[tensorboard_callback],
    verbose=0
)

print(f'\nTensorBoard logs written to {log_dir}')
print('Launch TensorBoard with: tensorboard --logdir=logs/fit')

After training, run tensorboard --logdir=logs/fit in your terminal and navigate to http://localhost:6006 in your browser. You'll see training/validation curves, histograms, and model graphs.

Custom Metrics & Monitoring

While Keras provides built-in metrics (accuracy, precision, recall), custom metrics track domain-specific performance. Subclass keras.metrics.Metric to create stateful metrics that accumulate results across batches.

Creating a Custom Metric

import tensorflow as tf
from tensorflow import keras

class MeanAbsoluteDifference(keras.metrics.Metric):
    def __init__(self, name='mean_abs_diff', **kwargs):
        super().__init__(name=name, **kwargs)
        # Create state variables
        self.total = self.add_weight(name='total', initializer='zeros')
        self.count = self.add_weight(name='count', initializer='zeros')
    
    def update_state(self, y_true, y_pred, sample_weight=None):
        # Accumulate absolute differences
        values = tf.abs(y_true - y_pred)
        self.total.assign_add(tf.reduce_sum(values))
        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))
    
    def result(self):
        # Compute final metric
        return self.total / self.count
    
    def reset_states(self):
        # Reset between epochs
        self.total.assign(0.0)
        self.count.assign(0.0)

# Test the custom metric
metric = MeanAbsoluteDifference()

# Simulate batch updates
y_true = tf.constant([[1.0], [2.0], [3.0]])
y_pred = tf.constant([[1.5], [2.2], [2.8]])
metric.update_state(y_true, y_pred)

print('Custom metric result:', metric.result().numpy())
metric.reset_states()
print('After reset:', metric.result().numpy())

Using Custom Metrics in Training

import tensorflow as tf
from tensorflow import keras
import numpy as np

# Custom metric (simplified version)
class MeanAbsError(keras.metrics.Metric):
    def __init__(self, name='mae', **kwargs):
        super().__init__(name=name, **kwargs)
        self.total = self.add_weight(name='total', initializer='zeros')
        self.count = self.add_weight(name='count', initializer='zeros')
    
    def update_state(self, y_true, y_pred, sample_weight=None):
        values = tf.abs(y_true - y_pred)
        self.total.assign_add(tf.reduce_sum(values))
        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))
    
    def result(self):
        return self.total / self.count
    
    def reset_states(self):
        self.total.assign(0.0)
        self.count.assign(0.0)

# Build model with custom metric
X = np.random.randn(100, 10).astype('float32')
y = np.random.randn(100, 1).astype('float32')

model = keras.Sequential([
    keras.layers.Dense(16, activation='relu', input_shape=(10,)),
    keras.layers.Dense(1)
])

model.compile(
    optimizer='adam',
    loss='mse',
    metrics=[MeanAbsError()]  # Add custom metric
)

# Train and monitor custom metric
history = model.fit(X, y, epochs=3, verbose=1)

Custom metrics appear in training logs and TensorBoard alongside built-in metrics. They're essential for business-specific KPIs (e.g., customer churn rate, conversion probability) that don't map to standard ML metrics.

Part 4: Practical Applications

Transfer Learning with Pretrained Models

Transfer learning reuses knowledge from models trained on large datasets (ImageNet with millions of images) for new tasks with limited data. The key insight: lower layers learn universal features (edges, shapes), while upper layers learn task-specific features. You reuse the universal parts!

Transfer Learning Concept

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# TRANSFER LEARNING WORKFLOW:
# Step 1: Load pretrained model trained on ImageNet (1 million images, 1000 classes)
# Step 2: Use its learned features (via base model)
# Step 3: Add custom classification head for YOUR task
# Step 4: Train only the head (base frozen)
# Step 5: Optionally fine-tune base with low learning rate

# LOAD PRETRAINED MOBILENETV2
# MobileNetV2: efficient, trained on ImageNet
# Parameters:
#   - weights='imagenet': loads pretrained ImageNet weights (first time downloads)
#   - weights=None: randomly initialized (for demo)
#   - include_top=False: exclude original 1000-class head, get features only
#   - input_shape: your image size (96x96 RGB)
base_model = keras.applications.MobileNetV2(
    weights=None,  # Change to 'imagenet' in production
    include_top=False,  # Get only feature extractor, not classifier
    input_shape=(96, 96, 3)  # Your image dimensions
)

# FREEZE base model
# trainable=False: don't update weights during training
# Why? Keep learned features from ImageNet, only train new head
base_model.trainable = False

print(f'Base model layers: {len(base_model.layers)}')
print(f'Base model parameters: {base_model.count_params():,}')
print(f'All frozen? {not any(layer.trainable for layer in base_model.layers)}')

# BUILD CUSTOM CLASSIFIER ON TOP
# Input: image 96x96x3
inputs = keras.Input(shape=(96, 96, 3))

# Pass through frozen base (training=False = use batch norm statistics from training)
# base_model output: spatial feature maps (e.g., 3x3x1280)
x = base_model(inputs, training=False)

# GLOBAL AVERAGE POOLING: Reduce spatial dimensions
# Input: (batch_size, 3, 3, 1280) feature maps
# Output: (batch_size, 1280) by averaging each channel across space
# Why? Reduces parameters, prevents overfitting, position-invariant
x = layers.GlobalAveragePooling2D()(x)

# DROPOUT: Regularization during training
# Randomly drop 30% of neurons to prevent overfitting
x = layers.Dropout(0.3)(x)

# CUSTOM CLASSIFIER HEAD: Your task-specific layer
# Output: 10 classes, softmax probabilities
outputs = layers.Dense(10, activation='softmax')(x)

# Create final model
model = keras.Model(inputs, outputs)

# COMPILE with Adam optimizer
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',  # For integer labels
    metrics=['accuracy']
)

print('\nTransfer Learning Model Architecture:')
print(f'Trainable parameters: {sum(tf.size(w).numpy() for w in model.trainable_weights):,}')
print(f'Frozen parameters: {sum(tf.size(w).numpy() for w in model.non_trainable_weights):,}')
print(f'Total parameters: {model.count_params():,}')

# TRAINING:
# X_train: (N, 96, 96, 3) images
# y_train: (N,) integer labels 0-9
# model.fit(X_train, y_train, epochs=10, validation_split=0.2)
# Only the Dense(10) and Dropout layers get updated!

What is GlobalAveragePooling2D?

import tensorflow as tf
import numpy as np

# Example: Feature maps from base model
# Shape: (batch_size=1, height=3, width=3, channels=1280)
# This is 3x3x1280, way too large to pass to Dense layer directly

feature_map = np.random.randn(1, 3, 3, 1280)
print('Feature map shape:', feature_map.shape)

# GLOBAL AVERAGE POOLING: Average across height and width for each channel
# Formula: output[c] = mean(feature_map[:, :, c]) for each channel c
gap = tf.keras.layers.GlobalAveragePooling2D()
output = gap(feature_map)
print('After GlobalAveragePooling2D:', output.shape)  # (1, 1280)

# Result: each channel is reduced to its average value
# 3x3x1280 → 1280 (1280× fewer parameters!)

# Alternative would be Flatten():
flatten = tf.keras.layers.Flatten()
flattened = flatten(feature_map)
print('With Flatten():', flattened.shape)  # (1, 28800) - way too big!

# GlobalAveragePooling2D is better:
# 1. Fewer parameters (1280 vs 28800)
# 2. More robust to spatial shifts
# 3. Better generalization

Fine-Tuning: Unlocking the Base Model

After training the head with frozen base, optionally unfreeze some base layers and fine-tune with very low learning rate:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Assume model built from previous example (base frozen)
# After training head for ~5 epochs...

# FINE-TUNING STRATEGY:
# Step 1: Unfreeze base model
base_model.trainable = True

# Step 2: Freeze early layers (keep general features)
# Only unfreeze last few layers (task-specific features)
# Example: MobileNetV2 has ~150 layers
for layer in base_model.layers[:-30]:  # Freeze all but last 30 layers
    layer.trainable = False

print(f'Total layers: {len(base_model.layers)}')
print(f'Frozen layers: {sum(1 for l in base_model.layers if not l.trainable)}')
print(f'Trainable layers: {sum(1 for l in base_model.layers if l.trainable)}')

# Step 3: Recompile with MUCH LOWER learning rate
# Fine-tuning uses ~1/100th the learning rate of initial training
# Why? Preserve learned features, only make small adjustments
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-5),  # 1e-5 vs 1e-3
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

print('Model ready for fine-tuning with low learning rate.')

# TRAINING:
# model.fit(X_train, y_train, epochs=10, validation_split=0.2)
# Now both the last 30 layers of base AND the Dense head will be updated!
# But at very small steps (lr = 1e-5)

Transfer Learning Best Practices & When to Use

import tensorflow as tf

# WHEN TO USE TRANSFER LEARNING:
print('Use transfer learning when:')
print('  - Limited training data (< 10,000 images)')
print('  - Task similar to ImageNet (object recognition)')
print('  - Computational resources are limited')
print('  - Need quick results')
print('')
print('Train from scratch when:')
print('  - Very large dataset (100k+ images)')
print('  - Task very different from ImageNet')
print('  - Have GPU/TPU resources')
print('  - Time available for long training')

# AVAILABLE PRETRAINED MODELS:
models_dict = {
    'MobileNetV2': 'Efficient, mobile-friendly, 3.5M params',
    'EfficientNetB0': 'Better accuracy-efficiency tradeoff, 5.3M params',
    'ResNet50': 'Accurate but large, 25.6M params',
    'InceptionV3': 'Very accurate, 27M params',
    'VGG16': 'Simple, 138M params (large!)',
}

print('\nPopular models and sizes:')
for name, desc in models_dict.items():
    print(f'  {name}: {desc}')

# RULE OF THUMB:
print('\nQuick decision:')
print('  Small/Medium data + accuracy-focused → EfficientNetB0')
print('  Small/Medium data + speed-focused → MobileNetV2')
print('  Large data → Train custom ResNet from scratch')
print('  Very small data (< 1000 images) → Transfer learning essential!')

Fine-tuning improves performance by adapting pretrained features to your specific domain. Use learning rates 10-100× smaller than initial training to preserve learned representations.

Computer Vision: CNN for Image Classification

Convolutional Neural Networks (CNNs) are the gold standard for image tasks. The core insight: images have spatial structure. CNNs leverage this by using convolutional layers that detect local patterns (edges, textures, shapes) without losing spatial information.

Understanding Convolution: How Filters Work

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# CONVOLUTION INTUITION:
# A filter (3x3) slides across an image (32x32), computing dot products
# Each position outputs a scalar → creates a feature map

# Example: Simple Sobel edge detection filter
# This filter detects vertical edges:
edge_filter = np.array([
    [-1, 0, 1],
    [-2, 0, 2],
    [-1, 0, 1]
], dtype='float32')

print('Vertical edge detection filter (Sobel):')
print(edge_filter)
print('')

# When Conv2D(16, (3,3)) is created:
# - 16 random filters are initialized
# - During training, they learn to detect useful features
# - Early layers learn simple patterns (edges, corners)
# - Middle layers learn textures (corners, corners)
# - Deep layers learn high-level objects (faces, wheels)

# PARAMETERS in Conv2D(32, (3,3)):
# - filters=32: number of different filters to learn
# - kernel_size=(3,3): filter dimensions
# - For RGB image, each filter learns 3x3x3 weights (3 color channels)
# - Total parameters per layer: 32 filters × 3×3×3 = 864 + 32 bias = 896

params_per_filter = 3 * 3 * 3  # kernel height × width × input channels
total_params = 32 * params_per_filter + 32  # 32 filters + bias per filter
print(f'Conv2D(32, (3,3)) on RGB image: {total_params} parameters')

# STRIDE and PADDING:
# Conv2D(32, (3,3), strides=2, padding='same')
# - strides=2: filter moves 2 pixels at a time (reduces spatial dims faster)
# - padding='same': pad input with zeros so output same size as input
# - padding='valid': no padding, output is smaller

# Output spatial dimensions formula:
# output_height = (input_height - kernel_size + 2*padding) / stride + 1
def output_size(input_size, kernel_size, stride, padding_type):
    padding = kernel_size - 1 if padding_type == 'same' else 0
    return (input_size - kernel_size + 2 * padding) // stride + 1

print(f'\n32x32 image with Conv2D(kernel=3, stride=1, padding=same):')
print(f'  Output size: {output_size(32, 3, 1, "same")}x{output_size(32, 3, 1, "same")}')

print(f'\n32x32 image with Conv2D(kernel=3, stride=2, padding=same):')
print(f'  Output size: {output_size(32, 3, 2, "same")}x{output_size(32, 3, 2, "same")}')

MaxPooling: Downsampling Feature Maps

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# MaxPooling example: take maximum value in 2x2 window
feature_map = np.array([
    [1, 3, 2, 5],
    [4, 2, 1, 3],
    [2, 5, 6, 1],
    [3, 2, 4, 7]
], dtype='float32')

print('Original 4x4 feature map:')
print(feature_map)

# Manual 2x2 MaxPooling:
# Top-left 2x2: max([1,3,4,2]) = 4
# Top-right 2x2: max([2,5,1,3]) = 5
# Bottom-left 2x2: max([2,5,3,2]) = 5
# Bottom-right 2x2: max([6,1,4,7]) = 7
pooled = np.array([
    [4, 5],
    [5, 7]
], dtype='float32')

print('\nAfter MaxPooling2D((2,2)):')
print(pooled)
print(f'Size reduced: 4x4 → 2x2 (75% fewer values)')

# Why MaxPooling?
print('\nBenefits of MaxPooling:')
print('  1. Reduces spatial dimensions → faster computation')
print('  2. Makes features translation-invariant (resistant to small shifts)')
print('  3. Keeps most important information (max value)')
print('  4. Prevents overfitting by reducing parameters')

# Alternative: AveragePooling2D computes mean instead of max
# Less common, but useful for some domains

Building a Complete CNN Architecture

import tensorflow as tf
from tensorflow import keras
from tensorflow import layers
import numpy as np

# BUILDING A CNN: Standard pattern is Conv → Activation → Pooling → Repeat

model = keras.Sequential([
    # INPUT: 32x32x3 RGB images (CIFAR-10 style)
    
    # BLOCK 1: Extract low-level features (edges, colors)
    # Conv2D(filters=32, kernel_size=3): 32 filters learning 3x3 patterns
    layers.Conv2D(
        filters=32,           # Learn 32 different 3x3 filters
        kernel_size=(3, 3),   # Each filter is 3x3
        padding='same',       # Keep spatial dimensions same
        activation='relu',    # ReLU: max(0, x)
        input_shape=(32, 32, 3)
    ),
    # After: 32x32x32 (height × width × num_filters)
    
    layers.MaxPooling2D(
        pool_size=(2, 2)      # Take max from each 2x2 window
    ),
    # After pooling: 16x16x32 (spatial dims halved)
    
    # BLOCK 2: Extract mid-level features (textures, shapes)
    layers.Conv2D(
        filters=64,           # Increase filter count as features get more complex
        kernel_size=(3, 3),
        padding='same',
        activation='relu'
    ),
    # After: 16x16x64
    
    layers.MaxPooling2D((2, 2)),
    # After: 8x8x64
    
    # BLOCK 3: Extract high-level features (objects, patterns)
    layers.Conv2D(
        filters=128,
        kernel_size=(3, 3),
        padding='same',
        activation='relu'
    ),
    # After: 8x8x128
    
    layers.MaxPooling2D((2, 2)),
    # After: 4x4x128
    
    # CLASSIFIER: Convert spatial features to class predictions
    
    # GlobalAveragePooling2D: Average across spatial dimensions
    # 4x4x128 → 128 (much smaller than Flatten: 4×4×128=2048)
    layers.GlobalAveragePooling2D(),
    
    # Dropout: Randomly deactivate 50% of neurons during training
    # Reduces co-adaptation, improves generalization
    layers.Dropout(rate=0.5),
    
    # Dense classifier: 128 features → 10 class probabilities
    layers.Dense(
        units=10,             # One unit per class
        activation='softmax'  # Output: class probabilities summing to 1
    )
])

# Compile
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',  # For integer labels 0-9
    metrics=['accuracy']
)

print('CNN Architecture Summary:')
model.summary()
print(f'\nTotal parameters: {model.count_params():,}')
print(f'Parameter breakdown:')
for layer in model.layers:
    if hasattr(layer, 'count_params'):
        print(f'  {layer.name}: {layer.count_params():,}')

Training CNN on Real Data

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Build model (from previous example)
model = keras.Sequential([
    layers.Conv2D(32, (3, 3), padding='same', activation='relu', 
                  input_shape=(32, 32, 3)),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), padding='same', activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(128, (3, 3), padding='same', activation='relu'),
    layers.GlobalAveragePooling2D(),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# OPTION 1: Load from tf.keras.datasets (MNIST, CIFAR-10, etc.)
# Each pixel value 0-255, normalized to 0-1
(X_train, y_train), (X_test, y_test) = keras.datasets.cifar10.load_data()
X_train = X_train / 255.0
X_test = X_test / 255.0
y_train = y_train.squeeze()  # Remove extra dimension
y_test = y_test.squeeze()

print(f'Training data shape: {X_train.shape}')
print(f'  Contains: {X_train.shape[0]} images of size {X_train.shape[1]}x{X_train.shape[2]}')
print(f'Training labels shape: {y_train.shape} (classes 0-9)')

# DATA AUGMENTATION: Create variations of images
# Improves generalization by increasing effective dataset size
augmentation = keras.Sequential([
    layers.RandomFlip('horizontal'),    # Flip 50% of images left-right
    layers.RandomRotation(0.1),         # Rotate by ±10%
    layers.RandomZoom(0.1),             # Zoom by ±10%
])

# TRAINING with augmentation
history = model.fit(
    # Apply augmentation on-the-fly during training
    # augmentation(X_train) would augment all images
    X_train,
    y_train,
    epochs=10,
    batch_size=128,
    validation_split=0.2,  # Use 20% for validation
    verbose=1
)

# EVALUATION on test set (unseen data)
test_loss, test_accuracy = model.evaluate(X_test, y_test, verbose=0)
print(f'\nTest accuracy: {test_accuracy:.2%}')
print(f'Test loss: {test_loss:.4f}')

Natural Language Processing Basics

Natural Language Processing (NLP) with deep learning involves three core steps: (1) Tokenization—converting text to sequences of integers representing words, (2) Embedding—mapping those integers to dense vectors capturing semantic meaning, and (3) Sequential modeling—using RNNs (LSTM, GRU) to process sequences while maintaining context. Keras provides TextVectorization for preprocessing and Embedding for learnable word representations.

Text Tokenization & Vectorization

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# TOKENIZATION INTUITION:
# Raw text: "I love deep learning"
# Tokenization: "I" → vocab_id 42, "love" → 156, "deep" → 89, "learning" → 234
# Result: [42, 156, 89, 234]

# TextVectorization does this automatically!
# Step 1: Create vectorizer and adapt to training data
texts = [
    "I love deep learning",
    "Deep learning is great",
    "I love machine learning",
    "Great neural networks"
]

vectorizer = keras.layers.TextVectorization(
    max_tokens=50,         # Keep only top 50 most frequent words
    output_sequence_length=10  # Pad/truncate to 10 tokens
)

# Analyze training texts to build vocabulary
vectorizer.adapt(texts)

# Check vocabulary
print('Vocabulary size:', len(vectorizer.get_vocabulary()))
print('Sample vocabulary:', vectorizer.get_vocabulary()[:10])

# Vectorize individual text
sample_text = "I love deep learning"
vectorized = vectorizer(sample_text)
print(f'\nOriginal: "{sample_text}"')
print(f'Vectorized: {vectorized.numpy()}')
print('(each number is token ID)')

# OPTIONS in TextVectorization:
print('\nTextVectorization parameters:')
print('  - max_tokens: Keep only top N most frequent words (vocabulary size)')
print('  - output_sequence_length: Pad to fixed length (handles variable-length inputs)')
print('  - standardize: Lowercase, remove punctuation (default) or custom function')
print('  - split: Split by whitespace (default) or custom regex')
print('  - ngrams: Also include bigrams (word pairs), trigrams, etc.')

Understanding Embeddings: From Tokens to Vectors

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# EMBEDDING INTUITION:
# Token ID 42 ("love") → 32-dimensional dense vector
# Similar words (e.g., "like", "adore") → similar vectors (close in space)
# This captures semantic relationships!

# Embedding layer: learnable lookup table
# Initialize: random 32-dimensional vectors
# During training: update vectors so related words get similar embeddings

vocab_size = 1000          # 1000 different words in vocabulary
embedding_dim = 32         # Each word → 32-d vector
seq_length = 20            # Each input: 20 token sequence

# Create embedding layer
embedding = layers.Embedding(
    input_dim=vocab_size,       # Vocabulary size
    output_dim=embedding_dim,   # Vector dimension per word
    input_length=seq_length     # Expected sequence length
)

# Example: Tokenized sequence of 5 words from vocab (IDs: 10, 20, 30, 40, 50)
token_sequence = np.array([[10, 20, 30, 40, 50]])

# Embedding lookup: convert tokens to vectors
embedded = embedding(token_sequence)
print('Input shape (token IDs):', token_sequence.shape)
print('  Batch of 1 sample, 5 tokens')

print('\nOutput shape (embedded vectors):', embedded.shape)
print('  Batch of 1 sample, 5 tokens, 32 dimensions each')

# INTERPRETATION:
# token_sequence[0,0] = 10 → embedding(10) = vector of 32 floats (e.g., [0.2, -0.5, 0.1, ...])
# During training, embedding(10) updates to capture meaning of word with ID 10

# ADVANTAGES of Embedding vs One-Hot:
print('\n\nEmbedding vs One-Hot Encoding:')
print('One-Hot: [0,0,1,0,0,0,0,0,0,0] ← 1000-dimensional, sparse, wastes memory')
print('Embedding: [0.2, -0.5, 0.1, ...] ← 32-dimensional, dense, efficient')
print(f'Size reduction: 1000 → {embedding_dim}x')

# HOW EMBEDDINGS ARE LEARNED:
print('\nEmbedding learning:')
print('1. Initialize: random vectors for each word')
print('2. Train model on downstream task (e.g., sentiment classification)')
print('3. Backprop updates embedding vectors')
print('4. Similar words naturally end up with similar vectors')
print('5. Example: "good" and "great" vectors become close')

Building an LSTM Text Classifier

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# PARAMETERS:
vocab_size = 1000          # Unique words in vocabulary
seq_length = 20            # Max sequence length
embedding_dim = 32         # Word vector dimension
lstm_units = 64            # Hidden state dimension in LSTM

# TEXT CLASSIFICATION MODEL:
model = keras.Sequential([
    # INPUT: Token sequences (batch_size, seq_length)
    # Example: [[42, 156, 89, 234, 0, 0, ...], ...]
    
    # LAYER 1: Embedding
    # Converts token IDs → dense vectors
    # Output: (batch_size, seq_length, embedding_dim)
    layers.Embedding(
        input_dim=vocab_size,      # Dictionary size
        output_dim=embedding_dim,  # Vector dimension
        input_length=seq_length
    ),
    # After: (batch_size, 20, 32)
    
    # LAYER 2: LSTM (Long Short-Term Memory)
    # Processes sequence while maintaining memory
    # Each LSTM cell:
    #   - Input: current token vector + previous hidden state
    #   - Output: new hidden state
    # return_sequences=False → output only FINAL hidden state
    layers.LSTM(
        units=lstm_units,            # Hidden state dimension
        return_sequences=False,      # Only return final state (not all states)
        dropout=0.2,                 # 20% dropout inside LSTM
        recurrent_dropout=0.2        # Dropout on recurrent connections
    ),
    # After: (batch_size, 64)
    # Single vector per sample containing sequence context
    
    # LAYER 3: Dropout
    # Prevent overfitting: randomly drop 30% of units
    layers.Dropout(rate=0.3),
    
    # LAYER 4: Output
    # Binary classification: 1 unit + sigmoid
    layers.Dense(
        units=1,              # 1 output (probability)
        activation='sigmoid'  # Squash to [0, 1]
    )
    # Output: (batch_size, 1) probability for each sample
])

# COMPILE:
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='binary_crossentropy',  # Binary classification loss
    metrics=['accuracy']
)

model.summary()
print('\nModel flow:')
print('Text → Tokenize → [42, 156, 89, 234, ...]')
print('       → Embed → [[0.2, -0.5, ...], [0.1, 0.8, ...], ...]')
print('       → LSTM → [final_context_vector]')
print('       → Dense(1) + sigmoid → 0.8 (probability of positive sentiment)')

LSTM Internals: How It Remembers

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# LSTM CELL INTERNALS:
# Each LSTM cell has:
# - hidden state h[t]: current context/memory
# - cell state c[t]: long-term memory
#
# Processing token at time t:
# 1. FORGET GATE: Decide what to forget from previous memory
# 2. INPUT GATE: Decide what new information to add
# 3. CELL UPDATE: Update internal state
# 4. OUTPUT GATE: Decide what to output as new hidden state

# Example: "I love deep learning" → sentiment = positive
# Token sequence: [I, love, deep, learning]
#
# Step 1: Process "I"
#   hidden_state = some vector
#
# Step 2: Process "love" (positive word)
#   FORGET: Keep most previous state
#   INPUT: Add strong positive signal
#   hidden_state updated to reflect "love"
#
# Step 3: Process "deep"
#   Accumulate context
#
# Step 4: Process "learning"
#   hidden_state now contains full context of all tokens
#   LSTM remembers "love" (positive) despite appearing at beginning!

# KEY INSIGHT: LSTM solves vanishing gradient problem
# Traditional RNN: gradients decay exponentially over long sequences
# LSTM: maintains constant error flow via cell state connections

print('LSTM Advantages:')
print('  1. Remembers long-range dependencies (words far apart)')
print('  2. Handles variable-length sequences efficiently')
print('  3. Solves vanishing gradient problem')
print('')
print('LSTM vs Traditional RNN:')
print('  RNN: small hidden state, prone to forgetting')
print('  LSTM: gated mechanism explicitly controls what to remember/forget')
print('  GRU: similar to LSTM but simpler (6 gates vs 3 gates in LSTM)')
print('')

# COMPARISON: Different architectures
architectures = {
    'LSTM': 'Full gating mechanism, 4 gates (forget, input, output, cell)',
    'GRU': 'Simplified, 2 gates (reset, update), fewer params than LSTM',
    'SimpleRNN': 'Basic recurrence, no gating, vanishing gradients',
    'Bidirectional': 'Process sequence left-to-right AND right-to-left',
}

print('RNN Architecture Comparison:')
for name, desc in architectures.items():
    print(f'  {name}: {desc}')

Training Text Classifier on Real Data

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow_datasets as tfds
import numpy as np

# Build model
vocab_size = 1000
seq_length = 20

model = keras.Sequential([
    layers.Embedding(vocab_size, 32, input_length=seq_length),
    layers.LSTM(64, return_sequences=False),
    layers.Dropout(0.3),
    layers.Dense(1, activation='sigmoid')
])

model.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)

# OPTION 1: IMDb movie reviews dataset (pre-tokenized)
# Load reviews pre-tokenized as integer sequences
(X_train, y_train), (X_test, y_test) = keras.datasets.imdb.load_data(num_words=vocab_size)

# Pad sequences to fixed length
X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=seq_length)
X_test = keras.preprocessing.sequence.pad_sequences(X_test, maxlen=seq_length)

print(f'Training data shape: {X_train.shape}')
print(f'  {X_train.shape[0]} reviews, each {X_train.shape[1]} tokens')
print(f'Labels: {y_train.shape} (0=negative, 1=positive sentiment)')

# TRAINING:
history = model.fit(
    X_train,
    y_train,
    epochs=5,
    batch_size=128,
    validation_split=0.2,  # 20% for validation
    verbose=1
)

# EVALUATION:
test_loss, test_accuracy = model.evaluate(X_test, y_test, verbose=0)
print(f'\nTest accuracy: {test_accuracy:.2%}')
print(f'Test loss: {test_loss:.4f}')

# PREDICTION on new text:
new_reviews = [
    [1, 45, 120, 50, 200, 0, 0, 0, 0, 0],  # Example: words with IDs [1, 45, 120, ...]
    [200, 50, 30, 15, 8, 2, 1, 0, 0, 0]
]

new_reviews = keras.preprocessing.sequence.pad_sequences(
    new_reviews, maxlen=seq_length
)

predictions = model.predict(new_reviews)
print(f'\nPredictions (probability of positive sentiment):')
for i, pred in enumerate(predictions):
    print(f'  Review {i+1}: {pred[0]:.2%}')

For real text data, use tf.keras.layers.TextVectorization to automatically build vocabulary and convert text to sequences.

Bidirectional LSTM for Better Context

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Bidirectional LSTM processes sequence forward and backward
model = keras.Sequential([
    layers.Embedding(1000, 32, input_length=20),
    layers.Bidirectional(layers.LSTM(32)),  # Wraps LSTM to process both directions
    layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()

print('\nBidirectional LSTM captures context from both past and future tokens.')

Bidirectional models improve performance by seeing the full context (past and future) but double the parameters and computation time.

Time Series Forecasting

Time series prediction uses historical data windows to forecast future values. Create windowed datasets by sliding a window across the series, then use dense layers, RNNs (LSTM/GRU), or CNNs for pattern recognition.

Creating Windowed Dataset

import tensorflow as tf
import numpy as np

# Generate synthetic time series (sine wave with noise)
time_steps = 200
series = np.sin(np.linspace(0, 10, time_steps)) + 0.1 * np.random.randn(time_steps)

# Create windowed dataset
window_size = 10
X_windows = []
y_targets = []

for i in range(len(series) - window_size):
    X_windows.append(series[i:i+window_size])
    y_targets.append(series[i+window_size])

X_windows = np.array(X_windows)
y_targets = np.array(y_targets)

print(f'Created {len(X_windows)} windows.')
print(f'Window shape: {X_windows.shape}')
print(f'Target shape: {y_targets.shape}')

Each window contains window_size historical values; the target is the next value. This converts a time series into a supervised learning problem: X (past) → y (future).

Training a Forecasting Model

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Synthetic time series
series = np.sin(np.linspace(0, 10, 200)) + 0.1 * np.random.randn(200)
window_size = 10

# Create windows
X = []
y = []
for i in range(len(series) - window_size):
    X.append(series[i:i+window_size])
    y.append(series[i+window_size])
X = np.array(X)
y = np.array(y)

# Build forecasting model
model = keras.Sequential([
    layers.Dense(32, activation='relu', input_shape=(window_size,)),
    layers.Dropout(0.2),
    layers.Dense(16, activation='relu'),
    layers.Dense(1)  # Single output: next value
])

model.compile(optimizer='adam', loss='mse', metrics=['mae'])

# Train
history = model.fit(X, y, epochs=10, batch_size=16, validation_split=0.2, verbose=0)

print('Time series model trained.')
print(f'Final validation MAE: {history.history["val_mae"][-1]:.4f}')

For more complex patterns, replace dense layers with LSTM or CNN layers. LSTMs excel at capturing long-term dependencies; CNNs are faster for local patterns.

Using LSTM for Time Series

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Data preparation (same as before)
series = np.sin(np.linspace(0, 10, 200)) + 0.1 * np.random.randn(200)
window_size = 10
X = []
y = []
for i in range(len(series) - window_size):
    X.append(series[i:i+window_size])
    y.append(series[i+window_size])
X = np.array(X).reshape(-1, window_size, 1)  # Reshape for LSTM: (samples, timesteps, features)
y = np.array(y)

# LSTM model
model = keras.Sequential([
    layers.LSTM(32, input_shape=(window_size, 1)),
    layers.Dense(16, activation='relu'),
    layers.Dense(1)
])

model.compile(optimizer='adam', loss='mse')

# Train
model.fit(X, y, epochs=5, batch_size=16, verbose=0)

print('LSTM time series model trained.')
print('LSTM input shape: (samples, timesteps, features)')

LSTM expects 3D input: (batch_size, timesteps, features). Reshape windows accordingly. For multivariate time series (multiple features), increase the last dimension.

Part 5: Advanced Topics

Attention Layers & MultiHeadAttention

Attention mechanisms allow models to focus on relevant parts of input sequences. Keras provides keras.layers.MultiHeadAttention, a production-ready implementation of scaled dot-product attention with multiple heads for parallel processing of different representation subspaces.

Basic Multi-Head Attention

The MultiHeadAttention layer computes attention weights over a sequence or between two different sequences:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Create multi-head attention layer
attention = layers.MultiHeadAttention(
    num_heads=8,      # Number of attention heads
    key_dim=64,       # Dimension of each head
    dropout=0.1       # Dropout for regularization
)

# Prepare inputs
batch_size = 32
seq_length = 20
feature_dim = 512

# Query, Key, Value tensors
query = tf.random.normal([batch_size, seq_length, feature_dim])
key = tf.random.normal([batch_size, seq_length, feature_dim])
value = tf.random.normal([batch_size, seq_length, feature_dim])

# Self-attention (query, key, value from same source)
attn_output = attention(query, value, key=key, return_attention_scores=False)
print(f'Attention output shape: {attn_output.shape}')  # [32, 20, 512]

# Get attention weights
attn_output, attn_weights = attention(query, value, key=key, return_attention_scores=True)
print(f'Attention weights shape: {attn_weights.shape}')  # [32, 8, 20, 20]

Cross-Attention: Use different sequences for Query vs Key/Value, useful for encoder-decoder architectures:

import tensorflow as tf
from tensorflow.keras import layers

attention = layers.MultiHeadAttention(
    num_heads=8,
    key_dim=64
)

# Encoder output (context for attention)
encoder_output = tf.random.normal([16, 30, 512])  # (batch, src_len, d_model)

# Decoder query
decoder_query = tf.random.normal([16, 20, 512])  # (batch, tgt_len, d_model)

# Cross-attention: Query from decoder, Key/Value from encoder
cross_attn = attention(
    query=decoder_query,
    value=encoder_output,
    key=encoder_output
)

print(f'Cross-attention output shape: {cross_attn.shape}')  # [16, 20, 512]

Attention Masking: Prevent attention to padding tokens or future positions:

import tensorflow as tf
from tensorflow.keras import layers
import numpy as np

attention = layers.MultiHeadAttention(
    num_heads=8,
    key_dim=64
)

# Input sequences
query = tf.random.normal([4, 10, 512])
key = tf.random.normal([4, 10, 512])
value = tf.random.normal([4, 10, 512])

# Create causal mask (prevent attention to future tokens)
mask = tf.linalg.band_part(tf.ones((10, 10)), -1, 0)  # Lower triangular
mask = (1.0 - mask) * -1e9  # Mask future positions with large negative values

# Apply attention with mask
attn_output = attention(
    query,
    value,
    key=key,
    attention_mask=mask
)
print(f'Masked attention output: {attn_output.shape}')  # [4, 10, 512]

Building Transformers with Keras

Combine MultiHeadAttention with feed-forward networks and normalization layers to build complete Transformer encoder-decoder architectures.

Transformer Encoder Block

A typical encoder block has self-attention and feed-forward layers with residual connections and layer normalization:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

class TransformerEncoderBlock(layers.Layer):
    def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
        super().__init__()
        self.att = layers.MultiHeadAttention(
            num_heads=num_heads,
            key_dim=embed_dim // num_heads,
            dropout=dropout
        )
        self.ffn = keras.Sequential([
            layers.Dense(ff_dim, activation='relu'),
            layers.Dense(embed_dim)
        ])
        self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)
        self.layernorm2 = layers.LayerNormalization(epsilon=1e-6)
        self.dropout1 = layers.Dropout(dropout)
        self.dropout2 = layers.Dropout(dropout)

    def call(self, inputs, training=False):
        # Self-attention block with residual connection
        attn_output = self.att(inputs, inputs, training=training)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(inputs + attn_output)

        # Feed-forward block with residual connection
        ffn_output = self.ffn(out1)
        ffn_output = self.dropout2(ffn_output, training=training)
        out2 = self.layernorm2(out1 + ffn_output)

        return out2

# Test the encoder block
encoder_block = TransformerEncoderBlock(
    embed_dim=512,
    num_heads=8,
    ff_dim=2048,
    dropout=0.1
)

input_seq = tf.random.normal([8, 20, 512])  # (batch, seq_len, embed_dim)
output = encoder_block(input_seq, training=True)
print(f'Encoder block output shape: {output.shape}')  # [8, 20, 512]

Complete Transformer Model

Stack encoder blocks and add embeddings for a complete sequence model:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

class PositionalEmbedding(layers.Layer):
    def __init__(self, max_len, d_model):
        super().__init__()
        self.d_model = d_model
        # Create positional encodings
        pe = np.zeros((max_len, d_model))
        position = np.arange(0, max_len, dtype=np.float32)[:, np.newaxis]
        div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
        pe[:, 0::2] = np.sin(position * div_term)
        pe[:, 1::2] = np.cos(position * div_term)
        self.pe = tf.constant(pe[np.newaxis, ...], dtype=tf.float32)

    def call(self, x):
        return x + self.pe[:, :tf.shape(x)[1], :]

class TransformerModel(keras.Model):
    def __init__(self, vocab_size, max_len, d_model, num_heads, num_layers, ff_dim):
        super().__init__()
        self.embedding = layers.Embedding(vocab_size, d_model)
        self.pos_embedding = PositionalEmbedding(max_len, d_model)
        self.encoder_blocks = [
            TransformerEncoderBlock(d_model, num_heads, ff_dim)
            for _ in range(num_layers)
        ]
        self.final_norm = layers.LayerNormalization(epsilon=1e-6)
        self.output_dense = layers.Dense(vocab_size)

    def call(self, inputs, training=False):
        # Embed and add positional encodings
        x = self.embedding(inputs)
        x = self.pos_embedding(x)

        # Pass through encoder blocks
        for encoder_block in self.encoder_blocks:
            x = encoder_block(x, training=training)

        # Final normalization and output
        x = self.final_norm(x)
        x = self.output_dense(x)
        return x

# Create and use model
model = TransformerModel(
    vocab_size=10000,
    max_len=100,
    d_model=512,
    num_heads=8,
    num_layers=6,
    ff_dim=2048
)

# Forward pass
input_ids = tf.random.uniform([8, 50], minval=0, maxval=10000, dtype=tf.int32)
output = model(input_ids, training=True)
print(f'Transformer output shape: {output.shape}')  # [8, 50, 10000]

# Count parameters
total_params = sum([tf.size(w).numpy() for w in model.trainable_weights])
print(f'Total parameters: {total_params / 1e6:.1f}M')

Vision Transformer (ViT) with Keras

Apply Transformers to image patches for state-of-the-art vision models:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

class PatchEmbedding(layers.Layer):
    def __init__(self, patch_size, embed_dim):
        super().__init__()
        self.patch_size = patch_size
        self.embed_dim = embed_dim
        self.projection = layers.Dense(embed_dim)

    def call(self, images):
        # images shape: (batch, height, width, channels)
        batch_size = tf.shape(images)[0]
        
        # Extract patches
        patches = tf.image.extract_patches(
            images,
            sizes=[1, self.patch_size, self.patch_size, 1],
            strides=[1, self.patch_size, self.patch_size, 1],
            rates=[1, 1, 1, 1],
            padding='VALID'
        )
        
        # Flatten patches
        num_patches = tf.shape(patches)[1] * tf.shape(patches)[2]
        patch_dim = tf.shape(patches)[-1]
        patches = tf.reshape(patches, [batch_size, num_patches, patch_dim])
        
        # Project to embedding dimension
        embeddings = self.projection(patches)
        return embeddings

class VisionTransformer(keras.Model):
    def __init__(self, image_size, patch_size, num_classes, d_model, num_heads, num_layers, ff_dim):
        super().__init__()
        num_patches = (image_size // patch_size) ** 2
        
        self.patch_embed = PatchEmbedding(patch_size, d_model)
        self.cls_token = self.add_weight(
            'cls_token',
            shape=[1, 1, d_model],
            initializer='random_normal'
        )
        self.pos_embed = self.add_weight(
            'pos_embed',
            shape=[1, num_patches + 1, d_model],
            initializer='random_normal'
        )
        
        self.encoder_blocks = [
            TransformerEncoderBlock(d_model, num_heads, ff_dim)
            for _ in range(num_layers)
        ]
        
        self.norm = layers.LayerNormalization(epsilon=1e-6)
        self.head = layers.Dense(num_classes)

    def call(self, images, training=False):
        batch_size = tf.shape(images)[0]
        
        # Embed patches
        x = self.patch_embed(images)
        
        # Add class token
        cls_tokens = tf.broadcast_to(self.cls_token, [batch_size, 1, tf.shape(x)[-1]])
        x = tf.concat([cls_tokens, x], axis=1)
        
        # Add positional embeddings
        x = x + self.pos_embed
        
        # Transformer encoder blocks
        for encoder_block in self.encoder_blocks:
            x = encoder_block(x, training=training)
        
        # Classification from [CLS] token
        x = self.norm(x)
        x = x[:, 0]  # Take [CLS] token
        x = self.head(x)
        return x

# Create ViT model
vit = VisionTransformer(
    image_size=224,
    patch_size=16,
    num_classes=1000,
    d_model=768,
    num_heads=12,
    num_layers=12,
    ff_dim=3072
)

# Forward pass
images = tf.random.normal([4, 224, 224, 3])
logits = vit(images, training=True)
print(f'ViT output shape: {logits.shape}')  # [4, 1000]

Distributed Training & Multi-GPU

Scale model training across multiple GPUs or TPUs using TensorFlow's distribution strategies. Keras models automatically support distributed training with minimal code changes.

Single-Machine Multi-GPU with MirroredStrategy

Replicate model on all GPUs and sync gradients after each batch:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Create distribution strategy
strategy = tf.distribute.MirroredStrategy()

print(f'Number of devices: {strategy.num_replicas_in_sync}')
print(f'Devices: {tf.config.list_physical_devices("GPU")}')

# Build model inside strategy scope
with strategy.scope():
    model = keras.Sequential([
        layers.Dense(64, activation='relu', input_shape=(20,)),
        layers.Dropout(0.2),
        layers.Dense(32, activation='relu'),
        layers.Dense(10, activation='softmax')
    ])
    
    model.compile(
        optimizer='adam',
        loss='categorical_crossentropy',
        metrics=['accuracy']
    )

# Data preparation
X = np.random.randn(1000, 20).astype('float32')
y = np.eye(10)[np.random.randint(0, 10, 1000)]

# Training automatically uses all GPUs
history = model.fit(
    X, y,
    epochs=5,
    batch_size=32,  # Per-GPU batch size
    verbose=1
)

print('Training complete with multi-GPU acceleration.')

Multi-Machine with ParameterServerStrategy

Distribute training across multiple machines with parameter servers:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Setup for multi-machine training
# Requires cluster configuration (TF_CONFIG environment variable)

if tf.config.list_physical_devices('GPU'):
    # Multi-GPU strategy
    strategy = tf.distribute.MultiWorkerMirroredStrategy()
else:
    # Fallback: single machine
    strategy = tf.distribute.get_strategy()

print(f'Devices: {strategy.num_replicas_in_sync}')

# Build and compile inside strategy scope
with strategy.scope():
    model = keras.Sequential([
        layers.Dense(128, activation='relu', input_shape=(100,)),
        layers.Dropout(0.3),
        layers.Dense(64, activation='relu'),
        layers.Dense(32, activation='relu'),
        layers.Dense(10, activation='softmax')
    ])
    
    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=0.001),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy']
    )

print('Model ready for multi-machine distributed training')
print('Set TF_CONFIG environment variable with cluster specification')

Custom Training Loop with Distribution

For fine-grained control, use strategy.run() with custom training steps:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

strategy = tf.distribute.MirroredStrategy()

# Data preparation
X = np.random.randn(1000, 32).astype('float32')
y = np.random.randint(0, 10, 1000)

dataset = tf.data.Dataset.from_tensor_slices((X, y))
dataset = dataset.shuffle(1000).batch(32)
dist_dataset = strategy.experimental_distribute_dataset(dataset)

# Build model
with strategy.scope():
    model = keras.Sequential([
        layers.Dense(64, activation='relu'),
        layers.Dense(10, activation='softmax')
    ])
    
    optimizer = keras.optimizers.Adam()
    loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=False)

# Custom training step
def train_step(batch_x, batch_y):
    with tf.GradientTape() as tape:
        logits = model(batch_x, training=True)
        loss_value = loss_fn(batch_y, logits)
        scaled_loss = loss_value / strategy.num_replicas_in_sync
    
    grads = tape.gradient(scaled_loss, model.trainable_weights)
    optimizer.apply_gradients(zip(grads, model.trainable_weights))
    
    return loss_value

@tf.function
def distributed_train_step(dist_inputs):
    per_replica_losses = strategy.run(train_step, args=(dist_inputs[0], dist_inputs[1]))
    return strategy.reduce(tf.distribute.ReduceOp.SUM, per_replica_losses, axis=None)

# Training loop
epochs = 3
for epoch in range(epochs):
    total_loss = 0.0
    num_batches = 0
    
    for dist_batch in dist_dataset:
        loss = distributed_train_step(dist_batch)
        total_loss += loss
        num_batches += 1
    
    avg_loss = total_loss / num_batches
    print(f'Epoch {epoch + 1}, Loss: {avg_loss:.4f}')

print('Custom distributed training complete')

Gradient Synchronization: TensorFlow automatically synchronizes gradients across replicas. Batch size is per-replica; total batch = batch_size × num_replicas.

Performance Optimization

TensorFlow offers multiple performance optimizations: @tf.function for graph compilation, mixed precision for faster GPU training, and distributed strategies for multi-GPU scaling. These techniques can deliver 2-10× speedups with minimal code changes.

Graph Compilation with @tf.function

Convert Python functions to optimized TensorFlow graphs for faster execution:

import tensorflow as tf
import time

# Regular Python function (eager execution)
def slow_function(x):
    return tf.reduce_sum(x * x)

# Graph-compiled function
@tf.function
def fast_function(x):
    return tf.reduce_sum(x * x)

# Benchmark
x = tf.random.normal([10000])

# Warm-up
fast_function(x)

# Time eager execution
start = time.time()
for _ in range(100):
    slow_function(x)
eager_time = time.time() - start

# Time graph execution
start = time.time()
for _ in range(100):
    fast_function(x)
graph_time = time.time() - start

print(f'Eager execution: {eager_time:.4f}s')
print(f'Graph execution: {graph_time:.4f}s')
print(f'Speedup: {eager_time/graph_time:.2f}x')

@tf.function traces the Python function once, builds a graph, and reuses it for subsequent calls—eliminating Python overhead. Use for training loops, inference functions, and data preprocessing.

Mixed Precision Training

Train with float16 for speed while keeping critical ops in float32 for stability:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, mixed_precision

# Enable mixed precision (requires compatible GPU)
# Note: This may not show benefits without GPU
mixed_precision.set_global_policy('mixed_float16')

print('Mixed precision policy:', mixed_precision.global_policy())

# Build model (automatically uses float16 where beneficial)
model = keras.Sequential([
    layers.Dense(128, activation='relu', input_shape=(32,)),
    layers.Dense(64, activation='relu'),
    layers.Dense(10)  # Output layer uses float32 for stability
])

# Loss scaling prevents underflow in gradients
# (handled automatically in model.compile with mixed precision)
model.compile(
    optimizer='adam',
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True)
)

print('Model built with mixed precision.')

# Reset to float32 for rest of examples
mixed_precision.set_global_policy('float32')

Mixed precision can deliver 2-3× speedups on modern GPUs (V100, A100) with minimal accuracy loss. TensorFlow handles loss scaling automatically to prevent gradient underflow.

Distributed Training with MirroredStrategy

Synchronous multi-GPU training with minimal code changes:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Create distribution strategy (auto-detects GPUs)
strategy = tf.distribute.MirroredStrategy()

print(f'Number of devices: {strategy.num_replicas_in_sync}')

# Build model inside strategy scope
with strategy.scope():
    model = keras.Sequential([
        layers.Dense(64, activation='relu', input_shape=(20,)),
        layers.Dense(32, activation='relu'),
        layers.Dense(1)
    ])
    
    model.compile(optimizer='adam', loss='mse')

print('Model created for distributed training.')
print('Note: Benefits appear with multiple GPUs; falls back to single device if unavailable.')

MirroredStrategy replicates model on all GPUs, splits batches across devices, and averages gradients. Near-linear scaling up to 8 GPUs for large batch training.

Model Interpretability

Understanding why models make predictions is critical for debugging, trust, and compliance. Grad-CAM (Gradient-weighted Class Activation Mapping) visualizes important regions in images for CNN decisions. For other interpretability methods, explore SHAP, LIME, and attention weights.

Grad-CAM Concept

Grad-CAM computes gradients of class predictions with respect to convolutional layer activations, highlighting important spatial regions:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Simple CNN for demonstration
model = keras.Sequential([
    layers.Conv2D(16, 3, activation='relu', input_shape=(32, 32, 3)),
    layers.Conv2D(32, 3, activation='relu', name='target_conv'),
    layers.GlobalAveragePooling2D(),
    layers.Dense(10, activation='softmax')
])

# Simplified Grad-CAM stub (conceptual demonstration)
def grad_cam_stub(model, img_tensor, class_index=0):
    """Simplified Grad-CAM for educational purposes."""
    # Get conv layer and predictions
    conv_layer = model.get_layer('target_conv')
    grad_model = keras.Model(
        inputs=model.inputs,
        outputs=[conv_layer.output, model.output]
    )
    
    # Compute gradients
    with tf.GradientTape() as tape:
        conv_outputs, predictions = grad_model(img_tensor)
        loss = predictions[:, class_index]
    
    # Get gradients of loss w.r.t. conv outputs
    grads = tape.gradient(loss, conv_outputs)
    
    # Weight feature maps by gradient importance
    weights = tf.reduce_mean(grads, axis=(0, 1, 2))
    cam = tf.reduce_sum(tf.multiply(weights, conv_outputs[0]), axis=-1)
    
    # Normalize heatmap
    cam = tf.maximum(cam, 0)
    cam = cam / tf.reduce_max(cam)
    
    return cam.numpy()

# Test with random image
test_img = tf.random.normal([1, 32, 32, 3])
heatmap = grad_cam_stub(model, test_img, class_index=0)

print('Grad-CAM heatmap shape:', heatmap.shape)
print('Heatmap highlights important regions for class prediction.')

In practice, overlay the heatmap on the original image using matplotlib's imshow with transparency. Bright regions indicate areas the model focused on for classification.

Deployment & Serving

Models are only valuable when deployed to production. TensorFlow offers multiple deployment options: TensorFlow Serving (high-throughput servers), TensorFlow Lite (mobile/edge), TensorFlow.js (web browsers), and cloud platforms (AWS SageMaker, Google AI Platform).

Saving for Deployment

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Build and train simple model
model = keras.Sequential([
    layers.Dense(32, activation='relu', input_shape=(16,)),
    layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy')

# Simulate training
X = np.random.randn(100, 16).astype('float32')
y = np.random.randint(0, 2, (100,)).astype('int32')
model.fit(X, y, epochs=2, verbose=0)

# Save as SavedModel (recommended for production)
model.save('production_model', save_format='tf')
print('Model saved in SavedModel format for TensorFlow Serving.')

# Load and verify
loaded = keras.models.load_model('production_model')
test_input = np.random.randn(5, 16).astype('float32')
predictions = loaded.predict(test_input, verbose=0)
print('Predictions shape:', predictions.shape)

SavedModel format includes architecture, weights, and computation graph—everything needed for inference in production environments.

TensorFlow Serving Overview

TensorFlow Serving is a high-performance serving system for production ML:

# Install TensorFlow Serving (Docker recommended)
docker pull tensorflow/serving

# Serve a SavedModel
docker run -p 8501:8501 \
  --mount type=bind,source=/path/to/production_model,target=/models/my_model \
  -e MODEL_NAME=my_model \
  -t tensorflow/serving

# Query the REST API
curl -X POST http://localhost:8501/v1/models/my_model:predict \
  -d '{"instances": [[1.0, 2.0, ..., 16.0]]}'

TensorFlow Serving handles versioning, batching, and GPU utilization automatically. It supports gRPC (low latency) and REST APIs (easy integration).

Best Practices & Next Steps

Mastering TensorFlow requires balancing theory, practice, and production awareness. Here's a comprehensive guide to solidify your foundation and advance your skills.

Production Deployment Checklist

1. Data Pipelines: Always use tf.data with prefetch for efficient I/O
2. Validation: Monitor training vs validation metrics—stop when validation degrades
3. Checkpointing: Save model checkpoints during training (use ModelCheckpoint)
4. TensorBoard: Log metrics early; visualize training curves to diagnose issues
5. Regularization: Apply dropout, L1/L2, or early stopping to combat overfitting
6. Testing: Evaluate on held-out test set; report confidence intervals for metrics
7. Versioning: Version models (semantic versioning) and track training config

Advanced Learning Paths

Computer Vision: Explore object detection (YOLO, Faster R-CNN), semantic segmentation (U-Net, Mask R-CNN), and generative models (GANs, diffusion models).

NLP & Transformers: Dive into Transformer architectures (BERT, GPT), use Hugging Face libraries with TensorFlow backend, or implement attention mechanisms from scratch.

Reinforcement Learning: Use TF-Agents for policy gradient methods, DQN, and actor-critic algorithms in game environments or robotics.

Optimization & Deployment: Profile with TensorBoard Profiler, quantize models for edge deployment (TensorFlow Lite), experiment with pruning and knowledge distillation.

Research: Implement papers from arXiv, contribute to TensorFlow addons, or build custom training loops for novel architectures.

Key Terms Glossary

Common Pitfalls Reference

TensorFlow & Keras Complete Reference

Comprehensive cheat sheet for essential TensorFlow and Keras APIs, code patterns, and best practices.

model = keras.Sequential([
    layers.Dense(64, activation='relu', input_shape=(20,)),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)
model.fit(X_train, y_train, epochs=10, 
          validation_split=0.2)

with tf.GradientTape() as tape:
    pred = model(x, training=True)
    loss = loss_fn(y, pred)
grads = tape.gradient(loss, 
                      model.trainable_variables)
opt.apply_gradients(
    zip(grads, model.trainable_variables)
)

inputs = keras.Input(shape=(16,))
x = layers.Dense(32, activation='relu')(inputs)
x = layers.Dropout(0.2)(x)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)