Chapter 11 - Multi-layer Artificial Neural Networks - Machine Learning for Intelligent Systems & Robotics

Prework¶

Import essentials¶

from IPython.display import Image
%matplotlib inline

Check package version (optional)¶

Add folder to path in order to load from the check_packages.py script.

import sys
sys.path.insert(0, '..')

Check recommended package versions.

# Package version check (Colab-safe: no external script required)
import importlib

def check_packages(d):
    for pkg_name, min_version in d.items():
        try:
            imported = importlib.import_module(pkg_name)
            version = getattr(imported, '__version__', 'unknown')
            print(f'[OK] {pkg_name} {version}')
        except ImportError:
            print(f'[MISSING] {pkg_name} is not installed')

d = {
    'numpy': '1.21.2',
    'matplotlib': '3.4.3',
    'sklearn': '1.0',
}
check_packages(d)

Watermark (optional)¶

Optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible.

You can install watermark Jupyter extension via

conda install watermark -c conda-forge

pip install watermark

For more information, please see: https://github.com/rasbt/watermark.

# 'watermark' is not preinstalled on Colab, so install it first if needed
import importlib
if importlib.util.find_spec('watermark') is None:
    %pip install -q watermark

%load_ext watermark
%watermark -a "Sebastian Raschka, Johnny Yu" -u -d -v -p numpy,pandas,matplotlib,scipy,sklearn

Table of Contents¶

11.1.Modeling complex functions with artificial neural networks¶

11.1.1.Single-layer neural network recap¶

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_01.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_01.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=600))

11.1.2.Introducing the multi-layer neural network architecture¶

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_02.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_02.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=600))

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_03.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_03.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=500))

11.1.3.Activating a neural network via forward propagation¶

No example code.

11.2.Classifying handwritten digits¶

11.2.1.Obtaining and preparing the MNIST dataset¶

The MNIST dataset is publicly available at http://yann.lecun.com/exdb/mnist/ and consists of the following four parts:

Training set images: train-images-idx3-ubyte.gz (9.9 MB, 47 MB unzipped, 60,000 examples)
Training set labels: train-labels-idx1-ubyte.gz (29 KB, 60 KB unzipped, 60,000 labels)
Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, 7.8 MB, 10,000 examples)
Test set labels: t10k-labels-idx1-ubyte.gz (5 KB, 10 KB unzipped, 10,000 labels)

The MNIST dataset was constructed from two datasets of the US National Institute of Standards and Technology (NIST). The training dataset consists of handwritten digits from 250 different people, 50 percent high school students, and 50 percent employees from the Census Bureau. Note that the test dataset contains handwritten digits from different people following the same split.

Use scikit-learn’s new fetch_openml function, which allows us to load the MNIST dataset more conveniently, instead of using numpy.

In scikit-learn, the fetch_openml function downloads the MNIST dataset from OpenML (https://www.openml.org/d/554) as pandas DataFrame and Series objects, which is why we use the .values attribute to obtain the underlying NumPy arrays. (If you are using a scikit-learn version older than 1.0, fetch_openml downloads NumPy arrays directly so you can omit using the .values attribute.)

Result explanation

X array consists of 70,000 images, an 784 (28×28) pixels for each image
- each pixel is represented by a grayscale intensity value
y array stores the corresponding 70,000 class labels

Here, fetch_openml already unrolled the 28×28 pixels into one-dimensional row vectors, which represent the rows in our X array (784 per row or image) above. The second array (y) returned by the fetch_openml function contains the corresponding target variable, the class labels (integers 0-9) of the handwritten digits.

from sklearn.datasets import fetch_openml


X, y = fetch_openml('mnist_784', version=1, return_X_y=True)
X = X.values
y = y.astype(int).values

print(X.shape)
print(y.shape)

Next, let’s normalize the pixels values in MNIST to the range –1 to 1 (originally 0 to 255 for grayscale intensity) via the following code line.

The reason behind normalizing the pixels is that gradient-based optimization is much more stable under these conditions.

X = ((X / 255.) - .5) * 2

To get an idea of how those images in MNIST look, let’s visualize examples of the digits 0-9 after reshaping the 784-pixel vectors from our feature matrix into the original 28×28 image that we can plot via Matplotlib’s imshow function.

Result explanation We should now see a plot of the 2×5 subfigures showing a representative image of each unique digit.

import matplotlib.pyplot as plt


# Visualize the first digit of each class
fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True)
ax = ax.flatten()
for i in range(10):
    img = X[y == i][0].reshape(28, 28)
    ax[i].imshow(img, cmap='Greys')

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
#plt.savefig('figures/11_4.png', dpi=300)
plt.show()

In addition, let’s also plot multiple examples of the same digit to see how different the handwriting for each really is.

Result explanation After executing the code, we should now see the first 25 variants of the digit 7.

# Visualize 25 different versions of "7"
fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True)
ax = ax.flatten()
for i in range(25):
    img = X[y == 7][i].reshape(28, 28)
    ax[i].imshow(img, cmap='Greys')

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
# plt.savefig('figures/11_5.png', dpi=300)
plt.show()

Finally, let’s divide the dataset into training, validation, and test subsets. The following code will split the dataset such that 55,000 images are used for training, 5,000 images for validation, and 10,000 images for testing.

from sklearn.model_selection import train_test_split


# Split into training, validation, and test set
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=10000, random_state=123, stratify=y)

X_train, X_valid, y_train, y_valid = train_test_split(
    X_temp, y_temp, test_size=5000, random_state=123, stratify=y_temp)


# optional to free up some memory by deleting non-used arrays:
del X_temp, y_temp, X, y

11.2.2.Implementing a multi-layer perceptron¶

We will now implement an MLP from scratch to classify the images in the MNIST dataset. To keep things simple, we will implement an MLP with only one hidden layer.

Let’s look at the following implementation of an MLP, starting with the two helper functions to compute the logistic sigmoid activation and to convert integer class label arrays to one-hot encoded labels.

The __init__ constructor instantiates the weight matrices and bias vectors for the hidden and the output layer.
- These are used in the forward method to make predictions.
The forward method takes in one or more training examples and returns the predictions.
- It returns both the activation values from the hidden layer (a_h) and the output layer (a_out).
- a_h: to optimize the model parameters; that is, the weight and bias units of the hidden and output layers
- a_out: represents the class-membership probabilities that we can convert to class labels
The backward method, which updates the weight and bias parameters of the neural network.
- Here for simplicity, we use MSE for the loss function.

Note that this code implementation of the NeuralNetMLP class differs from the familiar scikit-learn API that is centered around the .fit() and .predict() methods. Instead, the main methods of the NeuralNetMLP class are the .forward() and .backward() methods. One of the reasons behind this is that it makes a complex neural network a bit easier to understand in terms of how the information flows through the networks.

import numpy as np


##########################
### MODEL
##########################

def sigmoid(z):                                        
    return 1. / (1. + np.exp(-z))


def int_to_onehot(y, num_labels):

    ary = np.zeros((y.shape[0], num_labels))
    for i, val in enumerate(y):
        ary[i, val] = 1

    return ary


class NeuralNetMLP:

    def __init__(self, num_features, num_hidden, num_classes, random_seed=123):
        super().__init__()
        
        self.num_classes = num_classes
        
        # hidden
        rng = np.random.RandomState(random_seed)
        
        self.weight_h = rng.normal(
            loc=0.0, scale=0.1, size=(num_hidden, num_features))
        self.bias_h = np.zeros(num_hidden)
        
        # output
        self.weight_out = rng.normal(
            loc=0.0, scale=0.1, size=(num_classes, num_hidden))
        self.bias_out = np.zeros(num_classes)
        
    def forward(self, x):
        # Hidden layer
        # input dim: [n_examples, n_features] dot [n_hidden, n_features].T
        # output dim: [n_examples, n_hidden]
        z_h = np.dot(x, self.weight_h.T) + self.bias_h
        a_h = sigmoid(z_h)

        # Output layer
        # input dim: [n_examples, n_hidden] dot [n_classes, n_hidden].T
        # output dim: [n_examples, n_classes]
        z_out = np.dot(a_h, self.weight_out.T) + self.bias_out
        a_out = sigmoid(z_out)
        return a_h, a_out

    def backward(self, x, a_h, a_out, y):  
    
        #########################
        ### Output layer weights
        #########################
        
        # onehot encoding
        y_onehot = int_to_onehot(y, self.num_classes)

        # Part 1: dLoss/dOutWeights
        ## = dLoss/dOutAct * dOutAct/dOutNet * dOutNet/dOutWeight
        ## where DeltaOut = dLoss/dOutAct * dOutAct/dOutNet
        ## for convenient re-use
        
        # input/output dim: [n_examples, n_classes]
        d_loss__d_a_out = 2.*(a_out - y_onehot) / y.shape[0]

        # input/output dim: [n_examples, n_classes]
        d_a_out__d_z_out = a_out * (1. - a_out) # sigmoid derivative

        # output dim: [n_examples, n_classes]
        delta_out = d_loss__d_a_out * d_a_out__d_z_out # "delta (rule) placeholder"

        # gradient for output weights
        
        # [n_examples, n_hidden]
        d_z_out__dw_out = a_h
        
        # input dim: [n_classes, n_examples] dot [n_examples, n_hidden]
        # output dim: [n_classes, n_hidden]
        d_loss__dw_out = np.dot(delta_out.T, d_z_out__dw_out)
        d_loss__db_out = np.sum(delta_out, axis=0)
        

        #################################        
        # Part 2: dLoss/dHiddenWeights
        ## = DeltaOut * dOutNet/dHiddenAct * dHiddenAct/dHiddenNet * dHiddenNet/dWeight
        
        # [n_classes, n_hidden]
        d_z_out__a_h = self.weight_out
        
        # output dim: [n_examples, n_hidden]
        d_loss__a_h = np.dot(delta_out, d_z_out__a_h)
        
        # [n_examples, n_hidden]
        d_a_h__d_z_h = a_h * (1. - a_h) # sigmoid derivative
        
        # [n_examples, n_features]
        d_z_h__d_w_h = x
        
        # output dim: [n_hidden, n_features]
        d_loss__d_w_h = np.dot((d_loss__a_h * d_a_h__d_z_h).T, d_z_h__d_w_h)
        d_loss__d_b_h = np.sum((d_loss__a_h * d_a_h__d_z_h), axis=0)

        return (d_loss__dw_out, d_loss__db_out, 
                d_loss__d_w_h, d_loss__d_b_h)

After we have implemented the NeuralNetMLP class, we use the following code to instantiate a new NeuralNetMLP object.

The model accepts MNIST images reshaped into 784-dimensional vectors (in the format of X_train, X_valid, or X_test, which we defined previously) for the 10 integer classes (digits 0-9). The hidden layer consists of 50 nodes. Also, as you may be able to tell from looking at the previously defined .forward() method, we use a sigmoid activation function after the first hidden layer and output layer to keep things simple.

model = NeuralNetMLP(num_features=28*28,
                     num_hidden=50,
                     num_classes=10)

The figure shows the summary of the neural network architecture that we instantiated above.

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_06.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_06.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=500))

11.2.3.Coding the neural network training loop¶

In this subsection, we are going to implement the training function that we can use to train the network on mini-batches of the data via backpropagation.

The first function we are going to define is a mini-batch generator, which takes in our dataset and divides it into mini-batches of a desired size for stochastic gradient descent training.

import numpy as np

num_epochs = 50
minibatch_size = 100


def minibatch_generator(X, y, minibatch_size):
    indices = np.arange(X.shape[0])
    np.random.shuffle(indices)

    for start_idx in range(0, indices.shape[0] - minibatch_size 
                           + 1, minibatch_size):
        batch_idx = indices[start_idx:start_idx + minibatch_size]
        
        yield X[batch_idx], y[batch_idx]

        
# Iterate over training epochs
for i in range(num_epochs):

    # Iterate over minibatches
    minibatch_gen = minibatch_generator(
        X_train, y_train, minibatch_size)
    
    for X_train_mini, y_train_mini in minibatch_gen:

        break
        
    break
    
# Print the dimension of the mini-batches to confirm the desired size
print(X_train_mini.shape)
print(y_train_mini.shape)

Loss function and performance metric¶

1) Intuitive way¶

Next, we have to define our loss function and performance metric that we can use to monitor the training process and evaluate the model. And test the preceding function and compute the initial validation set MSE and accuracy of the model we instantiated in the previous section.

Note that model.forward() returns the hidden and output layer activations. Remember that we have 10 output nodes (one corresponding to each unique class label). Hence, when computing the MSE, we first converted the class labels into one-hot encoded class labels in the mse_loss() function. In practice, it does not make a difference whether we average over the row or the columns of the squared-difference matrix first, so we simply call np.mean() without any axis specification so that it returns a scalar.

The output layer activations, since we used the logistic sigmoid function, are values in the range [0, 1]. For each input, the output layer produces 10 values in the range [0, 1], so we used the np.argmax() function to select the index position of the largest value, which yields the predicted class label. We then compared the true labels with the predicted class labels to compute the accuracy via the accuracy() function we defined.

Result explanation

As we can see from the output, the accuracy is not very high. However, given that we have a balanced dataset with 10 classes, a prediction accuracy of approximately 10 percent is what we would expect for an untrained model producing random predictions.

def mse_loss(targets, probas, num_labels=10):
    onehot_targets = int_to_onehot(targets, num_labels=num_labels)
    return np.mean((onehot_targets - probas)**2)


def accuracy(targets, predicted_labels):
    return np.mean(predicted_labels == targets) 


# Test the preceding functions (before training)
_, probas = model.forward(X_valid)
mse = mse_loss(y_valid, probas)

predicted_labels = np.argmax(probas, axis=1)
acc = accuracy(y_valid, predicted_labels)

print(f'Initial validation MSE: {mse:.1f}')
print(f'Initial validation accuracy: {acc*100:.1f}%')

2) Memory-efficient way¶

In practice, our computer memory is usually a limiting factor for how much data the model can ingest in one forward pass (due to the large matrix multiplications). Hence, we are defining our MSE and accuracy computation based on our previous mini-batch generator. The following function will compute the MSE and accuracy incrementally by iterating over the dataset one mini-batch at a time to be more memory-efficient.

Result explanation

As we can see from the results, our generator approach produces the same results as the previously defined MSE and accuracy functions, except for a small rounding error in the MSE (0.27 versus 0.28), which is negligible for our purposes.

def compute_mse_and_acc(nnet, X, y, num_labels=10, minibatch_size=100):
    mse, correct_pred, num_examples = 0., 0, 0
    minibatch_gen = minibatch_generator(X, y, minibatch_size)
        
    for i, (features, targets) in enumerate(minibatch_gen):

        _, probas = nnet.forward(features)
        predicted_labels = np.argmax(probas, axis=1)
        
        onehot_targets = int_to_onehot(targets, num_labels=num_labels)
        loss = np.mean((onehot_targets - probas)**2)
        correct_pred += (predicted_labels == targets).sum()
        
        num_examples += targets.shape[0]
        mse += loss

    mse = mse/i
    acc = correct_pred/num_examples
    return mse, acc

# test the preceding functions (before training)
mse, acc = compute_mse_and_acc(model, X_valid, y_valid)
print(f'Initial valid MSE: {mse:.1f}')
print(f'Initial valid accuracy: {acc*100:.1f}%')

Training model¶

Let’s now implement the code to train our model. On a high level, the train() function iterates over multiple epochs, and in each epoch, it used the previously defined minibatch_generator() function to iterate over the whole training set in mini-batches for stochastic gradient descent training. Inside the mini-batch generator for loop, we obtain the outputs from the model, a_h and a_out, via its .forward() method. Then, we compute the loss gradients via the model’s .backward() method. Using the loss gradients, we update the weights by adding the negative gradient multiplied by the learning rate. For example, to update the model weights of the hidden layer, we defined the following line:

model.weight_h -= learning_rate * d_loss__d_w_h

For a single weight, $w_j$ , this corresponds to the following partial derivative-based update:

w_j:=w_j-\eta \frac{\partial L}{\partial w_j}

(1)

def train(model, X_train, y_train, X_valid, y_valid, num_epochs,
          learning_rate=0.1):
    
    epoch_loss = []
    epoch_train_acc = []
    epoch_valid_acc = []
    
    for e in range(num_epochs):

        # iterate over minibatches
        minibatch_gen = minibatch_generator(
            X_train, y_train, minibatch_size)

        for X_train_mini, y_train_mini in minibatch_gen:
            
            #### Compute outputs ####
            a_h, a_out = model.forward(X_train_mini)

            #### Compute gradients ####
            d_loss__d_w_out, d_loss__d_b_out, d_loss__d_w_h, d_loss__d_b_h = \
                model.backward(X_train_mini, a_h, a_out, y_train_mini)

            #### Update weights ####
            model.weight_h -= learning_rate * d_loss__d_w_h
            model.bias_h -= learning_rate * d_loss__d_b_h
            model.weight_out -= learning_rate * d_loss__d_w_out
            model.bias_out -= learning_rate * d_loss__d_b_out
        
        #### Epoch Logging ####        
        train_mse, train_acc = compute_mse_and_acc(model, X_train, y_train)
        valid_mse, valid_acc = compute_mse_and_acc(model, X_valid, y_valid)
        train_acc, valid_acc = train_acc*100, valid_acc*100
        epoch_train_acc.append(train_acc)
        epoch_valid_acc.append(valid_acc)
        epoch_loss.append(train_mse)
        print(f'Epoch: {e+1:03d}/{num_epochs:03d} '
              f'| Train MSE: {train_mse:.2f} '
              f'| Train Acc: {train_acc:.2f}% '
              f'| Valid Acc: {valid_acc:.2f}%')

    return epoch_loss, epoch_train_acc, epoch_valid_acc

Let’s now execute this function to train our model for 50 epochs.

Result explanation

The reason why we print all this output is that, in NN training, it is really useful to compare training and validation accuracy. This helps us judge whether the network model performs well, given the architecture and hyperparameters. For example, if we observe a low training and validation accuracy, there is likely an issue with the training dataset, or the hyperparameters’ settings are not ideal.

In general, training (deep) NNs is relatively expensive compared with the other models we’ve discussed so far. Thus, we want to stop it early in certain circumstances and start over with different hyperparameter settings. On the other hand, if we find that it increasingly tends to overfit the training data (noticeable by an increasing gap between training and validation dataset performance), we may want to stop the training early, as well.

np.random.seed(123) # for the training set shuffling

epoch_loss, epoch_train_acc, epoch_valid_acc = train(
    model, X_train, y_train, X_valid, y_valid,
    num_epochs=50, learning_rate=0.1)

11.2.4.Evaluating the neural network performance¶

Let’s look at the performance of the model that we trained in the previous subsection. In train(), we collected the training loss and the training and validation accuracy for each epoch so that we can visualize the results using Matplotlib. Let’s look at the training MSE loss first.

Result explanation

As we can see, the loss decreased substantially during the first 10 epochs and seems to slowly converge in the last 10 epochs. However, the small slope between epoch 40 and epoch 50 indicates that the loss would further decrease with training over additional epochs.

plt.plot(range(len(epoch_loss)), epoch_loss)
plt.ylabel('Mean squared error')
plt.xlabel('Epoch')
#plt.savefig('figures/11_07.png', dpi=300)
plt.show()

Next, let’s take a look at the training and validation accuracy.

Result explanation

The plot reveals that the gap between training and validation accuracy increases as we train for more epochs. At approximately the 25th epoch, the training and validation accuracy values are almost qual, and then, the network starts to slightly overfit the training data (where the two lines start to separate).

plt.plot(range(len(epoch_train_acc)), epoch_train_acc,
         label='Training')
plt.plot(range(len(epoch_valid_acc)), epoch_valid_acc,
         label='Validation')
plt.ylabel('Accuracy')
plt.xlabel('Epochs')
plt.legend(loc='lower right')
#plt.savefig('figures/11_08.png', dpi=300)
plt.show()

Finally, let’s evaluate the generalization performance of the model by calculating the prediction accuracy on the test dataset.

Result explanation

We can see that the test accuracy is very close to the validation set accuracy corresponding to the last epoch (94.74%). Moreover, the respective training accuracy is only minimally higher at 95.59%, reaffirming that our model only slightly overfits the training data.

To further fine-tune the model, we could change the number of hidden units, the learning rate, etc.

test_mse, test_acc = compute_mse_and_acc(model, X_test, y_test)
print(f'Test accuracy: {test_acc*100:.2f}%')

Lastly, let’s take a look at some of the images that our MLP struggles with by extracting and plotting the first 25 misclassified samples from the test set.

Result explanation

We should now see a 5×5 subplot matrix where the first number in the subtitles indicates the plot index, the second number represents the true class label (True), and the third number stands for the predicted class label (Predicted).

As we can see in the result figure, among others, the network finds $7 \mathrm{~s}$ challenging when they include a horizontal line as in examples 19 and 20. Looking back at an earlier figure in this chapter where we plotted different training examples of the number 7, we can hypothesize that the handwritten digit 7 with a horizontal line is underrepresented in our dataset and is often misclassified.

X_test_subset = X_test[:1000, :]
y_test_subset = y_test[:1000]

_, probas = model.forward(X_test_subset)
test_pred = np.argmax(probas, axis=1)

misclassified_images = X_test_subset[y_test_subset != test_pred][:25]
misclassified_labels = test_pred[y_test_subset != test_pred][:25]
correct_labels = y_test_subset[y_test_subset != test_pred][:25]

# Plot the result
fig, ax = plt.subplots(nrows=5, ncols=5, 
                       sharex=True, sharey=True, figsize=(8, 8))
ax = ax.flatten()
for i in range(25):
    img = misclassified_images[i].reshape(28, 28)
    ax[i].imshow(img, cmap='Greys', interpolation='nearest')
    ax[i].set_title(f'{i+1}) '
                    f'True: {correct_labels[i]}\n'
                    f' Predicted: {misclassified_labels[i]}')

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
#plt.savefig('figures/11_09.png', dpi=300)
plt.show()

11.3.Training an artificial neural network¶

11.3.1.Computing the loss function¶

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_10.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_10.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=300))

11.3.2.Developing your intuition for backpropagation¶

No code example.

11.3.3.Training neural networks via backpropagation¶

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_11.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_11.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=400))

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_12.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_12.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=500))

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_13.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_13.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=500))

11.4.Convergence in neural networks¶

import os, urllib.request
os.makedirs('figures', exist_ok=True)
fig_path = 'figures/11_14.png'
if not os.path.exists(fig_path):
    url = 'https://raw.githubusercontent.com/rasbt/machine-learning-book/main/ch11/figures/11_14.png'
    try:
        urllib.request.urlretrieve(url, fig_path)
    except Exception as e:
        print(f'Could not download figure: {e}')
if os.path.exists(fig_path):
    display(Image(filename=fig_path, width=500))

Readers may ignore the next cell.

# Optional utility script from the book's own repo (not included here) that
# converts this notebook to a .py file. Skipped safely when not present.
import os
script = '../.convert_notebook_to_script.py'
if os.path.exists(script):
    get_ipython().system('python {script} --input ch11-UWFLecture.ipynb --output ch11-UWFLecture.py')
else:
    print('Skipping: convert_notebook_to_script.py not present in this environment.')