A Tour of Machine Learning Classifiers Using Scikit-Learn¶

Package version checks¶

Add folder to path in order to load from the check_packages.py script:

Check recommended package versions:

Chapter 3 - A Tour of Machine Learning Classifiers Using Scikit-Learn¶

Overview¶

• Choosing a classification algorithm

• First steps with scikit-learn

  • Training a perceptron via scikit-learn

• Modeling class probabilities via logistic regression

  • Logistic regression intuition and conditional probabilities

  • Learning the weights of the logistic loss function

  • Training a logistic regression model with scikit-learn

  • Tackling overfitting via regularization

• Maximum margin classification with support vector machines

  • Maximum margin intuition

  • Dealing with the nonlinearly separable case using slack variables

  • Alternative implementations in scikit-learn

• Solving nonlinear problems using a kernel SVM

  • Using the kernel trick to find separating hyperplanes in higher dimensional space

• Decision tree learning

  • Maximizing information gain – getting the most bang for the buck

  • Building a decision tree

  • Combining weak to strong learners via random forests

• K-nearest neighbors – a lazy learning algorithm

• Summary

Choosing a classification algorithm¶

...

First steps with scikit-learn¶

Loading the Iris dataset from scikit-learn¶

The Iris dataset is a well-known dataset in the field of machine learning, and scikit-learn provides an easy way to load it.

Here, we’re loading the dataset and assigning the features (petal length and petal width) to the variable X, and the target (the type of Iris plant) to the variable y.

We’re also printing out the unique class labels for the target variable to confirm that they are as expected:

• 0: Iris-Setosa

• 1: Iris-Versicolor

• 2: Iris-Virginica

Loading the Iris dataset from scikit-learn. Here, the third column represents the petal length, and the fourth column the petal width of the flower examples. The classes are already converted to integer labels where 0=Iris-Setosa, 1=Iris-Versicolor, 2=Iris-Virginica.

The code below uses the train_test_split() function from scikit-learn’s model_selection module to split the Iris dataset into a training set (70% of the data) and a test set (30% of the data).

The first four input parameters X, y, test_size, and random_state are required. They are the feature matrix, the label vector, the size of the test set, and the seed for the random number generator, respectively.

The stratify parameter is used to ensure that the class labels are distributed in the same way across both the training and test sets. In this case, it is set to the label vector y to ensure that the class distribution in the test and training sets is the same as in the original dataset.

Splitting data into 70% training and 30% test data:

The np.bincount function returns the number of occurrences of each value in an array of non-negative integers. By passing in the y and y_train and y_test variables, we are able to see the number of samples that belong to each class in the original dataset and in the training and test sets, respectively. This is useful to check whether the train-test split preserves the class distribution from the original dataset or not.

This code snippet is using the StandardScaler class from the sklearn.preprocessing module to standardize the training and test feature sets, X_train and X_test.

• First, we create an object of the StandardScaler class, which will be used to standardize the features.

• Next, we use the fit method to compute the mean and standard deviation of X_train and store it in the sc object.

• The transform method is then used to standardize the training and test data using the mean and standard deviation that was computed from the training data.

• The result is stored in new variables X_train_std and X_test_std respectively, which contain the standardized feature sets.

Standardization is a common preprocessing step in machine learning and is used to scale the features so that they have zero mean and unit variance. This is important as many machine learning algorithms perform better or converge faster when the features are on a similar scale.

Standardizing the features:

Training a perceptron via scikit-learn¶

This code snippet is using the Perceptron class from the sklearn.linear_model module to train a perceptron model on the standardized training data, X_train_std and y_train.

First, we create an object of the Perceptron class, which will be used to train the perceptron model. The eta0 parameter is set to 0.1 which is the learning rate of the algorithm. And random_state is set to 1, which means the starting point of the random number generator is fixed, so that the results are reproducible. Next, we use the fit method to train the perceptron model on the standardized training data. The fit method learns the weights of the model that minimize the classification error on the training set. Once the fit method completes, the trained perceptron model can be used to make predictions on new samples.

This code snippet uses the predict method of the Perceptron class to predict the class labels of the samples in the standardized test dataset, X_test_std. The predicted class labels are stored in the variable y_pred.

The next line calculates the number of misclassifications by summing up the number of instances where the true class label, y_test, is not equal to the predicted class label, y_pred. The result is printed to the console.

This gives us an idea of the performance of the model on unseen data, which is important for evaluating the model’s generalization performance.

This code snippet is using the predict method from the Perceptron class to predict the class labels of the test set, X_test_std, and store the predictions in the variable y_pred. Then it is calculating the number of misclassifications by comparing the predicted class labels, y_pred, with the true class labels, y_test, using the not equal operator (!=) and taking the sum of the resulting Boolean array using the sum method.

Additionally, it is using the accuracy_score function from the sklearn.metrics module to compute the classification accuracy and prints it using the string formatting operator %. The accuracy score is the ratio of correctly predicted samples to the total number of test samples.

This code snippet is using the score method of the Perceptron class to evaluate the classification accuracy of the model on the standardized test feature set and true labels.

The score method returns the mean accuracy on the given test data and labels. In this case, the test data is X_test_std and the true labels are y_test. The mean accuracy is then printed using the print function and formatted to show 3 decimal places using the %.3f format specifier.

This function creates a plot of decision regions for a given classifier, with the option to highlight a test set of data. This function uses the matplotlib library to create the plot and customize its appearance, including setting the color map, markers, and labels. The function takes four parameters:

• X which is the feature set

• y the target variable

• classifier the classifier to be plotted

• test_idx indices of examples in the test set, which would be highlighted in the plot

• resolution the granularity of the grid for plotting the decision regions

It creates a scatter plot of the training set, with different markers and colors for each class. And also it creates a contourf plot of the decision regions of the classifier and if test_idx is provided it’ll also highlight the test set examples.

Training a perceptron model using the standardized training data:

This code is creating a plot of the decision regions for the perceptron model that has been trained on the standardized training data. It first creates a combined set of the standardized training and test data (X_combined_std) and the corresponding class labels (y_combined) by stacking the training and test data vertically and horizontally, respectively. Then it calls the plot_decision_regions function that we saw earlier, passing it the combined data, the trained perceptron model, and the indices of the test examples.

The plot_decision_regions function is then used to create a 2D scatter plot of the training and test examples, where the decision regions of the perceptron model are color-coded. The decision regions are created by training the perceptron model on a dense grid of points that covers the feature space of the data and then coloring each grid point according to the class label predicted by the model.

The plt.xlabel, plt.ylabel, plt.legend and plt.tight_layout() are used to add axis labels and a legend to the plot, and adjust the layout of the plot so that it fits nicely within the boundaries of the plotting area.

The last line plt.show() is used to display the plot on the screen.

Modeling class probabilities via logistic regression¶

...

Logistic regression intuition and conditional probabilities¶

This code is plotting the sigmoid function which is commonly used as an activation function in logistic regression and other neural network models.

The sigmoid function is defined as $\frac{1}{(1+e^{-z})}$ where $z$ is the input. The function maps any input value to a value between 0 and 1. It is often used as the output of a model to represent the probability of a certain class.

Here, the code first imports the necessary libraries, numpy and matplotlib, for numerical computations and plotting, respectively. Then it defines a function sigmoid(z) which takes in input z and returns the value of the sigmoid function evaluated at that point.

The code then creates an array z of values ranging from -7 to 7 with a step of 0.1 using the np.arange() function. It then applies the sigmoid function to each value in the array z and assigns the result to the variable sigma_z.

The code then plots the values of z against sigma_z using the plt.plot() function. It also adds a vertical line at the value of 0 on the x-axis using the plt.axvline() function. It then sets the limits for the y-axis to be between -0.1 and 1.1 using the plt.ylim() function. The x-axis and y-axis are labeled ‘z’ and ‘$\sigma (z)$’ respectively using plt.xlabel() and plt.ylabel().

It then sets the y-axis ticks to be at the values 0.0, 0.5, 1.0 using plt.yticks(). Finally, it shows the gridlines on the y-axis using ax.yaxis.grid(True) and tight the layout with plt.tight_layout()

The last line of the code with plt.show() is used to display the plot. The commented out line is used to save the plot as an image file.

Learning the weights of the logistic loss function¶

This code is creating a plot that illustrates the log-loss function for logistic regression. The log-loss function is used to measure the performance of the logistic regression model. It is a measure of how well the predicted probability estimates for a set of samples compare to the true class labels.

The code first defines two functions, loss_1 and loss_0, which calculate the log-loss for the cases when the true class label is 1 and 0, respectively. These functions make use of the sigmoid function, which is defined earlier in the code and takes an input z and returns the sigmoid of z.

The variable z is created using the numpy function arange which creates a range of values from -10 to 10 in increments of 0.1. The variable sigma_z is then created by passing z through the sigmoid function.

Then the code creates two lists, c1 and c0, by applying the loss_1 and loss_0 functions to each value in z respectively.

Then the code creates a plot with two lines, one for c1 and one for c0. The c1 line shows the log-loss for the case when the true class label is 1, and the c0 line shows the log-loss for the case when the true class label is 0. The line for c1 is plotted as solid line and the line for c0 is plotted as dashed line.

The code then sets the y-axis limit from 0.0 to 5.1 and the x-axis limit from 0 to 1. Then it sets x-axis label as ‘$\sigma(z)$’ and y-axis label as ‘L(w, b)’. Then it adds a legend to the plot, locates it at the best position and tightens the layout.

Finally, it shows the plot.

The LogisticRegressionGD class is an implementation of logistic regression using gradient descent. Logistic regression is a supervised learning algorithm that is used for classification problems. It is a linear model that is used to estimate the probability of a binary outcome (e.g. 0 or 1, True or False) based on one or more input features.

The class has three parameters: eta, n_iter, and random_state. eta is the learning rate, which controls the step size at which the algorithm makes updates to the model weights. n_iter is the number of passes over the training dataset, also known as epochs. random_state is a seed for the random number generator used to initialize the model weights.

The class has several methods:

• __init__(self, eta=0.01, n_iter=50, random_state=1): This is the constructor for the class, which sets the initial values for the parameters and initializes the attributes w_, b_, and losses_. w_ will store the model weights, b_ will store the bias term, and losses_ will store the loss function values in each epoch.

• fit(self, X, y): This method takes as input the training data (X) and the target values (y) and fits the model to the data. The method first initializes the model weights and bias term using a random number generator. Then, it performs the gradient descent algorithm for the number of iterations specified by n_iter. In each iteration, the method calculates the net input, the output of the model, the errors, the gradient of the loss function with respect to the weights and bias, and updates the weights and bias. Finally, it records the loss function value for that iteration.

• net_input(self, X): This method calculates the net input to the model by taking the dot product of the input data (X) and the model weights (self.w_), and adding the bias term (self.b_).

• activation(self, z): This method calculates the sigmoid activation function of the net input. The sigmoid function maps any input value to a value between 0 and 1, and is often used as the activation function in logistic regression.

• predict(self, X): This method takes as input new data (X) and returns the predicted class labels. It does this by calculating the model’s output using the sigmoid activation function and returning 1 if the output is greater than or equal to 0.5, and 0 otherwise.

In summary, this class is an implementation of logistic regression algorithm using gradient descent optimization. It starts by initializing the weights, bias and losses list, then in each iteration, it uses the net_input, activation and errors to calculate the gradient of the loss function with respect to the weights and bias and updates them. It also records the loss function value for that iteration. Finally, the predict method use the learned model to predict the class label of new input data based on the learned weights and bias

This code is creating a logistic regression model using gradient descent optimization, and then visualizing the decision boundaries produced by the model on a 2D plot of petal length and width of iris flowers that are labeled either 0 or 1.

The logistic regression model is created using the LogisticRegressionGD class, which is instantiated by passing in the learning rate eta, the number of passes over the training data n_iter, and a random seed random_state.

The model is then fit to a subset of the iris flower dataset, which only contains the examples with labels 0 or 1. The plot_decision_regions function is used to visualize the decision boundaries of the fitted logistic regression model on a scatter plot of the standardized petal length and width features.

Training a logistic regression model with scikit-learn¶

The following code imports the LogisticRegression class from the sklearn.linear_model module and creates an instance of the class named lr.

The model is trained on the standardized training data X_train_std and y_train using the fit method. The trained model is then used to create a plot of the decision regions on the combined standardized dataset X_combined_std and y_combined, where the test examples are highlighted by their indices test_idx.

The x-axis is labeled as ‘Petal length [standardized]’ and the y-axis is labeled as ‘Petal width [standardized]’. A legend is also added to the plot and the layout of the plot is made tight to fit all the elements properly. Finally, the plot is displayed using plt.show().

This line of code is using the predict_proba method of a Logistic Regression classifier object lr to make class predictions for the first 4 samples of the standardized test data X_test_std. The predict_proba method returns the predicted class probabilities for each sample, i.e., the estimated probability that the sample belongs to each of the possible classes. The result of the method will be a matrix of shape [4, n_classes], where n_classes is the number of classes in the target variable.

This line of code is computing the sum of the predicted class probabilities for each sample in the X_test_std[:3,:] array along the row axis (axis=1). The output will be an array with shape (3,) representing the total predicted probability of the samples being in any of the classes. This is useful when the sum of predicted probabilities is equal to one, which can indicate that the model’s prediction is well-calibrated.

The code lr.predict_proba(X_test_std[:3, :]).argmax(axis=1) returns the class label indices of the maximum predicted probability for the samples in X_test_std[:3, :]. This can be seen as a prediction of the class labels for the first three samples in the standardized test set. The argmax function returns the indices of the maximum values along the specified axis, which is set to axis=1 to return the indices of the maximum values for each sample.

The line lr.predict(X_test_std[:3, :]) uses the trained logistic regression model lr to predict the class labels of the first three samples in the standardized test dataset X_test_std. The predict method outputs the class label predictions as a 1D numpy array.

This line of code is using the trained Logistic Regression model lr to make a single prediction for the first sample of the standardized test data X_test_std[0, :]. The sample is first reshaped into a 2D array with a single row using the reshape method (reshape(1, -1)), as the predict method expects a 2D array as input. The resulting prediction is the class label that the model assigns to this sample based on the decision boundaries learned during the training phase.

The value -1 in the reshape method indicates that the size of that dimension should be inferred automatically based on the total size of the tensor and the size of the remaining dimensions.

For example, if you have a one-dimensional array a with shape (10,), you can use a.reshape(-1, 1) to convert it into a two-dimensional array with shape (10, 1). Similarly, you can use a.reshape(1, -1) to convert it into a two-dimensional array with shape (1, 10). The size of the first dimension is set to 1, and the size of the second dimension is inferred to be 10.

Tackling overfitting via regularization¶

Maximum margin classification with support vector machines¶

Maximum margin intuition¶

...

Dealing with the nonlinearly separable case using slack variables¶

This code imports the SVM (Support Vector Machine) classifier from scikit-learn library, and creates an instance of the classifier with a linear kernel, C=1.0, and random_state=1. The fit method trains the SVM on the standardized training data. Then, the function plot_decision_regions is used to visualize the decision boundary and how well the SVM classifier separates the classes in the combined dataset (training and test data). The axes of the plot are labeled as “Petal length [standardized]” and “Petal width [standardized]”, and the legend is located in the upper left corner. The plot is displayed using the show method.

Alternative implementations in scikit-learn¶

In this code, three classifiers are being created: a perceptron, a logistic regression, and a support vector machine (SVM). The classifiers are being created using the SGDClassifier class from the scikit-learn library, which is an implementation of stochastic gradient descent (SGD) for linear classification models. The loss parameter is used to specify the type of loss function that the classifier should use. For the perceptron, the 'perceptron' loss function is specified, which is the standard loss function for the perceptron algorithm. For logistic regression, the 'log' loss function is specified, which corresponds to the logistic loss function. Finally, for the SVM, the 'hinge' loss function is specified, which is the standard loss function for the hinge loss SVM algorithm.

Solving non-linear problems using a kernel SVM¶

This code creates a scatter plot that visualizes a synthetic dataset, generated using the numpy library, where the samples belong to two classes. The samples are generated using the np.random.randn method and are stored in the X_xor numpy array. The class labels are assigned based on the logical XOR of the two features of each sample, stored in the y_xor numpy array. The scatter plot visualizes the samples from class 1 as square markers and the samples from class 0 as circle markers. The plot limits are set to [-3, 3] for both the x- and y-axis, and the x- and y-axis are labeled as “Feature 1” and “Feature 2”, respectively. The plot includes a legend to distinguish the two classes.

Using the kernel trick to find separating hyperplanes in higher dimensional space¶

This code trains a radial basis function (RBF) support vector machine (SVM) model using the scikit-learn library. The SVC class is imported from the sklearn.svm module, and an instance of the class is created with the kernel parameter set to 'rbf' to specify that the RBF kernel should be used. The random_state parameter is set to 1 to ensure that the same random seed is used every time the code is run, which makes the results reproducible. The gamma parameter controls the width of the Gaussian RBF kernel and is set to 0.1, and the C parameter controls the regularization strength and is set to 10.0. The fit method is then called to train the model using the training data, X_xor, and the corresponding class labels, y_xor. Finally, the trained model is used to make predictions on the data and visualize the decision boundary using the plot_decision_regions function. The legend function is used to add a legend to the plot, and the tight_layout function is used to optimize the layout of the plot. The show function is called to display the plot.

The code imports the SVC (Support Vector Classification) class from the scikit-learn library, creates an instance of the class, and trains the model on the training data (X_train_std and y_train). The kernel argument is set to 'rbf' which stands for radial basis function, a common choice for non-linear problems. The gamma and C arguments set the width of the Gaussian radial basis function (gamma) and the regularization parameter (C), respectively.

The fit method trains the SVM model on the training data. The plot_decision_regions function is then used to visualize the decision boundaries of the trained model. The X_combined_std and y_combined input arguments represent the feature and target variables of the entire dataset (both training and test data), while the test_idx argument indicates which samples belong to the test set.

Finally, the plot is labeled with x and y-axis labels and a legend, and is shown to the user.

This code is creating another SVM classifier using a radial basis function (RBF) kernel, with a different value of the gamma parameter. The gamma parameter is used to control the shape of the Gaussian RBF kernel. A large gamma value creates a very narrow Gaussian that can fit tightly around individual data points, while a small gamma value creates a wider Gaussian that may not fit the data as well, but is more robust to noise.

In this example, we have set the value of gamma to 100.0, which is significantly larger than the value of 0.2 used in the previous code. This results in a much more complex model, which is likely overfitting the training data, and may not generalize well to unseen data.

The decision boundary for this new classifier is then plotted and displayed using the plot_decision_regions function.

Decision tree learning¶

Maximizing information gain - getting the most bang for the buck¶

Building a decision tree¶

Combining weak to strong learners via random forests¶

K-nearest neighbors - a lazy learning algorithm¶

Summary¶

...

Readers may ignore the next cell.