Chapter 8 - Applying Machine Learning To Sentiment Analysis¶

Prework¶

Import essentials¶

Check package version (optional)¶

Add folder to path in order to load from the check_packages.py script.

Check recommended package versions.

Watermark (optional)¶

Optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible.

You can install watermark Jupyter extension via

conda install watermark -c conda-forge  

or

pip install watermark   

For more information, please see: https://github.com/rasbt/watermark.

Table of Contents¶

• 8.1.Preparing the IMDb movie review data for text processing

  • 8.1.1.Obtaining the IMDb movie review dataset

  • 8.1.2.Preprocessing the movie dataset into more convenient format

• 8.2.Introducing the bag-of-words model

  • 8.2.1.Transforming words into feature vectors

  • 8.2.2.Assessing word relevancy via term frequency-inverse document frequency

  • 8.2.3.Cleaning text data

  • 8.2.4.Processing documents into tokens

• 8.3.Training a logistic regression model for document classification

• 8.4.Working with bigger data – online algorithms and out-of-core learning

• 8.5.Topic modeling

  • 8.5.1.Decomposing text documents with Latent Dirichlet Allocation

  • 8.5.2.Latent Dirichlet Allocation with scikit-learn

8.1.Preparing the IMDb movie review data for text processing¶

We are going to work with a dataset of 50,000 movie reviews from the Internet Movie Database (IMDb) and build a predictor that can distinguish between positive and negative reviews.

8.1.1.Obtaining the IMDb movie review dataset¶

The IMDB movie review set can be downloaded from http://ai.stanford.edu/~amaas/data/sentiment/. After downloading the dataset, decompress the files.

• If you are working with Linux or MacOS X, open a new terminal window, cd into the download directory and execute: tar -zxf aclImdb_v1.tar.gz

• If you are working with Windows, download an archiver such as 7Zip to extract the files from the download archive.

8.1.2.Preprocessing the movie dataset into more convenient format¶

In the following section, we will be reading the movie reviews into a pandas DataFrame object, which can take up to 10 minutes on a standard desktop computer.

To visualize the progress and estimated time until completion, we will use the Python Progress Indicator (PyPrind, https://pypi.python.org/pypi/PyPrind/) package.

We first initialized a new progress bar object, pbar, with 50,000 iterations, which was the number of documents we were going to read in. Using the nested for loops, we iterated over the train and test subdirectories in the main aclImdb directory and read the individual text files from the pos and neg subdirectories that we eventually appended to the df pandas DataFrame, together with an integer class label (1 = positive and 0 = negative).

Since the class labels in the assembled dataset are sorted, we will now shuffle the DataFrame using the permutation function from the np.random submodule—this will be useful for splitting the dataset into training and test datasets in later sections, when we will stream the data from our local drive directly.

Optional: Saving the assembled data as CSV file for convenience:

Quickly confirm that we have successfully saved the data in the right format by reading in the CSV and printing an excerpt of the first three examples:

Note¶

If you have problems with creating the movie_data.csv, you can find a download a zip archive at https://github.com/rasbt/machine-learning-book/tree/main/ch08/

8.2.Introducing the bag-of-words model¶

• Bag-of-words model: allows us to represent text as numerical feature vectors.

• The idea behind bag-of-words:

  1. We create a vocabulary of unique tokens

     • Ex: words, from the entire set of documents, as tokens

  2. We construct a feature vector from each document that contains the counts of how often each word occurs in the particular document.

Since the unique words in each document represent only a small subset of all the words in the bag-of-words vocabulary, the feature vectors will mostly consist of zeros, which is why we call them sparse.

8.2.1.Transforming documents into feature vectors¶

To construct a bag-of-words model based on the word counts in the respective documents, we can use the CountVectorizer class implemented in scikit-learn.

• CountVectorizer: takes an array of text data and constructs the bag-of-words model for us.

  • text data can be documents or sentences

By calling the fit_transform method on CountVectorizer, we constructed the vocabulary of the bag-of-words model and transformed the following three sentences into sparse feature vectors:

• 'The sun is shining'

• 'The weather is sweet'

• 'The sun is shining, the weather is sweet, and one and one is two'

Result explanation

As you can see from executing the preceding command, the vocabulary is stored in a Python dictionary that maps the unique words to integer indices.

• For example, 'the' is at the index position 6; 'sun' is at the index position 4.

Next, let’s print the feature vectors that we just created.

Result explanation

Each index position in the feature vectors shown here corresponds to the integer values that are stored as dictionary items in the CountVectorizer vocabulary.

• For example, 'and' is at the index position 0, which means the first column of the array represents the count of 'and'. And the first column [0 0 2] means 'and' occurs 0 times in the first document; occurs 0 times in the second document; and occurs 2 times in the third document.

• values in the feature vector = raw term frequencies = $t f(t, d)$: the number of times a term, $t$, occurs in a document, $d$.

Note that, in the bag-of-words model, the word or term order in a sentence or document does not matter. The order in which the term frequencies appear in the feature vector is derived from the vocabulary indices, which are usually assigned alphabetically.

N-gram models

The choice of the number, n, in the n-gram model depends on the particular application.

For example, n-grams of size 3 and 4 yield good performances in the anti-spam filtering of email messages.

Concept of n-gram representation,:

• 1-gram: “the”, “sun”, “is”, “shining”

• 2-gram: “the sun”, “sun is”, “is shining”

• CountVectorizer class in scikit-learn uses 1-gram by default.

8.2.2.Assessing word relevancy via term frequency-inverse document frequency¶

When we are analyzing text data, we often encounter words that occur across multiple documents from both classes. These frequently occurring words typically don’t contain useful or discriminatory information.

• term frequency-inverse document frequency (tf-idf)

  • Downweigh these frequently occurring words in the feature vectors.

  • Defined as the product of the term frequency and the inverse document frequency:

    $$t f-i d f(t, d)=t f(t, d) \times i d f(t, d)$$

    (1)

  • $t f(t, d)$ = term frequency

  • $idf(t, d)$ = inverse document frequency:

    $$idf(t, d)=\log \frac{n_d}{1+d f(d, t)}$$

    (2)

  • $n_d$ : total number of documents

  • $d f(d, t)$ : number of documents, $d$, that contain the term $t$

  • Adding the constant 1 to the denominator (optional): serves the purpose of assigning a non-zero value to terms that occur in none of the training examples.

  • The $log$: to ensure that low document frequencies are not given too much weight.

Code implementation of tf-idf: TfidfTransformer¶

The scikit-learn library implements yet another transformer, the TfidfTransformer class, which takes the raw term frequencies from the CountVectorizer class as input and transforms them into tf-idfs.

Result explanation

As you saw in the previous subsection, the word 'is' (the second column of the array) had the largest term frequency in the third document, being the most frequently occurring word. However, after transforming the same feature vector into tf-idfs, the word 'is' is now associated with a relatively small tf-idf (0.45) in the third document, since it is also present in the first and second document and thus is unlikely to contain any useful discriminatory information.

Math of TfidfTransformer¶

However, if we’d manually calculated the tf-idfs of the individual terms in our feature vectors, we would have noticed that TfidfTransformer calculates the tf-idfs slightly differently compared to the standard textbook equations that we defined previously.

The equation for the inverse document frequency implemented in scikit-learn is computed as follows:

Similarly, the tf-idf computed in scikit-learn deviates slightly from the default equation we defined earlier:

Note that the " +1 " in the previous idf equation is due to setting smooth_idf=True in the previous code example, which is helpful for assigning zero weight (that is, $i d f(t, d)=\log (1)=0$ ) to terms that occur in all documents.

While it is also more typical to normalize the raw term frequencies before calculating the tf-idfs, the TfidfTransformer class normalizes the tf-idfs directly. By default (norm= '12'), scikit-learn’s TfidfTransformer applies the L2-normalization, which returns a vector of length 1 by dividing an unnormalized feature vector, $v$, by its L2-norm:

Math example of TfidfTransformer¶

To make sure that we understand how TfidfTransformer works, let’s walk through an example and calculate the tf-idf of the word 'is' in the third document. The word 'is' has a term frequency of 3 $(t f=3)$ in the third document, and the document frequency of this term is 3 since the term 'is' occurs in all three documents $(d f=3)$. Thus, we can calculate the inverse document frequency as follows:

Now, in order to calculate the tf-idf, we simply need to add 1 to the inverse document frequency and multiply it by the term frequency:

If we repeated this calculation for all terms in the third document, we’d obtain the following tf-idf vectors: [3.39,3.0,3.39,1.29,1.29,1.29,2.0,1.69,1.29]. However, notice that the values in this feature vector are different from the values that we obtained from TfidfTransformer that we used previously.

The final step that we are missing in this tf-idf calculation is the L2-normalization, which can be applied as follows:

8.2.3.Cleaning text data¶

Before we build our bag-of-words model, it is to clean the text data by stripping it of all unwanted characters. To illustrate why this is important, let’s display the last 50 characters from the first document in the reshuffled movie review dataset.

Result explanation

As you can see here, the text contains HTML markup as well as punctuation and other non-letter characters. While HTML markup does not contain many useful semantics, punctuation marks can represent useful, additional information in certain NLP contexts. However, for simplicity, we will now remove all punctuation marks except for emoticon characters, such as :), since those are certainly useful for sentiment analysis.

To accomplish this task, we will use Python’s regular expression (regex) library, re.

Via the first regex, <[^>]*>, in the preceding code section, we tried to remove all of the HTML markup from the movie reviews. Although many programmers generally advise against the use of regex to parse HTML, this regex should be sufficient to clean this particular dataset. Since we are only interested in removing HTML markup and do not plan to use the HTML markup further, using regex to do the job should be acceptable. However, if you prefer to use sophisticated tools for removing HTML markup from text, you can take a look at Python’s HTML parser module, which is described at https://docs.python.org/3/library/html.parser.html.

After we removed the HTML markup, we used a slightly more complex regex to find emoticons, which we temporarily stored as emoticons. Next, we removed all non-word characters from the text via the regex [\W]+ and converted the text into lowercase characters. Eventually, we added the temporarily stored emoticons to the end of the processed document string. Additionally, we removed the nose character (- in :-)) from the emoticons for consistency.

Although the addition of the emoticon characters to the end of the cleaned document strings may not look like the most elegant approach, we must note that the order of the words doesn’t matter in our bag-of-words model if our vocabulary consists of only one-word tokens. But before we talk more about the splitting of documents into individual terms, words, or tokens, let’s confirm that our preprocessor function works correctly.

Lastly, since we will make use of the cleaned text data over and over again during the next sections, let’s now apply our preprocessor function to all the movie reviews in our DataFrame.

8.2.4.Processing documents into tokens¶

After successfully preparing the movie review dataset, we now need to think about how to split the text corpora into individual elements. $\rightarrow$ tokenizing documents

Two ways to do the tokenization

One way to tokenize documents is to split them into individual words by splitting the cleaned documents at their whitespace characters.

In the context of tokenization, another useful technique is word stemming, which is the process of transforming a word into its root form. It allows us to map related words to the same stem. The Natural Language Toolkit (NLTK, http://www.nltk.org) for Python implements the Porter stemming algorithm, which we will use in the following code section. To install the NLTK, you can simply execute conda install nltk or pip install nltk.

Split the documents into individual words by splitting the cleaned documents at their whitespace characters.

Using the PorterStemmer from the nltk package, we modified our tokenizer function to reduce words to their root form, which was illustrated by the simple preceding example where the word 'running' was stemmed to its root form 'run'.

Result explanation

While stemming can create non-real words, such as 'thu' (from 'thus'), as shown in the previous example, a technique called lemmatization aims to obtain the canonical (grammatically correct) forms of individual words—the so-called lemmas. However, lemmatization is computationally more difficult and expensive compared to stemming and, in practice, it has been observed that stemming and lemmatization have little impact on the performance of text classification.

Before we jump into training a machine learning model using the bag-of-words model, let’s briefly talk about another useful topic called stop word removal. Stop words are simply those words that are extremely common in all sorts of texts and probably bear no (or only a little) useful information that can be used to distinguish between different classes of documents.

• Examples of stop words are is, and, has, and like.

Removing stop words can be useful if we are working with raw or normalized term frequencies rather than tf-idfs, which already downweight the frequently occurring words.

To remove stop words from the movie reviews, we will use the set of 127 English stop words that is available from the NLTK library, which can be obtained by calling the nltk.download function.

After we download the stop words set, we can load and apply the English stop word set.

8.3.Training a logistic regression model for document classification¶

In this section, we will train a logistic regression model to classify the movie reviews into positive and negative reviews based on the bag-of-words model. First, we will divide the DataFrame of cleaned text documents into 25,000 documents for training and 25,000 documents for testing.

Next, we will use a GridSearchCV object to find the optimal set of parameters for our logistic regression model using 5-fold stratified cross-validation. The following bullet list is the details of the following code.

• TfidfVectorizer

  • CountVectorizer + TfidfTransformer (introduced in previous section)

• param_grid consisted of two parameter dictionaries:

  • 1st dictionary: TfidfVectorizer with its default settings (use_idf=True, smooth_idf=True, and norm= '12' ) to calculate the tf-idfs.

  • 2nd dictionary: set those parameters to use_idf=False, smooth_idf=False, and norm=None in order to train a model based on raw term frequencies.

• LogisticRegression (classifier): uses the LIBLINEAR solver as it can perform better than the default choice ('lbfgs') for relatively large datasets.

• GridSearchCV: we restricted ourselves to a limited number of parameter combinations, since the number of feature vectors, as well as the large vocabulary, can make the grid search computationally quite expensive. (Using a standard desktop computer, our grid search may take 5-10 minutes to complete.)

  • cv=5 : 5-fold stratified cross-validation

  • n_jobs=-1: highly recommend setting, to utilize all available cores on your machine and speed up the grid search

After the grid search has finished, we can print the best parameter set.

Results explanation

As you can see in the preceding output, we obtained the best grid search results using the regular tokenizer without Porter stemming, no stop word library, and tf-idfs in combination with a logistic regression classifier that uses L2-regularization with the regularization strength $\mathrm{C}$ of 10.0.

Using the best model from this grid search, let’s print the average 5-fold cross-validation accuracy scores on the training dataset and the classification accuracy on the test dataset.

Results explanation

The results reveal that our machine learning model can predict whether a movie review is positive or negative with 90 percent accuracy.

Start comment:¶

Please note that gs_lr_tfidf.best_score_ is the average k-fold cross-validation score. I.e., if we have a GridSearchCV object with 5-fold cross-validation (like the one above), the best_score_ attribute returns the average score over the 5-folds of the best model. To illustrate this with an example:

By executing the code above, we created a simple data set of random integers that shall represent our class labels. Next, we fed the indices of 5 cross-validation folds (cv3_idx) to the cross_val_score scorer, which returned 5 accuracy scores -- these are the 5 accuracy values for the 5 test folds.

Next, let us use the GridSearchCV object and feed it the same 5 cross-validation sets (via the pre-generated cv3_idx indices):

As we can see, the scores for the 5 folds are exactly the same as the ones from cross_val_score earlier.

Now, the best_score_ attribute of the GridSearchCV object, which becomes available after fitting, returns the average accuracy score of the best model:

As we can see, the result above is consistent with the average score computed with cross_val_score.

End comment.¶

8.4.Working with bigger data - online algorithms and out-of-core learning¶

If you executed the code examples in the previous section, you may have noticed that it could be computationally quite expensive to construct the feature vectors for the 50,000-movie review dataset during a grid search. In many real-world applications, it is not uncommon to work with even larger datasets that can exceed our computer’s memory. In order to deal with large dataset, we will now apply a technique called out-of-core learning, which allows us to work with such large datasets by fitting the classifier incrementally on smaller batches of a dataset.

Stochastic gradient descent (SGD) is an optimization algorithm that updates the model’s weights using one example at a time. In this section, we will make use of the partial_fit function of SGDClassifier in scikit-learn to stream the documents directly from our local drive and train a logistic regression model using small mini-batches of documents.

First, we will define a tokenizer function that cleans the unprocessed text data from the movie_data.csv file that we constructed at the beginning of this chapter and separates it into word tokens while removing stop words.

Next, we will define a generator function, stream_docs, that reads in and returns one document at a time.

To verify that our stream_docs function works correctly, let’s read in the first document from the movie_data.csv file, which should return:

• a tuple consisting of the review text ("In 1974, the teenager ...")

• the corresponding class label (1)

We will now define a function, get_minibatch, that will take a document stream from the stream_docs function (introduced in the later code block) and return a particular number of documents specified by the size parameter.

Unfortunately, we can’t use CountVectorizer for out-of-core learning since it requires holding the complete vocabulary in memory. Also, TfidfVectorizer needs to keep all the feature vectors of the training dataset in memory to calculate the inverse document frequencies. However, another useful vectorizer for text processing implemented in scikit-learn is HashingVectorizer. HashingVectorizer is data-independent and makes use of the hashing trick via the 32-bit MurmurHash3 function by Austin Appleby (you can find more information about MurmurHash at MurmurHash).

We initialized HashingVectorizer with our tokenizer function and set the number of features to 2**21. Furthermore, we reinitialized a logistic regression classifier by setting the loss parameter of SGDClassifier to 'log'. Note that by choosing a large number of features in HashingVectorizer, we reduce the chance of causing hash collisions, but we also increase the number of coefficients in our logistic regression model.

Now comes the really interesting part—having set up all the complementary functions, we can start the out-of-core learning using the following code.

Again, we made use of the PyPrind package to estimate the progress of our learning algorithm. We initialized the progress bar object with 45 iterations and, in the following for loop, we iterated over 45 mini-batches of documents where each mini-batch consists of 1,000 documents.

Having completed the incremental learning process, we will use the last 5,000 documents to evaluate the performance of our model.

Results explanation

As you can see, the accuracy of the model is approximately 87 percent, slightly below the accuracy that we achieved in the previous section using the grid search for hyperparameter tuning. However, out-of-core learning is very memory efficient, and it took less than a minute to complete.

Finally, we can use the last 5,000 documents to update our model.

8.5.Topic modeling¶

Topic modeling describes the broad task of assigning topics to unlabeled text documents.

• For example, a typical application is the categorization of documents in a large text corpus of newspaper articles. In applications of topic modeling, we then aim to assign category labels to those articles, for example, sports, finance, world news, politics, and local news.

Thus, in the context of the broad categories of machine learning, we can consider topic modeling as a clustering task, a subcategory of unsupervised learning.

In this section, we will discuss a popular technique for topic modeling called latent Dirichlet allocation (LDA). However, note that while latent Dirichlet allocation is often abbreviated as LDA, it is not to be confused with linear discriminant analysis, a supervised dimensionality reduction technique.

8.5.1.Decomposing text documents with Latent Dirichlet Allocation¶

Since the mathematics behind LDA is quite involved and requires knowledge of Bayesian inference, we will approach this topic from a practitioner’s perspective and interpret LDA using layman’s terms.

LDA is a generative probabilistic model that tries to find groups of words that appear frequently together across different documents. These frequently appearing words represent our topics, assuming that each document is a mixture of different words. The input to an LDA is the bag-of-words model that we discussed earlier in this chapter.

Given a bag-of-words matrix as input, LDA decomposes it into two new matrices:

• A document-to-topic matrix

• A word-to-topic matrix

$\rightarrow$ optimal bag-of-words matrix (with lowest possible error) = document-to-topic matrix $\times$ word-to-topic matrix

In practice, we are interested in those topics that LDA found in the bag-of-words matrix. The only downside may be that we must define the number of topics beforehand—the number of topics is a hyperparameter of LDA that has to be specified manually.

8.5.2.Latent Dirichlet Allocation with scikit-learn¶

In this subsection, we will use the LatentDirichletAllocation class implemented in scikit-learn to decompose the movie review dataset and categorize it into different topics. In the following example, we will restrict the analysis to 10 different topics, but readers are encouraged to experiment with the hyperparameters of the algorithm to further explore the topics that can be found in this dataset.

First, we are going to load the dataset into a pandas DataFrame using the local movie_data.csv file of the movie reviews that we created at the beginning of this chapter.

Next, we are going to use the already familiar CountVectorizer to create the bag-of-words matrix as input to the LDA.

• top_words='english'

  • Use scikit-learn’s built-in English stop word library for convenience.

• max_df=.1

  • Set the maximum document frequency of words to be considered to 10% to exclude words that occur too frequently across documents.

  • This makes it less likely to be associated with a specific topic category of a given document.

• max_features=5000

  • Limit the number of words to be considered to the most frequently occurring 5,000 words, which can limit the dimensionality of this dataset to improve the inference performed by LDA.

However, both max_d f=.1 and max_features=5000 are hyperparameter values chosen arbitrarily, and readers are encouraged to tune them while comparing the results.

The following code example demonstrates how to fit a LatentDirichletAllocation estimator to the bag-of-words matrix and infer the 10 different topics from the documents (n_components=10).

By setting learning_method='batch', we let the lda estimator do its estimation based on all available training data (the bag-of-words matrix) in one iteration, which is slower than the alternative 'online' learning method, but can lead to more accurate results (setting learning_method='online' is analogous to online or mini-batch learning).

After fitting the LDA, we now have access to the components_ attribute of the lda instance, which stores a matrix containing the word importance (10, 5000) for each of the 10 topics in increasing order.

To analyze the results, let’s print the five most important words for each of the 10 topics. Note that the word importance values are ranked in increasing order. Thus, to print the top five words, we need to sort the topic array in reverse order.

Based on reading the five most important words for each topic, you may guess that the LDA identified the following topics:

1. Generally bad movies (not really a topic category)

2. Movies about families

3. War movies

4. Art movies

5. Crime movies

6. Horror movies

7. Comedy movie reviews

8. Movies somehow related to TV shows

9. Movies based on books

10. Action movies

To confirm that the categories make sense based on the reviews, let’s plot three movies from the horror movie category (horror movies belong to category 6 at index position 5).

Results explanation

We printed the first 300 characters from the top three horror movies. The reviews—even though we don’t know which exact movie they belong to—sound like reviews of horror movies (however, one might argue that Horror movie #2 could also be a good fit for topic category 1: Generally bad movies).

Readers may ignore the next cell.