Twitter Text Analysis: A Friendly Introduction, Part 2

The code for this blog post is available in this GitHub repository. You can also follow along in this Collab notebook!

Introduction

This blog post follows up on an introduction, part 1, in which I showed how to conduct exploratory data analysis of the Sentiment140 data. In this second part, we will dive into Twitter sentiment analysis. Starting with an introduction of Large Language Models, we will go through implementing word embeddings, splitting training and test sets, and finally building a classifier to identify positive and negative sentiments. If you are interested in further exploring NLP techniques, don’t hesitate to sign up for the Python Text Analysis workshop series with D-Lab.

BERT

Bidirectional Encoder Representation from Transformers (BERT) is one of the state-of-the-art pretrained Language Models. It was first trained on a massive amount of data which facilitates fine-tuning on later downstream tasks, such as sentiment analysis. That is to say, we don’t need to start from scratch but build upon an existing model and customize it for our own purposes. 

We will first install the transformers package provided by Hugging Face. This package is the standard way to use large language models in Python. Then, we load the tokenizer corresponding to the BERT “base” model, which is one of two variants of the BERT model (there is also a “large” model). 

Tokenization

Instead of passing all tweets (1.6 million!) into our model, which will take too long to train on, we will use only 1% of the data in this tutorial. It should be noted that the label for negative tweets is changed to “1”; without this change, it may trigger a bug during training.

Next, we iterate through the tweet column and tokenize each tweet into individual tokens. It should be noted that BERT does not simply convert a word into a token. Very often, a single word is tokenized into multiple parts; for instance, “surfing” is decomposed into the two tokens “surf” and “-ing”. See this blog post for a detailed introduction of BERT’s tokenizer. 

We can plot the distribution of tweet lengths. It seems that all tweets have a length lower than 125 tokens. We will then set 125 as MAX_LEN, a parameter that is required for training. 

Train-test split

Next, we will split our dataset into three subsets: the training data, test data, and validation data. Slicing 10% or 20% of the data for testing is most commonly used. Creating a validation dataset is necessary, as our classifier may perform exceptionally well on training but perform less satisfactory when evaluated on a separate dataset containing out-of-distribution tweets. 

Training

The final step before training is to transform the tokens into word embeddings and create the classifier. We won’t go deep into this part as we follow the standard procedure for training a machine learning model. Throughout the whole process, we have many training decisions to make, e.g. the size of the embedding, batch size, and etc. All decisions will influence the training time and, most importantly, the accuracy of classification. The code we use here is from this Kaggle post. Feel free to take a look at it! Run the “Build the classifier” code chunk to get everything done before training. 

We have a number of additional hyperparameters to specify for training including:

• Batch Size: A batch size of 16, 32, or 64 is generally recommended. You can think of a batch as passing a sample to our learning algorithm in search of the optimal solution; after each batch a loss will be calculated and our learning algorithm will be updated.

• Number of epochs: This is the number of times the entire dataset is passed to our classifier.  The number of epochs varies depending on the task. We don’t have a huge dataset here, so a small number of epochs should return a good optimization result. 

• Learning rate: Determines the step size our learning algorithm will move forward with. If it is too large, the algorithm might bounce back and forth without hitting the optimum. If it is too small, it may take too many steps until converging to the optimum.

All of these hyperparameters are not fixed but often adjusted to fit any given downstream task. 

Once everything’s settled, we can start training. Click on the “Training” code chunk and pour yourself a cup of coffee. It may take 40 mins to run! Our classifier will spit out both training and validation accuracy for each epoch. 

Take a look at the result. We can of course visualize the accuracy data, simply aligning them together on a line plot. The training accuracy steadily increases, exactly as we would expect, and eventually reaches almost 100% after eight epochs of passes. The validation accuracy is not bad either, but it remains at around 81% without any increase. On the other hand, the loss of validation gets larger. It might be the case that our learning algorithm is overfitting. Considering the fact that the task itself isn’t that complex, the validation accuracy is still good enough. Nevertheless, to have a better performance on validation, we can always tweak a few things:

• Increase the size of the dataset (we are merely using 1% of data).

• Tweak the learning rate: [5e-5, 4e-5, 3e-5, 2e-5, 1e-4] are commonly used learning rates for the Adam optimizer. 

• Tweak training settings, or adjust batch and epoch size. 

Feel free to play with these hyperparameters and see if they end up with a better result! This tutorial is simply going through several important steps of building a classifier, leaving many for you to explore on your own. Again, if you enjoy reading this tutorial and would like to learn more about either Text Analysis or Deep Learning, sign up for our D-Lab workshops!