Forecasting Social Outcomes with Deep Neural Networks

Forecasting Social Outcomes with Deep Neural Networks

Deep learning has captured the world’s attention. It is the engine behind many of the last decade’s transformative advances, including Large Language Models (LLMs), biological prediction with AlphaFold, autonomous vehicles, and generative imaging. These developments have understandably resulted in massive interest and investment in deep learning based artificial intelligence (AI), including by those who may not understand the underlying technology or may overestimate its capability. While some of this excitement is uninformed “AI hype,” the mathematical properties of neural networks combined with their demonstrated success across domains indicate that there is something to this excitement, especially for those of us who use data to solve problems and understand the world.

Many social scientists, however, remain skeptical. As a result, computational social science training often reinforces a deep learning gap, instead focusing on tools like decision trees and penalized regression models, which have a stronger track record in the field. However, neglecting deep learning, which has seen wider success across more domains, may limit the creative potential of a new generation of social scientists. 

In this blog post, I provide a tutorial for social scientists curious about, intimidated by, and even skeptical of deep learning. I will walk you through a successful application of a deep neural network, a multi-layer perceptron (MLP), to a social science prediction task: forecasting population-level mortality rates.

Before you begin

This tutorial will use Python, specifically the Keras framework within the TensorFlow package. Neural networks can be trained in R, but the infrastructure is much less developed. If you are new to Python, this book by Jean Mark Gawron, Python for Social Science, can help with installation and other preliminaries [1]. TensorFlow also has several useful learning resources [2]. If you are new to deep neural networks and want to learn more, there are many very helpful resources out there, but I recommend Deep Learning with Python by Francois Chollet as an introduction [3]. 

Data and code for this tutorial are available at this GitHub repo. This data is provided for demonstration purposes only and should not be used for any secondary research. If you are interested in continuing to work with the data, please download directly from the Human Mortality Database as per their data sharing policy [4]. 

The training data has around 360,000 rows and 5 columns. The test data has around 79,000 rows and 5 columns. The first four columns are the features we’ll use to predict mortality and, in order, contain information about country, gender, year, and age. The final column contains mortality rates. Country and gender are represented with integers, but corresponding country codes can be recovered from the geos_key.npy file in the Drive folder. For gender, 0 = female and 1 = male.

Setting up training infrastructure

To start, you’ll want to install and import the following packages.

Additionally, you’ll want to rename the module you’ll use to create model layers to something less verbose.

Data preparation functions

The data you downloaded is already “prepared” in the sense that it is split into train and test sets, but it is not yet ready to be passed to the neural network. We’ll create two functions that will finalize preparations. 

The first function, get_data(), will select a row from the data, normalize or transform fields, and return the transformed features (or covariates, predictors) and target (or outcome, dependent variable). 

The second, called prep_data(), will convert the data to an object the model can handle, called a “tensor,” and repeatedly yield batches of 256 rows in the data. These two functions will create a prefetch dataset object, which will not actually carry out the sampling and preparation until the model is training. 

Model training functions

The network represented below has five intermediate or “hidden” layers, each with 128 neurons. Deep learning models have many hyperparameters, settings or configuration choices that control how the model is trained, that can be adjusted and tuned to improve model performance. In other words, modelers often have many decisions to make, although there are rules of thumb and best practices that can make some of these choices easier [3]. The code below shows one way of setting up our model, including some deep learning best practices, but you can also alter these decisions and see how performance is affected. Some of the key model elements/decisions to be aware of are:

• Setting up categorical and continuous variables

• year - a float32 numeric input

• age_embed, gender_embed, and geography_embed - our categorical features (we choose to treat age as categorical), whose values are converted to 5D vectors

• The transitions and transformations between layers

• We choose ‘tanh’ activation functions. Another common activation function is ‘relu’. These allow deep learning models to capture non-linear dynamics.

• BatchNormalization() layers adjust layer outputs back to reasonable values for the model.

• Dropout() layers prevent the model from fitting too closely to training data, helping it to generalize to unseen data

• The optimization specifications

• We choose the mean-squared error as the loss value to be optimized.

• We choose Adam as the optimizer (a common choice for many models).

For the sake of this tutorial, don’t worry too much about the details. Instead, I would focus on getting familiar with how networks are broadly set up in Keras and the general types of decisions you’ll need to make.

Finally, we set up a function that will execute our training process. We are running the create_model() function, which compiles our model and prepares it to take in batches of data. The model.fit() function takes in both our prepared training data and our test data. This allows us to get information during training about how our model is fitting on our training data, but also how it’s fitting on unseen data. This can clue us in to potential overfitting if our reported training MSE is much lower than our test or validation MSE. The verbose=2 argument in model.fit() allows us to watch the training process.

Training the model

Train models and observe progress

Now that we have the backbone of our code written, we get to do the fun part: watch our model learn in real time! We’ll start by loading in and preparing our data. 

And we run our model.

What you should see printed as the model trains is an evaluation of performance after each of our 20 epochs. An epoch represents one complete pass through all of the data, where the optimization algorithm, called backpropagation, is carried out on each batch of 256 rows until all of the data has been sampled and the model parameters are updated accordingly. In other words, after one epoch, the model will have seen all of the data and adjusted its parameters in response [5].

After a few seconds, we will see the results from the first epoch. This includes information about the training loss (mean-squared error), the validation loss (mean-squared error on test data), and the learning rate, which affects the size of adjustments the model makes to its parameters on each optimization step. We want to pay attention to the trajectory of loss values as the model does its 19 other passes through the data. Ideally, both the training and validation loss values will decrease with each epoch. Your output should look something like the following.

This training trajectory is a little noisy–loss values do not descend every epoch–but that’s pretty normal. Overall, the trend looks strong enough that this variation isn’t concerning. There’s also little evidence of overfitting. Stopping a bit earlier might have been beneficial, since improvements taper off toward the end of training. Still, the progress overall looks excellent.

Plot results

Let’s now plot the predictions from our model against actual mortality rates for one of the countries in our dataset. We’ll start by quickly preparing input features to be fed into the model for prediction. 

Then, let’s filter to one country, one gender, and one year and plot mortality rate predictions across age. We’ll plot for Australian males in 2015.

The results look pretty good – especially considering we are nine years out from the training data. The forecasted rates are pretty close to the observed rates in 2015, though they do struggle more to predict higher rates at old ages.

Conclusion

In this tutorial, we prepared input data for a deep neural network, set up the network, and trained a model to forecast age-specific mortality rates for many populations. We learned how to interpret the outputs of the training process and observed, both with descending error values during training and our visualization, that the model successfully learned to predict unseen mortality data. 

Aggregated mortality rates are unique social outcomes in that they tend to be easier to predict than noisier outcomes like migration or unemployment rates. We might expect a deep learning model to do well since there are much simpler time series models that also do a decent job of predicting mortality [6]. 

However, deep neural networks also have the capacity to model more complicated social processes and better predict outcomes than traditional statistical models for tasks beyond mortality. There is emerging research using deep learning and AI to both generate information-rich model inputs and to directly predict complex social behaviors, though it seems that much of this work is happening outside of social science departments [7,8]. This tutorial aimed to introduce social scientists to the process of training a deep neural network, with the hope that greater familiarity with deep learning, AI, and machine learning will empower us to contribute meaningfully to the changing conversation around social prediction.

References

1. Python for Social Science

2. TensorFlow

3. Deep Learning with Python

4. Human Mortality Database

5. In this case, it actually is not entirely true that the model will see ALL of the samples in each epoch since we are sampling the data with replacement, but the vast, vast majority of the data will be seen by the model. 

6. Though it is also surprising that deep networks do better than classic approaches for this problem given that the mortality time series are relatively short and the complexity here arises from interactions between only four features. For more on why the deep learning approach does better for this problem, check out my preprint

7. Google Population Dynamics Foundation Model

8. For example, faculty affiliates of the Berkeley Artificial Intelligence Research (BAIR) Lab, most of whom are engineering or computer science professors, are doing some very interesting work on social prediction with deep learning tools.