The Measuring Hate Speech Project
Overview
Since the launch of Donald Trump’s presidential campaign, reports of hate speech targeting various minority groups have risen dramatically (Ansari 2016). Although this surge is well-reported (ADL 2017; SPLC 2016), it remains difficult to quantify the magnitude of the problem or even properly classify hate speech (Silva et al. 2016). Keyword searches and dictionary methods are often imprecise and overly blunt tools for detecting the nuance and complexity of hate speech.
Our goal is to apply data science to develop new ways of characterizing and studying hate speech. We have developed comprehensive, groundbreaking method to measure hate speech with precision while mitigating the influence of human bias.
Our method leverages two distinct fields: measurement theory and deep learning. Using a combination of qualitative and quantitative methods, we have developed a measurement scale of hate speech. This scale is a continuous measure of the hatefulness of a social media comment, providing more nuance than a simple binary label of hate speech. Furthermore, measurement theory provides the tools to assess how annotator perspective shapes labeled data sets, allowing us to debias hate speech scores.
This work has culminated in the Measuring Hate Speech corpus, a dataset of roughly 40,000 social media comments, annotated by nearly 10,000 annotators, on 10 different survey items. Furthermore, we have developed a suite of large language models capable of predicting the measured hate speech score from an inputted social media comment.
We have published several papers in high-impact conferences on this work, and continue to develop our methodology and computational tools. Our methods, code, and data are freely available to the public.
Manuscripts
Data and Resources