Dealing with Annotator Disagreement in Hate Speech Classification

Hate speech detection plays a vital role in maintaining a safe and respectful environment, especially on social media platforms. To achieve accurate automatic hate speech detection, it is crucial to have a sufficient amount of well-labeled training data. Large language models (LLMs) such as BERT (Devlin et al., 2019) have demonstrated state-of-the-art performance in many NLP tasks, including hate speech detection. These models rely heavily on high-quality, accurately labeled datasets to train effectively. Therefore, ensuring that the training data is both fair and precise is essential for leveraging the full potential of these advanced models. However, earlier research on hate speech detection often lacks clarity in detailing their annotation processes, which can impact the quality of the datasets used for training. Many tasks in natural language processing (NLP) are subjective, meaning there can be a variety of valid perspectives on what the appropriate data labels should be. This is particularly true for tasks like hate speech detection, where individuals often hold differing opinions on what content should be labeled as hateful (Talat, 2016; Salminen et al., 2019; Davani et al, 2021).