Political Bias Found in Sentiment Analysis Models: Annotator Leanings Impact Results

A recent study reveals that sentiment analysis (SA) models can inherit political biases from human annotators, even when instructions aim to prevent it. This finding raises concerns about the reliability of SA research, especially in politically sensitive areas. The research highlights that supervised models trained on human-annotated datasets are susceptible to mirroring the biases of those annotators. The modified model, trained on a dataset pruned of texts containing politicians' names, exhibited significantly lower bias than the primary model. However, the model cannot be fully isolated from the influence of certain mentions that may affect its output. For instance, while identifying mentions of Jarosław Kaczyński [jarɔˈswaf kaˈt͡ʂɨɲskʲi], snippets related to his twin brother, Lech Kaczyński [ˈlɛx kaˈt͡ʂɨɲskʲi], might be included, potentially influencing the model's predictions. The study found that the bias was not explained by public trust in politicians or the valence of training data tweets. The moderate intraclass correlation coefficient (0.6) indicated limited agreement between annotators, suggesting the bias wasn't inherent to the text but stemmed from subjective perception. Researchers suggest that when annotators encountered text mentioning a politician, they tended to label it according to their own political orientation, which the model then learned. A post-hoc survey of 15 out of 20 annotators showed results generally consistent with the observed bias. As language models evolve, their understanding relies less on specific entities and more on abstract concepts. This shift could increase the risk of bias toward abstract concepts like "anarchism" or "democracy" if such biases are present in the training data. The study cautions against using SA models for research and advises careful interpretation of existing results. Lexicon-based SA systems, which rely on lists of emotionally loaded words annotated separately, are considered less susceptible to bias propagation but may sacrifice accuracy. The study concludes that the research community should perceive machine learning-based sentiment analysis models as biased until proven otherwise and consider exploring alternative approaches. The main limitation of the current study is its focus on a single sentiment analysis model and a specific dataset largely composed of political texts in Polish. The generalizability of the findings cannot be stated with certainty, although should be taken into consideration.