15 June 2025
Recent advancements in the field of Natural Language Processing (NLP) have underscored the importance of annotator disagreement, redefining it as a meaningful source of information regarding the task, the data and the annotators, rather than dismissing it as noise. Research has shown that Large Language Models (LLMs) may exhibit biases that align with dominant Western perspectives exposing inequalities that could negatively impact underrepresented communities, whose voices are often drowned out by majority opinions. As LLMs evolve alongside humans, aligning them with human preferences becomes a crucial aspect of their design process.
In this study, EMERGE partners from the University of Pisa present a Multi-Perspective framework for stance detection that explicitly incorporates annotation diversity by using soft labels derived from both human and large language model (LLM) annotations. Building on a stance detection dataset focused on controversial topics, the authors augment it with document summaries and new LLM-generated labels. They then compare two approaches: a baseline using aggregated hard labels, and a multi-perspective model trained on disaggregated soft labels that capture annotation distributions. Their findings show that multi-perspective models consistently outperform traditional baselines (higher F1-scores), with lower model confidence, reflecting task subjectivity. This work highlights the importance of modeling disagreement and promotes a shift toward more inclusive, perspective-aware NLP systems.
Read the paper in the link below.
More Information
