12:00
The study proposes a new evaluation method and model to address annotation disagreements in identifying unstated arguments (enthymemes) within politically contentious discourse.
This paper, based on tweet data related to vaccination and immigration—each tweet independently annotated by five raters—finds that existing fine-tuned models consistently achieve an F1 score of around 0.67 on the enthymeme detection task. To address this, the authors propose abandoning the single-label evaluation paradigm in favor of a new metric that captures the distribution of annotators’ opinions, and they design a model capable of directly predicting this distribution, thereby better capturing human judgmental divergence without compromising the F1 score.
Why it mattersThe paper squarely confronts the long-neglected reality in NLP that “annotator disagreement is an essential characteristic”—particularly in politically sensitive contexts, where identifying enthymemes relies heavily on subjective reasoning. Its methodological shift—from point prediction to distributional prediction—and its empirical demonstration of the limitations of discourse‑structural analysis carry clear implications for the design of content‑moderation systems: models trained on majority‑vote criteria may systematically overlook implicit arguments that require deep contextual inference.