Argument Quality in the Eye of the Annotator: Understanding the Challenges of Annotation Through Eye Tracking
Type
conference contribution
Date Issued
2026
Author(s)
Abstract
In Natural Language Processing, the quality of data annotation directly affects the performance of tools. In the subfield of Computational Argumentation, the annotation of argumentative text is a challenging task due to its subjective nature and high cognitive demand for both experts and novices. In this paper, we provide a systematic overview of these challenges and propose the use of eye tracking as a method that allows us to better understand the experience of annotators. We report on a randomised crossover experiment with 40 participants combining eye movement, self-reported workload using NASA-TLX, and task-specific questionnaires. We demonstrate a rapid decrease in attention throughout the annotation task and during the familiarisation of the annotators with the guidelines. We show that annotators perceive longer guidelines as more mentally demanding, yet more helpful and clear than shorter ones. Our findings show the feasibility of eye tracking for the understanding of the annotation challenges.
Funding(s)
This study was funded by the Behavioral Lab of the University of St. Gallen.
Language
English (United States)
Keywords
Computational Argumentation
Natural Language Processing
Eye Tracking
Text Annotation
HSG Classification
contribution to scientific community
Refereed
Yes
Pages
16
Subject(s)
Division(s)