Rositsa Ivanova
Last Name
Ivanova
First name
Rositsa
Email
rositsa.ivanova@unisg.ch
1 results
Now showing 1 - 1 of 1
- Some of the metrics are blocked by yourconsent settings
Item type:Publication, Argument Quality in the Eye of the Annotator: Understanding the Challenges of Annotation Through Eye Tracking(2026); In Natural Language Processing, the quality of data annotation directly affects the performance of tools. In the subfield of Computational Argumentation, the annotation of argumentative text is a challenging task due to its subjective nature and high cognitive demand for both experts and novices. In this paper, we provide a systematic overview of these challenges and propose the use of eye tracking as a method that allows us to better understand the experience of annotators. We report on a randomised crossover experiment with 40 participants combining eye movement, self-reported workload using NASA-TLX, and task-specific questionnaires. We demonstrate a rapid decrease in attention throughout the annotation task and during the familiarisation of the annotators with the guidelines. We show that annotators perceive longer guidelines as more mentally demanding, yet more helpful and clear than shorter ones. Our findings show the feasibility of eye tracking for the understanding of the annotation challenges.Type:conference contribution