ClearSkies: A Preliminary Study of Gaze-Mapped Scene Segmentation in Training Aircraft Cockpits
Type
conference paper
Date Issued
2026-06-01
Author(s)
Abstract
In pilot training, deviation from standard procedures is a significant concern. To provide student pilots with objective feedback in postflight debriefing, we captured pilots’ view and gaze with the Pupil Core eye-tracker. Then we conducted a preliminary evaluation to test the feasibility of existing scene segmentation models for gazemapping1. We used an OpenCV baseline model for coarse inside vs. outside-analysis, a fine-tuned Detectron2 model for specific instrument segmentation, and Segment Anything Models (SAM 2 and SAM 3) for human-in-the-loop analysis. The baseline was fast but fragile, failing in common flight scenarios; the Detectron2 model was powerful but inflexible and unsuitable for general use; and SAM 3 was promising, offering generalizability for post-flight analysis despite noisy digital displays. A qualitative preliminary evaluation of SAM with Visual Flight Rules shows that it can be beneficial in eye movement analysis. We identified poor data quality in bright cockpit environments and ergonomics as main limitations.
Keywords
scene segmentation
computer vision
eye tracking
pilot training
HSG Classification
contribution to scientific community
Refereed
Yes
Subject(s)
Division(s)
File(s)![Thumbnail Image]()
Name
2026_Oes_Bektas_ETRA_ClearSkies.pdf
Size
1.22 MB
Format
Adobe PDF
Checksum (MD5)
7aa459d04de79a9fca493c04e7fbb6a1