Simon Mayer

Thumbnail Image
Title
Prof. Dr.
Last Name
Mayer
First name
Simon
Email
simon.mayer@unisg.ch
ORCID

Settings

Now showing 1 - 2 of 2
  • Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    Temporal Scene Understanding using Contextually Unique Identification
    (2024-10-28)
    Jha, Sanjiv  
    ;
    Garcia, Kimberly  
    ;
    Antille, Yasmine  
    ;
    Solèr, Marc
    ;
    Padua, Simon
    Humans can easily comprehend and explain the dynamics of a scene by observing the evolution of relationships among identified objects over time-while research towards Hybrid Intelligence promotes the integration of human and machine capabilities, this ability is currently beyond the capabilities of automated systems. Toward realizing it in automated scene understanding systems, we present interpretable Object Identification using Contextual Information System (OIC). Given a video, OIC detects, identifies, and tracks objects and their relationships over time to answer questions about the analyzed scene. Moreover, OIC makes predictions and infers the actors' intentions in a scene. To achieve this, our approach generates a scene graph containing classified objects and their semantic relationships. It then computes a Frame Graph by adding Contextually Unique IDentifiers (CUIDs) to each of the detected objects in the scene graph; the CUIDs permit tracking multiple object instances over time, even if the objects are visually identical. The CUIDs are then used to connect objects across a sequence of Frame Graphs, generating a Temporal Graph. This graph is exported as a cue for a pretrained Large Language Model to provide assistance and answer user questions. OIC's modular architecture enables simple comprehension and swapping of its components, making OIC more interpretable and maintainable than end-to-end scene understanding systems. Our quantitative and qualitative evaluation results demonstrate the effectiveness of OIC as a viable next step toward interpretable automated scene understanding systems.
    Type:conference paper
    DOI:10.1109/ICTAI62512.2024.00145
    URI:https://www.alexandria.unisg.ch/handle/20.500.14171/120884
    Scopus© Citations 5
  • Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    Temporal Scene Understanding using Contextually Unique Identification
    (IEEE, 2024-10-28)
    Sanjiv Jha  
    ;
    Garcia, Kimberly  
    ;
    Antille, Yasmine  
    ;
    Solèr, Marc
    ;
    Padua, Simon
    Humans can easily comprehend and explain the dynamics of a scene by observing the evolution of relationships among identified objects over time-while research towards Hybrid Intelligence promotes the integration of human and machine capabilities, this ability is currently beyond the capabilities of automated systems. Toward realizing it in automated scene understanding systems, we present interpretable Object Identification using Contextual Information System (OIC). Given a video, OIC detects, identifies, and tracks objects and their relationships over time to answer questions about the analyzed scene. Moreover, OIC makes predictions and infers the actors' intentions in a scene. To achieve this, our approach generates a scene graph containing classified objects and their semantic relationships. It then computes a Frame Graph by adding Contextually Unique IDentifiers (CUIDs) to each of the detected objects in the scene graph; the CUIDs permit tracking multiple object instances over time, even if the objects are visually identical. The CUIDs are then used to connect objects across a sequence of Frame Graphs, generating a Temporal Graph. This graph is exported as a cue for a pretrained Large Language Model to provide assistance and answer user questions. OIC's modular architecture enables simple comprehension and swapping of its components, making OIC more interpretable and maintainable than end-to-end scene understanding systems. Our quantitative and qualitative evaluation results demonstrate the effectiveness of OIC as a viable next step toward interpretable automated scene understanding systems.
    Type:conference paper
    Journal:2024 IEEE 36th International Conference on Tools with Artificial Intelligence (ICTAI)
    DOI:10.1109/ICTAI62512.2024.00145
    URI:https://www.alexandria.unisg.ch/handle/20.500.14171/122105
    Scopus© Citations 5