JUIC-IoT: Just-In-Time User Interfaces for Interacting with IoT Devices in Mixed Reality
Type
conference contribution
Date Issued
2025-10-12
Author(s)
Research Team
Interactions Research Group (https://interactions.ics.unisg.ch)
Abstract
The number of deployed Internet of Things (IoT) devices is continuously increasing. While Mixed Reality (MR) allows hands-free interaction, creating MR User Interfaces (UI) for each IoT device is challenging, as often a separate interface has to be designed for each individual device. Additionally, approaches for automatic MR UI generation often still require manual developer intervention. To address these issues, we propose the JUIC-IoT system, which automatically assembles Just-in-Time MR UIs for IoT devices based on the machine-understandable format W3C Web of Things Thing Description (TD). JUIC-IoT detects an IoT device with object recognition, uses its TD to prompt an LLM for automatically selecting appropriate UI components, and then assembles a UI for interacting with the device. Our evaluation of JUIC-IoT shows us that the choice of LLM and the TD of a device are more crucial than the formulation of the input prompts for obtaining a usable UI. JUIC-IoT represents a step towards dynamic UI generation, thereby enabling intuitive interactions with IoT devices.
Language
English
Keywords
mixed reality
internet of things
innovation
web of things
adaptive user interfaces
AI
Generative AI
LLM
HSG Classification
contribution to scientific community
Book title
Companion of the the 2025 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp Companion ’25)
Publisher
Association for Computing Machinery
Publisher place
New York, NY, USA
Pages
7
Event Title
2025 ACM International Joint Conference on Pervasive and Ubiquitous Computing
Event Location
Espoo, Finland
Event Date
October 12–16, 2025
Subject(s)
Division(s)
Contact Email Address
jannis.strecker-bischoff@unisg.ch
File(s)![Thumbnail Image]()
Name
Automatic_MR_Interfaces_UbiComp25.pdf
Size
3.25 MB
Format
Adobe PDF
Checksum (MD5)
2519ed15e16faf70602eefadbd2c5749