User-driven Mixed Reality Adaptation
Title: User-driven Mixed Reality Adaptation (10 min presentation)
Abstract: Digital content in mixed reality must continuously adapt to the user’s changing context, including the real-world environment, the user’s current task, and evolving information and interaction needs. Correctly anticipating these needs is inherently difficult: they are highly personal and depend on many factors. Purely automatic approaches risk adaptation errors and user frustration, while manual customization, such as spatial rearrangement of content, tends to produce suboptimal outcomes in particular when it comes to aspects such as efficiency or ergonomics. These are difficult to assess for end-users who typically lack a structured understanding of both, the MR design space as well as usability criteria. Moreover, continuously adjusting interface parameters, such as spatial placement or visibility, is both cumbersome for users and disruptive to their primary task. In this talk, I present first results from our ongoing work on user-driven adaptation of mixed-reality interfaces. Our goal is to enable users to intuitively steer interface adaptation without interrupting their ongoing activity or breaking immersion. To this end, we develop a natural-language interface through which users can express high-level adaptation instructions in an unconstrained way. The central challenge is that such instructions are naturally vague and potentially incomplete, making it difficult to translate them into actionable adaptation behavior. We address this by developing a structured intermediate representation that captures the semantics of user