Intent-driven mixed reality adaptation
Title: User-driven Mixed Reality Adaptation (10 min presentation)
Abstract: Digital content in mixed reality must continuously adapt to the user’s changing context, including the real-world environment, the user’s current task, and evolving information and interaction needs. Correctly anticipating these needs is inherently difficult: they are highly personal and depend on many factors. Purely automatic approaches risk adaptation errors and user frustration, while manual customization, such as spatial rearrangement of content, tends to produce suboptimal outcomes in particular when it comes to aspects such as efficiency or ergonomics. These are difficult to assess for end-users who typically lack a structured understanding of both, the MR design space as well as usability criteria. Moreover, continuously adjusting interface parameters, such as spatial placement or visibility, is both cumbersome for users and disruptive to their primary task. In this talk, I present first results from our ongoing work on intent-driven adaptation of mixed-reality interfaces. Our goal is to enable users to steer interface adaptation without interrupting their ongoing activity or breaking immersion. To this end, we develop a natural-language interface through which users can express their high-level adaptation intents in an unconstrained way. The central challenge is that such intents are naturally vague and potentially incomplete, making it difficult to translate them directly into actionable adaptation behavior. We address this by developing a structured intermediate representation that captures the user’s adaptation intent in a structured form which can be connected to a multi-objective optimization problem. We use a large language model to cast the user’s utterance into this representation by reasoning about the spoken intent in the context of the user’s real-world and interface environment, as well as any previous adaptation behavior that was defined. We then translate the representation into an adaptation policy using the adaptive user interfaces toolkit and execute it in the MR application. Our sysem supports users in iterative refinements on a previously defined policy (e.g., adding constraints when an interaction problem arises), as well as complete policy changes (e.g., when changing tasks) and enables the system to ask clarification questions and give feedback. We evaluate our approach across multiple use cases in which participants perform representative real-world tasks while interacting with digital content, comparing our system against fully automatic and fully manual baselines on measures of user satisfaction, intent alignment, and perceived control.