Immersive Notification Framework:Adaptive & Plausible Notifications inVirtual Reality

André ZennerGerman Research Center forArtificial Intelligence (DFKI)Saarland Informatics CampusSaarbrücken, Germanyandre.zenner@dfki.de

Donald DegraenIntel Visual Computing InstituteSaarland Informatics CampusSaarbrücken, Germanydonald.degraen@intel-vci.uni-saarland.de

Marco SpeicherGerman Research Center forArtificial Intelligence (DFKI)Saarland Informatics CampusSaarbrücken, Germanymarco.speicher@dfki.de

Florian DaiberGerman Research Center forArtificial Intelligence (DFKI)Saarland Informatics CampusSaarbrücken, Germanyflorian.daiber@dfki.de

Sören KlingnerGerman Research Center forArtificial Intelligence (DFKI)Saarland Informatics CampusSaarbrücken, Germanysoeren.klingner@dfki.de

Antonio KrügerGerman Research Center forArtificial Intelligence (DFKI)Saarland Informatics CampusSaarbrücken, Germanykrueger@dfki.de

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).CHI’18 Extended Abstracts, April 21–26, 2018, Montreal, QC, CanadaACM 978-1-4503-5621-3/18/04.https://doi.org/10.1145/3170427.3188505

AbstractNotifications in everyday virtual reality (VR) applicationsare currently realized by displaying generic pop-ups withinthe immersive virtual environment (IVE) containing themessage of the sender. However, this approach tends tobreak the immersion of the user. In order to preserve theimmersion and the suspension of disbelief, we proposeto adapt the method of notification to the current situationof the user in the IVE and the messages’ priority. We pro-pose the concept of adaptive and immersive notifications inVR and introduce an open-source framework which imple-ments our approach. The framework aims to serve as aneasy-to-extend code base for developers of everyday VRapplications. As an example, we implemented a messagingapplication that can be used by a non-immersed person tosend text messages to an immersed user. We describe theconcept and our open-source framework and discuss ideasfor future work.

Author KeywordsVirtual Reality; Virtual Environments; User Notification;Presence; Immersion; Framework.

ACM Classification KeywordsH.5.1 [Information interfaces and presentation (e.g., HCI)]:Multimedia Information Systems - Artificial, augmented andvirtual realities

CHI 2018 Late-Breaking Abstract CHI 2018, April 21–26, 2018, Montréal, QC, Canada

LBW609, Page 1

IntroductionVirtual Reality (VR) is currently evolving into an everydaytechnology with the potential to be used in many differentfields such as education, training, therapy, and entertain-ment [1]. Applications simulating immersive virtual envi-ronments (IVEs) are becoming increasingly sophisticatedand interactive. Core concepts therein are immersion andthe feeling of presence, which describe the acceptance ofa user’s state of mind relating the virtual world as the oneactually being and acting in. Presence consists of two fac-tors known as place illusion and plausibility illusion. Duringthe existence of both, users show realistic responses to theIVE [3]. For an immersive experience, the VR applicationand system are supposed to stimulate the user’s senses ina continuous and plausible way with stimuli from the IVE.This preserves immersion and allows the user to experi-ence the IVE without being reminded of the physical world,which could impair the sense of presence.

!

Generic Pop-Up

Immersive Notifications

Figure 1: Immersive notificationsin VR: An immersed user receivesa message sent by anon-immersed person. Instead ofbeing disturbed by a genericpop-up displaying the information,a mounted messenger approachesthe user in the medieval VRexperience and hands her a letterwith the text. When shopping in avirtual department store, however,the message might be displayed onthe screens exhibited in theelectronics department of the VRstore.

While being immersed, however, non-immersed people inthe real surroundings of the user might want to get in con-tact with the immersed user, or the VR user might need tobe made aware of specific notifications, e.g. from a smart-phone or the environment. In these cases, information orig-inating from a real-world event is supposed to be presentedin a conceptually different world – inside the IVE.

To notify the VR user, a non-immersed person might nudgethe user and talk to her. Being very interruptive, this severelydistracts the immersed user from the IVE, causing breaks intheir sense of presence. To prevent this, an alternative ap-proach is to send a message via a software application,e.g. on the smartphone, to the immersed user. Today’sstate-of-the-art systems already show support for this bydisplaying the received message as a common virtual no-tification overlay, e.g. a generic pop-up with the notification

text as sketched in Figure 1, in the user’s field of view. Thisapproach is not optimal with regard to the user’s sense ofpresence, as the sudden appearance of a generic text pop-up is in many contexts unexpected, unrealistic and inappro-priate. As a result, the plausibility of the IVE – and thus acrucial component of presence – might suffer.

In this paper, we introduce an alternative concept to notifyan immersed user of real-world information. To preservethe immersive experience in the IVE, it seems sensible thata VR user relaxing on a virtual beach should be notified ina different manner than a user fully engaged in an actiongame. Thus, we propose a general framework that allowsfor an easy inclusion of adaptive notifications in VR applica-tions. It takes into account the importance of the incominginformation, the current situation of the immersed user inthe IVE and a set of immersive notification animations de-fined by the application developers to trigger the notifica-tion that suits the virtual situation best. Instead of genericpop-ups, these immersive notifications are supposed to bescripted events in the virtual environment that match the vir-tual context and represent realistic, appropriate and plausi-ble ways of notification in the context of the IVE. We presenta first implementation of the notification framework whichserves as a basis for future integration in VR applicationsand which can be extended by the community. Moreover,we demonstrate an actual instantiation of the concept bydiscussing a VR messaging system using the framework.To conclude, we summarize directions for future work andfor the evaluation of the concept.

Related WorkThis section discusses work related to our approach con-cerning the concept of presence and user notification.

CHI 2018 Late-Breaking Abstract CHI 2018, April 21–26, 2018, Montréal, QC, Canada

LBW609, Page 2

Immersion & PresenceAs central aspects of VR experiences, presence depictsthe user’s feeling of being and acting inside the IVE andimmersion expresses how well the system stimulates theuser’s senses. Being a property of the entire VR system,immersion is described from a technical perspective, whilepresence is a function of the user and the utilized system’simmersion [1, 5]. Immersion allows a user to experiencepresence but does not guarantee it, as the actual feel-ing of presence also depends on the user’s psychologicalstate and may vary over time. In certain circumstances,e.g. when a person not part of the virtual world speaks toan immersed user, breaks in presence can occur that end auser’s feeling of presence, which are to be avoided [1, 4].

Previous research identified two key aspects crucial forthe feeling of presence: place illusion and plausibility il-lusion [3]. Our concept introduced here aims to preventbreaks in presence by maintaining the plausibility of the IVEwhen presenting real-world information. In contrast to naiveapproaches, e.g. directly interacting with the VR user fromoutside the IVE or showing generic pop-up messages, weaim to adapt information presentation to the virtual context.

User NotificationAppropriate notification of users is a requirement of manyreal-world systems and has been extensively studied in thepast [2]. Our conceptual approach for seamless user notifi-cation in IVEs is related to real-world approaches known asambient notifications [6]. This concept has been a focus ofpast research and utilizes elements embedded in the user’senvironment, such as lights or sounds [6], to guide a user’sattention and to provide information in an unobtrusive way.Some commercial VR systems and applications, such as

the HTC Vive1 and the Samsung Gear VR2, implement fea-tures which allow an immersed user to receive specific noti-fications, e.g. text messages, phone calls, calendar events,or notifications from various other applications supportedthrough additional software3. However, they typically relyon the naive approach of presenting a generic pop-up el-ement to the user. The appearance of the message is notadapted, nor is the form of information presentation tailoredto the user’s current situation in the IVE. This can lead tofrustration and increases the risk of breaks in presence.

Some research efforts on immersive notifications for VRand corresponding frameworks for application developersexist4. However, to the best of our knowledge, no scientificpublications nor any open-source frameworks exist thatrefrain from using generic pop-up overlays and concentrateon immersive user notifications able to adapt to the user’svirtual context as proposed in this work.

Immersive Notification FrameworkAs users immersed in different VR applications can expe-rience fundamentally different IVEs, generic solutions toprovide notifications are suboptimal. In the following, wepropose to include the design of immersive notifications inthe development process of VR applications.

ConceptWe propose that immersive applications should implementnotification events that fit the virtual world experienced bythe user. When receiving a message while being immersedin an application with, for example, a medieval setting, thevirtual avatar of a mounted messenger could approach theuser to hand her a letter containing the message sent by a

1https://www.vive.com/2http://www.samsung.com/global/galaxy/gear-vr/3https://github.com/ThomasGaubert/zephyr4http://nishantdp.com/projects/NotifiVR.html

CHI 2018 Late-Breaking Abstract CHI 2018, April 21–26, 2018, Montréal, QC, Canada

LBW609, Page 3

non-immersed person. In a sci-fi game, the mounted mes-senger could be replaced by a futuristic communicationdrone. In contrast to a suddenly appearing generic pop-up,we hypothesize that such scripted animations can maintainthe plausibility of the IVE and prevent a break in presence –a hypothesis to be investigated in our future evaluation.

Figure 2: The accompanyingAndroid app. Non-immersed userscan send messages with a custompriority to the immersed user overthe network.

Aside from an adaptation to the general setting of the IVE,we additionally propose to adapt notifications to the user’svirtual situation and current psychological state. Context in-formation about the user might, for example, include stresslevel or concentration, and immersive notifications will needto adjust to that as well. As an example, an obtrusive butplausible event might be necessary to immersively guidethe user’s attention to an important message while theuser plays an engaging game in the IVE. Here, a virtualpower blackout could be a way to force the user to pausethe gameplay and reactivate a virtual power generator onwhich the user finds the message sent to her. In contrast,while relaxing on a virtual beach, it might suffice to let a vir-tual bird tweet the important message from the sky. Theinformation about the context of the user in the IVE, e.g. ex-perienced stress, could either be deduced from the user’scurrent progress in the VR application, i.e. the currentlyexperienced level or room, or could be gathered with appro-priate hardware monitoring physiological signs.

ImplementationOur open-source framework is implemented for the Unityengine and is publicly available on GitHub5. The involvedcomponents are depicted in Figure 3. To showcase a VRmessaging app scenario, the package comes with a simpleAndroid application, depicted in Figure 2, that allows non-immersed users to send a text message over the network.Before sending the message, the user can assign a custom

5https://github.com/AndreZenner/notifications-framework

Unity VR Application

Immersive

Notification

Framework

Context Function

Set of Notifications

External Application

(e.g. Phone-App)

{Message, Priority}

{Message,

Notification to Trigger}

Immersive Notification Framework

Figure 3: Conceptual architecture of the framework.

priority p ranging from 0 (= lowest) to 1 (= highest). Themessage is sent to the IP address of the immersed user’sPC running the Unity VR application with the framework.

The second component, depicted in green in Figure 3, isthe VR application. The application provides a Contextfunction that outputs a context rating c between 0 (= calmcontext) and 1 (= engaging context), summarizing the cur-rent situation of the immersed user. In addition, developerscan specify the amount of different levels of notification Nconsidered in their application in the third component, thecentral Notifications Manager. For example, a setting ofN = 3 might represent the 3 levels unobtrusive, mediumand obtrusive. As a next step, developers link their im-plementations of immersive notification animations to theNotifications Manager component. For each of the N lev-els of notification, multiple animations can be defined andlinked. The specified Selection Method then defines howthe framework chooses between those of the same level,e.g. randomly or sequentially. Given an incoming messagewith priority p and the user’s virtual context information c,a decision function chooses the actual level of notifica-tion to trigger. Our adaptive implementation assumes thatfor a specific message priority, calm contexts require less

CHI 2018 Late-Breaking Abstract CHI 2018, April 21–26, 2018, Montréal, QC, Canada

LBW609, Page 4

obtrusive notifications than engaging virtual contexts, andthat increased priority leads to more obtrusive notificationsin the IVE. It thus computes the final notification rating asr = (p+ c)/2 with r ∈ [0, 1]. The rating r is used to decidethe notification level to trigger the correct animation. Forthis, the index i ∈ [0, 1, ..., N − 1] of the notification level totrigger is computed as i = br·Nc (and i = N−1 for r = 1).When triggering, the framework passes the message to thechosen animation in order to immersively display it.

Figure 4: Overview of the virtualsupermarket scene.

Figure 5: Level 0 - Unobtrusive:The incoming message isdisplayed on a public display in thevirtual supermarket.

ExampleTo test our concept, we used the framework to integratethe described messaging system in a VR game for futureevaluation. The game requires the user to collect items ina small supermarket, which is shown in Figure 4. As theuser is only allowed to collect specific items for a limitedamount of time, we can control the difficulty of the game,and consider this as the stress or context of the user. Here,we use 3 levels of notification (unobtrusive, medium andobtrusive), as depicted in Figure 5, Figure 6 and Figure 7,to adaptively notify the user.

DiscussionThe introduced framework provides a starting point for fu-ture study on immersive notifications. We aim to study howcontext-aware and immersive presentation of real-worldinformation can maintain plausibility to prevent breaks inpresence. As plausibility was found to be a crucial com-ponent for presence in IVEs [3], future investigations willconsider lessons learned from notification research in realenvironments to study their effectiveness in IVEs. Basedon our framework and the supporting idea of adaptive andimmersive VR notifications, we see several directions tocontinue from here.

Evaluation of the Concept & FrameworkAs a next step, we will perform a user study investigat-ing our hypothesis that generic pop-up overlays in VR areprone to inducing breaks in presence. Comparing the cur-rent state of the art with our adaptive approach, we aim toevaluate the influence of immersive context-adapted notifi-cations in VR on the experienced presence. For this evalua-tion, we aim to use the supermarket game introduced in theprevious section and consider appropriate notifications.

Extension of the Concept & FrameworkBesides receiving real-world text notifications, the frame-work offers potential for further extension. An importantfeature we consider is the ability for the VR user to respondto incoming messages. Additionally, a forwarding system forapplication notifications from a smartphone and the inclu-sion of direct communication events, such as phone callsor doorbells with intercom systems, introduce very interest-ing potential scenarios to explore. In these events, userscould talk to virtual avatars of the person calling or ring-ing, embedded in the virtual setting and story. Moreover,everyday events occurring in the user’s real surroundingscould be translated to the virtual context. For example, thebeeping of a timer in the user’s kitchen could trigger a vir-tual fire alarm to remind the user. Furthermore, the physicalpresence of persons or pets could be visualized with virtualavatars or objects, matching the aesthetics of the IVE.

Guidelines for Immersive & Adaptive Notifications in IVEsThe introduced concept requires VR developers to thinkof suitable notification animations that fit their IVE and dif-ferent contexts the user might be in, starting in the designstage. Integrating the framework in different VR applicationscovering various domains (e.g. games, education, architec-ture, simulation, training, or therapy) and various scenarios(e.g. IVEs that are contemporary, medieval, ancient, futuris-

CHI 2018 Late-Breaking Abstract CHI 2018, April 21–26, 2018, Montréal, QC, Canada

LBW609, Page 5

tic, or fantastic) allows for the investigation of different typesof immersive notifications. Systematic research in this di-rection, informed by past results on mobile, desktop, wear-able and ambient notification in real environments, couldlead to the derivation of general guidelines. These will helpdevelopers to find, design and integrate appropriate notifi-cations in future VR applications more easily and might leadto a higher prominence of the concept in general.

Figure 6: Level 1 - Medium:Additionally to Level 0, the lightsare dimmed while spots point tothe display.

Figure 7: Level 2 - Obtrusive:Finally, a cage is added to blockaccess to the shelves.

ConclusionWe introduced our approach of immersive notifications forVR. Our method is motivated by the unsatisfying state-of-the-art approach of displaying generic notification pop-upsthat do not adapt to the experienced IVE, implemented incurrent VR systems. To maintain the feeling of presence,our method aims to preserve plausibility when providingthe user with real-world information. We propose to notifyimmersed users with plausible animations and interactionsin the IVE, that adapt to i) the general setting of the virtualworld (i.e. aesthetics, story, environment) and ii) the currentcontext of the immersed user (i.e. how engaging the user’svirtual situation is). We provide the concept of a frameworkto include immersive notifications of incoming real-worldmessages in VR applications. In addition, we describe ouropen-source implementation of it, available for the Unityengine, exemplarily showcase a basic VR messaging sys-tem based on it, and discuss several examples of plausibleand adaptive VR notifications. Finally, we outline startingpoints for follow-up research to evaluate our method, pro-vide ideas of meaningful future framework extensions anddiscuss where further research on this topic can lead.

AcknowledgementsThis project has received funding from the EU’s Horizon2020 research and innovation programme, under the MarieSkłodowska-Curie grant agreement No 642841 (DISTRO).

REFERENCES1. Jason Jerald. 2016. The VR Book: Human-Centered

Design for Virtual Reality. Association for ComputingMachinery and Morgan & Claypool, New York, NY,USA.

2. Alireza Sahami Shirazi, Niels Henze, Tilman Dingler,Martin Pielot, Dominik Weber, and Albrecht Schmidt.2014. Large-scale Assessment of Mobile Notifications.In Proc. CHI. ACM, New York, NY, USA, 3055–3064.DOI:http://dx.doi.org/10.1145/2556288.2557189

3. Mel Slater. 2009. Place Illusion and Plausibility CanLead to Realistic Behaviour in Immersive VirtualEnvironments. Philosophical Transactions of the RoyalSociety of London B: Biological Sciences 364, 1535(2009), 3549–3557. DOI:http://dx.doi.org/10.1098/rstb.2009.0138

4. Mel Slater and Anthony Steed. 2000. A VirtualPresence Counter. Presence: Teleoperators and VirtualEnvironments 9, 5 (2000), 413–434. DOI:http://dx.doi.org/10.1162/105474600566925

5. Mel Slater and Sylvia Wilbur. 1997. A Framework forImmersive Virtual Environments (FIVE): Speculationson the Role of Presence in Virtual Environments.Presence: Teleoperators and Virtual Environments 6, 6(1997), 603–616. DOI:http://dx.doi.org/10.1162/pres.1997.6.6.603

6. Frederik Wiehr, Alexandra Voit, Dominik Weber, SvenGehring, Christoph Witte, Daniel Kärcher, Niels Henze,and Antonio Krüger. 2016. Challenges in Designingand Implementing Adaptive Ambient NotificationEnvironments. In Proc. UbiComp Adjunct. ACM, NewYork, NY, USA, 1578–1583. DOI:http://dx.doi.org/10.1145/2968219.2968552

CHI 2018 Late-Breaking Abstract CHI 2018, April 21–26, 2018, Montréal, QC, Canada

LBW609, Page 6