Context-adaptive Multimodal User Interfaces∗

Sierro HervéUniversity of Fribourg

1700 FribourgSwitzerland

herve.sierro@unifr.ch

ABSTRACTIn ubiquitous computing environment, context-awareness hasbecome essential for allowing applications to deliver adaptedinformation to users. Such mobile applications have to learnabout the context in which they evolve to adapt their inter-faces and modalities used to communicate with users. Un-derstanding what context means is essential in order to useit effectively and make easier the development of context-aware applications. This article gives an overview of ap-proaches used to develop adaptive-applications regardingdifferent context definitions, discusses about adaptive-appli-cation properties and investigates in existing context-awarenessapplications. The most important observations are that theexecution of adaptive-applications is represented by a close-loop system representing the context evolution and that takeinto account all environment information represents a chal-lenge when developing such applications.

General TermsMobile adaptive applications to context

KeywordsContext Awareness; Ubiquitous Computing; Adaptive Ap-plication; Mobile devices

1. INTRODUCTIONToday, the widespread deployment of mobile devices bringsup a new type of software called adaptive-applications. Onecharacteristic of such applications is the capability of takeinto account the context in which they evolve. We distincttwo manners used by an application to modify its comport-ment: first, how the interface is build, and second, modali-ties used to enable communication between the user and theapplication. For example, in a vehicle, spontaneous speechseems to be a good option to communicate since hands are

∗Seminar on ”Multimodal Interaction on Mobiles De-vices” : https://diuf.unifr.ch/main/diva/teaching/seminars/seminar-multimodal-interaction-mobiles-devices

occupied. In other hand, in public area, gestures or devicevibrations could be used as a communication way.The development of such context-aware applications maybe motivated by the wish to deliver adapted information tothe user. To accomplish this in a right way, an applica-tion must consider everything in its environment that hasan impact on its properties and its behaviours. In 2009,Salehie and Tahvildari define self-adaptive-software as ”aclosed-loop system with feedback from the self and the con-text” [10]. Consequently, in an analogue view, self-adaptiveapplications may constantly analyse the context to operateadaptations at run-time. Accomplish changes at run-timemay be costly, therefore it is important to detect changesand take appropriate decisions [10].The context of an application can be represented by a num-ber of ways. Indeed, regarding existing literatures, differentmeanings of context appear. According to them, the con-text may represent: the device on which the applicationis executed (e.g.: notebooks, PDAs, Smart phones) [9, 7],the user’s activity when the application is running [3, 4] orthe application usage in which authors introduce techniquesthat are able to learn how a user interacts with an appli-cation [6, 11]. According to the previous definition of anadaptive-application, all these features form the context ofan application in the sense that all can have an impact onits properties and its behaviours.This paper aims to make an overview of the state-of-the-art in context-aware applications area. We first discuss oncontext meanings and introduce the context categorization.From there, we go to the applications section where we talkabout properties of context-aware applications. Finally, wesurvey existing context-aware applications implementing indifferent domains.

2. CONTEXTIn this section, because understanding the context facilitatesthe building context-aware applications, we first try to givea definition of the term context based on existing literature.Secondly, we discuss about different types of context, brieflygiving existing solutions to capture each type of context in-formation and then talk about adaptation possibilities.

2.1 Context definitionsThe first context-aware application was realised by Want etal. (1992) with the introduction of an Active Badge LocationSystem which was able to determine the current location ofa user. During the 90s, other location-based applicationsthat were delivering information depending the user’s loca-tion appeared (e.g.: tour guides). Quickly, researchers feltthe need to define the sense of the word context. Followingdefinitions are taken from a publication of Dey and Abowd[5] which aim to define notions of context and context-awarecomputing. Shilit and Theimer (1992) first use the termcontext-aware and define the context as location, identitiesof nearby people and objects, and changes to those objects.Ryan et al. (1997) describe the context as the user’s loca-tion, environment, identity and time. Another interestingdefinition is given by Dey (1998), he views the context asthe user’s emotional state, focus of attention, location andorientation, date and time, as well as objects and peoplein the user’s environment. Furthermore, synonyms for theterm context have been given: user’s environment that theuser’s computer knows about (Brown, 1996), situation of theuser (Franklin and Flaschbart, 1998), state of the applica-tion’s surroundings (Ward et al., 1997), application settings(Rodden et al., 1998), aspects of the current situation (Hullet al., 1997).These context definitions are too specific to be applied inpractice. A more generic definition given by Shilit et al.(1994) seeming more appropriate, emphasizes that the exe-cution environment is constantly changing. Finally, one ofthe most used definitions is given by Dey and Abowd whichdefine the context as follows: ”any information that can beused to characterize the situation of an entity. An entity isa person, place, or object that is considered relevant to theinteraction between a user and an application, including theuser and applications themselves” [5].Later, Burnett (2003) and Gustavsen (2002) introduce thenotion of internal and external context dimensions to clas-sify context types. The internal dimension concerns thecontext specified by users or learned by analysing users in-teractions (e.g.: user’s goals, user’s emotional state, workcontext) whereas the external dimension represents the con-text that can be captured by hardware sensors (e.g.: move-ment, touch, location, sound) [1].

2.2 Context typesIn previous section, we give a global definition of the termcontext. Now, it seems important to precisely define infor-mation that could be considered as context during the de-velopment of adaptive-applications.For application developers, understanding the context willfacilitate the choice of what type of context will be usedin their application. Then, understanding how the contextcan be used will facilitate the choice of what context-awarebehaviours an application will support. As mentioned in lit-erature, ”require users explicitly to express all informationrelevant to a given situation” [5] seems not plausible, conse-quently, automate the collect of context information appearsto be a more appropriate solution. Today, the mobility ofusers and the diversity of devices lead developers to designapplications that adapt appropriately, with the intention tobest integrate the user-application interaction [5].For this purpose, we propose the following context types:device, activity and usage learning. In the following, we

firstly aim to define how each particular context can be cap-tured and secondly what adaptations can be made on theapplication. Examples of applications considering one typeof context or combinations of these are introduced in section3.

2.2.1 DevicesDue to the diversity of devices, a mobile application mustbe able to propose several versions. Indeed, today, mobiledevices capabilities diverge according to users needs. Eachdevice has its proper characteristics and designing and devel-oping mobile applications is not trivial. As characteristics,we can cite the operating system on which the applicationwill be deployed and all hardware capabilities (e.g.: screen,GPS, memory, network connectivity).All these features can play a role when developing applica-tions. Imagine you develop an android application, you musttake into account that both tablet owners and smart-phoneowners are potential users. Furthermore, your applicationuses the Internet connection, what is going on if no connec-tion is available? Or, some devices have a small memorywhile others have a larger memory, which information willbe stored on the device? As application developer, this kindof remarks and questions constitute a first type of contextto take into consideration when designing a mobile appli-cation. Developing an efficiency context-aware application”is able to more than double the battery lifetime in real lifescenarios” [9].Devices context generally concerns the construction of mo-bile application interfaces across platforms. In literatureconcerning adaptive user interfaces on mobile devices, so-lutions as Appcelerator, PhoneGap, Titanium, Rhomobile,Smartface, Formotus are cited and provided ”the applica-tion developer a unified language for developing applicationsacross platforms” [7]. However, aforementioned frameworksdo not stop the need to create multiple versions of an ap-plication. Consequently, series of research adopted model-based techniques - based on XML language as an intermedi-ate language - which aim to separate the presentation modelfrom the logic model [7]. Finally, note that developmentframeworks like android or iOS propose functionalities toget technical information of the device (e.g.: get screen size,get memory usage, get sensors list).

2.2.2 ActivityAdapting a mobile application according to the current useractivity represents another type of context to be consider-ate when designing mobile applications. Unlike the first typeof context and more than adapt the user interface, this con-text understanding aims to adapt the modalities (e.g.: voice,speech synthesis, device vibrations, gesture) used as a com-munication way between the user and the application. As asimple example, consider the case of using a mobile applica-tion in a vehicle, while hands being occupied, the voice couldbe a way to communicate with the device and the interfaceapplication could be simplified to facilitate the visualizationof information.When developing an adaptive-application, developers mustdetermine what context does the application require andwhen does the application make use of the context. There-fore, we have two solutions to allow adaptations. First, theapplication is able to sense the environment to determinethe current activity. In this solution, developers determine

which sensors (e.g.: GPS, accelerometer) are relevant to cap-ture the activity and define the collection time and the fre-quency. On the other hand, the choice can be left to theuser to decide which profile to use according to its activity(e.g.: normal, driving, sport, night).

2.2.3 Usage learningUsage learning context aims to define user preferences whichare an important mechanism to apply proper adaptations.In literature, this mechanism is often related to the mod-elling the user concept [8] and is able to provide user per-sonalization. In mobile applications domain, the user modelcould be used in many ways: the prediction of user needsaccording to past user behaviours, the recommendation ofinteresting information according to past user behavioursor to user profiles, or the selection of information that areinteresting to a user [8]. These types of information haveprincipally an impact on application interfaces and the con-tent proposed by the application.Due to the mobility of devices, modelling the user in mobileenvironment seems more difficult than modelling the user indesktop environments. Tengfei Bao et al. [2] separate exist-ing works in three categories. The first category manuallymodels the context based on domain knowledge, the secondmodels context though supervised learning approaches, andthe third models context though unsupervised learning ap-proaches. These three categories differ from each other inthe degree of dependence of a given domain. Unsupervisedlearning approaches need no domain knowledge to learn asituation and supervised learning need less domain knowl-edge than manual solutions.

3. ADAPTIVE APPLICATIONSIn previous section, we define the term context and sepa-rate the context in three categories. In this section, we aimto present existing adaptive-applications which take into ac-count these different context types. Firstly, we try to defineproperties of adaptive-applications and features to considerwhen developing such applications.

3.1 Enabling adaptivityThe main characteristic of adaptive-applications is that adap-tations are made in run-time. Fulfill adaptations at run-timemay be time-consuming and costly. Therefore, take adapteddecisions to consume less energy and to make rapid changes,should be the goal of an adaptation mechanism. Relatedto adaptive-software, these previous goals can be reachedtrough ”monitoring the software system (self ) and its envi-ronment (context) to detect changes, make appropriate de-cisions, and act accordingly” [10].These objectives become even more important in a mobileenvironment. By self, authors refer to the application bodythat can be divided in multiple layers and context refers toour previous definition given in section 2.1. Self and contextcan respectively be related to Self-Awareness and Context-Awareness principles. The first means that the applicationis aware of its self states and behaviours and the secondmeans that the application is aware of its context [10].As application developer, understanding the context of a fu-ture application and how the context will be used, representsthe first step in developing an adaptive-application. Then,designing the application must always be done keeping in

mind that the software will adapted it self and be context-aware. The efficiency of adaptations will both depend of theapplication structure that permits rapid adaptations and theability to collect context information.During the development phase, developers determine the ap-plication needs based on the application context in order tobuild an adaptable application. During the operating phase,the application deals with context data to properly applyadaptations. As already mentioned, an adaptive-applicationcan be view as a closed-loop system.

Ac#ng&

Deciding&Detec#ng&

Monitoring&

Effectors&

Context4Aware&!

&&&&Ac#vity:!sensors!devices!(GPS,!!!!!!!!!!!accelerometer,!...)!

&&&Usage&Learning:!algorithm!based!

&&&&&&&Devices:!frameworks,!!OS func?onali?es!

dyna

mic(

sta+

c(

Requests!

Symptoms!

Events! Ac?ons!

Decisions!

Interface! Process!Data!Flow!

Figure 1: Extended MAPE-K loop

Figure 1 presents a detailed view of processes, sensors, effec-tors and predefined types of context related to adaptive soft-ware, which can be applied to mobile adaptive-applications.Four different processes are represented:

• The monitoring process collects information of the threetypes of context previously describe in section 2.2 throughtechniques related to each context type.

• The detecting process analyses information context col-lected by the monitoring process and, in modelling theuser situations, information provided by historical/logdata of the application.

• The deciding process decides what part of the appli-cation needs to change and determines the best wayadaptations must be done. This process could be ableto compare different adaptation ways.

• The acting process applies adaptations determined bythe previous process

According to our previous categorisation of the context, thismechanism could be view as a cycle, specially regarding theactivity and the usage learning context, which are in per-petual change. Concerning the device context, it will be lessvariable since an application cannot move through devices.

3.2 Related workThis section aims to present existing mobile adaptive-ap-plications. By mobile applications, we mean applicationsexecuting in mobile devices (e.g.: smart-phone, PDAs, othermobile systems).

3.2.1 Spatiotemporal Structure LearningHosub Lee et al. developed ”a user interface prototype forthe Android smart-phone, which recommends a number ofapplications to best match the user’s context” [6]. In theirapplication, the context is represented by the time, the user’slocation, the weather, user’s emotions and user’s activities(i.e.: walk, stop, running, riding in a car). Time, loca-tion, weather and activities are recognized through inter-nal/external sensors (e.g.: GPS, accelerometer, Web data,

camera, microphone), the emotional state is manually changedby the user through an interface and the history of pastcontext information are calculate by a learning and infer-ence algorithm extended from a Naive Bayesian Classifier.An Inference Engine is used to deliver a set of applicationpropositions to the user. Then, the Inference Feedback Pro-cessing Unit determines whether the user chooses one of theproposed applications. Finally, this last information is usedto refine the learning algorithm.Finally, authors reported that a study ”showed 69 percentaccuracy in predicting the user’s application usage” [6].

3.2.2 Device-Oriented Adaptive User InterfaceBo-Fu Liu et al. propose a cross-platform development solu-tion permitting developers to easily implement adaptive userinterfaces in mobile environment. Through a WYSIWYG(what-you-see-is-what-you-get) solution, developers are al-lowed to design the application interface. Then, according tothe device capabilities (i.e. devices context) and some poli-cies, what authors call the platform-neutral language couldbe translate ”into an appropriate application for the targetdevice” [7].Context information related to devices capabilities are storedin a database which correspond at the beginning ”of rulesfrom relevant expertise” [7]. Then, when context informa-tion are collected and stored, the database is able to learnnew rules.

3.2.3 Adaptive Interaction in the VehicleFischer P. et al. propose a system architecture adapting theuser interface and modalities used for the user-applicationcommunication in a vehicle. Due to the wish of reducingthe cognitive load involved by the driving activity (i.e. ac-tivity context), speech is used as input and output chan-nels. The user can ask questions as ”Where can I findIndian food nearby” using speech. ”Most recent informa-tion from the Internet can be queried by natural languagequestion-answering dialogs using frame-based dialog man-agement” [4]. Two types of preferences are stored: explicitpreferences are given by the driver through a profile screenand implicit preferences are created through a fuzzy rec-ommendation approach and are updated whenever the userperformed actions with proposed items (i.e. usage context).A preference model stores these user actions and choices.This model is both used to be able to deliver interesting in-formation to the current user based on old searches (content-based) and to suggest interesting information to other usersthat could have a similar preference model (collaborative).

4. CONCLUSIONIn this paper, two topics had been addressed. First, wegave an overview of existing context definitions based onliterature and define what could be represented as context.Then, we separated ”any information that can be used tocharacterize the situation of an entity” in three categories:device, activity and usage learning. We highlighted the factthat each type of context has its own properties and solu-tions to detect them differ from each other (i.e.: devices -frameworks, activity - sensors, usage learning - learning al-gorithms). Furthermore, we emphasized that each contexttype has impacts on different application parts (i.e.: devicesand usage learning - interface, activity - interface and com-munication modalities).

Second, we introduced the topic of adaptive-applications.We first highlighted that self-adaptive applications may con-stantly analyse the context to operate adaptations at run-time. Moreover, we presented a global view of the processenabling adaptations when developing adaptive-applications.This closed-loop system aims to detect context changes, makeappropriate decisions and is separated in four phases: moni-toring context data, detecting adaptations to apply, decidingthe best adaptation and acting in applying the adaptation.Furthermore, designing an adaptive-application must alwaysbe done keeping in mind that the software will adapted itselfand be context-aware. Then, examples of existing applica-tions have been covered.Finally, today, devices capabilities and users mobility be-come increasingly important and more and more informa-tion about the user and the context in which the applica-tion evolves are available to developers. Futures challengeswill be to merge all context information collected from verydifferent type of context to be able to apply the best adapta-tions and improve as much as possible devices capabilities.

5. REFERENCES[1] M. Baldauf. A survey on context-aware systems.

Personal and Ubiquitous Computing, 5(1):4–7, 2001.

[2] T. Bao, H. Cao, E. Chen, J. Tian, and H. Xiong. Anunsupervised approach to modeling personalizedcontexts of mobile users. IEEE, InternationalConference on Data Mining, pages 38–47, 2010.

[3] G. Cai and Y. Xue. Activity-oriented context-awareadaptation assisting mobile geo-spatial activities.Proceedings of the 11th international conference onIntelligent user interfaces, pages 354–356, 2006.

[4] P. Fischer and A. Nurnberger. Adaptive andmultimodal interaction in the vehicle. IEEE, Systems,Man and Cybernetics, pages 1512–1516, Oct. 2008.

[5] D. A. K. and A. G. D. Towards a betterunderstanding of context and context-awareness. CHI2000 Workshop on the What, Who, Where, When,and How of Context-Awareness, pages 304–307, 2000.

[6] H. Lee, Y. S. Choi, and Y.-J. Kim. An adaptive userinterface based on spatiotemporal structure learning.IEEE, Communications Magazine, 49(6):118–124,2011.

[7] B.-F. Liu, S.-C. Chou, Y.-T. Lin, and J.-Y. Lin.Toward easy delivery of device-oriented adaptive userinterface on mobile devices. NISS, 1:80–85, 2011.

[8] G. D. Magoulas and D. Dimakopoulos. An adaptivefuzzy model forpersonalization withevolvable userprofles. International Symposium on Evolving FuzzySystems, pages 336–341, Sept. 2006.

[9] D. Preuveneers and Y. Berbers. Towardscontext-aware and resource-driven self-adaptation formobile handheld applications. Proceedings of the 2007ACM symposium on Applied computing, pages1165–1170, 2007.

[10] M. Salehie and L. Tahvildari. Self-adaptive software:Landscape and research challenges. ACM TAAS, 4(2),2009.

[11] K. S. V., S. S. G., and B. D. S. Learning stylerecognition using artificial neural network for adaptiveuser interface in e-learning. IEEE ICCIC, pages 1–5,Dec. 2011.