Towards Multimodal Adaptive User Interfaces

download Towards Multimodal Adaptive User Interfaces

of 38

  • date post

    31-Dec-2016
  • Category

    Documents

  • view

    217
  • download

    2

Embed Size (px)

Transcript of Towards Multimodal Adaptive User Interfaces

  • MASARYK UNIVERSITYFACULTY OF INFORMATICS

    }w !"#$%&'()+,-./012345

  • Declaration

    Hereby I declare, that this paper is my original authorial work, which Ihave worked out by my own. All sources, references and literature used orexcerpted during elaboration of this work are properly cited and listed incomplete reference to the due source.

    Advisor: doc. Ing. Jir Sochor, CSc.Co-advisor: RNDr. Radek Oslejsek, Ph.D.

    ii

  • Acknowledgement

    I would like to thank my girlfriend, since she is my muse, my inspiration.She taught me importance of emotions in communications, particularly onthe following statement: Its Not What You Say Its How You Say It!

    iii

  • Contents

    1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 State of the art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

    2.1 Adaptable User Interfaces . . . . . . . . . . . . . . . . . . . . 62.2 Adaptive User Interfaces . . . . . . . . . . . . . . . . . . . . . 7

    2.2.1 Adaptive User Interface in ERP System . . . . . . . . 92.3 Utilisation of Human-Factors Methods and Devices . . . . . 9

    2.3.1 Eye-tracking . . . . . . . . . . . . . . . . . . . . . . . . 102.3.2 Electroencephalography . . . . . . . . . . . . . . . . . . 11

    2.4 Multimodal User Interfaces . . . . . . . . . . . . . . . . . . . 122.5 Multimodal Adaptive User Interfaces . . . . . . . . . . . . . 13

    3 Achieved Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.1 OpenOffice.org Interceptor (OOI) . . . . . . . . . . . . . . . . 143.2 Boulevard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

    3.2.1 Boulevard in OpenOffice.org . . . . . . . . . . . . . . 173.2.2 Proof-of-Concept Usability Study . . . . . . . . . . . 17

    3.3 Boulevard in ERP system . . . . . . . . . . . . . . . . . . . . . 183.3.1 Adaptive Disclosure with Ephemeral Visualization . 18

    4 Aims of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.1 Emotiv EPOC . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.2 The Eye Tribe . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.3 Plan of Research . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

    4.3.1 Research Methodology . . . . . . . . . . . . . . . . . . 224.4 Proposed Plan of Work . . . . . . . . . . . . . . . . . . . . . . 23

    A List of Authors Results in the Field . . . . . . . . . . . . . . . . . 33A.1 Published Publications . . . . . . . . . . . . . . . . . . . . . . 33A.2 Submitted Publications . . . . . . . . . . . . . . . . . . . . . 33A.3 Participation in Projects . . . . . . . . . . . . . . . . . . . . . 34A.4 Patent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34A.5 Citations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

    B Attached Publications . . . . . . . . . . . . . . . . . . . . . . . . . 35

    1

  • Chapter 1

    Introduction

    With the improvement of technology, Human-Computer Interaction (HCI) isbecoming more important. User interface has been recognized as one of themost important elements of software. End-user applications, such as spread-sheet and word processor, are growing in terms of the offered functionality,which induces growing complexity of user interfaces. Most users use only asmall fraction of the provided functionality and the functionality used byparticular users is very different. This is sometimes referred to as softwarebloat, creeping featurism or feature war. One of the means to address thisissue is user interface personalization. A properly personalized user inter-face improves users satisfaction and performance, compared to traditionalmanually designed one size fits all interfaces.

    There are two basic kinds of personalization: adaptable and adaptive.Both adaptable and adaptive approaches try to improve usability throughpersonalization of software from its default configuration. The adaptableapproach means personalization performed by the user, whereas the adap-tive approach stands for personalization performed by the computer. Theadaptable personalization is widely integrated in todays applications, e.g.Microsoft Office and OpenOffice.org. However, only a small number of userspersonalize, mostly because it is too hard to accomplish.

    We focus on adaptive user interfaces (AUI), where the user interfaceis personalized by the computer without direct users intervention. Thereis usually some kind of intelligence involved. The main advantage ofthe adaptive approach over the adaptable one is that no skills are requiredfrom a user to do the personalization. Such adaptive personalization can beperformed more often, without consuming the users time and energy. Bothadaptable and adaptive approaches have their advantages and disadvan-tages, however, we aim to provide the best of the both worlds since we tryto support both approaches.

    Leaders in computing innovations, such as Microsoft, noticed and rec-ognized the phenomenon, too. They have performed many attempts toimprove user interfaces. In Microsoft Office 97, the smart agent Clippy

    2

  • 1. INTRODUCTION

    the well-known AUI and a target of various jokes, was introduced. Afterthat, Clippy was replaced by Smart Menus in Microsoft Office 2003. ThenMicrosoft conducted an extensive and unpublished research which outcomewas the Ribbon User Interface, introduced in Microsoft Office 2007. It is astatic user interface with context behavior, no more WIMP-based (Window,Icon, Menu and Pointing device) user interface.

    Ability of computers to interpret, process, and simulate human affects, isgrowing. Computers are able to read and interpret Electroencephalography(EEG) and Electrocardiography (ECG). Nowadays, computers are able todetermine current users emotional state even through observation of theusers face or voice. When computers will consider the current users emo-tional state, the interaction between human and computer will become morenatural. The so-called Affective computing is becoming more widespreadand is expected to be even more prevalent in near future, which is supportedby a recent introduction of Intel RealSense. As a leader in computing inno-vations, the company designed so small a 3D Depth and 2D camera module,that it can be used in tablet, ultrabook, notebook and other mobile devicesin the future. RealSense also includes algorithms to deliver users emotionalstate from face expressions captured by 3D camera. The importance of usersemotions can be presented on the following statement:

    The Kometa Brno won again.

    Was this statement intended to be perceived as pleased or disappointed?This cant be decided without additional information such as facial expres-sion or emotion of voice of the person who said it.

    Eye tracking is used to determine where someone is looking (point ofgaze). It utilizes a device (Eye tracker) and appropriate method to captureand interpret eye movements for analyzing users behavior or as an real-timeinput to the human-computer dialogue. Eye tracking can be used in variousapplications typically described as active or passive. Passive applicationsinclude analysis of design or layout, where active applications involve devicecontrol, for example aiming in games or hands-free typing (usually by userswith disabilities).

    In our previous work we have developed a novel adaptive interfacecalled Boulevard, a panel container with a user interface automaticallypersonalized to an individual user. Our user interface automatically observesthe users behavior and personalizes the user interface to the preferredfunctionality and the preferred interaction style used to activate a particularuser command. Also parameters applied to user commands are consideredand personalized.

    3

  • 1. INTRODUCTION

    Our main research goal is discussed in this proposal. In brief, our goal isto enhance Boulevard by utilization of affective computing and eye track-ing, since considering userss emotional state and points of gaze may playimportant role for adaptations in user interface.

    This proposal is organized as follows: in Chapter 2 we present anddiscuss basic approaches in the field on published related work. Chapter 3describes our results in the field and Chapter 4 proposes our research.

    4

  • Chapter 2

    State of the art

    The todays desktop applications are usually very complex and offer muchfunctionality. Unfortunately, the excess of provided functionality was identi-fied as a substantial source of dissatisfaction by many users [1]. The usersusually use only a small subset of the offered functionality and there areconsiderable differences between users in the functionality preferred byusers, which was described in various studies (e.g. [2, 3]). Such dispropor-tion between provided and actually used functionality induces a need of thepersonalization of the user interfaces.

    Ceaparu et al. conducted a study on determining the causes of usersfrustration [4]. They presented very discouraging results when they showedthat users experience frustration on a frequent basis. The applications inwhich the frustrating experiences happened most frequently were identifiedas web browsing, e-mail, and word processing. The most reported causesof frustration were error messages, dropped network connections, longdownload times, and hard-to-find features. The time lost due to reportedissues ranged between 47-53% of time spent on a computer.

    Our interest focuses on word-processing applications, as they are usedby many users and various types of users, from beginners to professionals.It is estimated that Microsoft Office is used by 500 million users and OpenOf-fice.org version 3 has been downloaded more than a hundred million times.More importantly, todays word processors are complex applications andth