Model-driven engineering of multimodal user interfaces

1 Polytechnic University of Valencia – Valencia, November 30th, 2006

Model-Driven Engineeringof User Interfaces

for Multimodal Web ApplicationsJean Vanderdonckt

Université catholique de Louvain (UCL)Louvain School of Management (LSM)

Information Systems Unit (ISYS)Belgian Laboratory of Computer-Human Interaction (BCHI)

http://www.isys.ucl.ac.be/bchi


What is a multimodal user interface?

• Some terminology: notion of mode– We have 5 Modes corresponding to our 5 human senses– The human body has five major senses which operate to gather

information from the world around us, sight, hearing, smell, taste, and touch. Any stimulus to one of the sense areas is detected by sensory nerves and is sent to the brain for interpretation.



• Combining several modes simultaneously: some are very hard!

[Tison,2003]



• Some terminology: notion of modality– Definition: a particular form of a mode– Example: for auditive mode, we have: speech, music, voice,

chorals, noise– User interface for each mode thus involves one or many

modalities– A system is said to be

• Monomodal iff only one modality is involved• Multimodal iff many modalities are involves

– Some modalities are still in their infancy:• Olfactive user interfaces• Aromatic user interfaces

[Bodnar,2004]



• Some terminology: notion of media– Definition: is a hardware/software device allowing interaction

between a user and a system according to a given set of modalities

– Example: for auditive mode, we have several media possible: speech, music, voice, chorals, noise

– User interface for each mode thus involves one or many modalities

[Bellik,2002]


Why are multimodal user interfaces important?

• Graphical user interfaces – classical human-computer interface paradigm– Users with limited literacy and typing skills– Handheld devices

• Need for hands and eyes free• Difficult to support mouse and keyboard interaction

– Web access anyplace, anywhere, anytime

• Need for newer, better interface paradigm– Multimodal user interface is a possible answer…– Adaptation to the context of use (user, platform, environment)– More natural and efficient human-computer interaction

[IntuiLab,2006]


Why are multimodal user interfaces challenging?

• For Web Engineering– Since presentation and navigation are no longer visual only, how can we

design vocal, tactile and multimodal presentation and navigation within a page or across pages?

• For Model-Driven Approach– So far, little research has been devoted to modelling the various aspects

involved in multimodal web applications. Therefore, there is a need for identifying the appropriate implementation characteristics which will be the turned into high level design options to be supported by model-to-model and model-to-code transformations

• For Human-Computer Interaction:– Developing a user interface for any combination of graphical, vocal, and

tactile interaction modalities remains a challenge because of synchronization issues in time and space

• For Usability Engineering– It is not because we are able to develop a user interface with any

combination of any modalities that the resulting user interface is usable or tractable for a end user. We know little about the usability guidelines to be respected in this area


Concepts: The CARE Properties

• From Amodeus European Project (1995!)– Complementarity– Assignment– Redundancy– Equivalence

Put that there

[Bolt,1979]


What are the appropriate abstractions?

• Design space that supports effectively and efficiently the progress of the development life cycle towards a final system while assuring some form of quality

• Design space consists of an n-dimensional space where each dimension is denoted by a single design option

• Each design option has a finite set of design values• E.g., label location (left-aligned, centered, right-aligned)


Why a design space?

• Advantages:– Clarifies the development process in terms of options– Facilitates the incorporation in the development life cycle of design

options as an abstraction covered by a software– Supports the tractability of more complex design problems

• Shortcomings:– In theory: design options could be very numerous (even infinite)– In practice: impossible to consider a large amount of design options

• Too complex or expensive to implement• Design options do not necessarily address users’ needs and

requirements• Out of the designer’s scope of understanding, imagination or

background• Design decision is not always clear (may violate some usability

principles or guidelines)


Multimodal Web Applications

• The languages in which they are implemented (e.g., XHTML, VoiceXML, X+V):– Restrict the amount of possible UIs to obtain– Set directly the CARE properties to Assignment, Equivalence and

Redundancy– Limited to graphic, voice, and tactile modalities

[W3C,2006]


Solution space

• Cameleon reference framework• UsiXML language:

– Graphical interaction– Vocal interaction– Multimodal interaction

• Design option-basedtransformational approach:– Step 1: Construct the Task

& Domain Models– Step 2: From Task & Domain to

Abstract User Interface (AUI)– Step 3: From AUI to Concrete User Interface (CUI)– Step 4: From CUI to FUI

[Stanciulescu et al.,2005]


Design options for graphical web UIs

Design options for graphical UIs

Sub-task presentation

Sub-task navigation

Concretization of

navigation & control Navigation

concretizationControl

concretization



• Sub-task presentation: specifies the appearance of each sub-task in the final user interface.

Sub-task presentation

separated

combined

Sub-task 1

Sub-task 2

Sub-task 3

one by one

combined box

tabbed dialog box

floatwindow

Sub-task 1 Sub-task 2 Sub-task 3Sub-task 1

Sub-task 1

Sub-task 2

Sub-task 3

many at once

floatwindow

Sub-task 1

Sub-task 2

Sub-task 3

all in one

areas with separators

group boxes

bulleted list

numberedlist

Sub-task 1

Sub-task 3

Sub-task 2

Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2

Sub-task 3

1. Sub-task 1

2. Sub-task 2

3. Sub-task 3

Abstract level

Concrete level


Sub-task presentation – example

• Separated sub-task presentation

T11 T12 T13

T1

AC11

AC12

AC13

AC1

T11 T12 T13

T1

AC11

AC12

AC13

NAC LHS RHS

NAC LHS RHS

• Combined sub-task presentation



• Navigation concretization: identifies the placement and the cardinality of the objects/logically grouped set of objects that ensure the navigation.

Navigation concretization

placement cardinality

local

Sub-task 1

global

Sub-task 1

Sub-task 2

Sub-task 3

simple

Sub-task 2 Sub-task 3Sub-task 1

multiple


[Stanciulescu,2006]


Sub-task navigation - example

• Local placement for navigation objects

AC1

T11 T12 T13

T1

AC11

AC12

AC13

AIC111 AIC121 AIC131

• Global placement for navigation object

AC1

T11 T12 T13

T1

AC11

AC12

AC13

AIC11

NAC LHS RHS

NAC LHS RHS



• Sub-task navigation: specifies the way in which the dialog control is transferred from one sub-task to another.

Sub-task navigation

asynchronous


sequential (synchronous)




• Concretization of navigation and control: specifies if the navigation and control are ensured or not by the same object.

Concretization of

navigation & control

separated combined



• Control concretization: specifies the placement and the cardinality of the objects that ensure the control.

Control concretizations

placement cardinality

local

Sub-task 1

global

Sub-task 1

Sub-task 2

Sub-task 3

multiple


simple



Design options for multimodal web UIs

• Consider the ergonomic criteria for the evaluation of human-computer interfaces– Compatibility, consistency, work load, adaptation, dialog control,

guidance, error management

• Associate a corresponding CARE property to each design value.

Design options for multimodal UIs

Prompt Grouping /Distinction of items

Immediatefeedback Guidance

[Scapin & Bastien,1997][Vanderdonckt,1995]



• Prompt: refers to the interaction channels available in order to lead the users to take specific actions whether it be data entry or other tasks.– vocal (A)– graphical (A)– multimodal (E, C, R)



• Grouping/Distinction of items: concerns the organization of information in relation to one another.– Input: any information input from the user to the system

• vocal (A)• graphical (A)• multimodal (E, C, R)

– Output: any information output from the system to the user• vocal (A)• graphical (A)• multimodal (E, C, R)



• Immediate feedback: concerns the system responses to users’ action– vocal (A) – graphical (A)– multimodal (E, C, R)



• Guidance: refers to the means available to advise, orient, inform, instruct and guide the users throughout their interactions with the system– Guidance for input: any guidance offered to the user in order to

guide him with the input• Textual (A), Iconic (A), Acoustic (A), Speech (A)• Multimodal (E, C, R)

– Guidance for output: any guidance offered to the user in order to guide him with the feedback

• Textual (A), Iconic (A), Acoustic (A), Speech (A)• Multimodal (E, C, R)


Multimodal example

• Multimodal text field– Prompt: multimodal (R)– Grouping for input: multimodal (E)– Immediate feedback: multimodal (R)– Guidance:

• Input: iconic (A)• Output: iconic (A)

Prompt: multimodal

Please say your name

(vocal +graphical)1

Input: multimodal

Juan Gonzalez

2

(vocal +graphical)

Juan Gonzalez

Guidance for input: iconic

Guidance for feedback: iconic

Feedback:multimodal

Your answer was: Juan Gonzalez

3

(graphical + vocal)

[Stanciulescu,2006]


Vocal Concrete Interaction Objects

• VocalContainers:– VocalGroup– VocalForm– VocalMenu– VocalConfirmation

• VocalIndividualComponents:– VocalOutput:

• VocalFeedback• VocalPrompt• VocalMenuItem

– VocalInput– VocalNavigation– Break– Exit

• EventTypes: error, help, noInput, noMatch


Vocal Concrete Interaction Objects

• Example: virtual ice cream shop


Multimodal Concrete Interaction Objects

• MultimodalCIO: combination of graphical CIO and vocalCIO•

outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance

groupBox + radioButtons

vocalMenu + vocalItems + vocalInput

groupBox + radioButtons + vocalMenu + vocalItems +

vocalInput + guidance

Graphical Vocal Multimodal


Concrete UI Relationships

• GraphicalRelationships– GraphicalTransition– GraphicalAdjacency– GraphicalContainment

• VocalRelationships– VocalTransition– VocalAdjacency– VocalContainment

• Synchronization: between vocalCIOs and graphicalCIOs


Tool support

• MultimodaliXML – five softwaremodules:– IdealXML– TransformiXML– GrafiXML– CFB Generator– XHTML+Voice Generator tool


Multimodal Concrete Interaction Objects

• Syntax example: multimodal text field • with equivalent input and redundant output


Specification of transformation

• Graph-based patterns• Syntactically typed patterns• Textual concrete syntax of patterns: UsiXML syntax


How transformation rules work…

• LHS: Find an occurrence of LHS into G• NAC: Check that NAC does not match into G. If there is a

match then skip to another occurrence of LHS • RHS: Replace LHS by RHS

Transformation rule

Initial Model

Resulting Model

[Limbourg,2004]


UsiXML syntax of transformation rules

NAC

NAC

RHS

mappings


Transformation rule

NAC:LHS:

RHS:

Multimodal prompt

Guidance for input

Multimodal input

Guidance for output

Vocal feedback


Case studies

• 2 case studies: Virtual Polling system and Car rental system


Conclusion

• Model-driven Engineering of Multimodal Web applications is possible

• Requirements fulfillment:– Support for multimodal input/output– CARE properties support for input/output modalities– Approach based on design space composed of 13 design options– Design space covers a subset of the total number of UIs which

can be developed with X+V

• Methodology covers a subset of the UIs which can be developed with X+V


Conclusion

• Advantages of combining design options into a design space:

Vertues of a design space

– Descriptive– Generative– Comparative

[Stanciulescu & Vanderdonckt,2006]

[Beaudouin-Lafon,2004]


Future work

• Investigate the implementation of a new transformation engine: any suggestion?

• Reinforce existing vocal components• Extend the ontology with new vocal components• Design space improvement• Extend transformation catalogs• Adaptation of multimodal web user interface to the context of

use


References

• See videos and papers on MultimodaliXML at:http://www.usixml.org/index.php?view=page&idpage=51

• Stanciulescu, A., Vanderdonckt, J., Design Options for Multimodal Web Applications, Proc. of 6th Int. Conf. on Computer-Aided Design of User Interfaces CADUI’2006 (Bucharest, 6-8 June 2006), Chapter 4, Springer-Verlag, Berlin, 2006, pp. 41-56

• Stanciulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F., A Transformational Approach for Multimodal Web User Interfaces based on UsiXML, Proc. of 7th ACM Conf. on Multimodal Interfaces ICMI’2005 (Trento, 4-6 October, 2005), ACM Press, New York, 2005, pp. 259-266


Thank you very much for your attention

For more information and downloading,http://www.isys.ucl.ac.be/bchi

http://www.usixml.orgUser Interface eXtensible Markup Language

http://www.similar.ccEuropean network on Multimodal UIs

Special thanks to all members of the team!

http://www.isys.ucl.ac.be/bchi

Model-driven engineering of multimodal user interfaces

Business

Transcript of Model-driven engineering of multimodal user interfaces