Model-based Design of Multimodal Interaction for Augmented...
16
. 1 22.06.2015 Sebastian Feuerstack, Állan C. M. de Oliveira, Mauro dos Santos Anjo, Regina B.Araujo, and Ednaldo B. Pizzolato Model-based Design of Multimodal Interaction for Augmented Reality Web Applications The Augmented “Drag-and-Drop”
Transcript of Model-based Design of Multimodal Interaction for Augmented...
. 1
22.06.2015
Sebastian Feuerstack, Állan C. M. de Oliveira, Mauro dos Santos Anjo, Regina B.Araujo, and Ednaldo B. Pizzolato
Model-based Design of Multimodal
Interaction for
Augmented Reality Web Applications
The Augmented “Drag-and-Drop”
Overall Research Question
2
22.06.2015
How to model
interactions that
span Modes and
Media ?
http://www.multi-access.de
Multimodal Interfaces
22.06.2015
Which Media ?
http://flic.kr/p/6eeLwg
Which Modes ?
http://flic.kr/p/6eeLwg http://flic.kr/p/7sZ6ho
3
Interaction Resources
22.06.2015
How ? Interaction Resources
4
Why modeling?
22.06.2015
Why modeling?
One specification that
is declarative and exact
discuss & execute & store & share
add new forms of interaction
no code but model changes
5
Related Work 6
22.06.2015
Use Case:
Augmented Drag and Drop
7
22.06.2015
Buying Furniture
Shopping Cart
Drag and Drop
Into my environment
How to model?
22.06.2015
8
Cameleon Reference Framework
W3C MBUI
Task Model
Abstract UI
Custom
Interaction Resources
Mappings
Abstract User Interface Model
9
22.06.2015
Abstract User Interface Model
22. Juni 2015 The Augmented “Drag-and-Drop
10
Abstract Behavior Model
22. Juni 2015
AUI:AIO:AIChoiceElement:
AISingleChoiceElement
initialized
listed
focus
suspended
organize
defocus
organized
suspend
focused
organize
Presenting
chosen
unchosen
drag
dragging
unchoose
drop
H
[in(focused)] choose
/ aios=find(parent.childs.chosen);
aios.all.unchoose
next||prev||parent
/aio=find(act);
aio.focus
Selectionpresent
11
Multimodal Mapping: The Mouse-based Drag-and-Drop
22.06.2015
C
dst=SingleChoice.focusedC
Tw<0,3s
sc.dragdst.drop(aios)
sc=SingleChoiceElement.focused
Tw<0,3sLeftButton.pressed
LeftButton.released
sc.parent.dropfail
12
IR:IN:Pointer
stopped
move stop
moving
IR:IN:Button
released
press release
pressed
IR:IN:Mouse Mouse
ButtonWheel Pointer
LeftButton RightButton
- x,y- z
Hand Gesture Resource Interactor
22.06.2015
IR:IN:HandGesture:PointingSelect
left_hand_appeared
thumb_down
Left Hand
left_hand_disappeared
selected
released
thumb_up
Command
Right Hand
NoLeftHandBothHands
NoRightHand
right_hand_disappearedright_hand_appeared
Pointing
move
moving
stopped
stop
move
NoHands
left_hand_appeared right_hand_appeared
left_hand_disappearedright_hand_disappeared
Cdst=SingleChoice.focused
C
Tw<0,3s
aois.dragdst.drop(aios)
sc=SingleChoice.focused
Tw<0,3sRightHand.cmd.selected
RightHand.cmd.released
Video: Connecting the Web with AR
14
22.06.2015
Cam
Cam Pointing
Command
Monitor
Jessica
Conclusions & Future Work
22.06.2015
• Detailed, declarative modeling of multimodal interactions
• We can combine modes and media by mappings and
bridge realities
• All models are interpreted and can be changed while
running
• What`s still missing is a tool / editor
• We intend to contribute to
• W3C-MMI WG
• W3C Ubiquitous Application Design CG
15
