Statechart modeling of interactive gesture-based applications
description
Transcript of Statechart modeling of interactive gesture-based applications
Statechart modeling of interactive gesture-based applications
Statechart modeling of interactivegesture-based applications
Romuald Deshayes Tom Mens
Software Engineering Lab
2010-2011
Deshayes Romuald – UMONS 1 / 20
Statechart modeling of interactive gesture-based applications
Table des matières
1 Problem Statement
2 Proof-of-concept application
3 Modeling interactive behaviour
4 Statechart models
5 The application framework
6 Conclusion and Future Work
Deshayes Romuald – UMONS 2 / 20
Statechart modeling of interactive gesture-based applications
Problem Statement
Outline : Problem Statement
Problem Statement
Deshayes Romuald – UMONS 3 / 20
Statechart modeling of interactive gesture-based applications
Problem Statement
Problem Statement
ChallengeDevelop intuitive interactive applications that are easy to maintain
Why is it a challenge ?Complex behaviourNondeterministic user actionsSame events lead to different actions
ContributionsGestural user interface for 3D objects manipulationModeling executable behaviour using a visual formalismValidation with an application framework
Deshayes Romuald – UMONS 4 / 20
Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Outline : Proof-of-concept application
Proof-of-concept application
Deshayes Romuald – UMONS 5 / 20
Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Proof-of-concept application
3D visual drawing toolUses gestures to create and manipulate 3D objectsOpenGL graphical libraryMicrosoft Kinect + NITE
Deshayes Romuald – UMONS 6 / 20
Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Small video
Deshayes Romuald – UMONS 7 / 20
Statechart modeling of interactive gesture-based applications
Modeling interactive behaviour
Outline : Modeling interactive behaviour
Modeling interactive behaviour
Deshayes Romuald – UMONS 8 / 20
Statechart modeling of interactive gesture-based applications
Modeling interactive behaviour
Modeling interactive behaviour
ContextHighly reactive event-driven systemsGesture-based interaction
Proposed solutionsVisual modeling language
Statecharts
Petri nets
Amenable to formal analysisEasier to evolveReduced complexity
Deshayes Romuald – UMONS 9 / 20
Statechart modeling of interactive gesture-based applications
Statechart models
Outline : Statechart models
Statechart models
Deshayes Romuald – UMONS 10 / 20
Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Hand
Deshayes Romuald – UMONS 11 / 20
Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Component
Deshayes Romuald – UMONS 12 / 20
Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Scene
Deshayes Romuald – UMONS 13 / 20
Statechart modeling of interactive gesture-based applications
The application framework
Outline : The application framework
The application framework
Deshayes Romuald – UMONS 14 / 20
Statechart modeling of interactive gesture-based applications
The application framework
The application framework
Framework for developing interactive applicationsExecutable behaviour specified using statechartsClient-server architecture
Deshayes Romuald – UMONS 15 / 20
Statechart modeling of interactive gesture-based applications
The application framework
The application framework II
FeaturesGeneric
Abstract classes
Observer design pattern
Communication between statechartsUses Java SwingStates library
Deshayes Romuald – UMONS 16 / 20
Statechart modeling of interactive gesture-based applications
The application framework
The application framework III
OpenGLWidgetsBodyAdapters
Framework
-body : BodyBoyEvent extends VirtualEvent
-open : boolean-noCompTouched : boolean
ComponentEvent
-position : double[]-speed : double[]-pixelpos : int[]-lefthand : boolean
Body
Main Application
<<library>>SwingStates
+addAsListenerOf(s : Server)BodyAdapter
-serverSocket : DatagramSocketServer extends Thread
+handMovedX(e : BodyEvent)+handMovedY(e : BodyEvent)+handMovedZ(e : BodyEvent)+handClosed(e : BodyEvent)+handOpened(e : BodyEvent)
<<Interface>>BodyListener extends
EventListener
-machine : StateMachineHand
-machine : StateMachine+addComponent(c : Component)+addPointer(p : Pointer)+addScene(s : Scene)
Controller
-open : boolean-machine : StateMachine
Pointer
-boundingBox : Rectangle-machine : StateMachine
Component
-machine : StateMachineScene
+extend() : void+retract() : void+drawMenu(gl : GL) : void
ExtendableMenu+changeActiveTexture() : void+drawButton(gl : GL) : void+startCounter() : void
GLButton
+addButton(button : GLButton) : void+addText(text : String) : void+drawMenu(gl : GL) : void+spawn() : void+reduce() : void
SpawningMenu
+addButton(button : GLBUtton) : void+drawMenu(gl : GL) : void
ButtonMenu +drawMenu(gl : GL) : void+onModel() : void
ImageMenu
-selected : booleanModel
StateMachine
MainScene
3DModel
1..*
models
0..1
r ight
1..*
1..*
0..1
left
0..1selectedModel
1..**
1
Visual Paradigm for UML Standard Edition(University of Mons)
Deshayes Romuald – UMONS 17 / 20
Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Outline : Conclusion and Future Work
Conclusion and Future Work
Deshayes Romuald – UMONS 18 / 20
Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Conclusion and Future Work
StatechartsAppropriate and scalable formalism for modeling theinteractive gesture-based behaviour ?
ImprovementsExternalise/decouple statechart representation from frameworkCompare statecharts with Petri nets
Deshayes Romuald – UMONS 19 / 20
Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Future Work II
Apply our approach to other HCI applicationsOther types of natural interfacesCompare/combine with other UIMLControlled experiments, carry out user studies, empiricalstudies
Deshayes Romuald – UMONS 20 / 20