Petrinect: Modeling gestural interactions with executable Petri nets

PetriNect : Modeling gestural interactions with executable Petri nets

PetriNect : Modeling gestural interactions withexecutable Petri nets

Romuald Deshayes1 Philippe Palanque2 Tom Mens1

Université de Mons – UMONS, [email protected]

Université de Toulouse, [email protected]

4 september 2012

8 dec 2011

Romuald Deshayes – UMONS 1 / 31


Table of contents

1 IntroductionGoalCase StudyTools and formalisms used

2 Petshop and ICO

3 Modeling of the case study

4 Discussion

5 Future Work and improvements

6 Conclusion



Introduction

Introduction

IntroductionGoalCase StudyTools and formalisms used



Introduction

What is it about ?

Formal specificationtechniques

VS

(Informal) Games



Introduction

Goal

Goal of the talk

Why and how to use formal specification techniques for interactionwith games ?

Why ?Because it is not morecomplicated thancodingAnd it is safer ! Gamesalso have the right tosecurity

How ?By using toolsallowing to executeformal modelsSee what’s next !



Introduction

Case Study

Case study

Pong GameAlmost like the classic gameUses KinectWith free hands only (gestural interaction)⇒ Very simple to play

Move the hand to control the paddleClose it to grab the ball

Interaction modeled with executable Petri-nets



Introduction

Case Study

Gestural interaction

Gestural Interaction ?In a general way : control a computer with the handsIn our case study : move or manipulate virtual objectsdisplayed on the screen with gestures

Possible GesturesFlyOverGrabReleaseDragZoom (in/out)



Introduction

Case Study

Kinect

Kinect : what and why ?3D sensorEquipped with powerful real-time tracking algorithmsDetects hand position in metric space (point in space)Homemade algorithm to detect if hand is open/closed

For our case study, we will use the position and status of the handsand the head



Introduction

Tools and formalisms used

Tools and formalisms used

Work done in IRIT LabPetri-net rulesFormalism called ICO based on Petri-netsTool called Petshop to create ICO’s and execute them



Petshop and ICO

Petshop and ICO

Petshop and ICO



Petshop and ICO

Petri-net

Directed bipartite graphNode correspond to Places and TransitionsDirected arcs connect places and transitionsTokens travel from place to placeState of the Petri net is represented by the position of thetokens in the placesNon-deterministic executionlocality in transitions



Petshop and ICO

ICO

ICO : main key conceptsDefined as a formal description technique designed to thespecification, modeling and implementation of interactivesystemsUses high-level Petri nets (HLPN) to describe interactivesystems’ behavioural aspectsIn HLPN, tokens can carry informationsIn ICO, tokens can store Java objects (e.g. Kinect data)In ICO, transitions can contain Java code (some processing onKinect data)



Petshop and ICO

ICO example



Petshop and ICO

Petshop

PetshopTool suite dedicated to the engineering of interactive systems.Supports ICOAllows the editing, simulation and dynamic execution ofmodels.

Use of petshop to create our models, link them together, executethem, and interact with Java code



Petshop and ICO

Petshop



Modeling of the case study







Models used to process kinect raw dataTransform data into abstract eventsEvents are interpreted by virtual objects models





Use Petshop to create a 3 layered ICO modelEncapsulate the data of hands and head of each player intotokens (position and state)Models are created to be independent of the number of players(ID for each player)Only one instance of each layer whatever the amount of virtualobjects to controlEach virtual object can be modeled using petshop (4th layer)




Raw data to positions

Layer 1 : Raw data to positionsReceives positions of hands and head of each player at regulartime intervals3 tokens are generated at each kinect frame refresh (1 for thehead, 1 for each hand)Informations about head and hand positions are stored in thetokensICO model joins data from head and hand and create a newtoken for each handTokens are sent to the next layer




Raw data to positions




Positions to gestures

Layer 2 : Positions to gestures

Receives absolute positions of hands (1 token per hand perplayer)Uses 2 consecutive tokens to calculate relative movementDetects state changes of the hands (open/closed)Sends open/close/move events to the 3rd layer




Positions to gestures




Gestures to events

Layer 3 : Gestures to events

Receives move/open/close events from layer 2Stores the state of each player in a 4 places sub Petri netJoins information about the player and events from layer 2 togenerate eventsGenerates 5 different events

GrabReleaseFlyoverDragZoom (in/out)

Sends events to modeled objects




Gestures to events



Discussion

Discussion

Discussion



Discussion

Discussion

Modeling has a lot of advantages :Reusable models, easy to maintain or extend (dynamicity)Models hide some technical complexityAmenable to formal analysisCan deal with Multi-userEasy to integrate with many different virtual objectsGoal : toolkit for gestural interaction



Future Work and improvements






Future work and improvements

Future Work and improvements :Specify the game (rules and graphics) with Petri netsPerformance issue

Threads tweaking (constant token flow)Compile for deployment

Extend the models to deal with more gesturesCreate a DSL over ICO to simplify the specification of virtualobjects behaviour



Conclusion

Conclusion

Conclusion



Conclusion

Conclusion

Wrap-upPossible to specify gestural interaction of a game with formalmethodsFormal methods might help for safety issues in gaming tooLot of other advantages to resort to executable modelinginstead of coding

Dynamicity lead to productivity increaseHides some technical detailsEasy to read and extend

Petshop is a cool tool suite, USE IT !



Conclusion

Thank you

Thank you for your attention !

Questions ?


Petrinect: Modeling gestural interactions with executable Petri nets

Education

Transcript of Petrinect: Modeling gestural interactions with executable Petri nets