Post on 02-Apr-2015
Gesture InterfacesDavide Spano
lucio.davide.spano@isti.cnr.itspano@di.unipi.it
CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 156124 Pisa, Italy
Davide Spano Gesture Interfaces
Gestures
16 May 2012
Davide Spano Gesture Interfaces
Remotes (1)Nintendo Wiimote, 2006
Buttons Three-axis linear accelerometer IR camera Three axis gyroscope extension (Wii Motion Plus)
Game market opened to casual gamers
16 May 2012
Davide Spano Gesture Interfaces
Remotes (2)Playstation Move, 2010
Buttons Spherical RGB light emitter plus camera Three-axis accelerometer and gyroscope Compass
16 May 2012
Davide Spano Gesture Interfaces
Floor Devices (1) Dance pads
Konami Dance Dance Revolution (1998) Huge directional pads Buttons
16 May 2012
Davide Spano Gesture Interfaces
Floor Devices (2)Wii Balance Board
Four pressure sensors Snowboard games Fitness games
16 May 2012
Davide Spano Gesture Interfaces
Motion Tracking (1)CamSpace, 2009
Object recognition No haptic feedback
16 May 2012
Davide Spano Gesture Interfaces
Motion Tracking (2)Microsoft Kinect, 2009
RGB camera Depth sensor Microphone array Still no haptic feedback
16 May 2012
Davide Spano Gesture Interfaces
MultitouchTwo or more touch-sensed position on the screen
Patents already on 1984 iPhone 2007 Support in Mac OS X, Windows 7, Ubuntu …
16 May 2012
Davide Spano Gesture Interfaces
Gestures Overview
Gesture types: Static Dynamic Spatio-temporal
Conceptual: Gestures are continuous Gestures exploit time and space as arguments Gestures enable single user parallel interaction Need for a suitable programming paradigm
16 May 2012
Davide Spano Gesture Interfaces
The Midas Touch
16 May 2012
Tracking EngagedOutOf
range
TrackingNotTracking
Mouse
Touches,Skeleton
Tracking
Accelerometers,Gyroscope
Davide Spano Gesture Interfaces
Current Programming Model
Do gestures really fit this pattern?
16 May 2012
Davide Spano Gesture Interfaces
Multitouch Frameworks
16 May 2012
Gesture Events
Touch Events
End
Move
Start
Rotate
Pinch
Slide
Davide Spano Gesture Interfaces
Example: Pinch for zoom (1)
16 May 2012
Davide Spano Gesture Interfaces
Example: Pinch for zoom (2)
16 May 2012
Start Move
Move End
Davide Spano Gesture Interfaces
Full body tracking
16 May 2012
Davide Spano Gesture Interfaces
Microsoft Kinect SDK
16 May 2012
Davide Spano Gesture Interfaces
Primesense NITE ControlsState of the Art toolkit for full body gesture interaction
Reusable controls (recognizers) for NUI
Control list Push detector Swipe detector Steady detector Wave detector Circle detector Selectable slider (1D/2D)
All based on the observer pattern
16 May 2012
Davide Spano Gesture Interfaces
Circle Control Sample
16 May 2012
Control Event Skeleton Tracking
Davide Spano Gesture Interfaces
One Event is not enough!
16 May 2012
Pan
MoveStart
Handler
End1 1
Gesture
Touch IDSamples
Handler
Handler
Handler
1
Davide Spano Gesture Interfaces
Gesture Definition Meta ModelDeclarative
Compositional Ground Terms (basic building blocks) Composition Operators
Behaviour Association Entire Gesture Any other sub-component
Abstract Gesture Recognition Support
Formally defined through Petri Nets
16 May 2012
Davide Spano Gesture Interfaces
Break…
16 May 2012
http://www.foxtrot.com/
Davide Spano Gesture Interfaces
Gesture Support AbstractionFeature: an observable property
Gesture Support: a set of features
Gesture Support State: feature values at a given time
Gesture Support Sequence: gesture support state values through the time
16 May 2012
Davide Spano Gesture Interfaces
Ground Terms and Composition OperatorsLow-level device events as ground terms
Optional conditions on support state
Composition operators
16 May 2012
Davide Spano Gesture Interfaces
Multitouch Gesture SupportFeatures:
2D position of n touches, in pixel coordinates If a touch is not currently detected current detection time
Multi-Touch recognition ground terms:
16 May 2012
Davide Spano Gesture Interfaces
Multitouch Gesture DescriptionTap
Double Tap
Pan
Slide
Pinch
Rotate
16 May 2012
Davide Spano Gesture Interfaces
Proton (1)Kin et al. Proton: Multitouch Gestures as Regular
Expressions, CHI 2012
Multitouch Gestures Described as Regular Expression
Ground Terms
Composition operators
*Matcher for handling ambiguity
One gesture at time assumption
Tablature graphical representation for gestures
16 May 2012
Davide Spano Gesture Interfaces
Proton (2)
16 May 2012
Davide Spano Gesture Interfaces
Full Body Gesture PlatformSkeleton tracked as a set of joints
2 user full tracking + 4 only body centre Features: Joint (user id, joint id, position, orientation) Current time
16 May 2012
Davide Spano Gesture Interfaces
Full Body Example (1)
16 May 2012
Front Not Front Right hand Both hands
Davide Spano Gesture Interfaces
Full Body Example
(𝐹𝑟𝑜𝑛𝑡≫ ( h𝑅𝑖𝑔 𝑡𝐻𝑎𝑛𝑑 [ ] h𝐵𝑜𝑡 𝐻𝑎𝑛𝑑𝑠 ) )∗¿
16 May 2012
Davide Spano Gesture Interfaces
A different programming modelDescribe declaratively the gesture
Create new ones using the building blocks Reuse existing ones through composition
Code the event handlers Attach them to the events at the desired granularity It is called only when it is needed (no further checks)
Temporal description and interface behaviour are completely separated Complex Gestures can be stored as classes They can still expose their low level events
16 May 2012
Davide Spano Gesture Interfaces
Library Support
16 May 2012
Davide Spano Gesture Interfaces
Multitouch with Full Body Experiment Hands point tracking
Basic Touches Description
16 May 2012
Davide Spano Gesture Interfaces
Proof of Concept Applications
16 May 2012
Davide Spano Gesture Interfaces
Demo VideosParallel Interaction with iPhone video
“Multitouch” with Kinect video
Same Gesture Definition!
16 May 2012
Davide Spano Gesture Interfaces
Conclusion
16 May 2012
http://www.mojocomic.com/
Davide Spano Gesture Interfaces
ReferencesSpano, L.D. A Model Based Approach for gesture interfaces
(EICS 2011) pdf
Kin K., Hartmann B., DeRose T., Agrawala M. Proton: Multitouch Gestures as Regular Expressions pdf
Kinect for Windows
OpenNI
Primesense NITE
16 May 2012