National Research Council CanadaConseil national de recherches CanadaNational Research Council CanadaConseil national de recherches Canada

CanadaCanada

Dmitry Gorodnichy, Gerhard Roth, Shahzad Mallik

www.cv.iit.nrc.ca/research/Nouse

Dmitry Gorodnichy, Gerhard Roth, Shahzad Mallik

www.cv.iit.nrc.ca/research/Nouse

Institute for Information TechnologyInstitut de technologie de l'informationInstitute for Information TechnologyInstitut de technologie de l'information

Computational Video GroupGroupe Vidéo InformatiqueComputational Video GroupGroupe Vidéo Informatique

NouseNouse “Use your “Use your NNose as Mose as Mouseouse” – ” – New Technology for Hands-free Games and InterfacesNew Technology for Hands-free Games and Interfaces

NouseNouse “Use your “Use your NNose as Mose as Mouseouse” – ” – New Technology for Hands-free Games and InterfacesNew Technology for Hands-free Games and Interfaces

((22)) Face-Tracking Based User Interfaces Face-Tracking Based User Interfaces 1. Replacing cumbersome track-ball (track-stick) on laptops.2. Extra degree of control (e.g. to switch the focus of attention).3. Hands free control (e.g. for handicap users).4. Interactive games: more physical, entertaining, 3D control, multiple-user

Fig.1. A user plays an aim-n-shoot Bubble-Frenzy game aiming the turret by pointing with her nose. (slight rotation of head allows to aim precisely in 180o range)

((33))

Fig. 2. Two users play a virtual ping-pong game, bouncing the ball with their heads. Image-based tracking allows one to track heads, however it doesn’t allow one to pin-point with head.

Key Issues and ApproachesKey Issues and Approaches 1.Speed (in real time). 2.Affordability (with cheap easy-to-install, but low-quality USB cams)

3.3.RobustnessRobustness (to normal head motion).4.PrecisionPrecision (with pixel precision)

Image-based Face Tracking: - Uses global facial cues: skin colour, head shape, head motion

- Doesn’t require high-quility images, robust, but not precisenot precise

((44))

Should be used for Should be used for preciseprecise tracking. However, it’s tracking. However, it’s not robustnot robust.. [ Bradsky, Toyama, Gee, Cipolla, Zelinsky, Matsumoto, Yang, Baluja, Newman, …]

…”still not ready for practical implementation”

- Feature f is associated with vector Vf (obtained by

centering a mask on the feature)

- Features are tracked by template matching with Vf in the local area of interest (calculated with image-

based cues)

- The pixel u=(i,j) which has the largest score s(Vu, Vf) is returned

Feature-based Face TrackingFeature-based Face Tracking

Fig. 3. Tracking eyes

(from [Gorodnichy97]).

Question: What features to use?Question: What features to use?

Proposition 1: Robust and precise tracking can be achieved by designing an invariant to head motion feature template.

((55))

Features are conventionally thought of as visually distinctive (ie with large I(f) ).

Hence, the commonly used features are edge-based, such as corners of brows, eyes, lips, nostrils etc.

They however are • not robust

• not always visible

Desired feature properties:1. Uniqueness: s(Vf, Vu) min

2. Robustness: s(Vft=0, Vf

t) max

3. Continuity (for sub-pixel accuracy): the closer a pixel u in an image is to the pixel corresponding to f, the larger the score between Vu and Vf

(Then evidence-based convolution can be applied to refine feature position u)

Edge-based Features Edge-based Features – not good

((66)) Convex-shape features Convex-shape features – much betterDefinition 1: Convex-shape feature is defined as an extremum of a convex-shape surface

• Shape-from-Shading theory shows, that these features exhibit the desired properties (for the fixed camera-user-light configuration)

Nose featureNose featureDefinition 2: Nose feature is the extremum of the 3D nose

surface curvature defined as z=f(x,y) in camera centered coordinate system.

Thus defined, Nose feature is • Very robust• Can be detected with sub-pixel precision • PLUS, It is always visible!

((77)) NouseNouse TM TM Face Tracking TechnologyFace Tracking Technology

• Based on tracking the convex-shape nose feature.• Enables precise hand-free 2D control in a) joystick or b) mouse modes.

• Allows aiming and drawing with the nose.

Just think of your nose as a chalk or a joystick handle!NB: Left/Right head motion is very natural and can be easily applied for

control, provided it can tracked precisely.

• Affordable and downloadable. Uses a generic USB camera!

Zero intialization of Nouse Using Nouse for Painting

((88)) Performance: Robustness & PrecisionPerformance: Robustness & Precision

The range of head motion tracked

‘No’ motion

‘Yes’ motion

Robustness to rotation Robustness to scale

Test: The user rotates his head only! (the shoulders do not move)

((99)) Demo: Range of Tracked MotionDemo: Range of Tracked Motion

((1010)) On Importance of Two Cameras On Importance of Two Cameras

For humans: it is much easier to track with two eyes than with one eye.

Not only extends tracking from 2D to 3D,but also makes tracking more precise and robust!

For computers however: … 1. The relationship between “eyes” is not known. 2. Tracking of features is not robust (to rotation and scale)

StereoTracker from CVG NRC: Tracks face in 3D with two USB cams to control a virtual man, by using 1) Projective Vision Theory

and 2) robust Nose Feature Tracking

((1111)) StereoTracking with USB webcamsStereoTracking with USB webcams

Stage 1: Self-calibration• The relationship between the cameras is represented using the Fundamental Matrix F: (uleft, F uright)=0

• F can be found automatically for any two

cameras by observing the same scene

using www.cv.iit.nrc.ca/research/PVT:

find cornersmatchingfilteringrobust solution with 7 selected corners (RANSAC)F

Stage 2: Feature selection and calibration verification• Select features in one image• Verify that the epipolar line

passes thru each feature in the second image

• Use nose tip feature and two other common features (eg brow corners)• More at www.visioninterface.org/vi2002

((1212)) Using Nose for StereoTrackingUsing Nose for StereoTracking

Proposition 2 : With F known, the tracked 3D feature is the one that minimizes the epipolar error defined by

Proposition 3 : First detect convex-shape nose feature. Then use rigidity constraint to find other features.

((1313)) Demo: Stereotracking at WorkDemo: Stereotracking at Work

Robustness to Scale and Rotation around all (!) axis of rotation

((1414)) ConclusionsConclusions

AcknowledgementsAcknowledgements• Nouse TM is trademark of Computational Video Group IIT NRC• BubbleFrenzy game is provided by www.extendedreality.com

• Nose is a very unique feature. Humans are lucky to have it!• Nose allows us to track a face very robustly and precisely. • Pointing with Nose is natural.

This makes 2D perceptual user interfaces a reality!• Nose helps recovering other facial features. • Two cameras (even bad webcams) make tracking more robust.

This makes 3D face tracking affordable, precise and robust. • Use your Nose as MouseUse your Nose as Mouse! – Use NouseUse Nouse!

NouseTM is open for public evaluation atwww.cv.iit.nrc.ca/research/Nouse