3D Head Pose Estimation in Monocular Video Sequences Using...

1

3D Head Pose Estimation in Monocular VideoSequences Using Deformable Surfaces and Radial

Basis FunctionsMichail Krinidis

�, Nikos Nikolaidis

�and Ioannis Pitas

�Abstract— This paper presents a novel approach for estimating��

head pose in single-view video sequences. Following initial-ization by a face detector, a tracking technique that utilizes a��

deformable surface model to approximate the facial imageintensity is used to track the face in the video sequence. Headpose estimation is performed by using a feature vector which isa byproduct of the equations that govern the deformation of thesurface model used in the tracking. The afore-mentioned vectoris used as input in a Radial Basis Function (RBF) interpolationnetwork in order to estimate the

��head pose. The proposed

method was applied to IDIAP head pose estimation database.The obtained results show that the method can estimate the headdirection vector with very good accuracy.

Index Terms— Head pose estimation,��

deformable models,Radial Basis Function Interpolation.

1

I. INTRODUCTION

Head pose estimation in video sequences is a frequentlyencountered task in many computer vision applications. Invideo surveillance [1], head pose combined with prior knowl-edge about the world, enables the analysis of person motionand intentions. Head pose is also indicative for the focus ofattention of people, a fact that is very important in human-computer interaction [2]. Head pose can moreover be used innavigation of �� games [3], in visual communications [4], ��face reconstruction [5], etc. Head pose estimation is also usedas a preprocessing step in face detection [6], face recognition[7] and facial expression analysis [8], since these tasks arevery sensitive to even minor head rotations. Thus, the exactknowledge of the face pose is an essential problem which canboost the performance of such applications. A number of headpose estimation algorithms [9], [10] operate on stereoscopicsequences. However, stereoscopic information might not beavailable in the above-mentioned applications. As a result,research on single-view head pose estimation has been on therise during the last years.

The basic challenge in head pose estimation from single-view videos is to derive fast algorithms that do not require

1Copyright (c) 2008 IEEE. Personal use of this material is permitted.However, permission to use this material for any other purposes must beobtained from the IEEE by sending an email to [email protected].�Aristotle University of Thessaloniki, Department of Informatics, Box 451,

54124 Thessaloniki, Greece, email: � mkrinidi, nikolaid, pitas @aiia.csd.auth.gr Fax/Tel ++ 30 231 099 63 04, http://www.aiia.csd.auth.gr/.This work has been conducted in conjunction with the ”SIMILAR”European Network of Excellence on Multimodal Interfaces of the ISTProgramme of the European Union (http://www.similar.cc).

extensive preprocessing of the video sequence. Low-resolutionimages, image clutter, partial occlusions, unconstrained mo-tion, varying illumination conditions and complex backgroundcan mislead the head pose estimation procedure. Dependingon the way the face is treated, existing methods can be broadlydivided in three categories: approaches based on facial features, model-based algorithms, appearance-based algorithms.

A comparison of existing head pose estimation algorithms isgiven in [11], [12], [13].

The use of the spatial arrangement of important facial fea-tures for face pose estimation has been investigated by manyresearchers [14], [15], [16], [17]. In these approaches, the ��face structure is exploited along with a priori anthropometricinformation in order to define the head pose. The elliptic shapeof the face and the ratio of the major and minor axes of thisellipse, the mouth-nose region geometry, the line connectingthe eye centers, the line connecting the mouth corners andthe face symmetry are some of the geometric features used toestimate the �� head pose. In [17], five facial features, i.e.,the eye centers, the mouth centers and the nose are localizedwithin the detected face region. A weighting strategy is appliedafter the detection of the facial features, so as to estimate thefinal location of the five components more accurately. Theface pose is estimated by exploiting a metric which is based oncomparing the location of the acquired facial features with thecorresponding locations on a frontal pose. In [14], the locationof facial feature points is combined with color informationin order to estimate the �� face pose. The skin and the hairregion of the face is extracted, based on a perceptually uniformcolor system. Facial feature detection is performed on the faceregion and the bounding boxes of eyes, eyebrows, mouth andnose are defined. Then, corner detection is applied on thesebounding boxes. The left-most and the right-most corner areselected as the feature points of each facial feature. The ��head pose is inferred from both the facial features and theskin and hair region of the face. This category of algorithmshas a major disadvantage: their performance depends on thesuccessful detection of facial features which remains a difficultproblem, especially in non-frontal faces.

In the last few years many efforts have been spent onmodel-based head pose estimation algorithms [18], [19], [20].The basic idea in this category of methods is to use an apriori known �� face model which is mapped onto the ��

2

images. Once �� - �� correspondences are found between theinput data and the face model, conventional pose estimationtechniques are exploited to provide the �� face pose. Themain problem in these algorithms is to find in a robust waycharacteristic facial features that can be used to define the bestmapping of the �� model to the �� face images. In [20],the �� model is a textured triangular mesh. The similaritybetween the rendered (projected) model and the input facialimage is evaluated through an appropriate metric. The poseof the model that gives the best match is the estimated headpose. In [19], a cubic explicit polynomial in �� is used tomorph a generic face into the specific face structure using asinput multiple views. The estimation of the head structure andpose is achieved through the iterative minimization of a metricbased on the distance map (constructed by using a vector-valued Euclidean distance function).

Appearance-based approaches [21], [22], [23], achieve satis-factory results even with low-resolution head images. In theseapproaches, instead of using facial landmarks or face models,the whole image of the face is used for pose classification.In [23], a neural network-based approach for �� head poseestimation from low-resolution facial images captured by apanoramic camera is presented. A multi-layer perceptron istrained for each pose angle (pan and tilt) by feeding it withpreprocessed facial images derived from a face detectionalgorithm. The preprocessing consists of either histogramnormalization or edge detection. In [12], [21], an algorithmthat couples head tracking and pose estimation in a mixed stateparticle filter framework is introduced. The method relies on aBayesian formulation and the goal is to estimate the pose bylearning discrete head poses from training sets. Texture andcolor features of the face regions were used as input to theparticle filter. Two different variants were tested in [12]. Thefirst tracks the head and then estimates the head pose and thesecond jointly tracks the head and estimates the �� head pose.Support Vector Regression (SVR) [24], i.e., Support VectorMachines where the output domain contains continuous realvalues has been also used in appearance-based approaches.In [25], [26], two Sobel operators (horizontal and vertical)were used to preprocess the training images and the twofiltered images were combined together. Principal ComponentAnalysis (PCA) is then performed on the filtered image inorder to reduce the dimensionality of the training examples(facial images of known pose angles). SVM regression wasutilized in order to construct two pose estimators, for the tiltand yaw angles. The input to SVM was the PCA vectors andthe output was the estimated face angle. Since the final aimof the paper was multi-view face detection, these angles weresubsequently used for choosing the appropriate face detectoramong a set of detectors designed to operate on a differentview angle interval. In [27], SVR is used to estimate head posefrom range images. A three-level discrete wavelet transform isapplied on all the training range images and the LL sub-band(which accentuates pose-specific details, suppresses individualfacial details, and is relatively invariant to facial expressions)is used as input to two support vector machines that are trainedusing labelled examples to estimate the tilt and yaw angles.

The single-view �� head pose estimation approach pro-

posed in this paper belongs to the appearance-based methods.The method utilizes the deformable intensity surface approachproposed in [28], [29] for image matching. According tohis approach, an image is represented as a �� surface inthe so-called XYI-space by combining its spatial (XY) andintensity (I) components. A deformable surface model, whosedeformation equation is solved through modal analysis, issubsequently used to approximate this surface. Modal analysisis a standard engineering technique that has been introducedin the field of computer vision and image analysis in [30].Modal analysis allows effective computations and providesclosed form solutions of the deformation process and hasbeen used in a variety of different applications for solvingmodel deformations, i.e. for analyzing non-rigid object motion[31], for the alignment of serially acquired slices [32], formultimodal brain image analysis [33], segmentation of ��objects [34], image compression [35] and �� object tracking[36].

In our case, such a deformable intensity surface is used toapproximate, in the XYI-space, image regions depicting faces.The generalized displacement vector, which is an intermediatestep of the deformation process, is subsequently used in anovel way i.e. for both tracking the head and estimating its�� pose in monocular video sequences. Similarly to [36],the tracking procedure is based on measuring and matchingfrom frame to frame the generalized displacement vector ofa deformable model placed on the face. The generalized dis-placement vector is also used to train three RBF interpolationnetworks into estimating the pan, tilt and roll angles of thehead, with respect to the camera image plane. The tilt and thepan angles represent the vertical and the horizontal inclinationof the face, whereas the roll angle represents the rotation of thehead on the image plane (Figure 1). The proposed algorithmwas tested on the IDIAP head pose database [12] whichconsists of video sequences that were acquired in naturalenvironments and contain large rotations of the face. Thedatabase includes head pose ground truth information. Theresults show that the proposed algorithm can estimate the�� orientation vector of the face with an average error of

degrees.

Pan

Roll

Tilt

Fig. 1. Pan, tilt and roll head pose angles.

The remainder of the paper is organized as follows. In

3

Section II, a brief description of the deformation procedurein the XYI-space is presented. The tracking algorithm and thederivation of the feature vector used for pose estimation areintroduced in Section III. In Section IV, Radial Basis Functioninterpolation is reviewed and its use for pose estimation isexplained. The performance of the proposed technique isstudied in Section V. Finally, conclusions are drawn in SectionVI.

II. A FACIAL IMAGE DEFORMABLE MODEL

In this section, the physics-based deformable surface modelthat is used along with modal analysis to approximate imageregions depicting faces in the XYI-space will be brieflyreviewed. As already mentioned in Section I this approachhas been introduced in [28], [29] and has been used in ourcase with small modifications, described in this Section. Thenovelty of our approach lies in the utilization of the so-called generalized displacement vector, involved in the modalanalysis, for tracking the face and estimating the pose angles,as will be described in Section III.

According to [28], [29] an image can be represented asan intensity surface �� by combining its intensity�� and spatial �� components (Figure 2). The corre-sponding space is called the XYI space and a deformable meshmodel is used to approximate this surface. Modal analysis [30]is used to solve the deformation equations.

(a) (b)

(c) (d)

Fig. 2. (a) Facial image, (b) intensity surface representation of the image,(c) deformed model approximating the intensity surface, (d) deformed modelapproximating the intensity surface (only 25 % of the coefficients were usedin the deformation procedure).

The deformable surface model consists of a uniform quadri-lateral mesh of �� "!#�%$ nodes, as illustrated in Figure3. In this section, we assume that � � , �%$ are equal to theimage region height and width (in pixels) respectively, so thateach image pixel corresponds to one mesh node. Each node isassumed to have a mass & and is connected to its neighbors

with perfect identical springs of stiffness ' having naturallength (*) and damping coefficient + . Under the influence ofinternal and external forces, the mass-spring system deformsto a �� mesh representation of the image intensity surface, ascan be seen in Figure 2c.

kc

m

Nh

Nw

lo

Fig. 3. Quadrilateral surface (mesh) model.

In our case, the initial and the final deformable surfacestates are known. The initial state is the initial (planar) modelconfiguration and the final state is the image intensity surface,shown in Figure 2b. Therefore, it can be assumed that aconstant force load , is applied to the surface model [33].Since we are not interested in the deformation dynamics, wecan deal with the static problem formulation:-/. �0,�1 (1)

where-

is the �2!3� stiffness matrix, ,4�65 ,�78�919191:��,<;>=@?is the �A!B� vector whose elements are the � 3D externalforce vectors applied to the model and

.is the �6!/� nodal

displacements vector given by:. �C5 . 7D�919191E� .GF �919191E� . ;>=@?H� (2)

where.GF �I5 J F�K L ��J F�K M ��J F�K N = is the displacement of the O -th

node.Instead of finding directly the equilibrium solution of (1),

one can transform it by a basis change [30]:. �QPSR. � ;UT:;WV9;GXY F TG7[Z]\ R.GF � (3)

where R. is referred to as the generalized displacement vector,R.GF is the O -th component of R. and P is a matrix of order� , whose columns are the eigenvectors Z F of the generalizedeigenproblem: - Z F �_^]`FDa Z F � (4)

where a is the mass matrix of the model. The O -th eigen-vector Z F , i.e., the O -th column of P is also called the O -thvibration mode and ^ F is the corresponding eigenvalue (alsocalled vibration frequency). Equation (3) is known as modalsuperposition equation.

4

In practice, we wish to approximate nodal displacements.

by b. , which is the truncated sum of the �Sc low-frequencyvibration modes, where �ScEd_� :.Be b. � ;UfY F TG7�Z F R.GF 1 (5)

The eigenvectors Z F , Og�Ch��919191E��4c , form the reduced modalbasis of the system. This is the major advantage of modalanalysis: it is solved in a subspace corresponding to the �Bctruncated low-frequency vibration modes of the deformablestructure [31], [33], [30]. The number of vibration modes�4c retained in the surface description is chosen so as toobtain a compact but adequately accurate deformable surfacerepresentation. A typical a priori value for �Bc , covering manytypes of standard deformations, is equal to one quarter of thetotal number of the vibration modes.

A significant advantage of this formulation, in the full aswell as in the truncated modal space, is that the vibrationmodes (eigenvectors) Z F and the frequencies (eigenvalues) ^ Fof a plane topology have an explicit formulation [31] andthey do not have to be computed using eigen-decompositiontechniques:^ ` � i��<i c �]� '& j�k�l m `on�p i��"�Eqsr k�l m `ontp i c��%$uqwv � (6)

x�y K y f � i��<i c �z��{}| k p i��~�D�#�sh8��"� {}| k p i c �~�D� c �sh8��%$ � (7)

where i��9h��919191E��"��Ch , i�co�6��9h��919191��%$s��h , ��h��919191��"� , �:cC��h��919191E��%$ , ^ ` � i��<i�c@��^ `� ;GX�� f ,x�y K y f � i��<i c � is the ��g�� c � -th element of matrix Z � i��<i c � , whereZ � i��<i�c@�z� Z � ;GX�� f .In the modal space, (1) can be written as:R- R. � R,�� (8)

where R- ��P ? - P and R,Q��P ? , , , being the externalforce vector. Hence, by using (3), (6) and (7), equation (8)is simplified to �� scalar equations:^]`F RJ F�K � � R� F�K � � (9)

where i/�� and RJ F�K � is the i -th component of the O -thvector of R. .

In our case, the components of the forces in , along the �and � axes are taken to be equal to zero, i.e.

� F�K L � � F�K M �� .On the other hand, the components of these forces alongthe � (intensity) axis are taken to be proportional to theEuclidean distance between the point �� ) of theintensity surface and the corresponding model node positionin its initial configuration �� , i.e., equal to the intensity�� of pixel �� : �� L�� 7��;GX�� M8K N � � ��6�� ,where

�� L�� 7��;GX�� M8K N is the component along the � axis of the��%�3h8��%$ r � -th element , � L�� 7��;GX�� M of vector , . Moreover,the model is not allowed to deform along the x and yaxes i.e. RJ F�K L � RJ F�K M �2� . Hence, the ��!s� generalizeddisplacement vector can be simplified by ignoring these com-ponents and constructing a � -dimensional vector that containsonly the � components: R. ��5 RJ:78�919191:� RJ F �919191E� RJ�;WV9;GX�=@?��

5 RJ:7 K N �919191:� RJ F�K N �919191E� RJ�;WV9;GX K N = ? . By using (3), (6), (7) and(9) along with the force values mentioned above, one canexplicitly compute RJ:� as follows:RJ � F�� 7��;GX�� ;WVy TG7 � ;GXy f TG7 ��g��:c*� x�y K y f ��O��<i��h r ^ ` ��O��<i��}� � ;WVy TG7 � ;GXy f TG7 x `y K y f ��O��<i�� 1

(10)

It should be noted that the deformable model achieves onlyan approximation of the intensity surface of the target image.

For the problem at hand, facial areas of the video sequenceare described in terms of the vibrations of an initial model.Figure 2 illustrates the approximation by a deformable surfacemodel (Figures 2b,c) of the intensity surface (Figure 2b) of the�� image of a facial image shown in Figure 2a. The size ofthe model (in nodes) that was used to parameterize the imagesurface was equal to the image size (in pixels).

The generalized displacement vector �. of equations (8), (10)is exploited, as will be shown in the following sections, inorder to track facial regions on �� images and estimate the ��head pose. A flow diagram of the proposed algorithm is shownin Figure 4. The details of the algorithm will be provided inthe following sections.

III. FACE TRACKING AND DERIVATION OF THE POSEFEATURE VECTOR

A real-time frontal face detection algorithm [37] is appliedon the first image of the video sequence in order to initializethe face tracking and pose estimation procedure. The face de-tection scheme is based on simple features that are reminiscentof Haar basis functions [38]. These features were extended in[37] to further reduce the number of false alarms. The outputof the face detection procedure is a window around the facecenter, i.e., around the nose, that tightly encloses the face area.

Subsequently, a region based tracking approach similar tothe one proposed in [36] is used to track the central faceregion. The main difference between the tracking algorithmused here and the one introduced in [36], is that the latteraims at tracking feature points (by utilizing information fromthe surrounding region), while the former is adapted to trackregions. Additional information for the tracking algorithmalong with numerous experimental results can be found in [36].The following assumptions are adopted by this algorithm: The face window (bounding box) is of constant size,

i.e., the person does not move significantly towards oraway from the camera. This is a realistic assumptionfor most applications that require head pose estimation(e.g. human-computer interaction, face recognition, gazeestimation, etc.). A part of the human face is always visible (imagesdepicting the back of the head are not handled) and noocclusions occur. Both these assumptions are also realis-tic for most applications. If needed the tracking algorithmcan be enhanced with occlusion handling mechanisms.

Region-based tracking is performed by applying the de-formable model described in the previous section on a smallwindow � (e.g. one of dimensions ��#!t�� pixels) around

5

Input frame

Find window positionwhose CFV matchesbest the CFV of thewindow in previousframe

Sufficientmatchingquality

New position of facein the current frame

Size of searcharea is increased

Is the size ofthe searcharea below athreshold

Human face isconsidered as lost

Head pose anglesfor this frame

Template Face Tracking

no

yes

no

Fa

ce

De

tec

tio

n

Frontal Face Detectionlocates the imageregions of humanfaces

CFV is computed forall the positionsof the window withinthe search area

yes

CFV of the new positionis concatenated withprevious head pose anlge

3 trained RBFsare fed with theenriched CFV

Head pose estimation

First frame

yes

no

Fig. 4. Flow diagram of the proposed head pose estimation algorithm.

the face center �W 4�¡�� and evaluating the generalizeddisplacement vector �. }�� of equation (8) for this window:�. ��]�C5 RJ 7 ��¢� RJ ` ��¢�919191:� RJ ;W£:;g¤ ��<= ? � (11)

where ¥ is the time instance and �"¦ , � § are the heightand width of the deformable surface model (equal to thedimensions of the window). We will call vector �. ¢�� thecharacteristic feature vector (CFV). It has been shown [36]that this vector is the combination of the output of variousline and edge detection masks applied on the tracking window.As a result, tracking by matching the CFV along images issufficiently robust to illumination changes.

In order to find the position �W �G7 ��c~��c*� of the center

of the �%¦¨!t� § face window � in the next frame � �G7 ,the algorithm computes the CFV �. �G7 �~'��(~� for all windowswhose center �~'��(~� lie within a search region © with height�%¦«ª�¬< and width � § ª�¬< , centered at coordinates �� inimage � �G7 . The new location of the center of � is found asthe location ��c~��c*� in the search region whose CFV is closerto that of � in the current frame. More specifically:� �G7 �� c �� c �«��®6¯�°�±g² l m��³ ��´��. �G7 �~'��(~�G�µ�. ��9´9�¢� (12)

where 'B¶/·��#� ;W£�¸º¹~» � 7` �o19191:�u��%19191:�u� r ;W£�¸º¹~» � 7` ¼ and(o¶½·��¾� ;g¤]¸º¹~» � 7` ��19191E� ��#19191:�"� r ;g¤«¸º¹~» � 7` ¼ and ´À¿�´denotes the Euclidean distance.

6

The choice of the Euclidean distance was based on aset of experiments which aimed at providing results for theperformance of the proposed tracking algorithm when differentmeasures are used in order to select the next position of theface. The Euclidean distance was found to perform better thanboth the normalized correlation and Á Â L�K M �¾Â �G7� K ³ Á , where Â L�K Mis given by:

Â L�K M � ;W£�;g¤Y F TG7�ÃÃ RJ F �� ÃÃ 1 (13)

The experiment is described in detail in Section V.Since the motion characteristics of the face to be tracked

might change over time, i.e. the face can speed up or slowdown at certain video frames, the algorithm uses a searchregion © of variable size. For each video frame, the algorithmtries to locate the new center of the face window � usinginitially a small search region (e.g. Ä�!ÅÄ ). However, if forthe best candidate position the Euclidean distance (equation(12)) is above a certain threshold, the algorithm increases thesearch region size, trying to find a better match (a matchcorresponding to a matching error below the threshold) in thelarger search area. If this is again not feasible, the size increasecontinues up to a certain maximum region size ( ©A�Æ�� ).Some tracking results on the IDIAP database are presented inFigure 5.

In addition to its use for tracking, the CFV of the facewindow � is used for deriving the head pose. The CFVcontains information about the central visible face region,i.e. the region around the nose in the frontal images or thecheek in profile images (Figure 5), since its elements arerelated to the displacements of the deformable surface modelwhich approximates the intensity surface in this area. As theface/head changes orientation in the �� space, its projectionon the image ( �� space) changes. Thus, the fixed-size window� includes the central part of the face in different perspectiveviews (Figure 5). Hence, this information can be used toderive the orientation of the face. The characteristics of thedeformable surface model used in the experimental setup,were set so that the model is a rigid one. Thus, the finalstate of the deformable surface was a smoothed version ofthe face intensity surface, in order to be insensitive to clutter,differences between faces of persons and varying illuminationconditions. By utilizing the truncated space of the modalanalysis, one can reduce the size of the CFV feature length to��Ç�È of its original size (in our case to h�� elements, downfrom ��B!0�� elements), without losing significantinformation. The information contained in the CFV was usedalong with appropriately trained RBF interpolation networksto derive pose information, as will be described in the nextSection.

IV. RADIAL BASIS FUNCTION INTERPOLATION

The design of an interpolation system can be seen as asurface fitting problem in a high-dimensional space [39]. Insuch a situation, learning is equivalent to finding a smoothsurface, which interpolates (or approximates) the training data.

Radial Basis Functions have been used in our case for thispurpose. RBFs were chosen for the following reasons: they have a simple structure, they have the property of “best approximation”, they have the property of best learning and reduced

sensitivity to the order of presentation of training data.Let us assume a set of �ÊÉ -dimensional vectors5 ËG7ÌË ` 19191ÍË:;u= , Ë F ¶¨ÎWÏ and a set of scalar values5 (Ð7Ñ( ` 19191�(@;>= , ( F ¶4Î , which correspond to samples of an

unknown function�ÓÒ ÎgÏÔ®ÕÎ , i.e.

� ��Ë F � ��( F . The RBFinterpolation solves the problem of finding a smooth functionb� ��Ë�� that satisfies the following relation:b� ��Ë F �]�0( F �ÖOg�Óh��919191E��/� (14)

i.e. interpolates ��Ë F ��( F � and hopefully approximates suffi-ciently well

� ��Ë�� elsewhere.Radial basis function interpolation is defined as:� ��Ë��z� ;Y 7Q× F © F ��Ë��¢� (15)

where © F are the chosen radial basis functions and × F arelinear weights used to combine the RBF.

In our case, isotropic Gaussian radial basis functions wereused: © ��´¢Ë�� . ´}`��z�_Ø}Ù�Ú�� h��Û ` ´¢Ë�� . ´}`�´9�¢1 (16)

The parameters to be learnt in such an RBF interpolation arethe weights × F in (15), the means

. Fand the standard devia-

tions Û F of the RBF © F . Many approaches for training an RBFinterpolation network have been proposed [39]. According toone of them, the vectors

. Fcan be chosen to be equal to the

vectors Ë F , i.e.. F �µË F , so that each RBF is centered at one

training sample Ë F . The standard deviation for all the Gaussianbasis functions can be set equal to [39]:Û/� ÜÝ �� (17)

where � is the number of the training data and Ü is the maxi-mum Euclidean distance between any two training samples Ë F ,Ë � . This choice of Û ensures that the RBFs are neither too flatnor too peaky and that they will “fill” the space sufficientlywell. Once the RBF have been fully defined, a straightforwardprocedure for solving for the weights × F in (15), so that thenetwork satisfies (14) is to use the pseudoinverse method [40]on the following linear system:Þ�ß ��à (18)

where the element á � F of R is equal to © F ��´¢Ë � �CË F ´ ` � ,vectors

ßand à contain the weights × F and the scalar values( F respectively and i��Oâ��h��919191E�� . Obviously, this linear

system has the following solution:ß � Þ � à�� (19)

where

Þ � is the pseudoinverse of

Þ.

Radial basis function interpolation was used in our case,to estimate values for the three pose angles (pan, tilt, roll) offrame �� by using as input the CFV �. of the corresponding

7

Fig. 5. Face tracking results obtained in video sequences depicting persons in various poses.

frame. Thus, the input vector in the training and testingprocedure was the CFV of the area over the face that wasderived by the tracking algorithm, whereas the output wasthe pose angle value. More specifically, three RBF networks,each handling a different angle (pan, tilt, roll) of the facepose, were used. The value range of each of these parametersvaried from 5 �>ã��w19191�ã��D= for pan, 5 �>ä��w19191�ä��D= for tilt and5 �>��w19191��D= for roll. During the training of the RBF system,i.e. during the evaluation of the mean and variance of theRBFs and the weights × F , a set of CFVs �. along with thecorresponding pose angle derived from the ground truth, wasused as input. This set was a subset of the CFV-pose anglepairs, derived from the training sequences and the ground truth.In accordance to the training procedure described above, thenetwork consisted of as many RBFs as the training samples�. . In other words, the number of RBFs was equal to thenumber of frames of the training sequences. To performtesting, an unlabelled feature vector Ë c �Æ�. f is used as aninput. The trained RBF system that handles a certain poseangle, interpolates the test data in the surface derived fromthe training data and estimates the pose angle value.

In order to increase the performance of the system in termsof pose angles estimation accuracy, the input vectors of theRBF interpolation were expanded to consist of a concatenationof the CFV �. with the pose angle of the previous frame. Morespecifically, the RBF interpolation networks were fed at timeinstant ¥ with vectors of the form:5��. Á å � 7�= ? (20)

where å � 7 denotes the pan, tilt or roll angle in the previousframe. During training the ground truth pose angles were used,whereas during testing, the estimates from the application ofthe system in the previous frame were inserted. The use of thepose angle of the previous frame into the input vectors intro-

duces the notion of coherence. Indeed, if videos of sufficientframe rate are available and the head movements are smooth,the pose angles in the current frame will not differ significantlyfrom those in the previous frame. Thus, by including thisinformation the system can cope more efficiently with trackingerrors. On the other hand, if abrupt head motions occur, thepose angles of the previous frame would be uncorrelated tothose of the current frame and this information will affect theresult in a negative way. However, such movements are ratherinfrequent and the user can hope that the tracking will succeedin following the target and thus, will provide correct valuesfor the CFV part of the input vector.

The head pose for the first frame during the testing pro-cedure was assumed to be known. More specifically, it wasassumed that on the first frame the person under examinationis looking straight ahead with no rotations in any of the threeaxes, so that all three angles (pan, roll, tilt) are equal to zero.This scenario is not unrealistic, since in many applicationsthe face is in a frontal/neutral pose at the beginning of thevideo sequence and many algorithms that are applied on facialimages adopt the same assumption. Moreover, the face detectorused to initialize the tracking and head pose estimation, is afrontal face detector and thus, this assumption is necessary forits proper operation.

V. PERFORMANCE EVALUATION

A. Evaluation Setup and Dataset

Experiments were conducted to evaluate the performanceof the proposed algorithm in video sequences recorded underrealistic conditions.

The aim of the first set of experiments was two-fold: toevaluate the performance of the tracking part of the algorithmand find a suitable metric for measuring the similarity of

8

TABLE IPrecision and recall of the region tracking algorithm when using Euclidean

distance, normalized correlation and absolute difference of æ�çè�é ê (13) in

order to match CFVs between consecutive frames.

Metric P REuclidean distance 96.9 95.2Normalized correlation 89.7 85.3ë æ�çè�é êgì æ ç@í�îï é ð ë 96.1 94.6

CFVs. The performance of the algorithm was evaluated at theobject level, i.e., on the basis of whether the entire object underexamination was correctly tracked or not. True positives ( ñuò ),false positives ( óâò ) and false negatives ( óâ� ) were obtainedusing manually extracted ground truth data. Cases where thebounding box of the tracked face region contained more thanÄD��È of the face, were considered as ñuò . When more than��È of the bounding box consisted of background area, theframe was considered as contributing to óâò . Situations wherethe tracking algorithm loses the target (i.e., it tracked thebackground), were considered as contributing to óâ� . Basedon these numbers, the well-known precision ( òô� ?�õ?�õ �:ö õ )and recall measures ( ©÷� ?�õ?�õ �:ö:; ) were calculated. Thealgorithm was tested in Ç indoor and outdoor video sequences( Ç�ä�� frames) with motions frequently encountered in trackingsituations, such as translation and rotation. The studio (in-door) sequences depict one person moving on a predefinedor random trajectory, under either optimal (uniform lighting)or suboptimal (hard shadows and bright/dark areas) lightingconditions created using the studio’s lighting equipment. Theoutdoor video sequences depict one person moving on arandom trajectory under realistic conditions. The results for the� different metrics, namely the Euclidean distance, the normal-ized correlation and the absolute difference of Â L�K M in (13) aresummarized in Table I. One can see that the Euclidean distanceachieves the best performance. In addition, these experimentalresults provide quantitative evidence regarding the satisfactoryperformance of the tracking algorithm. Since the trackingalgorithm is not the focus of this paper and the algorithmused here resembles the one proposed in [36], no additionalexperiments were performed towards this direction. Readersare encouraged to consult [36] for more tracking experimentswith various types of motion and lighting conditions.

In the second set of experiments, the proposed method wasused to estimate the �� head pose on parts of the IDIAPvideo database [12]. This database was selected because itcontains ground truth data and because the video sequenceswere recorded under realistic conditions and depict realisticsituations. Each of the �� video clips lasts from

to h��

minutes and has a resolution of ��ä��W!w��ø�ø pixels, ��Ç frames persecond. There are two parts of the database. The one depictsan office environment and the other a meeting environment.The main scenario of the IDIAP database is that subjects actas in their daily life in their office or in a meeting. In total,h�ä different subjects participate in the video database. Thedatabase contains head pose ground truth in the form of pan,tilt and roll angles (i.e. Euler angles with respect to the cameracoordinate system) for each frame of the video sequences.

Eleven of the video sequences were used for training theRBF interpolation network and the rest for testing the system.In total, ã�Ç�� frames were used to train the system. Thus,an equal amount of RBFs were utilized. For each RBF, oneneeds to store its center (in our case a h�� -dimensional vectorfor the input and one value for the output) the standarddeviation (same for all RBFs) and the corresponding weight.The parameters of the deformable surface model were definedso as to give a smooth representation of the face intensitysurface, i.e. a ratio �ù �úh�� ( ' being the stiffness of thesprings and & the mass of the nodes) was used. Thus, thefinal state of the deformable surface was a smoothed version ofthe face intensity surface, in order to be insensitive to clutter,differences between faces of persons and varying lightingconditions.

One minute, i.e. h8Ç�� frames, were selected from each of theh8� test video sequences. The selection of the part of each videosequence that was used in the experiments was not random.We tried to choose h8Ç�� frames depicting rich person activity,covering as many head pose angles as possible. Face detectionwas performed in the first frame of the selected part of eachvideo sequence and the tracking technique described in SectionIII was applied to the rest of the frames in order to track theposition of the face. For each frame, the CFV was calculatedin order to be further used either for training or testing theRBF network. The tracking results showed that the proposedtracking algorithm offers satisfying results. However, in someframes the tracking algorithm can lose part of the target, i.e.the face rectangle might contain part of the background. Thiscan happen either when the movement of the face is abrupt andextreme or in instances that do not involve such movements. Inthe first case, the head pose of the previous frame does not helpin the estimation of the pose angle in the current frame and thealgorithm might provide estimates with significant error (e.g.ø�û ). In the second case, the head pose of the previous framecan improve the estimation in the current frame. In both cases,the tracking algorithm usually recovers within a few frames.

The metrics used for the evaluation of the proposed algo-rithm was the error (absolute difference) in degrees betweenthe ground truth and the estimated value for pan (PE), tilt (TE)and roll (RE) angles. Moreover, the head pose defines a vectorin the �� space which indicates where the head is pointingat. The angle (DE) between the pointing vector defined bythe head pose ground truth and the pose estimated by theproposed system was used as a pose estimation error measure,as proposed in [12]. DE is given by the following equation:��ü�� h�ø��pþý +}ÿ��

�Y F TG7 ��D��O��¬D��O��¢� (21)

where ý +}ÿ�� is the inverse cosine, ��O�� and ��¬D��O�� are the O -thcomponents of the pointing vectors �� and ��¬ respectively,which are constructed from the ground truth data and theestimated pose respectively, as follows [12]:

�>�C5 k�l m ��å��¢�9� k�l m ��å ��«{}| k ��å��¢��{}| k ��å ��«{}| k ��å��<=�� (22)

where 3��·�� ¼ , å�� and å are the pan and tilt angles ofthe face. Because this vector depends only on the pan and

9

tilt values, another error metric was also used. This metricdenoted by AE, is the angle between the unit length vectors� , � ¬ defined by rotating the “neutral” direction vector � y �5 h��D= ? by the estimated and ground truth pan, tilt and rollangles å�� , å , åDª as follows:

� � Þ �� Þ �� Þ � ¸ � y � (23)

where

Þ�� ,Þ�� and

Þ� ¸ are the rotation matrices for the pan,

tilt and roll axes defined in Table II.

B. Experimental Results

Figure 6 depicts the estimated pan, tilt and roll angles alongwith the corresponding ground truth values over time for threeof the test video sequences. Moreover, the absolute error indegrees between the three estimated angles and the groundtruth values, namely the values of PE, TE, RE over time,are presented in Figures 7. One can see that the proposedalgorithm can estimate the �� head pose with very goodaccuracy. In certain cases, when the movement of the faceis extreme and sudden, or in large rotation angles, the erroris bigger than usual. The angle AE between the �� directionvector defined by the estimated angles and the direction vectordefined by the ground truth is shown in Figure 8. For example,the absolute error averaged over all frames for the first testvideo sequence (Figure 8a) is h�1 ÄDä degrees and its variance is��1 ä�ã degrees.

Tables III, IV and V present the mean, variance and medianvalues (with respect to time) of the aforementioned errors forfive of the test video sequences. Average values over all videosequences are also provided. One can see that the averagevalue (over all sequences) of AE is ��1 �� degrees and theaverage value of variance is ��1 ��h degrees. The average valuesfor the other error metrics are also very small. Thus it isevident that the proposed system can accurately estimate the�� face pose in a video sequence.

The values of PE, TE, RE, DE errors obtained by the bestalgorithm in [12] are reported in Table VI for comparisonpurposes. One can see, that the proposed algorithm achievesfar better results than the method in [12]. For example, themean error in the pointing vector (DE) for the proposedalgorithm is ��1 ø�û , a large improvement over the error of ��h�1 ��ûachieved by [12].

In terms of computational complexity, ��1 � seconds per frameare required for the tracking procedure and ��1 ��Ç secondsper frame for the pose estimation procedure on an IntelPentium

( ��1 ��h GHz) processor PC with h�1 Ç GB of RAM.

Thus the algorithm requires in total ��1 ��Ç seconds per framein this modest hardware configuration and without any codeoptimization.

VI. CONCLUSION

A �� head pose estimation algorithm based on the useof a parameterized �� physics-based deformable model wasproposed in this paper. In this approach, the intensity surfaceof the facial area is represented by a �� physics-baseddeformable model. We have shown how to tailor the modeldeformation equations to efficiently track the human face in

TABLE VIMean, variance and median values for the PE, TE, RE and DE errors (in

degrees) achieved on the IDIAP database by the best algorithm presented in

[12].

PE TE RE DEmean 8.7 19.1 9.7 21.3

variance 9.1 15.41 7.1 15.2median 6.2 14.0 8.6 14.1

a video sequence and concurrently feed three properly trainedRBF interpolation networks that estimate the pan, tilt and rollangles of the face. Results obtained on the IDIAP databaseshow that the proposed method produces accurate results.

Future work includes the use of SVM regression instead ofRBF interpolation networks in order to estimate the �� facepose. Additionally, we plan to test our system on an even largerdata set and explore the influence of the adopted classifier intoour method’s performance.

REFERENCES

[1] M. Doi and Y. Aoki, “Real-time video surveillance system using omni-directional image sensor and controllable camera,” in Proceedings ofSPIE, Real-Time Imaging VII, vol. 5012, April 2003, pp. 1–9.

[2] U. Weidenbacher, G. Layher, P. Bayerl, and H. Neumann, “Detectionof head pose and gaze direction for human-computer interaction,” inPerception and Interactive Technologies, Springer Berlin / Heidelberg,vol. 4021/2006, June 2006, pp. 9–19.

[3] P. Lu, X. Zeng, X. Huang, and Y. Wang, “Navigation in 3d game bymarkov model based head pose estimating,” in Proceedings of the ThirdInternational Conference on Image and Graphics (ICIG’04), December2004, pp. 493–496.

[4] B. Yip and J. Jin, “Pose determination and viewpoint determinationof human head in video conferencing based on head movement,” inProceedings of the 10th International Multimedia Modelling Conference,Brisbane, Australia, January 2004, pp. 130–135.

[5] ——, “3d reconstruction of a human face with monocular camera basedon head movement,” in Proceedings of the Pan-Sydney area workshopon Visual information processing (VIP 2003), Darlinghurst, Australia,2003, pp. 99 – 103.

[6] J. Ng and S. Gong, “Multi-view face detection and pose estimation usinga composite support vector machine across the view sphere,” in IEEEInternational Workshop on Recognition, Analysis, and Tracking of Facesand Gestures in Real-Time Systems, Corfu, Greece, September 1999, pp.14 – 21.

[7] H. Song, U. Yang, J. Kim, and K. Sohn, “A 3d head pose estimationfor face recognition,” in Proceedings of the IASTED InternationalConference on Signal and Image Processing (SIP 2003), Honolulu,USA, August 13-15 2003, pp. 255–260.

[8] C. Chien, Y. Chang, and Y. Chen, “Facial expression analysis undervarious head poses,” in Proceedings of Third IEEE Pacific Rim Con-ference on Multimedia, vol. 2532/2002, Taiwan, December 16-18 2002,pp. 199–212.

[9] E. Seemann, K. Nickel, and R. Stiefelhagen, “Head pose estimationusing stereo vision for human-robot interaction,” in Proc. of the SixthInternational Conference on Automatic Face and Gesture Recognition(AFGR04), Seoul, Korea, May 17-19 2004, pp. 626–631.

[10] R. Yang and Z. Zhang, “Model-based head pose tracking with stereovision,” in Proc. of the Sixth International Conference on AutomaticFace and Gesture Recognition (AFGR02), D.C., USA, May 2002, pp.255–260.

[11] L. M. Brown and Y.-L. Tian, “Comparative study of coarse headpose estimation,” in IEEE Workshop on Motion and Video Computing,December 2002, pp. 125– 130.

[12] S. Ba and J.-M. Odobez, “Evaluation of multiple cue head poseestimation algorithms in natural environments,” in IEEE InternationalConference on Multimedia and Expo (ICME), Amsterdam, July 2005,pp. 1330–1333.

10

TABLE IIRotation matrices for the pan, tilt and roll axes.

�� T �� 7 ) ))�� ! " � �� )# �! " � �� $% �� T �� A)& �! " � �� ) 7 )� �! " � �� A)'�� $% �� ¸ T �� ¸�� ! " � � ¸��A) �! " � � ¸�� ¸��A)) ) 7 $%

200 300 400 500 600 700 800−60

−40

−20

0

20

40

60

80

frame index

degr

ees

estimated angleground truth

0 100 200 300 400 500 600 700−60

−40

−20

0

20

40

60

80

frame indexde

gree

s


0 50 100 150 200 250 300 350 400−60

−40

−20

0

20

40

60

80

frame index

degr

ees


(a) (b) (c)

0 200 400 600 800 1000 1200−40

−30

−20

−10

0

10

20

30

40

frame index

degr

ees


0 100 200 300 400 500 600 700−30

−20

−10

0

10

20

30

frame index

degr

ees


0 50 100 150 200 250 300 350 400−30

−20

−10

0

10

20

30

frame index

degr

ees


(d) (e) (f)

0 200 400 600 800 1000 1200−15

−10

−5

0

5

10

15

time in frames

degr

ees


0 100 200 300 400 500 600 700−10

−5

0

5

10

15

20

frame index

degr

ees


0 50 100 150 200 250 300 350 400−15

−10

−5

0

5

10

frame index

degr

ees


(g) (h) (i)

Fig. 6. The angles (in degrees) estimated by the proposed system and the corresponding ground truth values for parts of three test video sequences thatcontain significant head motion: (a)-(c) pan angles, (d)-(f) tilt angles and (g)-(i) roll angles.

TABLE IIIMean values for the PE, TE, RE, DE and AE errors (in degrees) achieved on the IDIAP database by the proposed algorithm.

Test Videos MeanPE TE RE DE AE

sequence 3 1.2665 1.0143 0.3245 2.2453 2.0133sequence 5 3.2214 4.3564 0.7980 3.9876 3.4331sequence 7 2.1234 1.4612 0.4125 3.0945 2.5503sequence 10 3.6889 3.5433 0.8764 4.3321 3.1902sequence 12 0.7650 2.3767 0.2984 5.6501 4.5774

average values (all sequences) 2.2114 2.5902 0.5328 3.8712 3.1997

[13] E. Murphy-Chutorian and M. Trivedi, “Head pose estimation in com-puter vision: A survey,” IEEE Transactions on Pattern Analysis and

Machine Intelligence, Accepted for future publication.

[14] Q. Chen, T. Shimada, H. Wu, and T. Shioyama, “Head pose estimation

11

0 200 400 600 800 1000 12000

2

4

6

8

10

12

frame index

erro

r in

deg

rees

0 100 200 300 400 500 600 7000

2

4

6

8

10

12

14

16

18

20

frame index

erro

r in

deg

rees

0 50 100 150 200 250 300 350 4000

5

10

15

20

25

30

frame index

erro

r in

deg

rees

(a) (b) (c)

0 200 400 600 800 1000 12000

1

2

3

4

5

6

7

8

9

10

frame index

erro

r in

deg

rees

0 100 200 300 400 500 600 7000

2

4

6

8

10

12

14

16

18

20

frame index

erro

r in

deg

rees

0 50 100 150 200 250 300 350 4000

2

4

6

8

10

12

frame index

erro

r in

deg

rees

(d) (e) (f)

0 200 400 600 800 1000 12000

0.5

1

1.5

2

2.5

3

frame index

erro

r in

deg

rees

0 100 200 300 400 500 600 7000

0.5

1

1.5

2

2.5

3

3.5

4

4.5

frame index

erro

r in

deg

rees

0 50 100 150 200 250 300 350 4000

0.5

1

1.5

2

2.5

3

3.5

frame index

erro

r in

deg

rees

(g) (h) (i)

Fig. 7. The absolute error in degrees between the angles estimated by the proposed system and the ground truth angles for parts of three test video sequencesthat contain significant head motion: (a)-(c) pan angles, (d)-(f) tilt angles and (g)-(i) roll angles.

0 200 400 600 800 1000 12000

1

2

3

4

5

6

7

8

9

frame index

erro

r in

deg

rees

0 100 200 300 400 500 600 7000

2

4

6

8

10

12

14

16

18

frame index

erro

r in

deg

rees

0 50 100 150 200 250 300 350 4000

5

10

15

20

25

frame index

erro

r in

deg

rees

(a) (b) (c)

Fig. 8. The angle AE in degrees between the (*) direction vector of the angles estimated by the proposed system and the direction vector of the groundtruth angles for parts of three test video sequences that contain significant head motion.

using both color and feature information,” in 15th International Con-ference on Pattern Recognition (ICPR’00), vol. II, Barcelona, Spain,

September 2000, pp. 842–845.[15] T. Huang, A. Bruckstein, R. Holt, and A. Netravali, “Uniqueness of 3d

12

TABLE IVVariance values for the PE, TE, RE, DE and AE errors (in degrees) achieved on the IDIAP database by the proposed algorithm.

Test Videos VariancePE TE RE DE AE



TABLE VMedian values for the PE, TE, RE, DE and AE errors (in degrees) achieved on the IDIAP database by the proposed algorithm.

Test Videos MedianPE TE RE DE AE



pose under weak perspective: A geometrical proof,” IEEE Transactionson Pattern Analysis and Machine Intelligence, vol. 17, no. 12, pp. 1220–1221, December 1995.

[16] Z. Liu and Z. Zhang, “Robust head motion computation by takingadvantage of physical properties,” in IEEE Workshop on Human Motion,Los Alamitos, CA, USA, December 2000, pp. 73–77.

[17] Y. Hu, L. Chen, Y. Zhou, and H. Zhang, “Estimating face pose byfacial asymmetry and geometry,” in IEEE Proceedings of the SixthInternational Conference on Automatic Face and Gesture Recognition(FGR 2004), Seoul, Korea, May 2004, pp. 651– 656.

[18] M. L. Cascia and V. Athitsos, “Fast, reliable head tracking under varyingillumination: An approach based on registration of texture-mapped 3dmodels,” IEEE Transactions on Pattern Analysis and Machine Intelli-gence, vol. 22, no. 4, pp. 322–336, April 2000.

[19] C. Zhang and F. S. Cohen, “3-d face structure extraction and recognitionfrom images using 3-d morphing and distance mapping,” IEEE Transac-tions on Image Processing, vol. 11, no. 11, pp. 1249–1259, November2002.

[20] B. Kwolek, “Model based facial pose tracking using a particle filter,” inIEEE Proceedings of the Geometric Modeling and Imaging New Trends(GMAI’06), July 2006, pp. 203– 208.

[21] S. O. Ba and J. Odobez, “A probabilistic framework for joint headtracking and pose estimation,” in Proceedings of the 17th InternationalConference on Pattern Recognition (ICPR04), vol. 4, 2004, pp. 264 –267.

[22] S. Z. Li, X. Lu, X. Hou, X. Peng, and Q. Cheng, “Learning multiviewface subspaces and facial pose estimation using independent componentanalysis,” IEEE Transactions on Image Processing, vol. 14, no. 6, pp.705–712, June 2005.

[23] R. Stiefelhagen, Y. Jie, and A. Waibel, “Simultaneous tracking ofhead poses in a panoramic view,” in Proceedings of 15th InternationalConference on Pattern Recognition, 2000, vol. 3, Barcelona, Spain,September 2000, pp. 722–725.

[24] V. Vapnik, The nature of statistical learning theory. Springer Verlag,1995.

[25] Y. Li, S. Gong, J. Sherrah, and H. Liddell, “Support vector machinebased multi-view face detection and recognition,” Image and VisionComputing, vol. 1, no. 5, p. 413427, May 2004.

[26] Y. Li, S. Gong, and H. Liddell, “Support vector regression and clas-sification based multi-view face detection and recognition,” in IEEEInternational Conference on Automatic Face and Gesture Recognition,Grenoble, France, March 2004, pp. 300–305.

[27] A. Rajwade and M. Levine, “Facial pose from 3d data,” Image andVision Computing, vol. 24, no. 8, pp. 849–856, August 2006.

[28] B. M. C. Nastar and A. Pentland, “Generalized image matching:Statistical learning of physically-based deformations,” in 4th EuropeanConference on Computer Vision (ECCV’96), vol. 1, Cambridge, UK,April 1996, pp. 589–598.

[29] B. Moghaddam, C. Nastar, and A. Pentland, “A bayesian similaritymeasure for direct image matching,” in International Conference onPattern Recognition (ICPR 1996), Vienna, Austria, August 1996, pp.350–358.

[30] A. Pentland and S. Sclaroff, “Closed-form solutions for physically basedshape modeling and recognition,” IEEE Transactions on Pattern Analysisand Machine Intelligence, vol. 13, no. 7, pp. 715–729, July 1991.

[31] C. Nastar and N. Ayache, “Frequency-based nonrigid motion analysis:Application to four dimensional medical images,” IEEE Transactions onPattern Analysis and Machine Intelligence, vol. 18, no. 11, pp. 1069–1079, November 1996.

[32] S. Krinidis, C. Nikou, and I. Pitas, “Reconstruction of serially acquiredslices using physics-based modelling,” IEEE Transactions on Informa-tion Technology in Biomedicine, vol. 7, no. 4, pp. 394–403, December2003.

[33] C. Nikou, G. Bueno, F. Heitz, and J. Armspach, “A joint physics-basedstatistical deformable model for multimodal brain image analysis,” IEEETransactions on Medical Imaging, vol. 20, no. 10, pp. 1026–1037,October 2001.

[34] C. Nastar and N. Ayache, “Fast segmentation, tracking, and analysisof deformable objects,” in Proceedings of the Fourth InternationalConference on Computer Vision (ICCV’93), Berlin, Germany, May1993, pp. 11–14.

[35] M. Krinidis, N. Nikolaidis, and I. Pitas, “The discrete modal transformand its application to lossy image compression,” Signal Processing:Image Communication, vol. 22, no. 5, pp. 480–504, June 2007.

[36] ——, “2d feature point selection and tracking using 3d physics-baseddeformable surfaces,” IEEE Transactions on Circuits and Systems forVideo Technology, vol. 17, no. 7, pp. 876 – 888, July 2007.

[37] R. Lienhart and J. Maydt, “An extended set of Haar-like features forrapid object detection,” in IEEE International Conference on ImageProcessing (ICIP02), Rochester, New York, USA, September 2002, pp.900–903.

[38] P. Viola and M. J. Jones, “Robust real-time object detection,” CambridgeResearch Laboratory, Tech. Rep. 01, 2001.

[39] S. Haykin, Neural Networks: A comprehensive Foundation, 2nd Edition.Englewood Cliffs: Macmillan Publishing Company, 1998.

[40] D. Broomhead and D. Lowe, “Multivariable functional interpolationand adaptive networks,” Complex Systems, vol. 2, pp. 321–355, 1998.

13

Michail Krinidis received the B.S. degree fromthe Department of Informatics, Aristotle Universityof Thessaloniki, Thessaloniki, Greece, in 2002. Heis currently pursuing a Ph.D. degree in the samedepartment while also serving as a teaching assistant.His current research interests lie in the areas of 2Dtracking, face detection and 3D head pose estimationin image sequences.

Nikos Nikolaidis received the Diploma of Electri-cal Engineering and the Ph.D. degree in electricalengineering from the Aristotle University of Thes-saloniki, Thessaloniki, Greece, in 1991 and 1997,respectively. From 1992 to 1996, he was TeachingAssistant at the Departments of Electrical Engi-neering and Informatics at the Aristotle Universityof Thessaloniki. From 1998 to 2002, he was aPostdoctoral Researcher and Teaching Assistant atthe Department of Informatics, Aristotle Universityof Thessaloniki, where he is currently an Assistant

Professor. He is the co-author of the book 3-D Image Processing Algorithms(New York: Wiley, 2000). He has co-authored 11 book chapters, 27 journalpapers, and 84 conference papers. His research interests include computergraphics, image and video processing and analysis, computer vision, copy-right protection of multimedia, and 3-D image processing. Dr. Nikolaidiscurrently serves as Associate Editor for the International Journal of InnovativeComputing Information and Control, the International Journal of InnovativeComputing Information and Control Express Letters and the EURASIPJournal on Image and Video Processing.

Ioannis Pitas received the Diploma of ElectricalEngineering in 1980 and the PhD degree in Elec-trical Engineering in 1985 both from the AristotleUniversity of Thessaloniki, Greece. Since 1994, hehas been a Professor at the Department of Informat-ics, Aristotle University of Thessaloniki. From 1980to 1993 he served as Scientific Assistant, Lecturer,Assistant Professor, and Associate Professor in theDepartment of Electrical and Computer Engineeringat the same University. He served as a VisitingResearch Associate or Visiting Assistant Professor

at several Universities. He has published 153 journal papers, 400 conferencepapers and contributed in 22 books in his areas of interest and edited or co-authored another 5. He has also been an invited speaker and/or member of theprogram committee of several scientific conferences and workshops. In thepast he served as Associate Editor or co-Editor of four international journalsand General or Technical Chair of three international conferences. His currentinterests are in the areas of digital image and video processing and analysis,multidimensional signal processing, watermarking and computer vision.

3D Head Pose Estimation in Monocular Video Sequences Using...

Documents

Transcript of 3D Head Pose Estimation in Monocular Video Sequences Using...