Research Article A Multimodal User Authentication System...

9
Research Article A Multimodal User Authentication System Using Faces and Gestures Hyunsoek Choi 1 and Hyeyoung Park 2 1 School of Electrical Engineering and Computer Science, Kyungpook National University, Deagu 702-701, Republic of Korea 2 School of Computer Science and Engineering, Kyungpook National University, Deagu 702-701, Republic of Korea Correspondence should be addressed to Hyeyoung Park; [email protected] Received 26 September 2014; Accepted 19 November 2014 Academic Editor: Sabah Mohammed Copyright © 2015 H. Choi and H. Park. is is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. As a novel approach to perform user authentication, we propose a multimodal biometric system that uses faces and gestures obtained from a single vision sensor. Unlike typical multimodal biometric systems using physical information, the proposed system utilizes gesture video signals combined with facial images. Whereas physical information such as face, fingerprints, and iris is fixed and not changeable, behavioral information such as gestures and signatures can be freely changed by the user, similar to a password. erefore, it can be a countermeasure when the physical information is exposed. We aim to investigate the potential possibility of using gestures as a signal for biometric system and the robustness of the proposed multimodal user authentication system. rough computational experiments on a public database, we confirm that gesture information can help to improve the authentication performance. 1. Introduction With the growing need for secure authentication methods, various biometric signals are being actively studied. One recent trend is the use of multimodal data for achieving high reliability [13]. However, in general, multimodal bio- metric systems require multiple sensors, which result in high developmental costs. As a new attempt for achieving high reliability and low cost, this paper proposes a novel multimodal biometric system that uses two heterogeneous biometric signals obtained from a single vision sensor: facial image and gesture video. Face is a representative of physical biometric signals, and many studies have been carried out on developing reliable face recognition systems [4, 5]. However, the performance of face recognition systems is easily influenced by various envi- ronmental factors such as illumination, expression, pose, and occlusion. Despite a significant number of studies conducted to overcome these limitations, face recognition systems are still vulnerable and need improvement. Multimodal fusion can be a good solution to overcome this vulnerability [68]; however, it incurs a high cost and causes inconvenience. e proposed method is a novel approach to resolve this problem. Gestures can also be used for user authentication. Ges- tures are a type of behavioral biometric signals that have recently been considered as good alternatives to physical biometric signals such as faces [9]. e biggest advantage of gestures is changeability by users. Even if physical biometric signals are stolen, users can not change their own physical signal. However, users can change the gesture signals easily like password. Along with the popularization of various IT devices such as smart phones, Kinect, and stereo cameras, a number of studies have been conducted to show that gestures can be used as a good behavioral biometric signal for user authentication. In earlier studies [1012], it was shown that accelerometer-based gesture recognition is feasible for user authentication in mobile devices. Also, in [13] the accelerometer and the gyroscope on mobile devices were combined for gesture-based user authentication. A novel multitouch gesture-based authentication technique was also proposed [14]. e gesture signal captured by Kinect was also used for user authentication [15, 16]. However, these conven- tional works require specific sensors such as accelerometer, gyroscope, and depth camera. Inspired by these previous studies, we propose to use gestures combined with face which can be obtained from Hindawi Publishing Corporation BioMed Research International Volume 2015, Article ID 343475, 8 pages http://dx.doi.org/10.1155/2015/343475

Transcript of Research Article A Multimodal User Authentication System...

Page 1: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

Research ArticleA Multimodal User Authentication System UsingFaces and Gestures

Hyunsoek Choi1 and Hyeyoung Park2

1School of Electrical Engineering and Computer Science Kyungpook National University Deagu 702-701 Republic of Korea2School of Computer Science and Engineering Kyungpook National University Deagu 702-701 Republic of Korea

Correspondence should be addressed to Hyeyoung Park hyparkknuackr

Received 26 September 2014 Accepted 19 November 2014

Academic Editor Sabah Mohammed

Copyright copy 2015 H Choi and H ParkThis is an open access article distributed under the Creative CommonsAttribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

As a novel approach to perform user authentication we propose a multimodal biometric system that uses faces and gesturesobtained from a single vision sensor Unlike typical multimodal biometric systems using physical information the proposed systemutilizes gesture video signals combined with facial images Whereas physical information such as face fingerprints and iris is fixedand not changeable behavioral information such as gestures and signatures can be freely changed by the user similar to a passwordTherefore it can be a countermeasure when the physical information is exposed We aim to investigate the potential possibility ofusing gestures as a signal for biometric system and the robustness of the proposedmultimodal user authentication systemThroughcomputational experiments on a public database we confirm that gesture information can help to improve the authenticationperformance

1 Introduction

With the growing need for secure authentication methodsvarious biometric signals are being actively studied Onerecent trend is the use of multimodal data for achievinghigh reliability [1ndash3] However in general multimodal bio-metric systems require multiple sensors which result inhigh developmental costs As a new attempt for achievinghigh reliability and low cost this paper proposes a novelmultimodal biometric system that uses two heterogeneousbiometric signals obtained from a single vision sensor facialimage and gesture video

Face is a representative of physical biometric signals andmany studies have been carried out on developing reliableface recognition systems [4 5] However the performance offace recognition systems is easily influenced by various envi-ronmental factors such as illumination expression pose andocclusion Despite a significant number of studies conductedto overcome these limitations face recognition systems arestill vulnerable and need improvement Multimodal fusioncan be a good solution to overcome this vulnerability [6ndash8]however it incurs a high cost and causes inconvenience Theproposedmethod is a novel approach to resolve this problem

Gestures can also be used for user authentication Ges-tures are a type of behavioral biometric signals that haverecently been considered as good alternatives to physicalbiometric signals such as faces [9] The biggest advantage ofgestures is changeability by users Even if physical biometricsignals are stolen users can not change their own physicalsignal However users can change the gesture signals easilylike password Along with the popularization of various ITdevices such as smart phones Kinect and stereo camerasa number of studies have been conducted to show thatgestures can be used as a good behavioral biometric signal foruser authentication In earlier studies [10ndash12] it was shownthat accelerometer-based gesture recognition is feasible foruser authentication in mobile devices Also in [13] theaccelerometer and the gyroscope on mobile devices werecombined for gesture-based user authentication A novelmultitouch gesture-based authentication technique was alsoproposed [14]The gesture signal captured by Kinect was alsoused for user authentication [15 16] However these conven-tional works require specific sensors such as accelerometergyroscope and depth camera

Inspired by these previous studies we propose to usegestures combined with face which can be obtained from

Hindawi Publishing CorporationBioMed Research InternationalVolume 2015 Article ID 343475 8 pageshttpdxdoiorg1011552015343475

2 BioMed Research International

Preprocessing

Face detection

Feature extraction

Frame difference

Feature extraction

Facerepresentation

Gesturerepresentation

Face similarity calculation

Decision module

Gesture similarity calculation

Decision criterion calculation Acceptreject

Registereduser DB

Face feature matrix

Gesture feature matrix

Figure 1Overall process of the proposedmultimodal biometric systemwhich combines face-based biometrics and gesture-based biometrics

mboxa single vision sensor for user authentication Theproposedmethod can be easily implemented tomany types ofIT equipment including smart TVs and game devices becauseit uses only a general vision sensor

One objective of the proposed method is to show thepossibility of gesture video as a biometric signal for userauthentication system Another one is to show the possibilityof combining two different biometric signals obtained by asingle vision sensor Although the signals are captured bythe same sensor in a single action they have virtually inde-pendent distributional properties which is desirable for mul-timodal combination Therefore we expect to improve theperformance of authentication systems using the proposedcombination plan with an insignificant increase in hardwarecost In addition to the benefit of low implementation costwe take advantage of the common properties of the twodifferent signals Noting that both face and gesture signals aregiven as RGB images we can use common image processingtechniques to extract efficient feature matrices from thetwo signals Furthermore we apply an appropriate distancemeasure to the feature matrices instead of typical distancemeasures A comprehensive description of the proposedsystem and its properties are addressed in the subsequentsections

2 Proposed Multimodal Biometric System

Figure 1 shows the overall structure of the proposed userauthentication system which is composed of three parts facerepresentation module gesture representation module and

decision module When a video stream that includes faceand hand gestures is provided simple preprocessing suchas image resizing and RGB-to-gray transformation is per-formed Then the face and gesture representation modulesextract facial and gesture information from the single videoand represent each of them using feature matrix respectivelyThe decision module uses the two feature matrices to deter-mine whether the given input is authentic or not

The proposed system operates in two different phasesdata registration phase and authentication phase In the dataregistration phase each gallery video is represented by twofeature matrices through the face and gesture representationmodules and it is added to user database in the form of twofeature matrices In the authentication phase a given probevideo initially goes through the representation modules to berepresented by two feature matricesThen the decisionmod-ule compares the probe feature matrices with the registeredgallery feature matrices to determine if the given probe datais authentic or not

Although detailed description of the representationmod-ules and decision module is given in Sections 3 and 4respectively we would like to note a main characteristicof the proposed system That is we obtain two biometricsignals from a single video stream and use a commonfeature extraction method for obtaining low-dimensionalfeatures from the two signals This not only reduces theimplementation cost but also makes the succeeding processsimple Because the two signals are represented by the samefeature descriptor they can be subjected to the same decisionmaking algorithms

BioMed Research International 3

Raw image Gray image(face detection)

Detected face HOG descriptor StackedHOG descriptor

1 times 31 vector

(resized into 32 times 32) (block size = 8 times 8)

(4 times 4 grid) (16 times 31 matrix)

16

bloc

k

middot middot middotmiddot middot middot

middot middot middot

middot middot middot

middot middot middot

Figure 2 Process of the face representation module

3 Data Representation Modules

31 Face Representation Module The face representationmodule detects a face in a given input video and represents itusing a featurematrixWe apply the Viola-Jones face detector[17] to locate the region of the facewithin an image It searchesfor a face in each frame starting with the first frame of thegiven input video until getting detection results from the facedetector

Once a face is detected the face area is resized to a32 times 32 pixel image and we divided face image into a 4 times 4grid with an 8 times 8 block size for local feature extractionAs a local feature descriptor we applied a histogram oforiented gradients (HOG) descriptor [18] We employ theVLFeat library [19] for obtaining a HOG descriptor inimplementation In the VLFeat library each local grid isrepresented by 31 dimensional feature vectors so that 16 times 31feature matrix F represents a face Figure 2 shows the processof the face representation module

32 Gesture Representation Module In the gesture repre-sentation module frame differencing is initially conductedbetween two consecutive image frames to capture the areawhere a gesture movement occurs It is also possible toeliminate the undesirable effect of the illumination changesand background using frame differencing Then we extractthe HOG descriptor from each image frame using the samealgorithm used in the face representation module Unlike theface representation module the difference image is dividedinto a 6 times 8 grid with a 40 times 40 block size

By stacking each HOG feature vector obtained from eachdifference image row by row we obtain a 119879 times 119863 featurematrix G for gesture data where 119879 denotes the number ofdifference images given by a gesture sequence and119863 denotesthe dimensionality of the feature vector obtained using theHOG descriptor Note that 119879 varies depending on the lengthof the input video whereas 119863 is fixed (1488 (= 6 times 8 times 31)in our actual implementation) Figure 3 shows the process ofthe gesture representation module

4 Decision Module and ProposedSimilarity Measure

Once a video signal (probe data) is represented by a pair oftwo feature matrices (FprbGprb) they are used as inputs withuser ID and a threshold 120579 for the decision module At first

the decision module finds a previously registered gallery data(FgalGgal) with given user ID Then it calculates distance offaces and gestures 119889(Fprb Fgal) and119889(GprbGgal) respectivelyAfter calculating the decision module calculates likelihoodratio to determine whether to accept or reject by decisioncriterion with a threshold 120579 To achieve a good authenticationperformance we focus on two core factors of the decisionmodule the distance measure and decision criterion

Note that columns and rows in the face feature matrixF and gesture feature matrix G have special characteristicsFor face feature matrix F each row vector corresponds tolocal grid in facial image and each column corresponds to ahistogram quantity of HOG feature descriptor (see Figure 2)For gesture feature matrixG each row vector corresponds toa frame in gesture video and each column corresponds to ahistogram quantity of HOG feature descriptor (see Figure 3)Therefore typical distance measures for vector data maycause some loss in the relation of time and spatial localityinformation We try to maintain the spatial locality of facialimage and the sequential relationship between the imageframes of the gesture video by using the matrix featuresdirectly without vectorization For this purpose we employthe matrix correlation distance proposed in our previousworks [20] which is a distancemeasure formatrix dataWhentwo 1198971times 1198972feature matrices X and Y are given the matrix

correlation distance is defined as

119889 (XY) = 1 minus (120588row (XY) + 120588col (XY)

2)

120588row (XY)

=1

1198971

1198971

sum

119894=1

sum1198972

119895=1(119909119894119895minus 119898119909) (119910119894119895minus 119898119910)

radicsum1198972

119895=1(119909119894119895minus 119898119909)2

sum1198972

119895=1(119910119894119895minus 119898119910)2

120588col (XY) =1

1198972

1198972

sum

119895=1

sum1198971

119894=1(119909119894119895minus 119898119909) (119910119894119895minus 119898119910)

radicsum1198971

119894=1(119909119894119895minus 119898119909)2

sum1198971

119894=1(119910119894119895minus 119898119910)2

(1)

where119898119909and119898

119910are the average of all the elements inX and

Y respectively The distance value 119889(XY) is in [0 2] whichis similar to the conventional correlation distanceWe shouldnote that the distance measure assumes that two matrices XandY have the same sizeTherefore in the case of gesture datawith various row sizes depending on the length of the videoan additional process is required to perform size alignment

4 BioMed Research International

48 b

lock

s

Raw image sequence Gray image sequence Frame differenceimage sequence

HOG descriptor StackedHOG descriptor

T

StackedHOG descriptor

(6 times 8 grid)

1 times 31 vector

(T times 1488 matrix)

1 times 1488 vector

(48 times 31 times T matrix)

T+1

frames

T+1

frames

Tfra

mes

Tfra

mes

Tfra

mes

(block size = 40 times 40)

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middotmiddot middot middotmiddot middot middotmiddot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Figure 3 Process of the gesture representation module

of two gesture feature matrices In this paper we apply adynamic time warping (DTW) algorithm [21] to align therows of matrices which is a technique to find an optimalalignment between two given sequences

After computing the distance values 119889119865= 119889(Fprb Fgal)

and 119889119866= 119889(GprbGgal) we need to make a decision of

acceptance using these values To do this we propose adecision criterion based on the likelihood ratio of the distancevalues which is defined by

119903119865119866(119889119865 119889119866) =119901 (Ω119860| 119889119865 119889119866)

119901 (Ω119868| 119889119865 119889119866)

=119901 (119889119865 119889119866| Ω119860) 119901 (Ω

119860)

119901 (119889119865 119889119866| Ω119868) 119901 (Ω

119868)

(2)

whereΩ119860denotes the class of distance values from authentic

data pairs and Ω119868denotes the class of distance values from

impostor data pairs Therefore 119903119865119866

indicates the ratio oflikelihood of whether the distance values (119889

119865 119889119866) originate

from an authentic data pair or an impostor data pair Inother words a large value of 119903

119865119866implies that the observed

distance (119889119865 119889119866) has a higher possibility of originating from

the population of authentic data pairsIn order to obtain an explicit function for calculating 119903

119865119866

we need to estimate the probability densities 119901(Ω119860| 119889119865 119889119866)

and 119901(Ω119868| 119889119865 119889119866) For real world implementation we

assume theGaussianmodel for119901(119889119865 119889119866| Ω119860) and119901(119889

119865 119889119866|

Ω119868) and estimate the parameters using gallery data Similarly

the prior probabilities 119901(Ω119860) and 119901(Ω

119868) are estimated too

Though the threshold 120579 is set for 1 typically it is changeable If120579 is high the number of false acceptances is decreased and thenumber of false rejections is increased If 120579 is low the reversephenomenon occurs In the experiments we measure theperformance of proposed authentication systemwith variable120579 A summarized description of decisionmodule is presentedin Algorithm 1

5 Experimental Results

In order to confirm the performance of proposed systemwe conducted experiments on the ChaLearn database [22]which was built for a gesture recognition competitionAlthough the data includes depth signals obtained from

Kinect we use only RGB signals because the proposedmethod is developed for a general vision sensor Figure 4shows some examples of the data From the whole dataset we prepared three setsmdashA B and Cmdashfor experimentsEach set is composed of 80 video samples from 20 subjectseach subject makes hisher own unique gesture four timesExperiments are carried out for each set separately using 4-fold cross-validation Three samples from each subject areused for gallery data and one sample is used for probe dataTherefore total 12 experiments were carried out

Before starting authentication we first need to estimatetwo conditional distributions 119901(119889

119865 119889119866| Ω119860) and 119901(119889

119865 119889119866|

Ω119868) which are used in decision criterion 119903

119865119866(119889119865 119889119866) For

each experiment we first make all possible data pairs fromgallery data and in order to obtain 1770 distance valuesamong which 60 values are from authentic pairs and 1710from impostor pairs The estimated pdf 119901(119889

119865 119889119866| Ω119860) and

119901(119889119865 119889119866| Ω119868) using these values are then applied to calcu-

late 119903119865119866(119889119865 119889119866) in the authentication phase For evaluating

authentication performance we compute distances betweengallery and probe data Since we have 20 probe samplesand 60 gallery samples there are 1200 distance values 60authentic values and 1140 impostor values The performanceis evaluated by the error rates (false acceptance and falserejection) of decision module for the 1200 values

We compared the performance of the decision moduleby changing modality and other conventional distance mea-sures In the unimodal case we use marginal distributionsuch as 119901(119889

119865| Ω119860) and 119901(119889

119866| Ω119860) for obtaining the

decision criterion We first compared the value of equalerror rate (EER) which is a typical measure for evaluatingauthentication systems EER is the value of error rate whenthe false acceptance rate (FAR) is equal to the false rejectionrate (FRR) Figure 5 shows the average EER over 4-foldcross-validation for each set A B and C As can be seenfrom Figure 5 gesture-based unimodal system shows slightlybetter performance than face-based unimodal system Alsothe proposed multimodal biometric system shows the bestresult

In Figure 6 we present the detection error tradeoff(DET) curves [23] for visualized comparison among differentmodalities with various distance measures The DET curveis a plot of error rates for binary classification systems inwhich the lower left curve implies the better performance

BioMed Research International 5

Input Feature matrices of face Fprb and gesture Gprb for aprobe video with user ID and a threshold 120579Output Authentication Result (AcceptReject)(1) Find a gallery data (FgalGgal) with user ID(2) Calculate the distance 119889

119865= 119889(Fprb Fgal) using (1)

(3) Align the gesture feature matrix Gprb and Ggal using DTWalgorithm(GprbGgal)

DTW997888997888997888997888rarr (Gprb Ggal)

Gprb and Ggal have same size(4) Calculate the distance 119889

119866= 119889(Gprb Ggal) using (1)

(5) Calculate the likelihood ratio 119903119865119866(119889119865 119889119866) using (2)

(6) if 119903119865119866(119889119865 119889119866) gt 120579 then

(7) Probe video is accepted(8) else(9) Probe video is rejected(10) end if

Algorithm 1 Pseudocode for the decision module

(a) (b)

Figure 4 Sample images from ChaLearn database (a) first frames of 20 selected users (b) image frames in a gesture video

7

6

5

4

3

2

1

0

EER

()

608 599

353

294

408

458

149

064

241

Unimodal(face)

Unimodal(gesture)

Multimodal(face + gesture)

Set ASet BSet C

Figure 5 Average EER () depending on biosignals using matrix correlation distance

6 BioMed Research International

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Manhattan distance

(a)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Euclidean distance

(b)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with matrix correlation distance

(c)

Figure 6 DET curves of authentication system with different modalities (a) Manhattan distance (b) Euclidean distance and (c) matrixcorrelation distance

BioMed Research International 7

60

55

50

45

40

35

30

25

20

15

10200 400 600 800 1000 1200 1400 1600 1800 2000

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

Manhattan distance between gallery and probe

ImpostorGenuine

(a)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

35

3

25

2

15

1

054 6 8 10 12 14 16 18

Euclidean distance between gallery and probe

ImpostorGenuine

(b)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

09

09

08

08

07

07

06

06

05

05

04

04

03

03

02

02

01

0

Matrix correlation distance between gallery and probe

ImpostorGenuine

(c)

Figure 7 Scatter plots of distance values between authentic pairs (I) as well as impostor pairs (◻) (a) Manhattan distance (b) Euclideandistance and (c) matrix correlation distance

As can be seen from Figure 6 the proposed multimodalbiometric system is superior to unimodal systems regardlessof the distance measures We can also observe that theperformance is dependent on the distance measures Forgesture conventional Manhattan distance and Euclideandistance give poor performance but the matrix correlationdistance shows improvement which is even better than faceThis effect is emphasized by the combination of face andgesture resulting in the remarkable improvement of DETcurves as shown in the solid curve of Figure 6(c)

Figure 7 shows the scatter plots of the distance values(119889119865 119889119866) in Ω

119860(I) as well as those in Ω

119868(◻) In this figure

we can observe that the discriminability is increased by usingmultimodality We also plot the marginal histogram of 119889

119865

and 119889119866on the corresponding axes The overlapped region of

histogram implies the region where decision error occurs Inthe case of a gesture we can see that the matrix correlationdistance can significantly decrease overlapped region Thismeans that matrix correlation distance is more appropriateto gesture data with our proposed feature representationAdditionally we can observe that the bivariate distributionsof (119889119865 119889119866) have the shape of ellipse which can justify our

Gaussian assumption for estimating the conditional distribu-tions 119901(119889

119865 119889119866| Ω119868) and 119901(119889

119865 119889119866| Ω119860) Moreover from

8 BioMed Research International

the shape of ellipse we can guess that the two modalities arealmost independent and this is supported by the fact that theaverage value of correlation coefficient is 019 This propertyis desirable for combining two biometric signals to constructmultimodal biometric system

6 Conclusion

In this paper we present a look into simple and efficientvision-based multimodal biometric system using heteroge-neous biometric signals By combining physical and behav-ioral biometric signals we can achieve a high degree ofreliability Because the proposed system uses a single visionsensor it can be easily implemented on commonly used smartdevices such as smart TVs More comprehensive study ondeveloping efficient feature extraction and classification willbe done for real world application of the proposal system

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research was partially supported by the DGIST RampDProgram of the Ministry of Education Science and Technol-ogy of Korea (13-IT-03) and Basic Science Research Programthrough the National Research Foundation of Korea (NRF)funded by theMinistry of Education Science andTechnology(NRF-2013R1A1A2061831)

References

[1] A Ross and A K Jain ldquoMultimodal biometrics an overviewrdquoin Proceedings of the 12th European Signal Processing Conferencepp 1221ndash1224 Vienna Austria September 2004

[2] K Bowyer K Chang and P Yan ldquoMulti-modal biometrics anoverviewrdquo in Proceedings of the 2nd Workshop on Multi-ModalUser Authentication Toulouse France May 2006

[3] A K Jain and A Kumar ldquoBiometric recognition an overviewrdquoin Second Generation Biometrics The Ethical Legal and SocialContext E Mordini and D Tzovaras Eds pp 49ndash79 SpringerAmsterdam The Netherlands 2012

[4] W Zhao R Chellappa P J Phillips and A Rosenfeld ldquoFacerecognition a literature surveyrdquo ACM Computing Surveys vol35 no 4 pp 399ndash458 2003

[5] R Jafri and H R Arabnia ldquoA survey of face recognitiontechniquesrdquo Journal of Information Processing Systems vol 5 no2 pp 41ndash68 2009

[6] I A Kakadiaris G Passalis T Theoharis G Toderici IKonstantinidis and NMurtuza ldquoMultimodal face recognitioncombination of geometry with physiological informationrdquo inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR rsquo05) vol 2 pp1022ndash1029 San Diego Calif USA June 2005

[7] K W Bowyer K Chang and P Flynn ldquoA survey of approachesand challenges in 3D and multi-modal 3D + 2D face recogni-tionrdquo Computer Vision and Image Understanding vol 101 no 1pp 1ndash15 2006

[8] K I Chang K W Bowyer and P J Flynn ldquoAn evaluationof multimodal 2D+3D face biometricsrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 27 no 4 pp 619ndash624 2005

[9] R V Yampolskiy and V Govindaraju ldquoBehavioural biometricsa survey and classificationrdquo International Journal of Biometricsvol 1 no 1 pp 81ndash113 2008

[10] J Liu ZWang L Zhong JWickramasuriya andVVasudevanldquouWave accelerometer-based personalized gesture recognitionand its applicationsrdquo in Proceedings of the 7th Annual IEEEInternational Conference on Pervasive Computing and Commu-nications (PerCom rsquo09) pp 1ndash9 Galveston Tex USA March2009

[11] G Bailador C Sanchez-Avila J Guerra-Casanova and A deSantos Sierra ldquoAnalysis of pattern recognition techniques for in-air signature biometricsrdquo Pattern Recognition vol 44 no 10-11pp 2468ndash2478 2011

[12] J Guerra-Casanova C Sanchez-Avila G Bailador and Ade Santos Sierra ldquoAuthentication in mobile devices throughhand gesture recognitionrdquo International Journal of InformationSecurity vol 11 no 2 pp 65ndash83 2012

[13] D Guse Gesture-based user authentication on mobile devicesusing accelerometer and gyroscope [Master thesis] Berlin Institueof Technology 2011

[14] N Sae-Bae K Ahmed K Isbister and N Memon ldquoBiometric-rich gestures a novel approach to authentication onmulti-touchdevicesrdquo in Proceedings of the 30th ACM Conference on HumanFactors in Computing Systems (CHI rsquo12) pp 977ndash986 AustinTex USA May 2012

[15] K Lai J Konrad and P Ishwar ldquoTowards gesture-based userauthenticationrdquo in Proceedings of the IEEE 9th InternationalConference on Advanced Video and Signal-Based Surveillance(AVSS rsquo12) pp 282ndash287 Beijing China September 2012

[16] J Wu J Konrad and P Ishwar ldquoThe value of multiple view-points in gesture-based user authenticationrdquo in Proceedingsof the IEEE Computer Society Conference on Computer Visionand Pattern RecognitionWorkshop pp 90ndash97 Columbus OhioUSA June 2014

[17] P Viola and M J Jones ldquoRobust real-time face detectionrdquoInternational Journal of Computer Vision vol 57 no 2 pp 137ndash154 2004

[18] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) pp 886ndash893 San Diego Calif USA June 2005

[19] A Vedaldi and B Fulkerson ldquoVLFeat An Open and PortableLibrary of Computer Vision Algorithmsrdquo httpwwwvlfeatorg

[20] H Choi J Seo andH Park ldquoMatrix correlation distance for 2Dimage classificationrdquo in Proceedings of the of ACM Symposiumon Applied Computing pp 1741ndash1742 Gyeongju Republic ofKorea March 2014

[21] M Muller ldquoDynamic time warpingrdquo in Information Retrievalfor Music andMotion M Muller Ed pp 69ndash84 Springer NewYork NY USA 2007

[22] ChaLearn ChaLearn Gesture Dataset (CGD 2011) 2012 httpgesturechalearnorgdata

[23] A Martin G Doddington and T Kamm ldquoThe DET curvein assessment of detection task performancerdquo in Proceedingsof the European Conference on Speech Communication andTechnology pp 1895ndash1898 Rhodes Greece September 1997

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology

Page 2: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

2 BioMed Research International

Preprocessing

Face detection

Feature extraction

Frame difference

Feature extraction

Facerepresentation

Gesturerepresentation

Face similarity calculation

Decision module

Gesture similarity calculation

Decision criterion calculation Acceptreject

Registereduser DB

Face feature matrix

Gesture feature matrix

Figure 1Overall process of the proposedmultimodal biometric systemwhich combines face-based biometrics and gesture-based biometrics

mboxa single vision sensor for user authentication Theproposedmethod can be easily implemented tomany types ofIT equipment including smart TVs and game devices becauseit uses only a general vision sensor

One objective of the proposed method is to show thepossibility of gesture video as a biometric signal for userauthentication system Another one is to show the possibilityof combining two different biometric signals obtained by asingle vision sensor Although the signals are captured bythe same sensor in a single action they have virtually inde-pendent distributional properties which is desirable for mul-timodal combination Therefore we expect to improve theperformance of authentication systems using the proposedcombination plan with an insignificant increase in hardwarecost In addition to the benefit of low implementation costwe take advantage of the common properties of the twodifferent signals Noting that both face and gesture signals aregiven as RGB images we can use common image processingtechniques to extract efficient feature matrices from thetwo signals Furthermore we apply an appropriate distancemeasure to the feature matrices instead of typical distancemeasures A comprehensive description of the proposedsystem and its properties are addressed in the subsequentsections

2 Proposed Multimodal Biometric System

Figure 1 shows the overall structure of the proposed userauthentication system which is composed of three parts facerepresentation module gesture representation module and

decision module When a video stream that includes faceand hand gestures is provided simple preprocessing suchas image resizing and RGB-to-gray transformation is per-formed Then the face and gesture representation modulesextract facial and gesture information from the single videoand represent each of them using feature matrix respectivelyThe decision module uses the two feature matrices to deter-mine whether the given input is authentic or not

The proposed system operates in two different phasesdata registration phase and authentication phase In the dataregistration phase each gallery video is represented by twofeature matrices through the face and gesture representationmodules and it is added to user database in the form of twofeature matrices In the authentication phase a given probevideo initially goes through the representation modules to berepresented by two feature matricesThen the decisionmod-ule compares the probe feature matrices with the registeredgallery feature matrices to determine if the given probe datais authentic or not

Although detailed description of the representationmod-ules and decision module is given in Sections 3 and 4respectively we would like to note a main characteristicof the proposed system That is we obtain two biometricsignals from a single video stream and use a commonfeature extraction method for obtaining low-dimensionalfeatures from the two signals This not only reduces theimplementation cost but also makes the succeeding processsimple Because the two signals are represented by the samefeature descriptor they can be subjected to the same decisionmaking algorithms

BioMed Research International 3

Raw image Gray image(face detection)

Detected face HOG descriptor StackedHOG descriptor

1 times 31 vector

(resized into 32 times 32) (block size = 8 times 8)

(4 times 4 grid) (16 times 31 matrix)

16

bloc

k

middot middot middotmiddot middot middot

middot middot middot

middot middot middot

middot middot middot

Figure 2 Process of the face representation module

3 Data Representation Modules

31 Face Representation Module The face representationmodule detects a face in a given input video and represents itusing a featurematrixWe apply the Viola-Jones face detector[17] to locate the region of the facewithin an image It searchesfor a face in each frame starting with the first frame of thegiven input video until getting detection results from the facedetector

Once a face is detected the face area is resized to a32 times 32 pixel image and we divided face image into a 4 times 4grid with an 8 times 8 block size for local feature extractionAs a local feature descriptor we applied a histogram oforiented gradients (HOG) descriptor [18] We employ theVLFeat library [19] for obtaining a HOG descriptor inimplementation In the VLFeat library each local grid isrepresented by 31 dimensional feature vectors so that 16 times 31feature matrix F represents a face Figure 2 shows the processof the face representation module

32 Gesture Representation Module In the gesture repre-sentation module frame differencing is initially conductedbetween two consecutive image frames to capture the areawhere a gesture movement occurs It is also possible toeliminate the undesirable effect of the illumination changesand background using frame differencing Then we extractthe HOG descriptor from each image frame using the samealgorithm used in the face representation module Unlike theface representation module the difference image is dividedinto a 6 times 8 grid with a 40 times 40 block size

By stacking each HOG feature vector obtained from eachdifference image row by row we obtain a 119879 times 119863 featurematrix G for gesture data where 119879 denotes the number ofdifference images given by a gesture sequence and119863 denotesthe dimensionality of the feature vector obtained using theHOG descriptor Note that 119879 varies depending on the lengthof the input video whereas 119863 is fixed (1488 (= 6 times 8 times 31)in our actual implementation) Figure 3 shows the process ofthe gesture representation module

4 Decision Module and ProposedSimilarity Measure

Once a video signal (probe data) is represented by a pair oftwo feature matrices (FprbGprb) they are used as inputs withuser ID and a threshold 120579 for the decision module At first

the decision module finds a previously registered gallery data(FgalGgal) with given user ID Then it calculates distance offaces and gestures 119889(Fprb Fgal) and119889(GprbGgal) respectivelyAfter calculating the decision module calculates likelihoodratio to determine whether to accept or reject by decisioncriterion with a threshold 120579 To achieve a good authenticationperformance we focus on two core factors of the decisionmodule the distance measure and decision criterion

Note that columns and rows in the face feature matrixF and gesture feature matrix G have special characteristicsFor face feature matrix F each row vector corresponds tolocal grid in facial image and each column corresponds to ahistogram quantity of HOG feature descriptor (see Figure 2)For gesture feature matrixG each row vector corresponds toa frame in gesture video and each column corresponds to ahistogram quantity of HOG feature descriptor (see Figure 3)Therefore typical distance measures for vector data maycause some loss in the relation of time and spatial localityinformation We try to maintain the spatial locality of facialimage and the sequential relationship between the imageframes of the gesture video by using the matrix featuresdirectly without vectorization For this purpose we employthe matrix correlation distance proposed in our previousworks [20] which is a distancemeasure formatrix dataWhentwo 1198971times 1198972feature matrices X and Y are given the matrix

correlation distance is defined as

119889 (XY) = 1 minus (120588row (XY) + 120588col (XY)

2)

120588row (XY)

=1

1198971

1198971

sum

119894=1

sum1198972

119895=1(119909119894119895minus 119898119909) (119910119894119895minus 119898119910)

radicsum1198972

119895=1(119909119894119895minus 119898119909)2

sum1198972

119895=1(119910119894119895minus 119898119910)2

120588col (XY) =1

1198972

1198972

sum

119895=1

sum1198971

119894=1(119909119894119895minus 119898119909) (119910119894119895minus 119898119910)

radicsum1198971

119894=1(119909119894119895minus 119898119909)2

sum1198971

119894=1(119910119894119895minus 119898119910)2

(1)

where119898119909and119898

119910are the average of all the elements inX and

Y respectively The distance value 119889(XY) is in [0 2] whichis similar to the conventional correlation distanceWe shouldnote that the distance measure assumes that two matrices XandY have the same sizeTherefore in the case of gesture datawith various row sizes depending on the length of the videoan additional process is required to perform size alignment

4 BioMed Research International

48 b

lock

s

Raw image sequence Gray image sequence Frame differenceimage sequence

HOG descriptor StackedHOG descriptor

T

StackedHOG descriptor

(6 times 8 grid)

1 times 31 vector

(T times 1488 matrix)

1 times 1488 vector

(48 times 31 times T matrix)

T+1

frames

T+1

frames

Tfra

mes

Tfra

mes

Tfra

mes

(block size = 40 times 40)

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middotmiddot middot middotmiddot middot middotmiddot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Figure 3 Process of the gesture representation module

of two gesture feature matrices In this paper we apply adynamic time warping (DTW) algorithm [21] to align therows of matrices which is a technique to find an optimalalignment between two given sequences

After computing the distance values 119889119865= 119889(Fprb Fgal)

and 119889119866= 119889(GprbGgal) we need to make a decision of

acceptance using these values To do this we propose adecision criterion based on the likelihood ratio of the distancevalues which is defined by

119903119865119866(119889119865 119889119866) =119901 (Ω119860| 119889119865 119889119866)

119901 (Ω119868| 119889119865 119889119866)

=119901 (119889119865 119889119866| Ω119860) 119901 (Ω

119860)

119901 (119889119865 119889119866| Ω119868) 119901 (Ω

119868)

(2)

whereΩ119860denotes the class of distance values from authentic

data pairs and Ω119868denotes the class of distance values from

impostor data pairs Therefore 119903119865119866

indicates the ratio oflikelihood of whether the distance values (119889

119865 119889119866) originate

from an authentic data pair or an impostor data pair Inother words a large value of 119903

119865119866implies that the observed

distance (119889119865 119889119866) has a higher possibility of originating from

the population of authentic data pairsIn order to obtain an explicit function for calculating 119903

119865119866

we need to estimate the probability densities 119901(Ω119860| 119889119865 119889119866)

and 119901(Ω119868| 119889119865 119889119866) For real world implementation we

assume theGaussianmodel for119901(119889119865 119889119866| Ω119860) and119901(119889

119865 119889119866|

Ω119868) and estimate the parameters using gallery data Similarly

the prior probabilities 119901(Ω119860) and 119901(Ω

119868) are estimated too

Though the threshold 120579 is set for 1 typically it is changeable If120579 is high the number of false acceptances is decreased and thenumber of false rejections is increased If 120579 is low the reversephenomenon occurs In the experiments we measure theperformance of proposed authentication systemwith variable120579 A summarized description of decisionmodule is presentedin Algorithm 1

5 Experimental Results

In order to confirm the performance of proposed systemwe conducted experiments on the ChaLearn database [22]which was built for a gesture recognition competitionAlthough the data includes depth signals obtained from

Kinect we use only RGB signals because the proposedmethod is developed for a general vision sensor Figure 4shows some examples of the data From the whole dataset we prepared three setsmdashA B and Cmdashfor experimentsEach set is composed of 80 video samples from 20 subjectseach subject makes hisher own unique gesture four timesExperiments are carried out for each set separately using 4-fold cross-validation Three samples from each subject areused for gallery data and one sample is used for probe dataTherefore total 12 experiments were carried out

Before starting authentication we first need to estimatetwo conditional distributions 119901(119889

119865 119889119866| Ω119860) and 119901(119889

119865 119889119866|

Ω119868) which are used in decision criterion 119903

119865119866(119889119865 119889119866) For

each experiment we first make all possible data pairs fromgallery data and in order to obtain 1770 distance valuesamong which 60 values are from authentic pairs and 1710from impostor pairs The estimated pdf 119901(119889

119865 119889119866| Ω119860) and

119901(119889119865 119889119866| Ω119868) using these values are then applied to calcu-

late 119903119865119866(119889119865 119889119866) in the authentication phase For evaluating

authentication performance we compute distances betweengallery and probe data Since we have 20 probe samplesand 60 gallery samples there are 1200 distance values 60authentic values and 1140 impostor values The performanceis evaluated by the error rates (false acceptance and falserejection) of decision module for the 1200 values

We compared the performance of the decision moduleby changing modality and other conventional distance mea-sures In the unimodal case we use marginal distributionsuch as 119901(119889

119865| Ω119860) and 119901(119889

119866| Ω119860) for obtaining the

decision criterion We first compared the value of equalerror rate (EER) which is a typical measure for evaluatingauthentication systems EER is the value of error rate whenthe false acceptance rate (FAR) is equal to the false rejectionrate (FRR) Figure 5 shows the average EER over 4-foldcross-validation for each set A B and C As can be seenfrom Figure 5 gesture-based unimodal system shows slightlybetter performance than face-based unimodal system Alsothe proposed multimodal biometric system shows the bestresult

In Figure 6 we present the detection error tradeoff(DET) curves [23] for visualized comparison among differentmodalities with various distance measures The DET curveis a plot of error rates for binary classification systems inwhich the lower left curve implies the better performance

BioMed Research International 5

Input Feature matrices of face Fprb and gesture Gprb for aprobe video with user ID and a threshold 120579Output Authentication Result (AcceptReject)(1) Find a gallery data (FgalGgal) with user ID(2) Calculate the distance 119889

119865= 119889(Fprb Fgal) using (1)

(3) Align the gesture feature matrix Gprb and Ggal using DTWalgorithm(GprbGgal)

DTW997888997888997888997888rarr (Gprb Ggal)

Gprb and Ggal have same size(4) Calculate the distance 119889

119866= 119889(Gprb Ggal) using (1)

(5) Calculate the likelihood ratio 119903119865119866(119889119865 119889119866) using (2)

(6) if 119903119865119866(119889119865 119889119866) gt 120579 then

(7) Probe video is accepted(8) else(9) Probe video is rejected(10) end if

Algorithm 1 Pseudocode for the decision module

(a) (b)

Figure 4 Sample images from ChaLearn database (a) first frames of 20 selected users (b) image frames in a gesture video

7

6

5

4

3

2

1

0

EER

()

608 599

353

294

408

458

149

064

241

Unimodal(face)

Unimodal(gesture)

Multimodal(face + gesture)

Set ASet BSet C

Figure 5 Average EER () depending on biosignals using matrix correlation distance

6 BioMed Research International

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Manhattan distance

(a)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Euclidean distance

(b)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with matrix correlation distance

(c)

Figure 6 DET curves of authentication system with different modalities (a) Manhattan distance (b) Euclidean distance and (c) matrixcorrelation distance

BioMed Research International 7

60

55

50

45

40

35

30

25

20

15

10200 400 600 800 1000 1200 1400 1600 1800 2000

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

Manhattan distance between gallery and probe

ImpostorGenuine

(a)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

35

3

25

2

15

1

054 6 8 10 12 14 16 18

Euclidean distance between gallery and probe

ImpostorGenuine

(b)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

09

09

08

08

07

07

06

06

05

05

04

04

03

03

02

02

01

0

Matrix correlation distance between gallery and probe

ImpostorGenuine

(c)

Figure 7 Scatter plots of distance values between authentic pairs (I) as well as impostor pairs (◻) (a) Manhattan distance (b) Euclideandistance and (c) matrix correlation distance

As can be seen from Figure 6 the proposed multimodalbiometric system is superior to unimodal systems regardlessof the distance measures We can also observe that theperformance is dependent on the distance measures Forgesture conventional Manhattan distance and Euclideandistance give poor performance but the matrix correlationdistance shows improvement which is even better than faceThis effect is emphasized by the combination of face andgesture resulting in the remarkable improvement of DETcurves as shown in the solid curve of Figure 6(c)

Figure 7 shows the scatter plots of the distance values(119889119865 119889119866) in Ω

119860(I) as well as those in Ω

119868(◻) In this figure

we can observe that the discriminability is increased by usingmultimodality We also plot the marginal histogram of 119889

119865

and 119889119866on the corresponding axes The overlapped region of

histogram implies the region where decision error occurs Inthe case of a gesture we can see that the matrix correlationdistance can significantly decrease overlapped region Thismeans that matrix correlation distance is more appropriateto gesture data with our proposed feature representationAdditionally we can observe that the bivariate distributionsof (119889119865 119889119866) have the shape of ellipse which can justify our

Gaussian assumption for estimating the conditional distribu-tions 119901(119889

119865 119889119866| Ω119868) and 119901(119889

119865 119889119866| Ω119860) Moreover from

8 BioMed Research International

the shape of ellipse we can guess that the two modalities arealmost independent and this is supported by the fact that theaverage value of correlation coefficient is 019 This propertyis desirable for combining two biometric signals to constructmultimodal biometric system

6 Conclusion

In this paper we present a look into simple and efficientvision-based multimodal biometric system using heteroge-neous biometric signals By combining physical and behav-ioral biometric signals we can achieve a high degree ofreliability Because the proposed system uses a single visionsensor it can be easily implemented on commonly used smartdevices such as smart TVs More comprehensive study ondeveloping efficient feature extraction and classification willbe done for real world application of the proposal system

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research was partially supported by the DGIST RampDProgram of the Ministry of Education Science and Technol-ogy of Korea (13-IT-03) and Basic Science Research Programthrough the National Research Foundation of Korea (NRF)funded by theMinistry of Education Science andTechnology(NRF-2013R1A1A2061831)

References

[1] A Ross and A K Jain ldquoMultimodal biometrics an overviewrdquoin Proceedings of the 12th European Signal Processing Conferencepp 1221ndash1224 Vienna Austria September 2004

[2] K Bowyer K Chang and P Yan ldquoMulti-modal biometrics anoverviewrdquo in Proceedings of the 2nd Workshop on Multi-ModalUser Authentication Toulouse France May 2006

[3] A K Jain and A Kumar ldquoBiometric recognition an overviewrdquoin Second Generation Biometrics The Ethical Legal and SocialContext E Mordini and D Tzovaras Eds pp 49ndash79 SpringerAmsterdam The Netherlands 2012

[4] W Zhao R Chellappa P J Phillips and A Rosenfeld ldquoFacerecognition a literature surveyrdquo ACM Computing Surveys vol35 no 4 pp 399ndash458 2003

[5] R Jafri and H R Arabnia ldquoA survey of face recognitiontechniquesrdquo Journal of Information Processing Systems vol 5 no2 pp 41ndash68 2009

[6] I A Kakadiaris G Passalis T Theoharis G Toderici IKonstantinidis and NMurtuza ldquoMultimodal face recognitioncombination of geometry with physiological informationrdquo inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR rsquo05) vol 2 pp1022ndash1029 San Diego Calif USA June 2005

[7] K W Bowyer K Chang and P Flynn ldquoA survey of approachesand challenges in 3D and multi-modal 3D + 2D face recogni-tionrdquo Computer Vision and Image Understanding vol 101 no 1pp 1ndash15 2006

[8] K I Chang K W Bowyer and P J Flynn ldquoAn evaluationof multimodal 2D+3D face biometricsrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 27 no 4 pp 619ndash624 2005

[9] R V Yampolskiy and V Govindaraju ldquoBehavioural biometricsa survey and classificationrdquo International Journal of Biometricsvol 1 no 1 pp 81ndash113 2008

[10] J Liu ZWang L Zhong JWickramasuriya andVVasudevanldquouWave accelerometer-based personalized gesture recognitionand its applicationsrdquo in Proceedings of the 7th Annual IEEEInternational Conference on Pervasive Computing and Commu-nications (PerCom rsquo09) pp 1ndash9 Galveston Tex USA March2009

[11] G Bailador C Sanchez-Avila J Guerra-Casanova and A deSantos Sierra ldquoAnalysis of pattern recognition techniques for in-air signature biometricsrdquo Pattern Recognition vol 44 no 10-11pp 2468ndash2478 2011

[12] J Guerra-Casanova C Sanchez-Avila G Bailador and Ade Santos Sierra ldquoAuthentication in mobile devices throughhand gesture recognitionrdquo International Journal of InformationSecurity vol 11 no 2 pp 65ndash83 2012

[13] D Guse Gesture-based user authentication on mobile devicesusing accelerometer and gyroscope [Master thesis] Berlin Institueof Technology 2011

[14] N Sae-Bae K Ahmed K Isbister and N Memon ldquoBiometric-rich gestures a novel approach to authentication onmulti-touchdevicesrdquo in Proceedings of the 30th ACM Conference on HumanFactors in Computing Systems (CHI rsquo12) pp 977ndash986 AustinTex USA May 2012

[15] K Lai J Konrad and P Ishwar ldquoTowards gesture-based userauthenticationrdquo in Proceedings of the IEEE 9th InternationalConference on Advanced Video and Signal-Based Surveillance(AVSS rsquo12) pp 282ndash287 Beijing China September 2012

[16] J Wu J Konrad and P Ishwar ldquoThe value of multiple view-points in gesture-based user authenticationrdquo in Proceedingsof the IEEE Computer Society Conference on Computer Visionand Pattern RecognitionWorkshop pp 90ndash97 Columbus OhioUSA June 2014

[17] P Viola and M J Jones ldquoRobust real-time face detectionrdquoInternational Journal of Computer Vision vol 57 no 2 pp 137ndash154 2004

[18] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) pp 886ndash893 San Diego Calif USA June 2005

[19] A Vedaldi and B Fulkerson ldquoVLFeat An Open and PortableLibrary of Computer Vision Algorithmsrdquo httpwwwvlfeatorg

[20] H Choi J Seo andH Park ldquoMatrix correlation distance for 2Dimage classificationrdquo in Proceedings of the of ACM Symposiumon Applied Computing pp 1741ndash1742 Gyeongju Republic ofKorea March 2014

[21] M Muller ldquoDynamic time warpingrdquo in Information Retrievalfor Music andMotion M Muller Ed pp 69ndash84 Springer NewYork NY USA 2007

[22] ChaLearn ChaLearn Gesture Dataset (CGD 2011) 2012 httpgesturechalearnorgdata

[23] A Martin G Doddington and T Kamm ldquoThe DET curvein assessment of detection task performancerdquo in Proceedingsof the European Conference on Speech Communication andTechnology pp 1895ndash1898 Rhodes Greece September 1997

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology

Page 3: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

BioMed Research International 3

Raw image Gray image(face detection)

Detected face HOG descriptor StackedHOG descriptor

1 times 31 vector

(resized into 32 times 32) (block size = 8 times 8)

(4 times 4 grid) (16 times 31 matrix)

16

bloc

k

middot middot middotmiddot middot middot

middot middot middot

middot middot middot

middot middot middot

Figure 2 Process of the face representation module

3 Data Representation Modules

31 Face Representation Module The face representationmodule detects a face in a given input video and represents itusing a featurematrixWe apply the Viola-Jones face detector[17] to locate the region of the facewithin an image It searchesfor a face in each frame starting with the first frame of thegiven input video until getting detection results from the facedetector

Once a face is detected the face area is resized to a32 times 32 pixel image and we divided face image into a 4 times 4grid with an 8 times 8 block size for local feature extractionAs a local feature descriptor we applied a histogram oforiented gradients (HOG) descriptor [18] We employ theVLFeat library [19] for obtaining a HOG descriptor inimplementation In the VLFeat library each local grid isrepresented by 31 dimensional feature vectors so that 16 times 31feature matrix F represents a face Figure 2 shows the processof the face representation module

32 Gesture Representation Module In the gesture repre-sentation module frame differencing is initially conductedbetween two consecutive image frames to capture the areawhere a gesture movement occurs It is also possible toeliminate the undesirable effect of the illumination changesand background using frame differencing Then we extractthe HOG descriptor from each image frame using the samealgorithm used in the face representation module Unlike theface representation module the difference image is dividedinto a 6 times 8 grid with a 40 times 40 block size

By stacking each HOG feature vector obtained from eachdifference image row by row we obtain a 119879 times 119863 featurematrix G for gesture data where 119879 denotes the number ofdifference images given by a gesture sequence and119863 denotesthe dimensionality of the feature vector obtained using theHOG descriptor Note that 119879 varies depending on the lengthof the input video whereas 119863 is fixed (1488 (= 6 times 8 times 31)in our actual implementation) Figure 3 shows the process ofthe gesture representation module

4 Decision Module and ProposedSimilarity Measure

Once a video signal (probe data) is represented by a pair oftwo feature matrices (FprbGprb) they are used as inputs withuser ID and a threshold 120579 for the decision module At first

the decision module finds a previously registered gallery data(FgalGgal) with given user ID Then it calculates distance offaces and gestures 119889(Fprb Fgal) and119889(GprbGgal) respectivelyAfter calculating the decision module calculates likelihoodratio to determine whether to accept or reject by decisioncriterion with a threshold 120579 To achieve a good authenticationperformance we focus on two core factors of the decisionmodule the distance measure and decision criterion

Note that columns and rows in the face feature matrixF and gesture feature matrix G have special characteristicsFor face feature matrix F each row vector corresponds tolocal grid in facial image and each column corresponds to ahistogram quantity of HOG feature descriptor (see Figure 2)For gesture feature matrixG each row vector corresponds toa frame in gesture video and each column corresponds to ahistogram quantity of HOG feature descriptor (see Figure 3)Therefore typical distance measures for vector data maycause some loss in the relation of time and spatial localityinformation We try to maintain the spatial locality of facialimage and the sequential relationship between the imageframes of the gesture video by using the matrix featuresdirectly without vectorization For this purpose we employthe matrix correlation distance proposed in our previousworks [20] which is a distancemeasure formatrix dataWhentwo 1198971times 1198972feature matrices X and Y are given the matrix

correlation distance is defined as

119889 (XY) = 1 minus (120588row (XY) + 120588col (XY)

2)

120588row (XY)

=1

1198971

1198971

sum

119894=1

sum1198972

119895=1(119909119894119895minus 119898119909) (119910119894119895minus 119898119910)

radicsum1198972

119895=1(119909119894119895minus 119898119909)2

sum1198972

119895=1(119910119894119895minus 119898119910)2

120588col (XY) =1

1198972

1198972

sum

119895=1

sum1198971

119894=1(119909119894119895minus 119898119909) (119910119894119895minus 119898119910)

radicsum1198971

119894=1(119909119894119895minus 119898119909)2

sum1198971

119894=1(119910119894119895minus 119898119910)2

(1)

where119898119909and119898

119910are the average of all the elements inX and

Y respectively The distance value 119889(XY) is in [0 2] whichis similar to the conventional correlation distanceWe shouldnote that the distance measure assumes that two matrices XandY have the same sizeTherefore in the case of gesture datawith various row sizes depending on the length of the videoan additional process is required to perform size alignment

4 BioMed Research International

48 b

lock

s

Raw image sequence Gray image sequence Frame differenceimage sequence

HOG descriptor StackedHOG descriptor

T

StackedHOG descriptor

(6 times 8 grid)

1 times 31 vector

(T times 1488 matrix)

1 times 1488 vector

(48 times 31 times T matrix)

T+1

frames

T+1

frames

Tfra

mes

Tfra

mes

Tfra

mes

(block size = 40 times 40)

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middotmiddot middot middotmiddot middot middotmiddot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Figure 3 Process of the gesture representation module

of two gesture feature matrices In this paper we apply adynamic time warping (DTW) algorithm [21] to align therows of matrices which is a technique to find an optimalalignment between two given sequences

After computing the distance values 119889119865= 119889(Fprb Fgal)

and 119889119866= 119889(GprbGgal) we need to make a decision of

acceptance using these values To do this we propose adecision criterion based on the likelihood ratio of the distancevalues which is defined by

119903119865119866(119889119865 119889119866) =119901 (Ω119860| 119889119865 119889119866)

119901 (Ω119868| 119889119865 119889119866)

=119901 (119889119865 119889119866| Ω119860) 119901 (Ω

119860)

119901 (119889119865 119889119866| Ω119868) 119901 (Ω

119868)

(2)

whereΩ119860denotes the class of distance values from authentic

data pairs and Ω119868denotes the class of distance values from

impostor data pairs Therefore 119903119865119866

indicates the ratio oflikelihood of whether the distance values (119889

119865 119889119866) originate

from an authentic data pair or an impostor data pair Inother words a large value of 119903

119865119866implies that the observed

distance (119889119865 119889119866) has a higher possibility of originating from

the population of authentic data pairsIn order to obtain an explicit function for calculating 119903

119865119866

we need to estimate the probability densities 119901(Ω119860| 119889119865 119889119866)

and 119901(Ω119868| 119889119865 119889119866) For real world implementation we

assume theGaussianmodel for119901(119889119865 119889119866| Ω119860) and119901(119889

119865 119889119866|

Ω119868) and estimate the parameters using gallery data Similarly

the prior probabilities 119901(Ω119860) and 119901(Ω

119868) are estimated too

Though the threshold 120579 is set for 1 typically it is changeable If120579 is high the number of false acceptances is decreased and thenumber of false rejections is increased If 120579 is low the reversephenomenon occurs In the experiments we measure theperformance of proposed authentication systemwith variable120579 A summarized description of decisionmodule is presentedin Algorithm 1

5 Experimental Results

In order to confirm the performance of proposed systemwe conducted experiments on the ChaLearn database [22]which was built for a gesture recognition competitionAlthough the data includes depth signals obtained from

Kinect we use only RGB signals because the proposedmethod is developed for a general vision sensor Figure 4shows some examples of the data From the whole dataset we prepared three setsmdashA B and Cmdashfor experimentsEach set is composed of 80 video samples from 20 subjectseach subject makes hisher own unique gesture four timesExperiments are carried out for each set separately using 4-fold cross-validation Three samples from each subject areused for gallery data and one sample is used for probe dataTherefore total 12 experiments were carried out

Before starting authentication we first need to estimatetwo conditional distributions 119901(119889

119865 119889119866| Ω119860) and 119901(119889

119865 119889119866|

Ω119868) which are used in decision criterion 119903

119865119866(119889119865 119889119866) For

each experiment we first make all possible data pairs fromgallery data and in order to obtain 1770 distance valuesamong which 60 values are from authentic pairs and 1710from impostor pairs The estimated pdf 119901(119889

119865 119889119866| Ω119860) and

119901(119889119865 119889119866| Ω119868) using these values are then applied to calcu-

late 119903119865119866(119889119865 119889119866) in the authentication phase For evaluating

authentication performance we compute distances betweengallery and probe data Since we have 20 probe samplesand 60 gallery samples there are 1200 distance values 60authentic values and 1140 impostor values The performanceis evaluated by the error rates (false acceptance and falserejection) of decision module for the 1200 values

We compared the performance of the decision moduleby changing modality and other conventional distance mea-sures In the unimodal case we use marginal distributionsuch as 119901(119889

119865| Ω119860) and 119901(119889

119866| Ω119860) for obtaining the

decision criterion We first compared the value of equalerror rate (EER) which is a typical measure for evaluatingauthentication systems EER is the value of error rate whenthe false acceptance rate (FAR) is equal to the false rejectionrate (FRR) Figure 5 shows the average EER over 4-foldcross-validation for each set A B and C As can be seenfrom Figure 5 gesture-based unimodal system shows slightlybetter performance than face-based unimodal system Alsothe proposed multimodal biometric system shows the bestresult

In Figure 6 we present the detection error tradeoff(DET) curves [23] for visualized comparison among differentmodalities with various distance measures The DET curveis a plot of error rates for binary classification systems inwhich the lower left curve implies the better performance

BioMed Research International 5

Input Feature matrices of face Fprb and gesture Gprb for aprobe video with user ID and a threshold 120579Output Authentication Result (AcceptReject)(1) Find a gallery data (FgalGgal) with user ID(2) Calculate the distance 119889

119865= 119889(Fprb Fgal) using (1)

(3) Align the gesture feature matrix Gprb and Ggal using DTWalgorithm(GprbGgal)

DTW997888997888997888997888rarr (Gprb Ggal)

Gprb and Ggal have same size(4) Calculate the distance 119889

119866= 119889(Gprb Ggal) using (1)

(5) Calculate the likelihood ratio 119903119865119866(119889119865 119889119866) using (2)

(6) if 119903119865119866(119889119865 119889119866) gt 120579 then

(7) Probe video is accepted(8) else(9) Probe video is rejected(10) end if

Algorithm 1 Pseudocode for the decision module

(a) (b)

Figure 4 Sample images from ChaLearn database (a) first frames of 20 selected users (b) image frames in a gesture video

7

6

5

4

3

2

1

0

EER

()

608 599

353

294

408

458

149

064

241

Unimodal(face)

Unimodal(gesture)

Multimodal(face + gesture)

Set ASet BSet C

Figure 5 Average EER () depending on biosignals using matrix correlation distance

6 BioMed Research International

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Manhattan distance

(a)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Euclidean distance

(b)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with matrix correlation distance

(c)

Figure 6 DET curves of authentication system with different modalities (a) Manhattan distance (b) Euclidean distance and (c) matrixcorrelation distance

BioMed Research International 7

60

55

50

45

40

35

30

25

20

15

10200 400 600 800 1000 1200 1400 1600 1800 2000

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

Manhattan distance between gallery and probe

ImpostorGenuine

(a)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

35

3

25

2

15

1

054 6 8 10 12 14 16 18

Euclidean distance between gallery and probe

ImpostorGenuine

(b)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

09

09

08

08

07

07

06

06

05

05

04

04

03

03

02

02

01

0

Matrix correlation distance between gallery and probe

ImpostorGenuine

(c)

Figure 7 Scatter plots of distance values between authentic pairs (I) as well as impostor pairs (◻) (a) Manhattan distance (b) Euclideandistance and (c) matrix correlation distance

As can be seen from Figure 6 the proposed multimodalbiometric system is superior to unimodal systems regardlessof the distance measures We can also observe that theperformance is dependent on the distance measures Forgesture conventional Manhattan distance and Euclideandistance give poor performance but the matrix correlationdistance shows improvement which is even better than faceThis effect is emphasized by the combination of face andgesture resulting in the remarkable improvement of DETcurves as shown in the solid curve of Figure 6(c)

Figure 7 shows the scatter plots of the distance values(119889119865 119889119866) in Ω

119860(I) as well as those in Ω

119868(◻) In this figure

we can observe that the discriminability is increased by usingmultimodality We also plot the marginal histogram of 119889

119865

and 119889119866on the corresponding axes The overlapped region of

histogram implies the region where decision error occurs Inthe case of a gesture we can see that the matrix correlationdistance can significantly decrease overlapped region Thismeans that matrix correlation distance is more appropriateto gesture data with our proposed feature representationAdditionally we can observe that the bivariate distributionsof (119889119865 119889119866) have the shape of ellipse which can justify our

Gaussian assumption for estimating the conditional distribu-tions 119901(119889

119865 119889119866| Ω119868) and 119901(119889

119865 119889119866| Ω119860) Moreover from

8 BioMed Research International

the shape of ellipse we can guess that the two modalities arealmost independent and this is supported by the fact that theaverage value of correlation coefficient is 019 This propertyis desirable for combining two biometric signals to constructmultimodal biometric system

6 Conclusion

In this paper we present a look into simple and efficientvision-based multimodal biometric system using heteroge-neous biometric signals By combining physical and behav-ioral biometric signals we can achieve a high degree ofreliability Because the proposed system uses a single visionsensor it can be easily implemented on commonly used smartdevices such as smart TVs More comprehensive study ondeveloping efficient feature extraction and classification willbe done for real world application of the proposal system

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research was partially supported by the DGIST RampDProgram of the Ministry of Education Science and Technol-ogy of Korea (13-IT-03) and Basic Science Research Programthrough the National Research Foundation of Korea (NRF)funded by theMinistry of Education Science andTechnology(NRF-2013R1A1A2061831)

References

[1] A Ross and A K Jain ldquoMultimodal biometrics an overviewrdquoin Proceedings of the 12th European Signal Processing Conferencepp 1221ndash1224 Vienna Austria September 2004

[2] K Bowyer K Chang and P Yan ldquoMulti-modal biometrics anoverviewrdquo in Proceedings of the 2nd Workshop on Multi-ModalUser Authentication Toulouse France May 2006

[3] A K Jain and A Kumar ldquoBiometric recognition an overviewrdquoin Second Generation Biometrics The Ethical Legal and SocialContext E Mordini and D Tzovaras Eds pp 49ndash79 SpringerAmsterdam The Netherlands 2012

[4] W Zhao R Chellappa P J Phillips and A Rosenfeld ldquoFacerecognition a literature surveyrdquo ACM Computing Surveys vol35 no 4 pp 399ndash458 2003

[5] R Jafri and H R Arabnia ldquoA survey of face recognitiontechniquesrdquo Journal of Information Processing Systems vol 5 no2 pp 41ndash68 2009

[6] I A Kakadiaris G Passalis T Theoharis G Toderici IKonstantinidis and NMurtuza ldquoMultimodal face recognitioncombination of geometry with physiological informationrdquo inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR rsquo05) vol 2 pp1022ndash1029 San Diego Calif USA June 2005

[7] K W Bowyer K Chang and P Flynn ldquoA survey of approachesand challenges in 3D and multi-modal 3D + 2D face recogni-tionrdquo Computer Vision and Image Understanding vol 101 no 1pp 1ndash15 2006

[8] K I Chang K W Bowyer and P J Flynn ldquoAn evaluationof multimodal 2D+3D face biometricsrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 27 no 4 pp 619ndash624 2005

[9] R V Yampolskiy and V Govindaraju ldquoBehavioural biometricsa survey and classificationrdquo International Journal of Biometricsvol 1 no 1 pp 81ndash113 2008

[10] J Liu ZWang L Zhong JWickramasuriya andVVasudevanldquouWave accelerometer-based personalized gesture recognitionand its applicationsrdquo in Proceedings of the 7th Annual IEEEInternational Conference on Pervasive Computing and Commu-nications (PerCom rsquo09) pp 1ndash9 Galveston Tex USA March2009

[11] G Bailador C Sanchez-Avila J Guerra-Casanova and A deSantos Sierra ldquoAnalysis of pattern recognition techniques for in-air signature biometricsrdquo Pattern Recognition vol 44 no 10-11pp 2468ndash2478 2011

[12] J Guerra-Casanova C Sanchez-Avila G Bailador and Ade Santos Sierra ldquoAuthentication in mobile devices throughhand gesture recognitionrdquo International Journal of InformationSecurity vol 11 no 2 pp 65ndash83 2012

[13] D Guse Gesture-based user authentication on mobile devicesusing accelerometer and gyroscope [Master thesis] Berlin Institueof Technology 2011

[14] N Sae-Bae K Ahmed K Isbister and N Memon ldquoBiometric-rich gestures a novel approach to authentication onmulti-touchdevicesrdquo in Proceedings of the 30th ACM Conference on HumanFactors in Computing Systems (CHI rsquo12) pp 977ndash986 AustinTex USA May 2012

[15] K Lai J Konrad and P Ishwar ldquoTowards gesture-based userauthenticationrdquo in Proceedings of the IEEE 9th InternationalConference on Advanced Video and Signal-Based Surveillance(AVSS rsquo12) pp 282ndash287 Beijing China September 2012

[16] J Wu J Konrad and P Ishwar ldquoThe value of multiple view-points in gesture-based user authenticationrdquo in Proceedingsof the IEEE Computer Society Conference on Computer Visionand Pattern RecognitionWorkshop pp 90ndash97 Columbus OhioUSA June 2014

[17] P Viola and M J Jones ldquoRobust real-time face detectionrdquoInternational Journal of Computer Vision vol 57 no 2 pp 137ndash154 2004

[18] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) pp 886ndash893 San Diego Calif USA June 2005

[19] A Vedaldi and B Fulkerson ldquoVLFeat An Open and PortableLibrary of Computer Vision Algorithmsrdquo httpwwwvlfeatorg

[20] H Choi J Seo andH Park ldquoMatrix correlation distance for 2Dimage classificationrdquo in Proceedings of the of ACM Symposiumon Applied Computing pp 1741ndash1742 Gyeongju Republic ofKorea March 2014

[21] M Muller ldquoDynamic time warpingrdquo in Information Retrievalfor Music andMotion M Muller Ed pp 69ndash84 Springer NewYork NY USA 2007

[22] ChaLearn ChaLearn Gesture Dataset (CGD 2011) 2012 httpgesturechalearnorgdata

[23] A Martin G Doddington and T Kamm ldquoThe DET curvein assessment of detection task performancerdquo in Proceedingsof the European Conference on Speech Communication andTechnology pp 1895ndash1898 Rhodes Greece September 1997

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology

Page 4: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

4 BioMed Research International

48 b

lock

s

Raw image sequence Gray image sequence Frame differenceimage sequence

HOG descriptor StackedHOG descriptor

T

StackedHOG descriptor

(6 times 8 grid)

1 times 31 vector

(T times 1488 matrix)

1 times 1488 vector

(48 times 31 times T matrix)

T+1

frames

T+1

frames

Tfra

mes

Tfra

mes

Tfra

mes

(block size = 40 times 40)

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middotmiddot middot middotmiddot middot middotmiddot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Figure 3 Process of the gesture representation module

of two gesture feature matrices In this paper we apply adynamic time warping (DTW) algorithm [21] to align therows of matrices which is a technique to find an optimalalignment between two given sequences

After computing the distance values 119889119865= 119889(Fprb Fgal)

and 119889119866= 119889(GprbGgal) we need to make a decision of

acceptance using these values To do this we propose adecision criterion based on the likelihood ratio of the distancevalues which is defined by

119903119865119866(119889119865 119889119866) =119901 (Ω119860| 119889119865 119889119866)

119901 (Ω119868| 119889119865 119889119866)

=119901 (119889119865 119889119866| Ω119860) 119901 (Ω

119860)

119901 (119889119865 119889119866| Ω119868) 119901 (Ω

119868)

(2)

whereΩ119860denotes the class of distance values from authentic

data pairs and Ω119868denotes the class of distance values from

impostor data pairs Therefore 119903119865119866

indicates the ratio oflikelihood of whether the distance values (119889

119865 119889119866) originate

from an authentic data pair or an impostor data pair Inother words a large value of 119903

119865119866implies that the observed

distance (119889119865 119889119866) has a higher possibility of originating from

the population of authentic data pairsIn order to obtain an explicit function for calculating 119903

119865119866

we need to estimate the probability densities 119901(Ω119860| 119889119865 119889119866)

and 119901(Ω119868| 119889119865 119889119866) For real world implementation we

assume theGaussianmodel for119901(119889119865 119889119866| Ω119860) and119901(119889

119865 119889119866|

Ω119868) and estimate the parameters using gallery data Similarly

the prior probabilities 119901(Ω119860) and 119901(Ω

119868) are estimated too

Though the threshold 120579 is set for 1 typically it is changeable If120579 is high the number of false acceptances is decreased and thenumber of false rejections is increased If 120579 is low the reversephenomenon occurs In the experiments we measure theperformance of proposed authentication systemwith variable120579 A summarized description of decisionmodule is presentedin Algorithm 1

5 Experimental Results

In order to confirm the performance of proposed systemwe conducted experiments on the ChaLearn database [22]which was built for a gesture recognition competitionAlthough the data includes depth signals obtained from

Kinect we use only RGB signals because the proposedmethod is developed for a general vision sensor Figure 4shows some examples of the data From the whole dataset we prepared three setsmdashA B and Cmdashfor experimentsEach set is composed of 80 video samples from 20 subjectseach subject makes hisher own unique gesture four timesExperiments are carried out for each set separately using 4-fold cross-validation Three samples from each subject areused for gallery data and one sample is used for probe dataTherefore total 12 experiments were carried out

Before starting authentication we first need to estimatetwo conditional distributions 119901(119889

119865 119889119866| Ω119860) and 119901(119889

119865 119889119866|

Ω119868) which are used in decision criterion 119903

119865119866(119889119865 119889119866) For

each experiment we first make all possible data pairs fromgallery data and in order to obtain 1770 distance valuesamong which 60 values are from authentic pairs and 1710from impostor pairs The estimated pdf 119901(119889

119865 119889119866| Ω119860) and

119901(119889119865 119889119866| Ω119868) using these values are then applied to calcu-

late 119903119865119866(119889119865 119889119866) in the authentication phase For evaluating

authentication performance we compute distances betweengallery and probe data Since we have 20 probe samplesand 60 gallery samples there are 1200 distance values 60authentic values and 1140 impostor values The performanceis evaluated by the error rates (false acceptance and falserejection) of decision module for the 1200 values

We compared the performance of the decision moduleby changing modality and other conventional distance mea-sures In the unimodal case we use marginal distributionsuch as 119901(119889

119865| Ω119860) and 119901(119889

119866| Ω119860) for obtaining the

decision criterion We first compared the value of equalerror rate (EER) which is a typical measure for evaluatingauthentication systems EER is the value of error rate whenthe false acceptance rate (FAR) is equal to the false rejectionrate (FRR) Figure 5 shows the average EER over 4-foldcross-validation for each set A B and C As can be seenfrom Figure 5 gesture-based unimodal system shows slightlybetter performance than face-based unimodal system Alsothe proposed multimodal biometric system shows the bestresult

In Figure 6 we present the detection error tradeoff(DET) curves [23] for visualized comparison among differentmodalities with various distance measures The DET curveis a plot of error rates for binary classification systems inwhich the lower left curve implies the better performance

BioMed Research International 5

Input Feature matrices of face Fprb and gesture Gprb for aprobe video with user ID and a threshold 120579Output Authentication Result (AcceptReject)(1) Find a gallery data (FgalGgal) with user ID(2) Calculate the distance 119889

119865= 119889(Fprb Fgal) using (1)

(3) Align the gesture feature matrix Gprb and Ggal using DTWalgorithm(GprbGgal)

DTW997888997888997888997888rarr (Gprb Ggal)

Gprb and Ggal have same size(4) Calculate the distance 119889

119866= 119889(Gprb Ggal) using (1)

(5) Calculate the likelihood ratio 119903119865119866(119889119865 119889119866) using (2)

(6) if 119903119865119866(119889119865 119889119866) gt 120579 then

(7) Probe video is accepted(8) else(9) Probe video is rejected(10) end if

Algorithm 1 Pseudocode for the decision module

(a) (b)

Figure 4 Sample images from ChaLearn database (a) first frames of 20 selected users (b) image frames in a gesture video

7

6

5

4

3

2

1

0

EER

()

608 599

353

294

408

458

149

064

241

Unimodal(face)

Unimodal(gesture)

Multimodal(face + gesture)

Set ASet BSet C

Figure 5 Average EER () depending on biosignals using matrix correlation distance

6 BioMed Research International

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Manhattan distance

(a)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Euclidean distance

(b)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with matrix correlation distance

(c)

Figure 6 DET curves of authentication system with different modalities (a) Manhattan distance (b) Euclidean distance and (c) matrixcorrelation distance

BioMed Research International 7

60

55

50

45

40

35

30

25

20

15

10200 400 600 800 1000 1200 1400 1600 1800 2000

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

Manhattan distance between gallery and probe

ImpostorGenuine

(a)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

35

3

25

2

15

1

054 6 8 10 12 14 16 18

Euclidean distance between gallery and probe

ImpostorGenuine

(b)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

09

09

08

08

07

07

06

06

05

05

04

04

03

03

02

02

01

0

Matrix correlation distance between gallery and probe

ImpostorGenuine

(c)

Figure 7 Scatter plots of distance values between authentic pairs (I) as well as impostor pairs (◻) (a) Manhattan distance (b) Euclideandistance and (c) matrix correlation distance

As can be seen from Figure 6 the proposed multimodalbiometric system is superior to unimodal systems regardlessof the distance measures We can also observe that theperformance is dependent on the distance measures Forgesture conventional Manhattan distance and Euclideandistance give poor performance but the matrix correlationdistance shows improvement which is even better than faceThis effect is emphasized by the combination of face andgesture resulting in the remarkable improvement of DETcurves as shown in the solid curve of Figure 6(c)

Figure 7 shows the scatter plots of the distance values(119889119865 119889119866) in Ω

119860(I) as well as those in Ω

119868(◻) In this figure

we can observe that the discriminability is increased by usingmultimodality We also plot the marginal histogram of 119889

119865

and 119889119866on the corresponding axes The overlapped region of

histogram implies the region where decision error occurs Inthe case of a gesture we can see that the matrix correlationdistance can significantly decrease overlapped region Thismeans that matrix correlation distance is more appropriateto gesture data with our proposed feature representationAdditionally we can observe that the bivariate distributionsof (119889119865 119889119866) have the shape of ellipse which can justify our

Gaussian assumption for estimating the conditional distribu-tions 119901(119889

119865 119889119866| Ω119868) and 119901(119889

119865 119889119866| Ω119860) Moreover from

8 BioMed Research International

the shape of ellipse we can guess that the two modalities arealmost independent and this is supported by the fact that theaverage value of correlation coefficient is 019 This propertyis desirable for combining two biometric signals to constructmultimodal biometric system

6 Conclusion

In this paper we present a look into simple and efficientvision-based multimodal biometric system using heteroge-neous biometric signals By combining physical and behav-ioral biometric signals we can achieve a high degree ofreliability Because the proposed system uses a single visionsensor it can be easily implemented on commonly used smartdevices such as smart TVs More comprehensive study ondeveloping efficient feature extraction and classification willbe done for real world application of the proposal system

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research was partially supported by the DGIST RampDProgram of the Ministry of Education Science and Technol-ogy of Korea (13-IT-03) and Basic Science Research Programthrough the National Research Foundation of Korea (NRF)funded by theMinistry of Education Science andTechnology(NRF-2013R1A1A2061831)

References

[1] A Ross and A K Jain ldquoMultimodal biometrics an overviewrdquoin Proceedings of the 12th European Signal Processing Conferencepp 1221ndash1224 Vienna Austria September 2004

[2] K Bowyer K Chang and P Yan ldquoMulti-modal biometrics anoverviewrdquo in Proceedings of the 2nd Workshop on Multi-ModalUser Authentication Toulouse France May 2006

[3] A K Jain and A Kumar ldquoBiometric recognition an overviewrdquoin Second Generation Biometrics The Ethical Legal and SocialContext E Mordini and D Tzovaras Eds pp 49ndash79 SpringerAmsterdam The Netherlands 2012

[4] W Zhao R Chellappa P J Phillips and A Rosenfeld ldquoFacerecognition a literature surveyrdquo ACM Computing Surveys vol35 no 4 pp 399ndash458 2003

[5] R Jafri and H R Arabnia ldquoA survey of face recognitiontechniquesrdquo Journal of Information Processing Systems vol 5 no2 pp 41ndash68 2009

[6] I A Kakadiaris G Passalis T Theoharis G Toderici IKonstantinidis and NMurtuza ldquoMultimodal face recognitioncombination of geometry with physiological informationrdquo inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR rsquo05) vol 2 pp1022ndash1029 San Diego Calif USA June 2005

[7] K W Bowyer K Chang and P Flynn ldquoA survey of approachesand challenges in 3D and multi-modal 3D + 2D face recogni-tionrdquo Computer Vision and Image Understanding vol 101 no 1pp 1ndash15 2006

[8] K I Chang K W Bowyer and P J Flynn ldquoAn evaluationof multimodal 2D+3D face biometricsrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 27 no 4 pp 619ndash624 2005

[9] R V Yampolskiy and V Govindaraju ldquoBehavioural biometricsa survey and classificationrdquo International Journal of Biometricsvol 1 no 1 pp 81ndash113 2008

[10] J Liu ZWang L Zhong JWickramasuriya andVVasudevanldquouWave accelerometer-based personalized gesture recognitionand its applicationsrdquo in Proceedings of the 7th Annual IEEEInternational Conference on Pervasive Computing and Commu-nications (PerCom rsquo09) pp 1ndash9 Galveston Tex USA March2009

[11] G Bailador C Sanchez-Avila J Guerra-Casanova and A deSantos Sierra ldquoAnalysis of pattern recognition techniques for in-air signature biometricsrdquo Pattern Recognition vol 44 no 10-11pp 2468ndash2478 2011

[12] J Guerra-Casanova C Sanchez-Avila G Bailador and Ade Santos Sierra ldquoAuthentication in mobile devices throughhand gesture recognitionrdquo International Journal of InformationSecurity vol 11 no 2 pp 65ndash83 2012

[13] D Guse Gesture-based user authentication on mobile devicesusing accelerometer and gyroscope [Master thesis] Berlin Institueof Technology 2011

[14] N Sae-Bae K Ahmed K Isbister and N Memon ldquoBiometric-rich gestures a novel approach to authentication onmulti-touchdevicesrdquo in Proceedings of the 30th ACM Conference on HumanFactors in Computing Systems (CHI rsquo12) pp 977ndash986 AustinTex USA May 2012

[15] K Lai J Konrad and P Ishwar ldquoTowards gesture-based userauthenticationrdquo in Proceedings of the IEEE 9th InternationalConference on Advanced Video and Signal-Based Surveillance(AVSS rsquo12) pp 282ndash287 Beijing China September 2012

[16] J Wu J Konrad and P Ishwar ldquoThe value of multiple view-points in gesture-based user authenticationrdquo in Proceedingsof the IEEE Computer Society Conference on Computer Visionand Pattern RecognitionWorkshop pp 90ndash97 Columbus OhioUSA June 2014

[17] P Viola and M J Jones ldquoRobust real-time face detectionrdquoInternational Journal of Computer Vision vol 57 no 2 pp 137ndash154 2004

[18] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) pp 886ndash893 San Diego Calif USA June 2005

[19] A Vedaldi and B Fulkerson ldquoVLFeat An Open and PortableLibrary of Computer Vision Algorithmsrdquo httpwwwvlfeatorg

[20] H Choi J Seo andH Park ldquoMatrix correlation distance for 2Dimage classificationrdquo in Proceedings of the of ACM Symposiumon Applied Computing pp 1741ndash1742 Gyeongju Republic ofKorea March 2014

[21] M Muller ldquoDynamic time warpingrdquo in Information Retrievalfor Music andMotion M Muller Ed pp 69ndash84 Springer NewYork NY USA 2007

[22] ChaLearn ChaLearn Gesture Dataset (CGD 2011) 2012 httpgesturechalearnorgdata

[23] A Martin G Doddington and T Kamm ldquoThe DET curvein assessment of detection task performancerdquo in Proceedingsof the European Conference on Speech Communication andTechnology pp 1895ndash1898 Rhodes Greece September 1997

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology

Page 5: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

BioMed Research International 5

Input Feature matrices of face Fprb and gesture Gprb for aprobe video with user ID and a threshold 120579Output Authentication Result (AcceptReject)(1) Find a gallery data (FgalGgal) with user ID(2) Calculate the distance 119889

119865= 119889(Fprb Fgal) using (1)

(3) Align the gesture feature matrix Gprb and Ggal using DTWalgorithm(GprbGgal)

DTW997888997888997888997888rarr (Gprb Ggal)

Gprb and Ggal have same size(4) Calculate the distance 119889

119866= 119889(Gprb Ggal) using (1)

(5) Calculate the likelihood ratio 119903119865119866(119889119865 119889119866) using (2)

(6) if 119903119865119866(119889119865 119889119866) gt 120579 then

(7) Probe video is accepted(8) else(9) Probe video is rejected(10) end if

Algorithm 1 Pseudocode for the decision module

(a) (b)

Figure 4 Sample images from ChaLearn database (a) first frames of 20 selected users (b) image frames in a gesture video

7

6

5

4

3

2

1

0

EER

()

608 599

353

294

408

458

149

064

241

Unimodal(face)

Unimodal(gesture)

Multimodal(face + gesture)

Set ASet BSet C

Figure 5 Average EER () depending on biosignals using matrix correlation distance

6 BioMed Research International

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Manhattan distance

(a)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Euclidean distance

(b)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with matrix correlation distance

(c)

Figure 6 DET curves of authentication system with different modalities (a) Manhattan distance (b) Euclidean distance and (c) matrixcorrelation distance

BioMed Research International 7

60

55

50

45

40

35

30

25

20

15

10200 400 600 800 1000 1200 1400 1600 1800 2000

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

Manhattan distance between gallery and probe

ImpostorGenuine

(a)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

35

3

25

2

15

1

054 6 8 10 12 14 16 18

Euclidean distance between gallery and probe

ImpostorGenuine

(b)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

09

09

08

08

07

07

06

06

05

05

04

04

03

03

02

02

01

0

Matrix correlation distance between gallery and probe

ImpostorGenuine

(c)

Figure 7 Scatter plots of distance values between authentic pairs (I) as well as impostor pairs (◻) (a) Manhattan distance (b) Euclideandistance and (c) matrix correlation distance

As can be seen from Figure 6 the proposed multimodalbiometric system is superior to unimodal systems regardlessof the distance measures We can also observe that theperformance is dependent on the distance measures Forgesture conventional Manhattan distance and Euclideandistance give poor performance but the matrix correlationdistance shows improvement which is even better than faceThis effect is emphasized by the combination of face andgesture resulting in the remarkable improvement of DETcurves as shown in the solid curve of Figure 6(c)

Figure 7 shows the scatter plots of the distance values(119889119865 119889119866) in Ω

119860(I) as well as those in Ω

119868(◻) In this figure

we can observe that the discriminability is increased by usingmultimodality We also plot the marginal histogram of 119889

119865

and 119889119866on the corresponding axes The overlapped region of

histogram implies the region where decision error occurs Inthe case of a gesture we can see that the matrix correlationdistance can significantly decrease overlapped region Thismeans that matrix correlation distance is more appropriateto gesture data with our proposed feature representationAdditionally we can observe that the bivariate distributionsof (119889119865 119889119866) have the shape of ellipse which can justify our

Gaussian assumption for estimating the conditional distribu-tions 119901(119889

119865 119889119866| Ω119868) and 119901(119889

119865 119889119866| Ω119860) Moreover from

8 BioMed Research International

the shape of ellipse we can guess that the two modalities arealmost independent and this is supported by the fact that theaverage value of correlation coefficient is 019 This propertyis desirable for combining two biometric signals to constructmultimodal biometric system

6 Conclusion

In this paper we present a look into simple and efficientvision-based multimodal biometric system using heteroge-neous biometric signals By combining physical and behav-ioral biometric signals we can achieve a high degree ofreliability Because the proposed system uses a single visionsensor it can be easily implemented on commonly used smartdevices such as smart TVs More comprehensive study ondeveloping efficient feature extraction and classification willbe done for real world application of the proposal system

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research was partially supported by the DGIST RampDProgram of the Ministry of Education Science and Technol-ogy of Korea (13-IT-03) and Basic Science Research Programthrough the National Research Foundation of Korea (NRF)funded by theMinistry of Education Science andTechnology(NRF-2013R1A1A2061831)

References

[1] A Ross and A K Jain ldquoMultimodal biometrics an overviewrdquoin Proceedings of the 12th European Signal Processing Conferencepp 1221ndash1224 Vienna Austria September 2004

[2] K Bowyer K Chang and P Yan ldquoMulti-modal biometrics anoverviewrdquo in Proceedings of the 2nd Workshop on Multi-ModalUser Authentication Toulouse France May 2006

[3] A K Jain and A Kumar ldquoBiometric recognition an overviewrdquoin Second Generation Biometrics The Ethical Legal and SocialContext E Mordini and D Tzovaras Eds pp 49ndash79 SpringerAmsterdam The Netherlands 2012

[4] W Zhao R Chellappa P J Phillips and A Rosenfeld ldquoFacerecognition a literature surveyrdquo ACM Computing Surveys vol35 no 4 pp 399ndash458 2003

[5] R Jafri and H R Arabnia ldquoA survey of face recognitiontechniquesrdquo Journal of Information Processing Systems vol 5 no2 pp 41ndash68 2009

[6] I A Kakadiaris G Passalis T Theoharis G Toderici IKonstantinidis and NMurtuza ldquoMultimodal face recognitioncombination of geometry with physiological informationrdquo inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR rsquo05) vol 2 pp1022ndash1029 San Diego Calif USA June 2005

[7] K W Bowyer K Chang and P Flynn ldquoA survey of approachesand challenges in 3D and multi-modal 3D + 2D face recogni-tionrdquo Computer Vision and Image Understanding vol 101 no 1pp 1ndash15 2006

[8] K I Chang K W Bowyer and P J Flynn ldquoAn evaluationof multimodal 2D+3D face biometricsrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 27 no 4 pp 619ndash624 2005

[9] R V Yampolskiy and V Govindaraju ldquoBehavioural biometricsa survey and classificationrdquo International Journal of Biometricsvol 1 no 1 pp 81ndash113 2008

[10] J Liu ZWang L Zhong JWickramasuriya andVVasudevanldquouWave accelerometer-based personalized gesture recognitionand its applicationsrdquo in Proceedings of the 7th Annual IEEEInternational Conference on Pervasive Computing and Commu-nications (PerCom rsquo09) pp 1ndash9 Galveston Tex USA March2009

[11] G Bailador C Sanchez-Avila J Guerra-Casanova and A deSantos Sierra ldquoAnalysis of pattern recognition techniques for in-air signature biometricsrdquo Pattern Recognition vol 44 no 10-11pp 2468ndash2478 2011

[12] J Guerra-Casanova C Sanchez-Avila G Bailador and Ade Santos Sierra ldquoAuthentication in mobile devices throughhand gesture recognitionrdquo International Journal of InformationSecurity vol 11 no 2 pp 65ndash83 2012

[13] D Guse Gesture-based user authentication on mobile devicesusing accelerometer and gyroscope [Master thesis] Berlin Institueof Technology 2011

[14] N Sae-Bae K Ahmed K Isbister and N Memon ldquoBiometric-rich gestures a novel approach to authentication onmulti-touchdevicesrdquo in Proceedings of the 30th ACM Conference on HumanFactors in Computing Systems (CHI rsquo12) pp 977ndash986 AustinTex USA May 2012

[15] K Lai J Konrad and P Ishwar ldquoTowards gesture-based userauthenticationrdquo in Proceedings of the IEEE 9th InternationalConference on Advanced Video and Signal-Based Surveillance(AVSS rsquo12) pp 282ndash287 Beijing China September 2012

[16] J Wu J Konrad and P Ishwar ldquoThe value of multiple view-points in gesture-based user authenticationrdquo in Proceedingsof the IEEE Computer Society Conference on Computer Visionand Pattern RecognitionWorkshop pp 90ndash97 Columbus OhioUSA June 2014

[17] P Viola and M J Jones ldquoRobust real-time face detectionrdquoInternational Journal of Computer Vision vol 57 no 2 pp 137ndash154 2004

[18] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) pp 886ndash893 San Diego Calif USA June 2005

[19] A Vedaldi and B Fulkerson ldquoVLFeat An Open and PortableLibrary of Computer Vision Algorithmsrdquo httpwwwvlfeatorg

[20] H Choi J Seo andH Park ldquoMatrix correlation distance for 2Dimage classificationrdquo in Proceedings of the of ACM Symposiumon Applied Computing pp 1741ndash1742 Gyeongju Republic ofKorea March 2014

[21] M Muller ldquoDynamic time warpingrdquo in Information Retrievalfor Music andMotion M Muller Ed pp 69ndash84 Springer NewYork NY USA 2007

[22] ChaLearn ChaLearn Gesture Dataset (CGD 2011) 2012 httpgesturechalearnorgdata

[23] A Martin G Doddington and T Kamm ldquoThe DET curvein assessment of detection task performancerdquo in Proceedingsof the European Conference on Speech Communication andTechnology pp 1895ndash1898 Rhodes Greece September 1997

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology

Page 6: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

6 BioMed Research International

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Manhattan distance

(a)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with Euclidean distance

(b)

40

20

10

5

2

1

05

02

01

Miss

pro

babi

lity

()

01 02 05 1 2 5 10 20 40

False alarm probability ()

FaceGestureFace + gesture

DET with matrix correlation distance

(c)

Figure 6 DET curves of authentication system with different modalities (a) Manhattan distance (b) Euclidean distance and (c) matrixcorrelation distance

BioMed Research International 7

60

55

50

45

40

35

30

25

20

15

10200 400 600 800 1000 1200 1400 1600 1800 2000

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

Manhattan distance between gallery and probe

ImpostorGenuine

(a)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

35

3

25

2

15

1

054 6 8 10 12 14 16 18

Euclidean distance between gallery and probe

ImpostorGenuine

(b)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

09

09

08

08

07

07

06

06

05

05

04

04

03

03

02

02

01

0

Matrix correlation distance between gallery and probe

ImpostorGenuine

(c)

Figure 7 Scatter plots of distance values between authentic pairs (I) as well as impostor pairs (◻) (a) Manhattan distance (b) Euclideandistance and (c) matrix correlation distance

As can be seen from Figure 6 the proposed multimodalbiometric system is superior to unimodal systems regardlessof the distance measures We can also observe that theperformance is dependent on the distance measures Forgesture conventional Manhattan distance and Euclideandistance give poor performance but the matrix correlationdistance shows improvement which is even better than faceThis effect is emphasized by the combination of face andgesture resulting in the remarkable improvement of DETcurves as shown in the solid curve of Figure 6(c)

Figure 7 shows the scatter plots of the distance values(119889119865 119889119866) in Ω

119860(I) as well as those in Ω

119868(◻) In this figure

we can observe that the discriminability is increased by usingmultimodality We also plot the marginal histogram of 119889

119865

and 119889119866on the corresponding axes The overlapped region of

histogram implies the region where decision error occurs Inthe case of a gesture we can see that the matrix correlationdistance can significantly decrease overlapped region Thismeans that matrix correlation distance is more appropriateto gesture data with our proposed feature representationAdditionally we can observe that the bivariate distributionsof (119889119865 119889119866) have the shape of ellipse which can justify our

Gaussian assumption for estimating the conditional distribu-tions 119901(119889

119865 119889119866| Ω119868) and 119901(119889

119865 119889119866| Ω119860) Moreover from

8 BioMed Research International

the shape of ellipse we can guess that the two modalities arealmost independent and this is supported by the fact that theaverage value of correlation coefficient is 019 This propertyis desirable for combining two biometric signals to constructmultimodal biometric system

6 Conclusion

In this paper we present a look into simple and efficientvision-based multimodal biometric system using heteroge-neous biometric signals By combining physical and behav-ioral biometric signals we can achieve a high degree ofreliability Because the proposed system uses a single visionsensor it can be easily implemented on commonly used smartdevices such as smart TVs More comprehensive study ondeveloping efficient feature extraction and classification willbe done for real world application of the proposal system

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research was partially supported by the DGIST RampDProgram of the Ministry of Education Science and Technol-ogy of Korea (13-IT-03) and Basic Science Research Programthrough the National Research Foundation of Korea (NRF)funded by theMinistry of Education Science andTechnology(NRF-2013R1A1A2061831)

References

[1] A Ross and A K Jain ldquoMultimodal biometrics an overviewrdquoin Proceedings of the 12th European Signal Processing Conferencepp 1221ndash1224 Vienna Austria September 2004

[2] K Bowyer K Chang and P Yan ldquoMulti-modal biometrics anoverviewrdquo in Proceedings of the 2nd Workshop on Multi-ModalUser Authentication Toulouse France May 2006

[3] A K Jain and A Kumar ldquoBiometric recognition an overviewrdquoin Second Generation Biometrics The Ethical Legal and SocialContext E Mordini and D Tzovaras Eds pp 49ndash79 SpringerAmsterdam The Netherlands 2012

[4] W Zhao R Chellappa P J Phillips and A Rosenfeld ldquoFacerecognition a literature surveyrdquo ACM Computing Surveys vol35 no 4 pp 399ndash458 2003

[5] R Jafri and H R Arabnia ldquoA survey of face recognitiontechniquesrdquo Journal of Information Processing Systems vol 5 no2 pp 41ndash68 2009

[6] I A Kakadiaris G Passalis T Theoharis G Toderici IKonstantinidis and NMurtuza ldquoMultimodal face recognitioncombination of geometry with physiological informationrdquo inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR rsquo05) vol 2 pp1022ndash1029 San Diego Calif USA June 2005

[7] K W Bowyer K Chang and P Flynn ldquoA survey of approachesand challenges in 3D and multi-modal 3D + 2D face recogni-tionrdquo Computer Vision and Image Understanding vol 101 no 1pp 1ndash15 2006

[8] K I Chang K W Bowyer and P J Flynn ldquoAn evaluationof multimodal 2D+3D face biometricsrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 27 no 4 pp 619ndash624 2005

[9] R V Yampolskiy and V Govindaraju ldquoBehavioural biometricsa survey and classificationrdquo International Journal of Biometricsvol 1 no 1 pp 81ndash113 2008

[10] J Liu ZWang L Zhong JWickramasuriya andVVasudevanldquouWave accelerometer-based personalized gesture recognitionand its applicationsrdquo in Proceedings of the 7th Annual IEEEInternational Conference on Pervasive Computing and Commu-nications (PerCom rsquo09) pp 1ndash9 Galveston Tex USA March2009

[11] G Bailador C Sanchez-Avila J Guerra-Casanova and A deSantos Sierra ldquoAnalysis of pattern recognition techniques for in-air signature biometricsrdquo Pattern Recognition vol 44 no 10-11pp 2468ndash2478 2011

[12] J Guerra-Casanova C Sanchez-Avila G Bailador and Ade Santos Sierra ldquoAuthentication in mobile devices throughhand gesture recognitionrdquo International Journal of InformationSecurity vol 11 no 2 pp 65ndash83 2012

[13] D Guse Gesture-based user authentication on mobile devicesusing accelerometer and gyroscope [Master thesis] Berlin Institueof Technology 2011

[14] N Sae-Bae K Ahmed K Isbister and N Memon ldquoBiometric-rich gestures a novel approach to authentication onmulti-touchdevicesrdquo in Proceedings of the 30th ACM Conference on HumanFactors in Computing Systems (CHI rsquo12) pp 977ndash986 AustinTex USA May 2012

[15] K Lai J Konrad and P Ishwar ldquoTowards gesture-based userauthenticationrdquo in Proceedings of the IEEE 9th InternationalConference on Advanced Video and Signal-Based Surveillance(AVSS rsquo12) pp 282ndash287 Beijing China September 2012

[16] J Wu J Konrad and P Ishwar ldquoThe value of multiple view-points in gesture-based user authenticationrdquo in Proceedingsof the IEEE Computer Society Conference on Computer Visionand Pattern RecognitionWorkshop pp 90ndash97 Columbus OhioUSA June 2014

[17] P Viola and M J Jones ldquoRobust real-time face detectionrdquoInternational Journal of Computer Vision vol 57 no 2 pp 137ndash154 2004

[18] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) pp 886ndash893 San Diego Calif USA June 2005

[19] A Vedaldi and B Fulkerson ldquoVLFeat An Open and PortableLibrary of Computer Vision Algorithmsrdquo httpwwwvlfeatorg

[20] H Choi J Seo andH Park ldquoMatrix correlation distance for 2Dimage classificationrdquo in Proceedings of the of ACM Symposiumon Applied Computing pp 1741ndash1742 Gyeongju Republic ofKorea March 2014

[21] M Muller ldquoDynamic time warpingrdquo in Information Retrievalfor Music andMotion M Muller Ed pp 69ndash84 Springer NewYork NY USA 2007

[22] ChaLearn ChaLearn Gesture Dataset (CGD 2011) 2012 httpgesturechalearnorgdata

[23] A Martin G Doddington and T Kamm ldquoThe DET curvein assessment of detection task performancerdquo in Proceedingsof the European Conference on Speech Communication andTechnology pp 1895ndash1898 Rhodes Greece September 1997

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology

Page 7: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

BioMed Research International 7

60

55

50

45

40

35

30

25

20

15

10200 400 600 800 1000 1200 1400 1600 1800 2000

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

Manhattan distance between gallery and probe

ImpostorGenuine

(a)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

35

3

25

2

15

1

054 6 8 10 12 14 16 18

Euclidean distance between gallery and probe

ImpostorGenuine

(b)

Distance of gesture (dG)

Dist

ance

of f

ace (

dF)

09

09

08

08

07

07

06

06

05

05

04

04

03

03

02

02

01

0

Matrix correlation distance between gallery and probe

ImpostorGenuine

(c)

Figure 7 Scatter plots of distance values between authentic pairs (I) as well as impostor pairs (◻) (a) Manhattan distance (b) Euclideandistance and (c) matrix correlation distance

As can be seen from Figure 6 the proposed multimodalbiometric system is superior to unimodal systems regardlessof the distance measures We can also observe that theperformance is dependent on the distance measures Forgesture conventional Manhattan distance and Euclideandistance give poor performance but the matrix correlationdistance shows improvement which is even better than faceThis effect is emphasized by the combination of face andgesture resulting in the remarkable improvement of DETcurves as shown in the solid curve of Figure 6(c)

Figure 7 shows the scatter plots of the distance values(119889119865 119889119866) in Ω

119860(I) as well as those in Ω

119868(◻) In this figure

we can observe that the discriminability is increased by usingmultimodality We also plot the marginal histogram of 119889

119865

and 119889119866on the corresponding axes The overlapped region of

histogram implies the region where decision error occurs Inthe case of a gesture we can see that the matrix correlationdistance can significantly decrease overlapped region Thismeans that matrix correlation distance is more appropriateto gesture data with our proposed feature representationAdditionally we can observe that the bivariate distributionsof (119889119865 119889119866) have the shape of ellipse which can justify our

Gaussian assumption for estimating the conditional distribu-tions 119901(119889

119865 119889119866| Ω119868) and 119901(119889

119865 119889119866| Ω119860) Moreover from

8 BioMed Research International

the shape of ellipse we can guess that the two modalities arealmost independent and this is supported by the fact that theaverage value of correlation coefficient is 019 This propertyis desirable for combining two biometric signals to constructmultimodal biometric system

6 Conclusion

In this paper we present a look into simple and efficientvision-based multimodal biometric system using heteroge-neous biometric signals By combining physical and behav-ioral biometric signals we can achieve a high degree ofreliability Because the proposed system uses a single visionsensor it can be easily implemented on commonly used smartdevices such as smart TVs More comprehensive study ondeveloping efficient feature extraction and classification willbe done for real world application of the proposal system

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research was partially supported by the DGIST RampDProgram of the Ministry of Education Science and Technol-ogy of Korea (13-IT-03) and Basic Science Research Programthrough the National Research Foundation of Korea (NRF)funded by theMinistry of Education Science andTechnology(NRF-2013R1A1A2061831)

References

[1] A Ross and A K Jain ldquoMultimodal biometrics an overviewrdquoin Proceedings of the 12th European Signal Processing Conferencepp 1221ndash1224 Vienna Austria September 2004

[2] K Bowyer K Chang and P Yan ldquoMulti-modal biometrics anoverviewrdquo in Proceedings of the 2nd Workshop on Multi-ModalUser Authentication Toulouse France May 2006

[3] A K Jain and A Kumar ldquoBiometric recognition an overviewrdquoin Second Generation Biometrics The Ethical Legal and SocialContext E Mordini and D Tzovaras Eds pp 49ndash79 SpringerAmsterdam The Netherlands 2012

[4] W Zhao R Chellappa P J Phillips and A Rosenfeld ldquoFacerecognition a literature surveyrdquo ACM Computing Surveys vol35 no 4 pp 399ndash458 2003

[5] R Jafri and H R Arabnia ldquoA survey of face recognitiontechniquesrdquo Journal of Information Processing Systems vol 5 no2 pp 41ndash68 2009

[6] I A Kakadiaris G Passalis T Theoharis G Toderici IKonstantinidis and NMurtuza ldquoMultimodal face recognitioncombination of geometry with physiological informationrdquo inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR rsquo05) vol 2 pp1022ndash1029 San Diego Calif USA June 2005

[7] K W Bowyer K Chang and P Flynn ldquoA survey of approachesand challenges in 3D and multi-modal 3D + 2D face recogni-tionrdquo Computer Vision and Image Understanding vol 101 no 1pp 1ndash15 2006

[8] K I Chang K W Bowyer and P J Flynn ldquoAn evaluationof multimodal 2D+3D face biometricsrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 27 no 4 pp 619ndash624 2005

[9] R V Yampolskiy and V Govindaraju ldquoBehavioural biometricsa survey and classificationrdquo International Journal of Biometricsvol 1 no 1 pp 81ndash113 2008

[10] J Liu ZWang L Zhong JWickramasuriya andVVasudevanldquouWave accelerometer-based personalized gesture recognitionand its applicationsrdquo in Proceedings of the 7th Annual IEEEInternational Conference on Pervasive Computing and Commu-nications (PerCom rsquo09) pp 1ndash9 Galveston Tex USA March2009

[11] G Bailador C Sanchez-Avila J Guerra-Casanova and A deSantos Sierra ldquoAnalysis of pattern recognition techniques for in-air signature biometricsrdquo Pattern Recognition vol 44 no 10-11pp 2468ndash2478 2011

[12] J Guerra-Casanova C Sanchez-Avila G Bailador and Ade Santos Sierra ldquoAuthentication in mobile devices throughhand gesture recognitionrdquo International Journal of InformationSecurity vol 11 no 2 pp 65ndash83 2012

[13] D Guse Gesture-based user authentication on mobile devicesusing accelerometer and gyroscope [Master thesis] Berlin Institueof Technology 2011

[14] N Sae-Bae K Ahmed K Isbister and N Memon ldquoBiometric-rich gestures a novel approach to authentication onmulti-touchdevicesrdquo in Proceedings of the 30th ACM Conference on HumanFactors in Computing Systems (CHI rsquo12) pp 977ndash986 AustinTex USA May 2012

[15] K Lai J Konrad and P Ishwar ldquoTowards gesture-based userauthenticationrdquo in Proceedings of the IEEE 9th InternationalConference on Advanced Video and Signal-Based Surveillance(AVSS rsquo12) pp 282ndash287 Beijing China September 2012

[16] J Wu J Konrad and P Ishwar ldquoThe value of multiple view-points in gesture-based user authenticationrdquo in Proceedingsof the IEEE Computer Society Conference on Computer Visionand Pattern RecognitionWorkshop pp 90ndash97 Columbus OhioUSA June 2014

[17] P Viola and M J Jones ldquoRobust real-time face detectionrdquoInternational Journal of Computer Vision vol 57 no 2 pp 137ndash154 2004

[18] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) pp 886ndash893 San Diego Calif USA June 2005

[19] A Vedaldi and B Fulkerson ldquoVLFeat An Open and PortableLibrary of Computer Vision Algorithmsrdquo httpwwwvlfeatorg

[20] H Choi J Seo andH Park ldquoMatrix correlation distance for 2Dimage classificationrdquo in Proceedings of the of ACM Symposiumon Applied Computing pp 1741ndash1742 Gyeongju Republic ofKorea March 2014

[21] M Muller ldquoDynamic time warpingrdquo in Information Retrievalfor Music andMotion M Muller Ed pp 69ndash84 Springer NewYork NY USA 2007

[22] ChaLearn ChaLearn Gesture Dataset (CGD 2011) 2012 httpgesturechalearnorgdata

[23] A Martin G Doddington and T Kamm ldquoThe DET curvein assessment of detection task performancerdquo in Proceedingsof the European Conference on Speech Communication andTechnology pp 1895ndash1898 Rhodes Greece September 1997

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology

Page 8: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

8 BioMed Research International

the shape of ellipse we can guess that the two modalities arealmost independent and this is supported by the fact that theaverage value of correlation coefficient is 019 This propertyis desirable for combining two biometric signals to constructmultimodal biometric system

6 Conclusion

In this paper we present a look into simple and efficientvision-based multimodal biometric system using heteroge-neous biometric signals By combining physical and behav-ioral biometric signals we can achieve a high degree ofreliability Because the proposed system uses a single visionsensor it can be easily implemented on commonly used smartdevices such as smart TVs More comprehensive study ondeveloping efficient feature extraction and classification willbe done for real world application of the proposal system

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research was partially supported by the DGIST RampDProgram of the Ministry of Education Science and Technol-ogy of Korea (13-IT-03) and Basic Science Research Programthrough the National Research Foundation of Korea (NRF)funded by theMinistry of Education Science andTechnology(NRF-2013R1A1A2061831)

References

[1] A Ross and A K Jain ldquoMultimodal biometrics an overviewrdquoin Proceedings of the 12th European Signal Processing Conferencepp 1221ndash1224 Vienna Austria September 2004

[2] K Bowyer K Chang and P Yan ldquoMulti-modal biometrics anoverviewrdquo in Proceedings of the 2nd Workshop on Multi-ModalUser Authentication Toulouse France May 2006

[3] A K Jain and A Kumar ldquoBiometric recognition an overviewrdquoin Second Generation Biometrics The Ethical Legal and SocialContext E Mordini and D Tzovaras Eds pp 49ndash79 SpringerAmsterdam The Netherlands 2012

[4] W Zhao R Chellappa P J Phillips and A Rosenfeld ldquoFacerecognition a literature surveyrdquo ACM Computing Surveys vol35 no 4 pp 399ndash458 2003

[5] R Jafri and H R Arabnia ldquoA survey of face recognitiontechniquesrdquo Journal of Information Processing Systems vol 5 no2 pp 41ndash68 2009

[6] I A Kakadiaris G Passalis T Theoharis G Toderici IKonstantinidis and NMurtuza ldquoMultimodal face recognitioncombination of geometry with physiological informationrdquo inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR rsquo05) vol 2 pp1022ndash1029 San Diego Calif USA June 2005

[7] K W Bowyer K Chang and P Flynn ldquoA survey of approachesand challenges in 3D and multi-modal 3D + 2D face recogni-tionrdquo Computer Vision and Image Understanding vol 101 no 1pp 1ndash15 2006

[8] K I Chang K W Bowyer and P J Flynn ldquoAn evaluationof multimodal 2D+3D face biometricsrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 27 no 4 pp 619ndash624 2005

[9] R V Yampolskiy and V Govindaraju ldquoBehavioural biometricsa survey and classificationrdquo International Journal of Biometricsvol 1 no 1 pp 81ndash113 2008

[10] J Liu ZWang L Zhong JWickramasuriya andVVasudevanldquouWave accelerometer-based personalized gesture recognitionand its applicationsrdquo in Proceedings of the 7th Annual IEEEInternational Conference on Pervasive Computing and Commu-nications (PerCom rsquo09) pp 1ndash9 Galveston Tex USA March2009

[11] G Bailador C Sanchez-Avila J Guerra-Casanova and A deSantos Sierra ldquoAnalysis of pattern recognition techniques for in-air signature biometricsrdquo Pattern Recognition vol 44 no 10-11pp 2468ndash2478 2011

[12] J Guerra-Casanova C Sanchez-Avila G Bailador and Ade Santos Sierra ldquoAuthentication in mobile devices throughhand gesture recognitionrdquo International Journal of InformationSecurity vol 11 no 2 pp 65ndash83 2012

[13] D Guse Gesture-based user authentication on mobile devicesusing accelerometer and gyroscope [Master thesis] Berlin Institueof Technology 2011

[14] N Sae-Bae K Ahmed K Isbister and N Memon ldquoBiometric-rich gestures a novel approach to authentication onmulti-touchdevicesrdquo in Proceedings of the 30th ACM Conference on HumanFactors in Computing Systems (CHI rsquo12) pp 977ndash986 AustinTex USA May 2012

[15] K Lai J Konrad and P Ishwar ldquoTowards gesture-based userauthenticationrdquo in Proceedings of the IEEE 9th InternationalConference on Advanced Video and Signal-Based Surveillance(AVSS rsquo12) pp 282ndash287 Beijing China September 2012

[16] J Wu J Konrad and P Ishwar ldquoThe value of multiple view-points in gesture-based user authenticationrdquo in Proceedingsof the IEEE Computer Society Conference on Computer Visionand Pattern RecognitionWorkshop pp 90ndash97 Columbus OhioUSA June 2014

[17] P Viola and M J Jones ldquoRobust real-time face detectionrdquoInternational Journal of Computer Vision vol 57 no 2 pp 137ndash154 2004

[18] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) pp 886ndash893 San Diego Calif USA June 2005

[19] A Vedaldi and B Fulkerson ldquoVLFeat An Open and PortableLibrary of Computer Vision Algorithmsrdquo httpwwwvlfeatorg

[20] H Choi J Seo andH Park ldquoMatrix correlation distance for 2Dimage classificationrdquo in Proceedings of the of ACM Symposiumon Applied Computing pp 1741ndash1742 Gyeongju Republic ofKorea March 2014

[21] M Muller ldquoDynamic time warpingrdquo in Information Retrievalfor Music andMotion M Muller Ed pp 69ndash84 Springer NewYork NY USA 2007

[22] ChaLearn ChaLearn Gesture Dataset (CGD 2011) 2012 httpgesturechalearnorgdata

[23] A Martin G Doddington and T Kamm ldquoThe DET curvein assessment of detection task performancerdquo in Proceedingsof the European Conference on Speech Communication andTechnology pp 1895ndash1898 Rhodes Greece September 1997

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology

Page 9: Research Article A Multimodal User Authentication System ...downloads.hindawi.com/journals/bmri/2015/343475.pdf · user authentication in mobile devices. Also, in [ ]the ... Face

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Anatomy Research International

PeptidesInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

International Journal of

Volume 2014

Zoology

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Molecular Biology International

GenomicsInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioinformaticsAdvances in

Marine BiologyJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Signal TransductionJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

BioMed Research International

Evolutionary BiologyInternational Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Biochemistry Research International

ArchaeaHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Genetics Research International

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Virolog y

Hindawi Publishing Corporationhttpwwwhindawicom

Nucleic AcidsJournal of

Volume 2014

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Enzyme Research

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

Microbiology