SHORT WAVELENGTH INFRARED FACE RECOGNITION FOR PERSONALIZATION

Jinwoo Kang, Amol Borkar; Angelique Yeung, Nancy Nong, Mark Smith, Monson Hayes

Center for Signal and Image ProcessingGeorgia Institute of Technology, Atlanta, GA 30332

ABSTRACT

The paper describes an application of practical technologies to im-plement a low cost, consumer grade, single chip biometric systembased on face recognition using infra-red imaging. The paper presentsa system that consists of three stages that contribute in the face de-tection and recognition process. Each stage is explained with itsindividual countribution alongside results of tests perforumed for thatstage. The system shows a high recognition rate when full frontalface images are fed to the system. The paper further discusses theapplication based approach in the automotive world with plans forfurther study. Recognition rates of the overall system are also pre-sented.

Index Terms- Face Detection, Face Recognition, Fisherfaces,Eigenfaces, Infrared, Intensity Based Segmentation.

1. INTRODUCTION

An interesting difference in this work compared with previous re-search in face recognition is the idea of what we termn consumergrade biomietrics. Biometrics as an identity management disciplineseeks to either identify or verify a person's identity. In most suchsystems the design goal is to minimize the probability of false ad-mission even at the expense of increasing the probability of false re-jection. The economics behind this in high security applications areobvious, and an'y system that incorrectly admits even a vanishinglysmall number of subjects will fail in the market place. However, themarket for ID management extends well beyond high security ap-plications. For example, proponents of pervasive and context awarecomputing have argued that determining a user's identity with sen-sors can greatly enhance the perceived value of an application byautomatically personalizing it. Examples include personalizing en-tertainment equipment to play favored music, targeting advertisingto a particular set of tastes, or adjusting the position of objects suchas car seats and mirrors. There are countless such applications thatare enabled by knowledge of the user's identity, but carry no signif-icant risk of personal, physical or economic harm should the user'sidentity be incorrectly determined. Unlike high security applicationswhere it is preferable to reject the identity of a person under any rea-sonable doubt, in these no risk security, consumer applications it ispreferable to always attempt to converge on an identity of the subject,even if it is wrong. Attributes that describe consumer grade biomet-rics are low cost, simple training, and operability without direct userinteraction.

The initial target application for the consumer grade face recognizer described here is in the automotive market. The novelty ofthis application is the ability to invisibly perform noninvasive real-time recognition of the subject in the driver's seat The intent is touse this data for personalization applications for the convenience ofthe driver, but poses no risk and is easily corrected if the identity isdetermined wrong.

Fig. 1. System flowchart.

Research has already been done in the field of face detectionand recognition; for detection, some of the computationally less in-tensive approaches use variations of skin color matching or shapedetection based on pre-defined databases. Other more intensive ap-proaches use Support Vector Machine, multi layer Probabilistic Neu-ral Networks and Wavelets. Since this also an application based ap-proach, some of the assumptions made are ad-hoc and cannot be gen-eralized towards a particular approach. [1], [2], [3]. In our system,short wavelength infrared (SWIR) of a wavelength of 940 nm waschosen as an illumination source. Prior research has been done ondetection using mid and long wave IR (thermal imaging) [4]. Solidstate SWIR emitters are inexpensive, and bright enough to providea fill in for shadows during daytime operation, or to provide totalface illumination at night. Although the illumination is invisible tothe subject being imaged, most CMOS based imagers have a highdegree of sensitivity at these wavelengths. A low cost system usingSWIR illuminators and a CMOS based imager with integrated elec-tronics to perform the recognition algorithms should be practical.

2. SYSTEM OVERVIEW

The goal is to provide a single integrated circuit biometric devicethat uses dedicated analog circuitry. Due to restrictions on the math-ematical capabilities of the analog hardware, proven algorithms oflow complexity such as intensity based segmentation, cross corre-lation, principal component analysis (PCA) and linear discriminantanalysis (LDA) are used since they will work well in conjunctionwith the provided SWIR band limited CMOS imager. As a result,the volume of silicon used in manufacturing will be far less than ofdigital hardware with very low production costs, e.g. US $5 or less.

To achieve this goal, the most indispensable tasks are includedin the system which is face detection, eye detection and recogni-tion. Figure 1 illustrates how the subsystems are interconnected inthe overall system. Cluster analysis in face detection part decideswhether the input video frame has a face If a face presents the faceregion is passed to eye detection otherwise the frame is disregarded.Eye detection performs boundary analysis on the face image and ex-tracts eye coordinates Accurate positioning of eyes is indispensableto make the face recognition work. The face image is preprocessedbased on the eye locations. And finally identification is made basedon the result of face recognition A detailed functional description of

1-4244-0481-9/06/$20.00 C2006 IEEE 2757 ICIP 2006

Fig. 3. Sample results of the face detection using thresholds.

tFig. 2. The original images on the top row and the thresholded binaryimages on the bottom row.

the algorithms for the subsystems is given in the following sectionsrelating them to previous work in each area.

In order to train and test each subsystem and the end to end sys-

tem, data was acquired in the form of video captures that simulated a

person sitting in the driver's seat of a vehicle, fastening the seat beltand starting to drive with mild body movements. Two videos were

acquired of each of the four subjects tested, with each video contain-ing approximately 1000 frames. The videos are in grey scale and theframe size is 320 by 240. A simple off the shelf imager coupled withan optical IR filter was used to obtain the video stream.

3. FACE DETECTION

Visible and SWIR light obey the Inverse Square Law which statesthat brightness of the light source is inversely proportional to thesquare of its distance [5]. Using this property, one can observe thatobjects closer to the light source will be more illuminated than thosefarther away. Consider a person located near the lighting source, thiswill cause the person to "glow" or be saturated as compared to thebackground. Our face detector is based on this phenomenon. We as-

sume that after converting a gray level image with a face to a binaryimage by thresholding, the largest cluster of white pixels (which are

corresponding to the intensity values larger than a threshold value inthe gray level image) is a face or an upper body including a face.

The face detection subsystem has three steps: thresholding, clus-ter analysis, and the final decision step. In the cluster analysis step, a

simple erosion and dilation is performed on the binary image to ridany stray pixels that could be present because of noise (Figure 2).Next, the various clusters are collected with centers computed foreach cluster using median information on each axis. The cluster thatis closest to the center of the frame is of interest to us. We choosethis cluster because the duiver of a vehicle is going to be located near

the center of the camera's view and sitting in an upright vertical po-sition. The median is preferred over the mean because the medianwill be directed towards the center of a concentration of pixels andis not easily steered away by stray pixels.

The selected cluster belongs to one of three categories under

the assumption mentioned above: face, upper body (including theface), or neither of them. The category that the cluster belongs to isdetermined in the final decision step with statistical measures of face

dimensions: height and width. In the first pass, the cluster height iscompared to it's statistical value to decide the category of the cluster.The details are shown below.

Video Total Frames .Fr Frames DetectionFrames with with with Rate

faces faces in- face (present coirectly correctly

detected detectl 1088 931 45 813 87.282 1128 1046 2 1044 99.813 750 690 1 1 678 98.264 1342 1126 2 11i6 99ii5 563 462 5 410 88.746 1324 1168 0 1146 98.127 747 676 21 618 91.42

Table 1. Face detection success rates.

Given that the classification results in a possible face, we use

the height of the cluster combined with the statistical measure of theaspect ratio to determine the width. The face region with the heightand width is extracted from the image. Given that the classificationresults in a possible upper body, the proper face region in the upper

body cluster is extracted as the pseudo code below. If no face isfound in either of two cases, we decide that there is no face in theimage.

if cluster i lassifed as body

face width = distance between points at location (meanheight 2 + top of cluster)

if face width < mean width + 3 x standard deviationFace region found;

endif

Approximately 200 face images were manually extracted fromthe 7 videos. The binary image threshold was set to onme standarddeviation less than the mean of the pixel intensity values of the faceimages, which allows us to segment images with low illumination.The face images also serve as training data to get statistics of facedimensions.

The 7 videos were tested to verify the accuracy of the face detection. To perform this task successfully, each frame was visuallyinspected by one of the authors. Table 1 shows the result for eachvideo. The detection rate varnes from 87.28rc to 99. 81%. Figure 3

shows sample results of the face detection.

4. EYE DETECTION

A large amount of work has been done on finding facial features likeeyes. One approach is to locate the eyes in images without locatingthe face [6]. Another approach is to use a combination of physio-

2758

select cluster heightCase > mean height + 2 x standard deviation

Possible upper body (including face);Case < mean height + 2x standard deviation

Possible face,Case default

Neither,end select

0.9

08

071

0.6

05

LDA with videos--LDA with CMUPCA with videosPCA with CMU

20 40 60 80Number of coefficients

Fig. 5. Recognition hit rates compari son.

Fig. 4. Eye Detection Example.

Video Face Correctly Incofectly Inco rectly Detectionframes fourd fourd iejected rate

eyes eyes eyes .%)1 662 393 249 20 59.372 374 268 98 8 71.663 564 441 99 24 78.194 375 242 121 12 64.53

Table 2. Results of the eye detection.

logical properties of the eyes, Kalman trackers to model eye/headdynamics, and a probabilistic appearance models to locate the eye

appearances [7].The eye location algorithm is implemented on images from which

the background is cropped out leaving only the face. The eye co-

ordinates are used to perform the required transformation on the faceimage. Thus, all the face images are centered based on the eye coor-dinates in the preprocessing step before they are sent into the recog-nition system.

Binary representations of the face images were utilized to out-line the face and eyes. The exterior boundary points of the binaryface were determined, as well as the boundaries of holes inside theface by using 8-connected neighborhood connectivity. It is assumedthat the eyes were the largest objects in the upper face region. There-fore, only the set of boundary points is chosen which contains thelargest number of points. Once the face is outlined the upper half ofthe face is analyzed to locate the eyes. Either the left eye or righteye is detected first. This can be determined from the x location ofthe first eye compared to the vertical midline of the face. Based on

which eye is detected first, the left upper region or the right upperregion of the face is utilized to search for the second eye (See Fig-ure 4). The accuracy of the eye detection was verified by the visualinspection of 4 test videos by one of the authors (Table 2).

5. FACE RECOGNITION

Among many approaches to the problem of face recognition, ap-

pearance based subspace analyses is one of the oldest approachesdelivering the most promising results. Two of the more popular ap-

pearance based subspace analysis are Eigenface methods which are

equivalent to Principal Component Anal sis (PCA) and Fisherfacemethods which are the combination of PCA and Linear Discriminant Analysis (LDA). PCA finds a set of the most representativeprojection vectors such that the projected samples retain most in-formation about the original samples [8]. On the other hand, LDAuses the class information to find a set of vectors that maximize thebetween-class scatter while minimizing the within-class scatter [9].

Before further explanation we will clarify the terms for the data

sets used in the recognizer. The recognizer finds an adequate sub-space using a set of face images labeled with the subject's identity,

which we will call the training set. After the subspace is trained,unlabeled face images (test set) are to be identified as one of thesubjects whose face images are given (gallery set).

In our application we use a gallery set whose images belong tothe subjects that are not in the training set since finding the subspacerequires eigensolvers which are computationally expensive. Sinceonly a small number of subjects are expected and it is hard to rely onthe subspace calculated based on the images.

An experiment was performed to determine if a training set builtfrom subject images not in the gallery set could result in recogni-tion performance close to that of a recognizer using a training setbuilt from actual gallery images. Frontal face images of CMU PIEdatabase [10] are used as the training set that is different from thegallery set. The training set has approximately 170 images for eachof the 68 subjects. 4 videos of 4 subjects in our data set are used forgallery set and the remaining 4 videos are used for test set. Eigenfacemethods and Fisherface methods are applied to both cases. Figure5 shows the recognition rates on various numbers of coefficients. Itshows that the recognition rate for the case when CMU PIE databaseis used as a training set is almost as good as the case when the galleryset is used as a training set for both of subspace methods. Since thediFference is less than 10%, we can say that the CMU PIE imagesare good enough as the training set of our application. The resultalso confirms that Fisherface methods perform better than Eigenfacemethods.

Before subspace methods are applied to the detected face imagesin the video frames, the images go through a preprocessing step. Itincludes geometrical transformation, masking and pixel value nor-

malization. Geometrical transformation processes the face imagesso that eyes are located in the predefined positions. Masking takespixels inside the face boundary. And the pixel values are normalizedso that they have 0 mean and I standard deviation.

Preprocessed images are projected on the Fisherface subspaceand minimum distance calculated between the projected test dataand the mean values of the projected gallery data of each subject isapplied to identify the test subjects of the images. The minimumdistance method is equivalent to the maximum likelihood decisionassuming that the distributions of all the subjects are equal and priorsare equal. The decisions on the frames from the beginning of thevideo to the current frame are used to vote the overall decision.

The projected test data are also used to decide whether the pre-processed image comes from a frontal pose or not If the image isnot a frontal face, the projected test data will have a distance from

origin of the subspace larger than that of a frontal face. A simplethreshold can be used to reject the image. This approach is similar tothe method mentioned in [8] but different. The authors suggest usingthe distance from a data point in the image space to the face space as

a measure for face detection. But what we use is the distance from a

point to the origin in the Fisherface subspace.

2759

(a) Ori2inal

a .eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

_reeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeIeeee

-~(b inr

Video Subject Subject Subject Subject Subjectin 1 2 3 4Video

1 1 0.618 0.050 0.222 0.0442 .2 0..........00.0570 0.163 0.2473 3 0.078 0.033 0.843 0.0284 4 0.027 0.099 0.281 0.556

Table 3. Voting ratios of 4 videos.

subject 1- - subject 2

subject 30.8 subject 4

0.6 V 1

0.4i /

0.2r 1

100 200 300 400 500 600 700 800

(a) Video 1L

-subject 1-subject 2subject 3-subject 4

(c) Video 3

subject 1subject 2subject 3subject 4

100 200 300 400 500 600 7

(b) Video 2

0.8._

0.6 , v.

0.4

0.2 .. ..-..-.-.. ....

200 400 600

(d) Video 4

Fig. 6. Change of voting ratios over the sequence of fram

6. SYSTEM RESULTS AND DISCUSSION

IVideo Itest 1 test 21 test 31 0.956 0.854 0.6182 0.662 0.591 0.5703 0.948 0.947 0.8434 0T969 04709 0r556

lable 4. Detection ratio comparison for additional tests.

would be unnoticeable. If the overall system has low false alarm rate,the difference between the results of test 2 and 3 would be insignifi-cant. From the results, we finally conclude that the face recognitionsubsystem shows a high detection rate and that the face and eye de-tection subsystems find faces with enough accuracy to get a correctidentification result in a small group of subjects.

In this paper a consumer grade biometric systerm based on facerecognition using infrared imaging is presented with a successful de-tection result ini a siimall group of subjects. This low cost approachis intended for practical, high volume applications where the dis-tance is minimum and controlled, such as automotive applicationsand hand held devices. The results can be further improved by in-

su1ect creasing the accuracy of the eye detection and the frontal pose de-sbjet2 tection. Future work will address how to deal with other sources ofsuject 4 ivariations such as illumination, facial expression, and occlusion.

7. REFERENCES

[1] Zhi-fang Liu, Zhi-sheng You, A. K. Jain, and Yun-qiong Wang,"Face detection and facial feature extraction in color image,"

800 in IJut. Conf. Computational Inutelligence and Multimedia Ap-plications, 2003, p. 126.

[2] Ming-Jung Seow, Deepthi Valaparla, and Vijayan K. Asari,es. "Neural network based skin color model for face detection,"

in Applied Imagery Patter Recognition Workshop, 2003, pp.141-145.

We performed an end to end evaluation of our system using the 8videos of 4 subjects, which was introduced in Section 2. Consideringeach subject as a member of a family of four, the goal is to recognizewhich one of the 4 is sitting in the driver's seat of a car. The over-all voting ratios on all the subjects at the final frames of videos areshown in Table 3. Since all videos have the maximum voting ratioson the corresponding subjects,it is concluded that all the subjects aresuccessfully identified at the end of videos. Specifically, video 1 and3 shows relatively high detection ratios compared to the others. TheGraphs shown in Figure 6 illustrate the change of voting results overall the video sequences. Video 1, 3 and 4 indicate that the votingratios are stabilized after the 400th frame approximately. Detectionratio of video 2 is high at the beginning of the sequence but then be-gins to decrease. To figure out the difference between the results, weexamined the videos and came to find that the subject in video 4 is infrontal position at the beginning of the video but it moves actively astime progresses. That explains why the detection ratio is decreasedover time.

Additional tests were conducted to evaluate the performance ofsubsystems (Table 4). In the first test, the recognition part of thesystem is only evaluated with manually specified eye locations onmanually determined frontal face frames. In the second test, per-formance of the overall system is measured on manually detectedfrontal face frames. And in the third test, performance of the overalls stem is assessed on all frames of videos The result of the firsttest is equivalent to the performance of the face recognition subsys-tem. If the face and eye detectors find the locations accurately in thepresence of face, the difference between the results of test 1 and 2

[3] Shinjiro Kawato and Jun Ohya, "Automatic skin-color distri-bution extraction for face detection and tracking," in Proc. Int.Conf. Signal Processing, 2000, vol. II, pp. 1415-1418.

[4] S.G. Kong, J. Heo, B.R. Abidi, J. Paik, and M.A. Abidi, "Re-cent advances in visual and infrared face recognition: a re-view," Computer Vision and Image Understanding, vol. 97,pp. 103-135, Jan. 2005.

[5] Rod Nave, "Inverse square law," http://hyperphysics.phy-astr gsu. edu/hbase/forces/isq. html.

[6] R. Kothari and J. Mitchell, "Detection of eye locations in un-constrained visual images," in IEEE Int. Conf inage Process-ing, 1996, p. 19A8.

[7] Antonio Haro, Myron Flickner, and Irfan Essa, "Detection andtracking eyes by using their physiological properties, dynam-ics, and appearance," in IEEE Conf Conputer Vision and Pat-tern Recognition, 2000, pp. 163-168.

[8] M. Turk and A. Pentland, "Eigenfaces for recognition," Jour-nal of Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, 1991.

[9] Peter N. Belhumeur, Joao P. Hespanha, and David J. Kriegman,'Eigenfaces vs. fisherfaces: Recognition using class specificlinear projection," IEEE Trans. Pattern Analysis and MachineInitelligence, vol. 19, no. 7, pp. 711-720, Jul. 1997.

[10] T. Sim, S. Baker, and M. Bsat, "The cmu pose, illumination,and expression database" IEEE Trans. Patter Analysis andMachine Intelligence, vol. 25, no. 12, pp. 1615-1618, Dec.2003.

2760

1

0.8

0.6

0.4

0.2

700 800

200 400 600 800