SeeingWith OPenCVFinding Faces in lmages

what this mouthful means. Figure 1shows an example of OPenCV's facedetector in action.

Given an image - which can comefrom a file or from live video - the facedetector examines each image locationand classi f ies i t as "Face" or "Not Face."Classification assumes a fixed scale forthe face, say 50 x 50 pixels. Since facesin an image might be smal ler or largerthan this, the classifier runs over theimage several t imes, to search forfaces across a range of scales. This mayseem l ike an enormous amount ofprocessing, but thanks to algor i thmictr icks (explained in the sidebar), c lassi f i -cation is very fast, even when it'sapplied at several scales.

The classifier uses data stored inan XML file to decide how to classifyeach image locat ion. The OPenCVdownload includes four flavors of XMLdata for frontal face detection, andone for orofile faces. lt also includesthree non-face XML files: one for fullbody (pedestrian) detection, one forupper body, and one for lower bodY.

You'll need to tell the classifierwhere to find the data file You

libnary for C/C++, ptqrcmrr€rs. ltcovercd the basics - downloadingand installing OpenCV, reading andwriting image files, capturing video,and working with the lPllmage datastructufe.

rFhis month, l ' l l show You how to

I use OpenCV to detect faces. l ' l lI exolain how the face detection

algorithm works, and give you tips forgetting the most out of it.

Background andPreliminaries

OpenCV uses a tYPe of facedetector cal led a Haar Cascadeclassi f ier. The sidebar, "How FaceDetection Works, or What's a HaarCascade Classifier, Anyhow?" explains

want i t to use. The one l ' l l beusing is called haarcascade-f rontalface-default .xml. In OPenCVversion 1 .0, it 's located at:

IoPE]rCv-RoOTl /data/haarcascades /haarcascade-f rontal f ace-def au1 t .xnl

where loPn'icv-noor] is the Pathto your OpenCV installation. Forexample, if you're on Windows XP

FIGURE 1. Face detection with OpenCVusing default parameters. The inputimage is lenaiPg, in the samPles/cdirectory.

48 srnvo oe.2ool

and you selected the default installa-t ion locat ion, you'd use:

iOPi l {CV-ROOTI - "C: Ptogram Fi lesr

@enCV"

( l f you're working with an older, 16-bi tversion of Windows, You'd use '\ ' asthe directory separator, instead of 'l '-)

It 's a good idea to locate the XMLfile you want to use and make sureyour path to it is correct before you

code the rest of your face detectronpro9ram.

You' l l a lso need an image toprocess. The image lena. jpg is a goodone to test with. lt 's located in theOpenCV samples/c directory. lf youcopy it to your program's workingdirectory, you' l l easi lY be able tocompare your Program's outPut withthe output from the code in Figure 2.

lmolementing FaceDetection, StEp by SteP

Figure 2 shows the source code toload an image from a f i le, detectfaces in it, and display the image wrthdetected faces out l ined in 9reen.Frgure 1 shows the display produced bythis program when it's run from thecommand l ine, using:

DeLectFaces lena. jPg

Initializing (and running) theDetector

The variable CvHaarClassi f iercascade * pcascade (Line 2) holds thedata from the XML file You locatedearl ier. To load the XML data intopCascade, you can use the cvloadofunct ion, as in Lines 11-13. cvloadois a general-purpose funct ion forloading data from files. lt takes up to

FIGURE 9. Source code to detectfaces in one image. Usaga DetectFaces <image file>.

three input parameters. For thisexample, you' l l only need the f i rstparameter. This is the path to anXML f i le containing a val id HaarCascade. Here, l 've loadedthe default frontal face detectorincluded with OpenCV. lf you'recoding in C, set the remainingparameters to 0. lf you're coding inC++, you can simply omit theunused parameters f rom yourfunct ion cal l .

Before detect ing faces inimages, you'll also need to instanti-ate a CvMemstorage object(pStorase, declared at Line 3). Thisis a memory buffer that expandsautomatically, as needed. The facedetector will put the list of detectedfaces into this buffer. Since thebuffer is expandable, you won'tneed to worry about overflowing it.All you'll have to do is create it (Line10), then release it when you'ref inished (Line 5a).

You'll often need to load datafrom files with OpenCV. Since it'seasy to get a path wrong, it 'sa good idea to insert a quick checkto make sure everything loadedand initialized properly. Lines '16-24

do a simple error check, print adiagnost ic message, and exi t i fin i t ia l izat ion fai ls.

Running the DetectorLines 27-32 call cvHaarDerect

objecrs ( ) to run the face detector.This function takes up to sevenparameters. The f i rst three arethe image pointer, XML data, andmemory buffer. The remaining fourparameters are set to their C++defaults. These last four parametersare described below, in the section,"Parameters

and Tuning."

Viewing the ResultsA quick way to check if your

program works is to display the resultsin an OpenCV window. You can createa d isp lay w indow us ing the:vNamedWindowo function, as in Line35. The f i rst parameter is a str ing,with a window name. The second,--lr_wrNDow_AurosrzE, is a flag that

123456l89

1 01 1121 31 A

1 5a o

I ]1 81 92 02 T2 22 31 A

2 52 62 12 82 93 03 1323 33 43 53 63 13 83 94 04 L

4 34 4c l

4 6

4 84 95 05 152535 4

/ / declarat ionsCvHaarClassifierCascade * pCascade = 0;CvMemstorage * pStorage = 0;. \ / q a d * n E : . a P a - f e 6 ^ .

i n t i ;

/ / Lhe face detector// e>pandable memory buffer/ / hsL of detected faces

/ / initializaLionsIpllrnage * pTnpInE = (argc > 1) ?

cvloadlrnage(argrvFl, CV_LOAD_IMAGE_COIOR) : 0;pstoraqre - clcreateMenistorage{0) ;pCascade = (CvHaarClassifierCascade *)cvload

( (OPNCV_ROOT " /data/haarcascades /haarcascade_frontal face_defauLt . xnl " ) ,0 , 0 , 0 ) ;

// valj-date that everything initialized properlyi f ( !p lnplmg l1 lpStorage l l lpCascade )t

pr int f ( " In i t ia l - izat ion fa i led: ?s \n, , ,( l p l np lmg )? ' d i dn ' t l oad image f i l e " :( lpCascade)? "didn' t load Haar cascade -- , ,

"make sure path is correct" :" fa i led to a l locate memory for data storage, , ) ;

e x i r ( - 1 ) ;

)

/ / detect faces in imagenFafeRpff Sao = . \ /H^Arf)ol-cef Ohiocf e

(plnplmg, pCascade, pstorage,1 .1 , / / i n c rease sea rch sca le by 1Og each pass3, / / drop groups of fewer tharl chree detectionsCV_IAAR_DO_CANM_PRUNING, / / skip regions unlikely to contain a facecvSize(0,O) ) ; / / use )o{L defaul t for smal lest search scale

/ / crea|e a window to display detected facescvNamedwindow ( "Haar Window,,, CV_WINDOW_AU|OSIZE) ;

/ / draw a rectangular outline around each detectionfor( i=Q; iq(pFaceRectseq? pFaceRectseq->total :0) ; i++ ){

CvRect * r = (CvRect*)cvcetseqElern(pFaceRectSeq, i) ;CvPoint pt1 = { r ->x, r ->y } ;CvPoint pt2 = 1 r->x + r->wid[h, r->y + r->height );c vRec tang le (p l np Img , p t1 , p t2 , CV_RGB(0 ,255 ,A ) , 3 , 4 , 0 l ;

)

/ / d i c n l : r r f ^ . 6 A a t d . r '- * - , * - - - - -10nscvShowlnr,age ( "Haar Window., plnplmgf ) ;cwva l tKey(0) ;cvDestroywindow( "Haar Window" ) ;

/ / clean up and release resourcescvReleaselmage ( &plnpIng ) ;i f (pCascade ) cvReleaseHaarClassi- f ierCascade ( &pcascade ) ;i f (pStorage ) cvRel-easeMemstorage ( &pstorage ) ;

te l ls the window to automatical lyresize itself to fit the image you give itto display.

To pass an image for display, callcvShowrmageO with the name you pre-viously assigned the window, and theimage you want it to display. Thec\,r'/airKeyO call at Line 48 pauses theapplication until you close the window.lf the window fails to close by clickingits close icon, click inside the window'sdisplay area, then press a keyboard key.Also, make sure your program callscvDestroywindowO (Line 49) to close

the window.Face detections are stored as a list

of CvRect srrucr pointers. Lines 38-44access each detection rectangle andadd its outline to the prnprms variable,wh ich ho lds the in -memory imageloaded from the f i le.

Releasing ResourcesLines 52-54 release the resources

used by the input image, the XMLdata, and the storage buffer. l fyou' l l be detect ing faces in mult ipleimages, you don't need to release the

r

j

l

))t f

o

SERVO O2.2oO7 49

FIGURE B. The Integral lmage trick. Afterintegrating, the pixel at (x,y) containsthe sum of a l l p ixe l va lues in the shadedrectangle. The sum of pixel values inrectangle D is (x4, y4) - (x2, Y2) - (x3,y3) + (x1, y1) .

OpenCV's face detector uses amethod that Paul Viola and MichaelJones published in 2001' Usually calledsimply the Viola-Jones method, oreven just Viola-Jones, this approach todetecting objects in images combinesfour key concepts:

. Simple rectangular features, calledHaar features.

. An Integral lmage for rapid featuredetection.

. The AdaBoost machine-learning method.

FIGURE C. The c lass i f ier cascade is achain of single-feature fi l ters. lmagesubregions that make it through theentire cascade are classified as "Face."

All others are classified as "Not Face."

. A cascaded classifier to combine manyfeatures efficiently.

The features that Viola and Jonesused are based on Haar wavelets. Haarwavelets are single-wavelength squarewaves (one high interval and one lowinterval). In two dimensions, a sguarewave is a pair of adjacent rectangles -

one light and one dark.The actual rectangle combinations

used for visual object detection are nottru€ Haar wavelets. lnstead, they containr€ctangle combinations better suited tovisual recognition tasks. Because of thatdifference, these features are called Haarfeatures, or Haarlike features, rather thanHaar wavelets. Figure A shows thefeatures that OpenCV uses.

The presence of a Haar feature isdetermined by subtracting the averagedark-region pixel value from the averagelight-region pixel value. lf the difference isabove a threshold (set during learning),that feature is said to be Present.

To determine the Presenc€ orabsence of hundreds ofHaar features at everYimage location and at sev-eral scales efficiently, Violaand Jones used a tech-nique called an Integrallmage. ln general, "inte'

grat ing" means addingsmall units together. In thiscas€, the small units arepixel values. The integralvalue for each Pixel is thesum of all the pixels aboveit and to its left. Starting atthe top left and traversingto the right and down, the

FIGURE A. Examples of the Haarfeatures used in OPenCV.

entire image can be integrated with atew integer operations Per Pixel.

As Figure 81 shows, after integra-tion, the value at each pixel location,(x,y) contains the sum of all Pixelvalues within a rectangular r€gion that

has one corner at the top left of theimage and the other at location (x,y)'

To find the average pixel value in thisrectangle, you'd only need to divide thevalue at (x,y) by the rectangle's area.

But what if You want to know thesummed values for some other rectangle,one that doesn't have one corner at theupper left of the image? Figure B9 showsthe solution to that problem. Supposeyou want the summed values in D. Youcan think of that as being the sum of pixel

values in the combined rectangle, A+B+C+D, minus the sums in rectangles A+Band A+C, plus the sum of pixel values inA. In other wordt

D = A+B+C+D - (A+B) - (A+C) + 4.

Conveniently, A+B+C+D is the Integrallmage's value at location 4 A+B is thevalue at location 2, A+C is the value atlocation 3, and A is the value at location1. So, with an lntegral lmage, you can findthe sum of pixel values for any rectanglein the original image with just threeinteger operations:

(x4,y4) - (x9, Y9) - (x3, Y3) + (xl, Y1).

To select the specific Haar featuresto use and to set threshold levels, Violaand Jones use a machine-learningmethod called AdaBoost. AdaBoostcombines many "weak" classifiers tocr€ate one "strong" classifier. "!(eak" heremeans the classifier only gets the rightanswer a little more often than randomguessing would. That's not very good. Butif you had a whole lot of these weakclassifiers and each one "pushed" thef inal answer a l i t t le b i t in the r ightdirection, you'd have a strong, combinedlorce for arriving at the correct solution.AdaBoost selects a set of weak

classifiers to combin€ and assignsa weight to each. This weightedcombination is the strongclassif ier.

Viola and Jones combinedweak classifiers as a filter chain,shown in Figure C, that's esPecial-

FIGURE D. The first twoHaar features in the or ig inalViola-Jones cascade.

E-trtr

lmagesubregion

n*Softr,rotr""e ,

\-7oao

ffinot r""" ,\_7 trtr

flafl50 sEnvo oe.2ool

ly elticient for classifying image regions.Each filter is a weak classifier consistingof one Haar feature. The threshold foreach filter is set low enough that itpasses all, or nearly all, face examples inthe training set. (The training set is a largedatabase of faces, maybe a thousand orso.) During use, if any one of these filtersfails to pass an image region, that region

XML data or the buffer unt i l af teryou're done detect ing faces.

Parameters and TuningT h ^ - ^ - . ^ - ^ . , ^ - - |I r tere dre 5everdt paramelers you

can adjust to tune the face detector foryour appl icat ion.

Minimum Detection ScaleThe seventh parameter in the cal l

to cvHaarDereeLobjecLsl) is the sizeof the smallest face to search for. In C,you can select the default for this bysett ing the scale to 0x0, as in Figure 2,L ine 32 . ( ln C++, s imp ly omi t th isparameter to use the default . ) Butwhat is the default? You can f ind outby opening the XML f i le you' l l be using.Look for the <size> tag. In the defaultfrontal face detector. it 's:

<s i ze> 24 24 < / s i ze> .

So, for this cascade, the default mini-mum scale is 24 x 24.

Depend ing on the reso lu t ionyou're using, this default s ize may be avery smal l port ion of your overal limage. A face image this smal l may notbe meaningful or useful , and detect ingit takes up CPU cycles you could use forother purposes. For these reasons -and also to minimize the number offace detections your own code needsto process - it 's best to set theminimum detect ion scale only as smal las you truly need.

To set the minimum scale higherthan the de fau l t va lue , se t th isparameter to the size you want. Agood rule of thumb is to use somefract ion of your input image's width orheight as the minimum scale - forexample, 1/4 of the rmage width. l fyou specify a minimum scale otherthan the default , be sure i ts aspectrat io ( the rat io of width to height) isthe same as the default 's. In this case.aspec t ra t io i s 1 :1 .

is immediately classified as "Not Face."When a filter passes an image region, itgoes to the next filter in the chain.lmage regions that pass through all filtersin the chain are classified as "Face."

Viola and Jones dubbed this filteringchain a cascade.

The order of filters in the cascade isdetermined by weights that AdaBoost

assigns. The more heavily weighted filterscome first to eliminate non-face imageregions as quickly as possible. Figure Dshows the first two features from theoriginal Viola-Jones cascade super-imposed on my face. The first one keys offthe cheek area being lighter than the eyeregion. The second uses the fact th6t th€bridge of the nose is lighter than the eyes.

Minimum Neighbors ThresholdOne of the things that happens

"behind the scenes" when you cal l theface detector is that each positive faceregion actual ly generates many hi tsfrom the Haar detector. Figure 3 showsOpenCV's internal rectangle l ist for theexample image, lena. jpg . The faceregion i tsel f generates the largestcluster of rectangles. These largelyoverlap. In addit ion, there's one smal ldetection to the (viewer's) left, andtwo larger detections slightly aboveand lef t of the main face cluster.

Usual ly, isolated detect ions arefalse detections, so it makes sense todiscard these. l t a lso makes senseto somehow merge the mul t ip ledetect ions for each face region into asingle detect ion. OpenCV does boththese before returning i ts l ist ofdetected faces. The merge step firstgroups rectangles that contain a largeamount of overlap, then f inds theaverage rectangle for the group. l tthen replaces al l rectangles in thegroup with the average rectangle.

Between isolated rectanglesa n d l a r g e g r o u p i n g s a r e s m a l l e rgroupings that may be faces, or maybe false detect ions. The minimum-neighbors threshold sets the cutofflevel for discarding or keepingrectangle groups based on howmany raw detect ions are in thegroup. The C++ default for thisparameter is three, which meansto merge groups of three ormore and discard groups withfewer rectangles. l f you f indthat your face detector is missinga lot of faces, you mighttry Iowering this threshold totwo or one.

lf you set it to 0, OpenCV willreturn the complete list of raw

FIGURE 3. OpenCV's internaldetection rectangles. To see these,

use min_neighbors = 0.

detect ions f rom the Haar classi f ier.Whi le you're tuning your face detector,i t 's helpful to do this just to see what 'sgo ing on ins rde OpenCV. V iewingthe raw detections will improve yourintui t ion about the effects of changingother parameters, which wi l l help youtune them.

Scale Increase RateThe fourth input parameter to

cvHaar DecectObj ecr- s I i Specif ies howquick ly OpenCV should increase thescale for face detections with each passi t makes over an image. Set t ing th ishigher makes the detector run faster(by running fewer passes) , but i f i t 'stoo h igh, you may jump too quick lybetween scales and miss faces. Thedefaul t in OpenCV is i .1 , in otherwords, scale increases by a factor of1 .1 (10%) each pass.

Canny Pruning FlagThe sixth parameter to cvHaar

DeLecLobjecrs ( ) is a f lag var iable.There are currently only two options: 0or cv_nAAR_Do_cANNy_pRUNrNG. lf theCanny Pruning opt ion is se lected, thedetector sk ips image regions that areunl ike ly to conta in a face, reducingcomputat ional overhead and possib ly

SERVO O2.2oO7 51

. OpenCV on Sourceforgehttp://sourc etorge.nd / groiecls /opencvlibrary

. Official OpenCV usergrouphfip:. / / lech.groups.yahoo.com/group/OpenCV

. G. Bradski, A. Kaehler, and V.Pisarevsky, "Learning-Based ComputerVision with Intel 's OPen SourceComputer Vision LibrarY," lntelTechnology Journal, Vol 9(1), MaY 19,9005. www.intel.com/technology/irl / 2OO5 / v ol u me09issu eOZ / a*Ot -learning-vision/ p01 -abstract.htm

. R.E. Schapire, "ABriet Introduction toBoosting," Joint Conference onArtificial lntelligence, Morgan Kauman,San Francisco, pp. 1401-1 406, 1999.

. P. Viola and M.J. Jones, "Rapid ObjectDetection using a Boosted Cascade ofSimple Features," CVPR, 9001.

e l im ina t i ng some fa l se de tec t i ons . Thereg ions to sk ip a re i den t i f i ed byrunn ing an edge de tec to r ( t he Cannyedge detector) over the image beforer r rnn ino the face de tec tOr .

Again, the choice of whether orno t t o se t t h i s f l ag i s a t r adeo f fbetween speed and detect ing moreIaces Set t ing th is f lag speeds processing , bu t mav cause you to m iss somefaces. In general , you can do wel l wi th

i t set , but i f you ' re havrng d i f f icu l tydetectrng faces, c lear ing thrs f lag maya l l ow you to de tec t more re l i ab l y .S e t t i n g t h e m i n i m u m - n e i g h b o r sthreshold to 0 so you can v iew the rawdetectror^s wr l l he lp you bet ter gauge

the e f f ec t o f us ing Canny P run ing .

The Haar CascadeThere a re seve ra l f r on ta l f ace

de tec to r cascades i n OpenCV. Thebest choice for you wi l l depend on yourset up. l t 's easy to swi tch betweenthem - l us t change the f i l e name. Why

no t t r y each?I t 's a lso possib le to create Your

own , cus tom XN/L f i l e us ing the

HaarT ra in ing app l i ca t i on , i n OpenCV 'sapps d i rec to ry . Us ing tha t app l i ca t i oni s beyond the scope o f t h i sar t ic le However, the inst ruct ions are

in OpenCV 's apps /haa r t ra in rng /docsd r recto ry

Coming Up ...Now that you've found a face, You

might want to fo l low i t around. Next

mon th , l ' l l show you how to use

Camsh i f t , OpenCV 's f ace t rack ingmethod , t o do l us t t ha t Be see rngyou I

Robin Hewitt is an independent soft'ware consultant working in the areas ofcomputer vision and robotics. She hasworked as a Computer Vision AlgorithmDeveloper at Evolution Robotics and is amember of 5O(3), a computer'visionresearch group at UC 9an Diego. She isone of the original develoPers ofSodaVision, an experimental face-recognition system at UC San Diego.SodaVision was built with OPenCV.

\

52 srnvo o2.2oo:'