Frontier of Face Recognition Technology...Frontal face image Mask Sunglasses Profile By enhancing...

Frontier of Face Recognition Technology

October 24 , 2019Hitoshi Imaoka, NEC Fellow

NEC Corporation

2 © NEC Corporation 2019

NEC Fellow

Hitoshi Imaoka

1997 Joined NEC corporation, human brain system

2002 Started face recognition research, and

commercialization of face recognition product “NeoFace”

2009~ Achieved the No.1 accuracy 5 times in face

recognition benchmark tests

2015~ Started new research areas

- medical imaging: endoscopic cancer detection

- remote gaze detection

- otoacoustic recognition

2019 the youngest NEC Fellow in company history

The “face” of face recognition


What is the hurdle in face recognition?


Accuracy

Social consensus


Test image

B

C

A

Question:Which of these three pictures is me?

Why is face recognition

is so difficult?


Test image

Different

Same

Different

Same

Different

Same

B

C

A

Even in this sample, a lot of problems include

- long term aging change

- facial view, expression, similar face etc.

However, why can people understand?


Why is face recognition so difficult?

Hair style

Eyebrows

Eye close and open

Wearing glassesNose has

little information

Mouth open and close, smile

Other variations• view and

illumination• aging change• facial expression

• makeup• identical twins• plastic surgery etc.

Beard

Most facial parts can be changed


Do you know “Bio-IDiom” ?


Fingerprint /palm print

Face

Oto-acoustic

Voice

Fingervein

Iris


NEC’s world No.1 biometric authentication

All are the results of contests by the National Institute of Standards and technology (NIST).

Finger printauthentication

FpVTE (2003,2012)SFSE (2004)

MINEX(2016,2016)ELFT (2007)

PFT/PFTII (2009,2013)

8 times

No.1

Iris recognition

IREX IXIris Exchange IX

(2018)

Also Iris recognition

No.1

FRVT (2019)

MBGC (2009)

MBE (2010)

FRVT (2013)

FIVE (2017)

face recognition

Achieved world No.1 5 times


Multi-modal authentication

Combination of various biometrics such as fingerprint, face, and iris.

1970 1980 1990 2000

History of NEC’s Biometric Authentication technology

• Wider range of application in our daily lives through enhancement of technology and combination of different modalities.

A pioneer with over 50 years of experience in R&D

1982: Metropolitan Police Department (Japan)

1971: Started researchfor fingerprint recognition

Early 1990’s: Criminal AFIS for states across US

Late 1990’s: Global deployment

1984: San Francisco Police Department

2008: Developed finger hybrid recognition technology

2003: Ranked No.1 in US Government benchmark testing

1989: Started researchfor face recognition

1996 Started 3D face recognition R&D

1999 Commercialization of face recognition

2003 Global deployment

2009, 2010, and 2013:Ranked No.1 in US Government benchmark testing

2017: Ranked No.1 for the 4th consecutive time in US Government benchmark testing

FingerprintRecognition

Face Recognition

2010


Over 1000 systems in over 70 countries globally

Systems for Immigration Control, National IDs, Entertainment, etc.

Global Deployment of NEC’s Solutions


Integrated platform

Expansion of market value through biometrics technology

Authentication Security

Enterprise

Mobile

City Group

Public Safety

Individual

Society

QR code payments

User authentication

when opening

accounts

Police, local governments,

shopping areas

Crime investigation

and surveillance

Apartment buildings,

hospitals

Safety and security

of residentsLarge-scale events

Sporting events

Boarding

procedures

Airports

Entire regions, including airports,

tourist sites, hotels, etc.

Regional

revitalization

through hospitality

Identity verification

when entering/

exiting

Major financial

institutions (demonstration experiments)

Hands-free

payments using

face recognition

Train stations, malls,

vending machines

Push advertising

Fast food retailers

Personalized

customer loyalty

programs

Citizen ID

Nations/governments

MarketingPayment Hospitality

Major retailers

Unmanned stores

Safe and secure

operations

sports.png


NEC's Face Recognition Activities


Policies for NEC's face recognition technology development efforts

Reliability

Provide customers with reliable

technology by the world's highest

accuracy

TransparencyClarify the level of

technology through third-party evaluations


Advantages of NEC's face recognition algorithm

Weighting variable w

Featu

re

extra

ctio

n

Loss function L

Input x∂L(x,w)

∂w

Feature extraction

space

Verify features

MATCH

Deep learning

……

highly accurate

fast search speed

small face

- min eye distance 10 pixel

robustness of facial variations

wearing glasses, masks etc.

profile view

Original loss function and network structure


enrolled image

Enhanced robustness against various changes

Mask Sunglasses ProfileFrontal face image

By enhancing the recognition accuracy for partially occluded faces

ex. wearing a mask or sunglasses, or are turned to one side

Images for verification


Demonstration Video: Security Gate Solution

NEC's face recognition ranked No.1 in latest NIST (US) benchmark test

Patrick Grother, Mei Ngan, Kayee Hanaoka, Face Recognition Vendor Test (FRVT), NIST Interagency

Report 8271(Sep 11,2019)Https(Link)://(Link)www.nist.gov/site

Results shown from NIST do not constitute an endorsement of any particular system, product, service, or

company by NIST."

※ Subsequent face images are derived from this report.

https://www.nist.gov/sites/default/files/documents/2019/09/11/nistir_8271_20190911.pdf

https://www.nist.gov/sites/default/files/documents/2019/09/11/nistir_8271_20190911.pdf


The purpose is to evaluate face recognition accuracy and search speed, using large-

scale data up to 12 million people enrollment

Overview of Face Recognition Vendor Test (FRVT) 2018

What is Face Recognition Vendor Test (FRVT) 2018?

Examples of

evaluation images※

Evaluator: U.S. National Institute of Standards and Technology (NIST)

Organization established to enhance technological innovation and industrial competitiveness

Completely blind test

Evaluations are

rigorous and fair


Face Recognition Vendor Test (FRVT) 2018* Participants

49 Leading organizations all over the world participate

Country/regionNumber of

organizationsParticipating organizations

U.S. 11Aware, Camvi, Ever AI, Frog wing, Incode, Micro Focus, Microsoft, Noblis, Rank One, Real

Networks, Remark AI, Shaman, Vigilant Solutions

China 11Anke, Dahua, Hikvision, megvii, Newland, Sensetime, SIAT CASIA, Tong Yi Trans,

Yisheng, Yitu

Europe 9Cogent Gemalto (France), Cognitec (Germany), Dermalog (Germany), Eyedea (Czech

Republic), Idemia (France), Innovatiscs (Slovakia), NeuroTechnology (Lithuania), NFS

(U.K.), Quantasoft (Czech Republic), Visidon (Finland)

Russia 8 3DiVi, Innovation Sys., NTechLab, Smilart, Synesis, Tevian, VisionLabs, Vocord

Japan 4 Ayonix, GLORY, NEC, Toshiba

Other 6Alchera (S. Korea), Gorilla (Taiwan), HB innovation (S. Korea), Imagus (Australia),

Lookman (India), Mvision, Nodeflux (Indonesia), Tiger IT (Bangladesh)

48 companies, 1 research institution, and no universities

Participation increased since the previous NIST FIVE benchmark test (up from 16 organizations to 49)

Names of participants and evaluation results are included in the report

FRVT2018 Overall evaluation

Comparison of Authentication Accuracy

Comparison of Authentication Accuracy and Search Speed


Evaluation Results: Comparison of recognition accuracy

NEC is the 1st rank of identification rate, whose error rate is less than 1%.

※ False negative identification rate at a false positive identification rate of 0.1% at 1.6 million registered people. Compared only by the highest identification accuracy algorithm in each organization. Including Research Institutes in Part of the "Vendor" Labeling

Identific

atio

nerro

r rate

(%)

100.0

90.0

80.0

70.0

60.0

50.0

40.0

30.0

20.0

10.0

0.0

NEC

Vendor

1

Vendor

2

Vendor

3

Vendor

4

Vendor

5

Vendor

6

Vendor

7

Vendor

8

Vendor

9

Vendor

11

Vendor

12

Vendor

13

Vendor

10

Vendor

14

Vendor

17

Vendor

16

Vendor

18

Vendor

19

Vendor

20

Vendor

21V

endor22

Vendor

23

Vendor

24

Vendor

25

Vendor

26

Vendor

27

Vendor

28

Vendor

29

Vendor

30

Vendor

31

Vendor

32

Vendor

33

Vendor

34

Vendor

35

Vendor

37

Vendor

36

Vendor

39

Vendor

38

Vendor

40

Vendor

41

Vendor

42

Vendor

43

Vendor

44

Vendor

45

Vendor

46

Vendor

47

Vendor

48

Vendor

15

Major differences among topvendorsNEC has about 1/3 the error rate of the runner-up

Effective for applications that require high reliability such as payments

NEC

Vendor

1

Vendor

2

Vendor

3

Vendor

4

Vendor

5

Vendor

6

5.0

4.0

3.0

2.0

1.0

0.00.4

1.21.4

2.02.3

2.9

3.8 3.9 4.0

4.6

Vendor

7

Vendor

8

Vendor

9


Evaluation results: Comparison of recognition accuracy and search speed

Identification error rate (%)

NEC

Vendor 19

Vendor 17

Vendor 6

Vendor 28

Vendor 46

Vendor 35

Vendor 38Vendor 39

Vendor 23

Vendor 21

Vendor 44

Vendor 16Vendor 12

Vendor 22

Vendor 3

Vendor 45

Vendor 11

Vendor 47

Vendor 36

Vendor 26

Vendor 15

Vendor 31

Vendor 24

Vendor 4

Vendor 37

Vendor 9

Vendor 48

Vendor 5

Vendor 25Vendor 27

Vendor 30

Vendor 43

Vendor 7

Vendor 13

Vendor 20

Vendor 14

Vendor 10

Vendor 33

Vendor 2

Vendor 42

Vendor 18

Vendor 1

Vendor 40

Vendor 29

Vendor 34

Vendor 41

Vendor 8

Vendor 32

0.1%1.0%10.0%100.0%

Compared only among the algorithm with the highest identification accuracy in

each organization. Including Research Institutes in Part of the "Vendor" Labeling

Searc

h s

peed (m

atc

hin

gs/s

ec)

1 million

10 million

100 million

1 billionComparing recognition Accuracy and Search Speed in NIST FRVT2018

NEC's high accuracy

algorithm is 230

million matchings/sec.

※ False negative identification rate at a false positive identification rate of 0.1% at 1.6 million registered people

NEC achieved overwhelming performance in terms of both recognition accuracy and search speed.


Evaluation Results: Impact of aging changes on accuracy

NEC algorithm is more accurate than other vendors for long-term aging changes

- Differences increase as aging increases

- Error rate 4 times lower than the runner-up

Maintains high recognitionaccuracyover long-term changes,such as passport authentication

※ False negative identification rate at a false positive identification rate of 0.1% at 3.1 million registered people

0

5

10

15

20

25

30

35

NEC

Identific

atio

nerro

r rate

(%)

※

0-2 3-4 5-6 7-8 9-10 13-12 13-14 15-18 (year)

Vendor E

Vendor F

Vendor D

Vendor B

Vendor A

Vendor C

2.7

2.11.71.51.30.9

0.7

1.32.3

1.1

3.44.6

5.87.1

9.0

11.5

* Face Image Source: Patrick Grother, Mei Ngan, Kayee Hanaoka, Face Recognition Vendor Test (FRVT), NIST Interagency Report 8271(Sep 11, 2019)


Evaluation Results: Impact of the number of registered people on accuracy

NEC’s technology can be applied to large-scale systems and maintain recognition accuracy

Similar looking

The difference increases as the numberof registered users increases

Vendor E

Vendor F

Vendor G

Vendor H

NEC

3.5

3.0

2.5

2.0

1.5

1.0

0.5

0.0

Number of registered people (million)

Identific

atio

nerro

r rate

(%) ※

0.64 1.6 3 6 12

0.40.4 0.4 0.5

1.8

1.3

1.10.9

0.8 0.5

※ False negative identification rate at a false positive identification rate of 0.3%

As the number of registered users increases,the number of similar looking people increases,making it difficult to recognize

Identification Error Rate is just 0.5%at 12 million registered people

* Face Image Source: Patrick Grother, Mei Ngan, Kayee Hanaoka, Face Recognition Vendor Test (FRVT), NIST Interagency Report 8271(Sep 11, 2019)


Face Recognition Vendor Test (FRVT) 2018 Summary

With the evolution of AI technology, many companies

have begun adopting face recognition

49 organizations from around the world participate

(48 companies, 1 research organization)

Large differences for the evaluation of ageing changes

As much as 4 times difference between 1st and 2nd

Error rate for 15-18 years: NEC 2. 7%, 2nd place: 11.5%

NEC continues to be in the top position with

identification accuracy Error rate 0.5% @ 12 million registered people

NEC is the top in search speed Search speed: 230 million matchings a second

NIST comments:

NEC algorithms

continue to have the

Most Accuratefollowing 2010 and 2013

NIST IR 8271 p.11Technical Summary

Quote from

※ Comparing the most accurate algorithms for each participating organization

Comparison between Computer and Human face recognition


Are you up for the challenge?

Face Recognitionby AI

Face Recognitionby Human


Global deployments ofNEC’s biometrics


Faster and securer immigration control with face recognition.

JFK International Airport, USA: Entry Control system

NEC press release https://www.nec.com/en/press/201605/global_20160505_01.html

Immigration Control

Enhanced security with face recognition at Automated Passport Control kiosks at JFK International Airport in New York.

NeoFace face recognition engine.

Reduce processing time, enhance customer service and safety of airport operations.

FaceRecognition

Matching of face data in the e-passportagainst the photo of the passengercaptured at the kiosk, to ensure thelegitimate owner of the passport isentering the country.

System delivered by Unisys and inoperation since January 2016.

Challenges

Solution

Results


Frictionless exit control system using face recognition.

USA: US Exit Control face recognition


Immigration Control

Enhanced security for exit control using face recognition.

Enable frictionless exit process.

Walkthrough face recognition system NeoFace Express

Reduce processing time, enhance customer service and safety of airport operations.

FaceRecognition

Exit control trial in several US airportsincluding Dulles International Airport.

Checking ageist passport DB (US citizens)

and entry data (visitors) to confirm theidentities of those leaving the country.

Enhance security and enable detection ofillegal overstays of foreign visitors.

Challenges

Solution

ResultsUS Customs and Border Protection

https://ja-jp.facebook.com/homelandsecurity/posts/us-customs-and-border-protection-cbp-is-the-dhs-component-responsible-for-securi/594616770550963/


Preventing illegal activities at airport customs control.

Brazil: Face recognition for 14 international airports

FaceRecognition

Detection of pre-registered individuals who have been registered for suspicious activities.

NeoFace Watch real-time face recognition system

Increase efficiently and rigor of customs control at airports.

Used to enhance efficiency andeffectiveness of customs operations in14 international airports in Brazil.

Screening of passengers as they walkpast the customs control area basedon a database of pre-registeredsuspects.

Challenges

Solution

Results

Enhanced Citizen Services



Social services for 1.3 billion with multi-modal biometrics.

India: Unique Identification Numbers Program

FaceRecognition

FingerprintRecognition

IrisRecognition

Accurate citizen data registered with the government.Equal and efficient social services for citizens.

Multi-modal biometric authentication system.

Equal services for its 1.3 billion citizens.

ID theft prevention

Multi-modal system combining face,fingerprint and iris recognition toidentify individuals.

Prevent duplicate registrations andensure all citizens have equal accessto services such as food supply,employment and tax payment.

Challenges

Solution

Results

Enhanced Citizen Services

http://www.google.co.jp/imgres?imgurl=http://newsonair.nic.in/feature-image/uid_logo.jpg&imgrefurl=http://newsonair.nic.in/AADHAAR-UID-Card-THE-NUMBERS-FOR-LIFE.asp&usg=__LtKpEwWGRAWj01ZaF6wBUAqt1uY=&h=402&w=633&sz=104&hl=ja&start=9&zoom=1&tbnid=7phuecH93sCe3M:&tbnh=87&tbnw=137&prev=/images?q=aadhaar&hl=ja&sa=N&gbv=2&tbs=isch:1&itbs=1


Other applications in Japan


Identity theft prevention

Japan: Concert Ticket Holder Identity Verification

NEC Online TV: https://www.nec.com/en/global/onlinetv/en/concert.html

Supports events of every size or scale. Reduces burden on audience by eliminating the need to bring photo IDs to the venue.

NeoFace face recognition engine for identity verification.

Risk of entry with resold tickets using borrowed or fake IDs.

Ticket resell prevention and smooth admission!FaceRecognition

Matching of faces captured using tablets at admission with the pre-registered face images of the ticket holders.

Supports events of any size and scale, including those with tens and thousands of audience through fan clubs.

Reduces admission time by up to 50% compared to visual inspection by concert staff, enabling smooth entry for the audience.

Challenges

Solution

Results

Mega Event


Terra-cotta soldier’s face recognition (TV program project)

Sculptures of the first emperor of China’s army

Buried over 8,000 soldier sculptures

Analyzed sculpture faces using face recognition software

All of their faces are unique


Save the memory project in the Great East Japan Earthquake

▌Great East Japan Earthquake and Tsunami

11th March 2011

Magnitude 9.0

20,000 dead and missing people


“Save the Memory Project” (collaboration of Ricoh and NEC)

▌Earthquake disaster reconstruction project

Rescue team collect albums and photographs

Volunteer washed and digitized photographs

Face recognition is used to search 150,000

photographs

Return photographs to the owner

Face recognition system assisted in returning

12% of the photographs to the owner

https://www.ricoh.com/-/Media/Ricoh/Sites/com/release/2015/pdf/0309_stmp_E.pdf


Applied biometrics technologies


Remote gaze detection

Understanding interest of person with remote camera

30-50 cm

Conventional methods

Ordinary camera:Capturing high resolution image

Special devices (IR, etc.):Detecting eyes for higher precision

Our technology

Key technology

Feature points detection of eyes,

which is developed for face recognition

Need to detect accurate eyes position (center of pupils, tail of eyes, etc.)in near distance

10m away

CamerasMultiple peoplein real time

Detects gazes accurately from low resolution images in remote distances


Demonstration Video : Gaze Detection Technology


5 persons 3 persons

Conventional method Our technology

High throughput even in crowd scene High accuracy in the scene with occlusion

Deriving the number of person in each patch and summing up them

Detecting each personand counting

Learn the number of persons in each patch image generated by simulation

Crowd behavior analysis

Recognize crowd accurately with existing surveillance camera


Demonstration Video: Crowd behavior analysis


Stress Estimation

Behavior Estimation

Understandingbehavior

SecuritySecret

computation

Social acceptabilitytechnology

Multi-modal

Personal verification

Multi-modalBiometrics

To the future application

Face recognition

Access control

Immigration Control

Terminal login

Warranty of safetyEnsuring fairness

Payments and Financial Transactions

Entertainment

Walk-through

City security

Healthcare


Summary

Security Privacy

Frontier of Face Recognition Technology...Frontal face image Mask Sunglasses Profile By enhancing...

Documents

Transcript of Frontier of Face Recognition Technology...Frontal face image Mask Sunglasses Profile By enhancing...