All Your Faces Belong To UsBuilding an Open Face Recognition Platform

Copyright © 2016 Accenture  All rights reserved. 2

About Us

Senior Technology ArchitectAccenture SpainGlobal Java Community LeadTwitter: @_deorsGitHub: deors

Senior Technology AnalystAccenture SpainGlobal Java SMETwitter: @locoporf1GitHub: locoporf1

Copyright © 2016 Accenture  All rights reserved. 3

Related Sessions in Java One 2016Monday, Sep 19 Tuesday, Sep 20 Wednesday, Sep 21Introduction to Java ME 8 [CON3189]

Java ME and Single-Board Computers for Creating Industrial Middleware [CON3187]

All Your Faces Belong to Us: Building an Open Face Recognition Platform [CON6217]

12:30 p.m. - 1:30 p.m 2:30 p.m. - 3:30 p.m 3:00 p.m. - 4:00 p.m

Hilton - Golden Gate 6/7/8 Hilton - Golden Gate 6/7/8 Hilton - Golden Gate 6/7/8

Kevin HookeJulio Palma

Jorge HidalgoJulio Palma

Jorge HidalgoMariano Rodriguez

Copyright © 2016 Accenture  All rights reserved. 4

• Internet of Things• Applications

• OpenCV library• Cascade Classifier for Object Detection

• Lbp vs Haar• Java API• Object Recognition Sequence Diagram

• MQTT protocol• Quality of Service

• Faces counter• Solution schema• Development environment

• Demo live!

Building an Open Face Recognition Platform

Copyright © 2016 Accenture  All rights reserved. 5

Internet Of Things

• Objects: Pressure sensor in tires Monitor implant in hearts GPS in cabs Thermometers in offices …

• With unique identifier• Connected to internet• Transfer data without requiring

human interaction

Copyright © 2016 Accenture  All rights reserved. 6

IoT Applications

• Smart Cities: smart street lights, traffic control, surveilance cameras, …

• Healthcare: remote monitoring, ambulance telemetry, drug tracking, …

• Smart Manufacturing: flow optimization, real time inventory, employee safety, …

• Wearable: entertaiment, fitness, smart watch, location & tracking, …

• Automotive: wire replacement, telemetry, predictive maintenance, …

• Building & Home automation: light & temperature control, access control, energy optimization, …

Smart homesAppliances

Wearables

Industrial internet

Healthcare Smart places

Service Industries

Smart citiesP

Copyright © 2016 Accenture  All rights reserved. 7

OpenCV Library

• Coded in C/C++• Free for academic or commercial use• Supports Windows, Linux, Mac OS, iOS and Android• Focused on real-time applications: optimized using hardware

accelaration and multi-core processing• C++, Phyton and Java interfaces• Current version: 2.4.13 (May, 2016)• Repository: https://github.com/opencv/opencv

Copyright © 2016 Accenture  All rights reserved. 8

Cascade Classifier

1. A XML file applied to every frame of the video stream to detect a particular object

2. OpenCV distribution contains some examples to detect eyes, cats, human faces, …

3. Can be generated to detect any object by training

Copyright © 2016 Accenture  All rights reserved. 9

Cascade Classifier Types

LBP Faster Calculations in integers Best option for

embedded/mobile

HAAR More accurate Calculations in floats

Falses Positives

Lost Frames / sec

LBP 0 2 5.72

HAAR 2 0 4.24

Copyright © 2016 Accenture  All rights reserved. 10

• org.opencv.core• Mat

• org.opencv.videoio• VideoCapture

• org.opencv.objdetect• CascadeClassifier

• org.opencv.imgproc• Imgproc

• org.opencv.imgcodecs• Imgcodecs

OpenCV Java API

Copyright © 2016 Accenture  All rights reserved. 11

Object Detection Sequence Diagram

read

VideoCapture

Mat

Imgproc

cvtColor

equalizeHist

CascadeClassfier

detectMultiScale

Imgcodecs

imwrite

new

new

rectangle

Copyright © 2016 Accenture  All rights reserved. 12

Low network bandwidth Low power usage Perfect for IoT Publish / Subscribe mechanism Can use WebSockets Downloads:

Client: http://www.eclipse.org/paho/ Broker:

http://projects.eclipse.org/projects/technology.mosquitto

MQTT Protocol

MQTT Broker

Topic

Device 1

publish

Device 2

Device N

subscribe

subscribe…

Copyright © 2016 Accenture  All rights reserved. 13

MQTT: Quality of Service

Publisher Broker Subscriber

publish

0: At most once Guarantees a best effort delivery It won’t be acknowledged by the

receiver or stored and redelivered by the sender

Fire and forgetpublish

Copyright © 2016 Accenture  All rights reserved. 14

MQTT: Quality of Service

Publisher Broker Subscriber

publish

1: At least once Guarantees that a message will be

delivered at least once to the receiver

But the message can also be delivered more than once

Application must be tolerating duplicates and process them accordingly

publish

puback

store message

delete message

Copyright © 2016 Accenture  All rights reserved. 15

MQTT: Quality of Service

Publisher Broker Subscriber

publish

1: Exactly once Guarantees that each message is

received only once by the counterpart

Safest and also the slowest quality of service level

publishpubrec

pubrel

pubcomp

store message

store message

Copyright © 2016 Accenture  All rights reserved. 16

Face Detector Device

• Raspberry PI (model 3)• Raspicam (V2) ~ 60 $}

Copyright © 2016 Accenture  All rights reserved. 17

Faces Dectector Solution Schema (I)

…

Face detector devices

publish

MQTT Broker Web Server

subscribe

MQTT MQTT

WS

Copyright © 2016 Accenture  All rights reserved. 18

Face Counter Solution Schema (II)

Java RuntimeFacesDetector App

OpenCV MQTT Paho

JNI

OpenCV

Raspicam drivers

publish subscribe

MQTT broker Java Runtime

Spring Boot

Monitor WebApp

Vaadin

Copyright © 2016 Accenture  All rights reserved. 19

Setting up a Raspberry PI: Install dependecies

sudo apt-get update

sudo apt-get install cmake ant build-essential pkg-config libpng12-0 libpng12-dev libpng++-dev libpng3 libpnglite-dev zlib1g-dbg zlib1g zlib1g-dev pngtools  libtiff4 libtiffxx0c2 libtiff-tools libjpeg8 libjpeg8-dev libjpeg8-dbg libjpeg-progs libavcodec-dev   libavformat-dev libgstreamer0.10-0-dbg libgstreamer0.10-0libgstreamer0.10-dev  libunicap2 libunicap2-dev libdc1394-22-dev libdc1394-22libdc1394-utils swig libv4l-0 libv4l-dev

Copyright © 2016 Accenture  All rights reserved. 20

Setting up a Raspberry PI: Enviroment variables

Edit ~/.bashrc file to add the following lines:export JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/export ANT_HOME=/usr/share/ant/

Copyright © 2016 Accenture  All rights reserved. 21

Setting up a Raspberry PI: Download OpenCV source code

wget https://codeload.github.com/Itseez/opencv/zip/3.1.0mv 3.1.0 opencv-3.1.0.zipunzip opencv-3.1.0.zip

Copyright © 2016 Accenture  All rights reserved. 22

Setting up a Raspberry PI: Create Makefile

cd opencv-3.1.0mkdir buildcd buildcmake -DCMAKE_BUILD_TYPE=RELEASE -DWITH_OPENCL=OFF -D BUILD_PERF_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DJAVA_INCLUDE_PATH=$JAVA_HOME/include -DJAVA_AWT_LIBRARY=$JAVA_HOME/jre/lib/amd64/libawt.so -DJAVA_JVM_LIBRARY=$JAVA_HOME/jre/lib/arm/server/libjvm.so -DCMAKE_INSTALL_PREFIX=/usr/local ..

Copyright © 2016 Accenture  All rights reserved. 23

Setting up a Raspberry PI: Compile, install and check• make install• ATTENTION!!! This process can take several hours!• Check if exist: build/libs/libopencv_java310.so and build/bin/opencv-

310.jar have been created!!!

Copyright © 2016 Accenture  All rights reserved. 24

Setting up a Raspberry PI: Install raspicam drivers• curl http://www.linux-projects.org/listing/uv4l_repo/lrkey.asc |

sudo agt-key add –• Edit file /etc/apt/source.list to add the following line: deb http://www.linux-projects.org/listing/uv4l_repo/raspbian/ jessie main• sudo apt-get update• sudo apt-get install uv4l uv4l-raspicam• sudo apt-get install uv4l-raspicam-extras• sudo raspi-config (Enable raspicam interface)• sudo rpi-update (Update firmware)

Copyright © 2016 Accenture  All rights reserved. 25

Setting up a Raspberry PI: Transfer files & run

• Using ftp client tool copy:• Faces detector properties and JAR files• MQTT Paho JAR file

• Edit properties file to customize• Run java -Djava.library.path=build/libs/libopencv_java310.so -jar

faces-counter.jar faces-counter.properties

Copyright © 2016 Accenture  All rights reserved. 26

Development Enviroment

Remote Mode (option I):• Oracle JDK • NetBeans• Putty• FileZilla (client)

Remote Mode (option II):• Oracle Java ME SDK• NetBeans

“Embedded” Mode• BlueJ or Geany• Oracle JDK

Copyright © 2016 Accenture  All rights reserved. 27

• Faces Detector: https://github.com/locoporf1/faces-detector

• WebApp Monitor: https://github.com/locoporf1/faces-detector-server

Face Detector App Source Code

28

Question & Answers

29

Live DEMO !!!