A Vision-Based System for Monitoring the Loss of

Attention in Automotive Drivers

UNDER THE GUIDANCE OF-: PROF. S. B. JAMGE M.E. ELECTRONICS

W.I.T. SOLAPUR2014-2015

Table of Content Introduction Problem Statement & Objective Required hardware and software Block Diagram & proposed method Opencv Haar classifier Template matching Flowchart and results. Future scope References.

Introduction Drowsiness is one of the major cause for the accidents. Every year

10 to 15% of the accidents are due to driver drowsiness. The drowsiness can be defined as an transition from awake state

to sleep state The aim of this project is to develop a drowsiness detection

system. The focus will be placed on designing a system that will monitor the open or closed state of the eyes in real-time.

By monitoring the eyes, it is believed that the symptoms of fatigue can be detected early enough to avoid a accident.

PROBLEM STATEMENT The proposed work is to alert the driver when he is drowsy or

distracted away from normal alert mode to non-alert mode by means of buzzer or vibration mechanism.

In this project real time data is collected by video camera and other micro electro mechanical system devices (MEMS). This data gives information about driving condition of the driver which acts as input to controller. The appropriate measures are taken by the controller to alert the driver.

OBJECTIVES Our proposed method is to design and develop a low cost system,

which is based on embedded platform for drowsiness detection. Most existing approaches for the drowsiness detection rely either

on eye closure or head nodding angles to determine the driver drowsiness or distraction level.

This method combines the both eye state and head position to detect the drowsiness of the driver.

Used hardware and software Hardware-

Raspberry Pi Board Camera Accelerometer Buzzer

Software- Raspbian OS OpenCV Library. QT as editor. Wiring PI library.

Generalized block diagram

MEMS

USB PORT

CAMERA

ARM ARCHITECTUR

E

POWER SUPPLY

BUZZER

DISPLAY

Proposed Method The whole system is built upon Raspberry Pi board, Raspbian OS, OpenCV

image processing library and QT as editor. The primary focus is given to the faster drowsiness detection and processing

of data. The system is used to detect the eyes whether they are closed or open in

real-time through the camera. Here accelerometer simulates the different head position. The buzzer is turned on when eyes are closed and head position are above

the threshold values of the accelerometer.

Raspbian Commands and Dependency Packages

Sudo raspi-config Sudo apt-get update Sudo apt-get upgrade Sudo rpi-update Sudo apt-get install qt4-dev-tools Image I/O packages-: libjpeg8-dev, libtiff4-dev, libjasper-dev, libpng12-dev. Video I/O packages-: libavcodec-dev, libavformat-dev, libswscale-dev, libv4l-

dev GTK development library for GUI support.

What is OpenCV? OpenCV is an open source computer vision library, designed for

computational efficiency and has a high focus on real-time image processing.

It was developed by the Intel Research Initiative along with the Media Labs of MIT University.

It provides simple to use computer vision platform, which allows us to build the sophisticated vision application quickly.

Used by most of the companies such as SONY,INTEL,MICROSOFT,SIEMENS and GOOGLE.

One of the most popular project based on the OpenCV is GOOGLE STREET VIEW.

Why OpenCV? Execution Speed is fast than Matlab. Consumes less memory, which is important for the embedded platform. Interpretation time is less, as Opencv is programmed using optimized C

language. Supports different processor family from INTEL,AMD,ARM,SPARC &

POWERPC. Cross platform support for different operating system.

Structure of the OpenCV

CVImage processing

and vision algorithm

MLLStatistical Classifiers and Clustering tools

HIGHGUIGUI, Image and video

I/O

CXCOREBasic structure and algorithm, XML support, drawing

functions.

Haar Cascade Classifier A Haar Classifier is a machine learning approach for visual object detection

originally developed by Viola & Jones. It was originally intended for facial recognition but can be used for any object.

Haar Cascade classifier has two stages-: training and detection of the object. The haar classifier will quickly reject regions that are highly unlikely to

contain the object, and the object detection can move on to the further region. It does this by making use of the cascade of classifiers.

The classifier outputs ‘1’ when it finds an object and ‘0’ otherwise. The necessary applications for implementing a Haar classifier are included

in OpenCV and these can be used to train a classifier for detecting objects in an image.

Cascade Classifier Training There are two types-: open_haartraining & open_transcascade. The

open_transcascade support both the haar and Local binary pattern (LBP). Steps in training and using a haar classifier-:

Collect positive and negative training images. Markup positive images using objectmarker utility. Create a .vec file using createsamples utility. Train the classifier using haartraining utility. Run the classifier using cvhaarDetectObject().

Cont’d Collect the training images- generally about 1000 images are required to

train the classifier. It may include both positive and negative images. Mark positive images- This step creates a data file containing the file name

and the location of the object in the image. The data file is created using the object marker utility.

Create the vector file- It can be created using the createsamples utility of the opencv, optionally we can set the –width and –height.

Train the classifier- The classifier is trained with the utility opencv_haartraining.

The final output will be .XML (extended markup language) file.

Cont’d The final output of the classifier after training is an .xml file which has vector

values of the objects to be detected. For example-: <threshold>5.6097120977938175e-003</threshold> <left_val>-0.7382487058639526</left_val> <right_val>0.3885168135166168</right_val>

Examples of positive images

Examples of negative images

Template matching Technique There are two components-: source image(I) and template image (T). The template matching is a technique for finding the area’s of an image that

match to a template image (patch). To identify the matching area we have to compare the template image

against the source image by sliding it. For each location Of T over I, we store the metric in the result matrix (R).

Each location (x,y) in (R) contain match metric.

Detailed process of template matching. Loads an input image and a image patch (template) Perform a template matching procedure by using the OpenCV function

‘matchTemplate’ Normalize the output of the matching procedure Localize the location with higher matching probability using minMaxLoc. Draw a rectangle around the area corresponding to the highest match

Method = CV_TM_SQDIFF

R(x,y) =

Method = CV_TM_SQDIFF_NORMED

R(x,y) =

Project implementation

Cont’d The system comprises of 3 phases- Capturing- includes video capturing through camera. Detection- It includes the detection, whether eyes are closed or open and

also calculates the head position. Correction- If eyes are closed and head position is above threshold value the

buzzer turns on to alert the driver. Here eyes are located with the help of trained cascade classifier.

Flowchart Start

Capture video

Divide video into frames

Locate eye region

Eyes detection

All feature

detected?

Save eye template

A

Fetch frames

No

Yes

B

Haar cascade samples of eyes

Template matching

Eye tracking

Drowsiness Detection

More frames

to process?

End

A B

yes

No

Fatigue/Non fatigue

Results and Discussion Different head position’s

Figure shows x=19, y=3, z=17 head position= right

Figure Shows x=58, y=56, z=20 Head position=left

Results of Eye Blink Detection

Eyes open Eyes Closed

Conclusion We have designed the drowsiness detection system, built upon raspberry pi

as development board. The system uses open source software called as OpenCV image processing

libraries, which is very much suitable for the embedded board. The Haar classifier are used for locating the eyes. Raspberry pi and opencv makes the overall system a low cost drowsiness

detection system.

Future Scope Use of DSP processor makes the system more suitable for faster processing

for the faster processing of data. Night vision or near infrared camera can be used instead standard camera

that can work even in low light condition and during night. An improvement in the algorithm can be made to detect the eyes much

faster and results would be more accurate.

References R. N. Khushaba, S. Kodagoda, S. Lal, and G. Dissanayake, “Driver drowsiness

classification using fuzzy wavelet-packetbased featureextraction algorithm,” IEEE Trans. Biomed Eng., vol. 58, no. 1, pp. 121–131, Jan. 2011.

[2] Y. Liang, M. L. Reyes, and J. D. Lee, “Real-time detection of driver cognitive distraction using support vector machines,”IEEE Trans. Transp.Syst., vol. 8, no. 2, pp. 340–350, Jun.2007.

[3] R. I. Hammoud, G. Witt, R. Dufour, A. Wilhelm, and T. Newman, “On driver eye closure recognition for commercial vehicles,” SAE Int. J. Commercial Veh., vol. 1, no. 1, pp.454–463, Apr. 2009.

[4] Goel, P, et.al. "Hybrid Approach of Haar Cascade Classifiers and Geometrical Properties of Facial Features Applied to Illumination Invariant Gender Classification System," Computing Sciences (ICCS), 2012 International Conference on, vol., no., pp.132, 136, 14-15 Sept. 2012.

[5] Eye Detection Using Morphological and Color Image Processing, Tanmay Rajpathaka, Ratnesh Kumar and Eric Schwartz 2009 Florida Conference on Recent Advances in Robotics, FCRAR 2009.

Thank You