Detection of Stress Using Image Processing and Machine ... · * Infosys, PocharamVillage Hyderabad...

Detection of Stress Using Image Processing and Machine Learning Techniques

Nisha Raichur #1, Nidhi Lonakadi*2, Priyanka Mural #3 Department of Information Science and Engineering, BVBCET, Hubli, India

* Tata Elxsi, ITPL main road, Hoodi Bengaluru,India 1 [email protected]

* Accenture, Ecospace Bellandur Bengaluru,India 2 [email protected]

* Infosys, PocharamVillage Hyderabad India 3 [email protected]

Abstract— Stress is a part of life it is an unpleasant state of emotional arousal that people experience in situations like working for long hours in front of computer. Computers have become a way of life, much life is spent on the computers and hence we are therefore more affected by the ups and downs that they cause us. One cannot just completely avoid their work on computers but one can at least control his/her usage when being alarmed about him being stressed at certain point of time. Monitoring the emotional status of a person who is working in front of a computer for longer duration is crucial for the safety of a person. In this work a real-time non-intrusive videos are captured, which detects the emotional status of a person by analysing the facial expression. We detect an individual emotion in each video frame and the decision on the stress level is made in sequential hours of the video captured. We employ a technique that allows us to train a model and analyze differences in predicting the features. Theano is a python framework which aims at improving both the execution time and development time of the linear regression model which is used here as a deep learning algorithm. The experimental results show that the developed system is well on data with the generic model of all ages.

Keyword - Stress, Facial expression, Theano, Framework, Deep learning

I. INTRODUCTION

Most of the researchers focused on detecting stress involved in a person, which causes in a person several emotional problems like anxiety, grief, low self-esteem and other mental health problems. Recent studies have shown that stress can also affect the aspects of your life, including your thinking ability and physical health. To reduce riskiness from being stress and affected with its adverse effects, it is crucial to detect such emotions and take certain actions to relax them.

Most of the previous work on stress detection is based on the digital signal processing, taking into consideration Galvanic skin response, blood volume, pupil dilation and skin temperature. And the other work on this issue is based on several physiological signals and visual features (eye closure, head movement) to monitor the stress in a person while he is working. However these measurements are intrusive and are less comfortable in real application.

In this work we develop a stress detection system based on the analysis of the facial expression. The system is non-intrusive and is able to run in real-time. A camera is used to capture the near frontal view of the person while he is working in front of the computer. The camera is mounted facing a person. Video captured is divided into three sections of equal length and set of equal number of image frames are extracted from each section correspondingly and are analysed. The image analysis includes the calculation of the variation in the position of the eyebrow from its mean position. The displacement of eyebrow from its position is calculated by scanning the image for the eyebrow co-ordinates. If the person is found stressed in the consecutive sections of the time intervals which was previously divided, the decision for stress detection is made for a person working in front of the computer

With the obtained results we employ the technique of deep learning which is a branch of machine learning which gives the computer an ability to learn without being explicitly programmed. Theano is a python framework which aims at improving both the execution time and development time of the linear regression model which is used here as a deep learning algorithm.

ISSN (Print) : 2319-8613 ISSN (Online) : 0975-4024 Nisha Raichur et al. / International Journal of Engineering and Technology (IJET)

DOI: 10.21817/ijet/2017/v9i3/170903S001 Vol 9 No 3S July 2017 1

In this sethat captimages th

The acquacquisitiobrightnesthe subsethe co-orthe eyebcalculatiopresence involved

The resulan area superviseused to plinear reg

This sectdetection

A. Task D

Differentdefine thstressed stress recrecognitiparticularvariation

B. Metho

1) Image

The faceextractedprocessintransform

Pixel tranpixel valuan input i

G (i, j) =paramete

ection we brietures the near hat are compe

uired video fraon and pre-pross and contrasequent modulerdinates of thebrow using thon of the disp

of emotion. .

lts that are obof machine l

ed learning algpredict the resgression algor

tion describesn target and th

Definition:

t people may he stress emot.To ease the pcognition. Eyeon. The eyebr person is us

n of the eyebro

odology:

e Pre-process

e acquisition md and the pre-png of the immation and the

nsformation aue depends onimage. The tw

= α . F (i, j) ers are said to

efly describe tfrontal view

ensated for the

ames are prococessing modust of the image. The stress d

e eyebrow. Thhe obtained placement. TThe integrate

btained using clearning that gorithm whichsults for the trithm using tra

III

s the implemene methodolog

behave or extion. And it iproblem, we ebrow which

brow movemeed to detect th

ows constantly

ing:

module proceprocessing of

mages includese other one is b

as an image prn only the corrwo commonly

+ β, The pacontrol the br

II. SYS

Fig. 1.Module

the hardware of a person

e illumination

cessed sequenule pre-proceses captured wdetection mod

he offline displeyebrow co-he classifier ed decision o

classification attempt to m

h analyzes theest dataset. Taining dataset

I. EMOTION

ntation detailsgies.

xpress differeis not easy trhave evolvedshows a rigid

ent has not ghe stress involy within a fixe

esses the videf the images fos two transfobinary transfo

rocessing funcresponding inused point pr

arameters α >rightness and t

STEM OVER

explaining the sy

and software while workineffects caused

ntially by the ssses the real ti

which are furthdule scans thelacement calc-ordinates whsub-module i

of individual

are further opmodel high-lee training dataheano being tthat produces

NAL STRESS

s of the indiv

ently under strain a model d with an algod transformatot any univerlved. We consed time interva

eo sequences for the subsequormations of ormation of th

ction is a technput pixel valurocessing are m

0 and β are the contrast of

RVIEW:

ystem overview

of the develong in front of d by the ambi

software modime captured iher convertede binary imageulation sub-m

hich is subseis trained offlframes eventu

ptimized usingevel abstractioa and producethe python pas a model to pr

S DETECTIO

idual modules

tress and it isoffline that c

orithm that relion is used asrsal pattern bsider that streal.

captured by uent analysis the extracted

e modified im

hnique of tranue, which helpmultiplication

often called f the image re

oped section. Tf the computerent lighting co

dules as illustrimages, which

d to binary imae from the ext

module calculaequently folloline are emplually determi

g machine leaons in data. s the inferred ackage has beredict the desi

ON:

s. This includ

s hard to findclassifies and lates to the fas the facial ar

but the variatiss is detected

the camera. in the further

d frame. Firsmage after pixe

nsforming a pips alter the brin and the addit

gain and the espectively

The camera isr. Camera caponditions

rated in Fig 1.h includes adjages for the atreme left top

ates the displaowed by the loyed to deterines the level

arning. Deep lLinear regresfunction, whi

een used in traired behaviou

des the definit

d a universal ppredicts if a

acial area of irea of interestion corresponif a person sh

The image frr modules is dst one being el transformat

ixel value. Eaightness and ction with a con

bias paramet

s mounted ptures the

. The face usting the

analysis in to record cement of

variance rmine the l of stress

earning is ssion is a ich can be aining the

ur.

ion of the

pattern to person is

interest in t in stress

nding to a hows high

frames are done. Pre-the pixel

tion.

ach output contrast of nstant.

ters, these



F (i, j) ->row and diverse. difficult.

Now the finding thset to 1 e

2) Eyebro

Several afeatures. scale (20top, this i

Binary cocolours uof 0 or 1.binary imrecorded is recordFigure 3 analysis (20, 181)

3) Stress

As defineyebrows

> source imagej-th column. That is if a pNow the imag

image is convhe threshold v

else 0, hence in

ow Detection:

approaches arWe investiga

00 x 200) pixeinvolves the a

onverted imagused for a bina. The bit valuemage scans th

for the first eded with its co

shows the notechnique, wh) represents th

Detection:

ed in the sects which are us

e pixel, G (i,j)The pixel tran

person is withge is filled wi

verted into binvalue the imagn this way the

:

re applied to eate approachesels. Pixel valuanalysis of eve

ge is a digitalary image are e 0 correspond

he image fromencountered pioordinates, whormalized fachich produceshe 20 Th row a

tion Task Defsed as the maj

)->output imansformation oh a darker comith primary co

nary image wge pixel valuee image is con

Fig 2

extract the diss based on theue analysis is ery pixel it enc

l image that hblack and wh

ds to black anm the top left ixel with the vhich is consid

cial image. Ths the (i, j) cooand the 181 Th

Fig. 3.Eyebrow

finition, stressjor facial area

age pixel. Wheof an image ismplexion the

olours (RGB).

which first incles are set that nverted into bin

2. Image Preproce

scriminative fe pixel analysithe method ocounters in ev

has only two hite. These bi-nd that of 1 cor

through eachvalue correspodered as the ehe image convordinates of thh column, wh

w Coordinates afte

s detection is a if interest in

ere the (i, j) ins done to mak

eyebrow and

ludes convertiis if the pixel nary form.

essing

features to leais of the imag

of analysing avery row of th

possible valu-tonal images rresponds to w

h row. The poonding to 0 thaeyebrow tip coverted to bina

he eyebrow ashich is the coor

er preprocessing

based on the its subsequen

ndicate that theke the model md its co-ordina

ing the image values > thre

arn the patternges which are an input imagee normalized

ues for each phas each pixe

white. The pixosition of the at is the first eo-ordinate of ary form is us depicted. Thrdinate of the

rigid transfornt process of an

e pixel is locamore generic ates detection

into greyscalshold the pixe

n of the differnormalized toe from the eximage.

ixel. Typicallels stored as a xel value analyrow and the c

encountered bthe normalizesed for the pi

he values of (itip of the eye

rmations shownalysis. The p

ated at i-th and more

n becomes

e , and by el value is

rent facial o standard treme left

ly the two single bit

ysis of the column is lack pixel ed image. ixel value i, j) being

ebrow.

wn by the previously



followeddetectiondisplacem

The first the obtainthe first ordinatescalculatio

The secoover a peshow var

4) Deep

The deepregressioobtained algorithmregressio

Linear rcorresponis of the f

X = { x1

Y = { y1

With thedataset co

To evaluexperime

A. Data

The during ththe subje

d processes usn methodologment and class

sub-module oned coordinatimage gives t

s (21, 202) , thon for each im

ond sub-modueriod of time iriance above t

Learning:

p learning modon algorithm,

results in thm is trained uson algorithm in

regression: usnding detectedform Y = mX

,x2...,xn} , X

,y2...,yn} , Y

se values of xonsisting of n

uate the perfental evaluatio

Collection

dataset consishe lab hours. Tects. None of

sed to pre-progy, which insifier which h

offline displactes in the prevthe x and y c

his sub-modulmage with resp

ule is calculatiis estimated bthe threshold i

dule is the finamodel and tehe previous sing the pythoncorporates th

sing the traind emotion of

X + c where in

represents the

represents the

xi and yi whicew input valu

Fig. 4.Graph s

I

formance of on. The results

sts of set of imThe videos wef the recorded

cess the inpunvolves sub-melps in stress d

cement calculavious steps, wco-ordinates ole gives the mepect to first im

ion of the varby calculating is classified by

al module thaesting the datamodules. Usi

on package Thhe gradient de

ning dataset an individual

e set of numer

e emotion of th

ch forms the trues for x to pre

showing displace

IV. EXPERIM

the developes are discussed

mage frames ere recorded dd subjects are

ut image for thmodules of detection.

ation, calculatwith respect to of the eyebrowean displacem

mage is calcula

riance. The eyits variability

y the sub-mod

at consists of thaset for prediing the supe

heano, which bscent algorith

which consil. The linear r

rical deviation

he correspond

raining dataseedict the corre

ement of eyebrow

MENTS AND

ed system, wd and justified

which are extduring the lab e professional

he eyebrow doffline displ

tes the displacits mean posi

w as (20, 181ment of the eyeated.

yebrow transfoy. And the set dule as stresse

he sub-moduliction. The traervised learninbuilds a modem which is us

sts of inputsregression mo

ns of eyebrow

ding xi.

et is used to tresponding y va

w with respect to t

D ANALYSIS

we collected d in this sectio

tracted from thours with a c actors. The

detection is fulacement cal

cement of the ition. In the F1) and the nexebrow and thi

ormation in eof images ov

ed over the par

les of training aining datasetng techniqueel prepared fosed to achieve

s of numericdel builds a l

w movements.

rain a model aalues.

time in minutes

S:

data and caon.

the videos reccamera placedsubjects durin

urther used in lculation, var

eyebrow positFig 3, the coorxt image gives way the disp

very subsequever a period ofrticular time in

dataset, learnt is prepared

e the linear rr prediction. T

e optimization

cal deviationsinear expressi

and tested usin

arried out qu

corded in the ud on a tripod ing the lab ho

the stress riation of

tion using rdinates in es the co-placement

ent image f time that nterval.

ning linear using the

regression The linear .

s and the ion which

ng the test

uantitative

university in front of ours work



continuouvideos recaptured the video

The is furtherthe entire

The Fig 5mean potraining adataset is

usly in front oecorded duringduring the lab

o every hour, h

whole dataser divided into e dataset and t

5 and Fig 6 shsition and theand the testings divides into t

of the computg the sessionsb hours for thhaving a total

t consists of vthe training a

the remaining

F

F

hows the datae standard devg data set. Thtraining and th

ter with the as are used forhe duration ofof 18 (6 * 3)

video recordinand the testingforms the test

Fig. 5. Data set alo

Fig. 6. Data set alo

a set along witviation. Thesee entire datasehe testing data

ssigned tasks.r the model adf 3 hours. Set frames during

ng and the corg datasets. Thting dataset us

ong with their ey

ong with their ey

th their eyebroe data sets foret consists of a sets

. These imagedaption in its of 6 frames o

g the session o

rresponding fhe training datsed to test the

ebrow co-ordinat

ebrow co-ordinat

ow co-ordinaterm the databa19 sets of im

es form the baanalysis. Vi

of an individuof three hours.

frames of (sevaset consists o

e trained mode

tes

tes

es, displacemease which are ages of 19 dif

asis of our daideo of an indual were extra.

veral individuaof all the samel.

ent of eyebrow primarily divfferent individ

ataset. The dividual is acted from

al), which mples from

w from its vided into duals. The



The trainlearning model prdataset bmodel Th

ning dataset itechnique therepared for pr

being a part ofheano precisel

is prepared ue linear regresrediction, is if the dataset ily predicts if a

Fig 7.The T

using the obtassion algorithmincorporated ts tested for tha person is str

Training and Test

ained results m is trained uto predict thhe desired resuessed.

ting Dataset

in the previousing the pythhe results forults using the

ous modules.hon package Tr the stress in

trained mode

Using the sTheano, whichn a person. Thel on GPU. Th

supervised h builds a he testing he trained



The modpoints bemodel aft

We devecomputerconsistin

The propeach perobtained this modparticipanrecognizepart of ou

[1] Toma3 DisDepre

[2] ACT [3] Austr

Emot[4] Hiche[5] Unive[6] H2C

Unsup

del trained usielow them as fter the trainin

eloped a monir. To assess

ng of 18 image

posed system irson for over

results from del with the nts and data, e his ongoing ur future work

as Simon Kruez2 ney Research Pitession from FaciaAustralia Austral

ralia David Vandyion Classificationem Sahli Lang Heersite de Montrea3J7 Canada Yo

pervised and Tran

ing the traininnot stressed. g predicts if th

V.

toring systemthe detection

es of an indivi

integrates imaa month .Th

image procestest data setthe main adstress in orde

k.

Iain Matthews3 ttsburgh PA USAal Actions and Volian Nat. Univ., Cyke Human-Centn via Data-Drivene, Dongmei Jiangal Yoshua Bengiooshua Bengio Densfer Learning ".

Fig 7.The Tra

ng dataset deThis model ishe set and the

CONCLUSI

m for detectingn performanceidual , hence t

age processinghese images wsing with suit. Although th

dded value of er to minimize

RYing Yang1 Min

A Fernando De laocal Prosody". Canberra. "Eye mtred Computing Ln Glottal Wavefog. "Multimodal Do Dept. IRO. "Leaept. IRO, Unive

aining of Deep Le

epicts the set s tested for thir correspondi

ION AND FU

g emotional ste, we conducotal of 19x18

g and deep leawere processetable inputs whe obtained r

f our work is e future health

REFERENCEnh Hoai Nguyen2a Torre2 1 Unive

movement analysisLab University of rms ". epression Recogn

arning Deep Archersite de Montre

earning Model

of points abohe set7, set8, sing person is s

UTURE WOR

tress of a perscted experime

images analy

arning to deteed in order t

were used to trresults are prachieved by

h risks. A stud

ES 2 Margara Tejeraersity of Pittsburg

s for depression df Canberra, Austra

nition with Dynamhitectures for AI"al. Montreal (Q

ove a line as set9, set10, sestressed.

RK:

son working cents on 19 coysed.

ct stress We hto extract a frain linear regreliminary wenabling end

dy on a larger

Padilla2 Feng Zhgh, 2 Carnegie M

detection". alian Capital Terr

mic Visual and A. C). "Deep Learn

stressed and et11. The deep

continuously iollected data

had collected feature (eyebrgression modeith limited nd users to sucr population w

hou2 Jeffrey F. CMellon University

ritory. "Depressio

Audio Cues ".

ning of Represe

the set of p learning

in front of sets each

images of row). The el and test number of ccessfully

will form a

Cohn1, 2 and y. "Detecting

on Detection

entations for



AUTHOR PROFILE

Nisha Raichur Software Engineer at Tata Elxsi, ITPL main road, Hoodi Bengaluru,India

Nidhi Lonakadi Software Engineer at Accenture, Ecospace Bellandur Bengaluru,India

Priyanka Mural Software Engineer at Infosys, PocharamVillage Hyderabad India



Detection of Stress Using Image Processing and Machine ... · * Infosys, PocharamVillage Hyderabad...

Documents

Transcript of Detection of Stress Using Image Processing and Machine ... · * Infosys, PocharamVillage Hyderabad...