Post on 18-Jun-2020
http://www.iaeme.com/IJMET/index.asp 126 editor@iaeme.com
International Journal of Computer Engineering & Technology (IJCET) Volume 8, Issue 5, Sep-Oct 2017, pp. 126–135, Article ID: IJCET_08_05_014
Available online at
http://www.iaeme.com/ijcet/issues.asp?JType=IJCET&VType=8&IType=5
Journal Impact Factor (2016): 9.3590(Calculated by GISI) www.jifactor.com
ISSN Print: 0976-6367 and ISSN Online: 0976–6375
© IAEME Publication
AUTOMATIC FACIAL EXPRESSION RELATED
EMOTION RECOGNITION USING MACHINE
LEARNING TECHNIQUES
V. Sathya
Research Scholar, A.V.V.M .Sri Pushpam College, Poondi, Tamilnadu, India
T.Chakravarthy
Associate Professor, Department of Computer Science, A.V.V.M Sri Pushpam College,
Poondi, Tamilnadu, India
ABSTRACT
Facial expression are commonly used in everyday human communication for
express the emotions. Emotions are reflected on the face, hand, body gesture and
voice to express our feelings. In human communication, the facial expression is
understanding of emotions help to achieve mutual sympathy. It is a nonverbal
communications. Computer vision based technology is placed an important role in
various applications especially in human emotion recognition process because
emotions are related to the peoples mental ability and thinking process[1]. More ever,
one single emotions leads to create the difficult health problems. Peoples affected by
single emotions due to their stress, over thinking, personal problems and so on. So,
their mental ability need to be maintained continoulsy for avoiding their health issues
which is done by linking the emotion recognition system with computer vision area
that effectively utilize the intelligent techniques [2]. The intelligent techniques analyze
the human emotions from different parameters such as facial expression had
electroencephalogram (EEG) brain activities with successful way. Among the
parameters, facial expression based emotion recognition process is one of the easiest
method because it does not require high cost, easy to capture the face expression [3]
with the help of the digital camera, minimize the computation complexity also the
impact of the facial expression is related with the brain activities and social impacts.
There are there are 100 types of facial expressions such as blinking, cheerless, coy,
blithe, deadpan, brooding, glowering, faint, grave, dejected, derisive, leering, moody,
hopeless, slack-jawed and so on. These facial expressions are derived from the basic
expressions such as Happy, Sad, Anger, Disgust, Surprise, Fear and Neutral.
Keywords: facial expression, emotion recognition, the non-local median filtering,
neural networks, hidden markov model.
Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques
http://www.iaeme.com/IJMET/index.asp 127 editor@iaeme.com
Cite this Article: V. Sathya and T.Chakravarthy, Automatic Facial Expression
Related Emotion Recognition Using Machine Learning Techniques, International
Journal of Computer Engineering & Technology, 8(5), 2017, pp. 126–135.
http://www.iaeme.com/ijcet/issues.asp?JType=IJCET&VType=8&IType=5
1. INTRODUCTION
Facial expression is one of the important computer vision based process that helps to detect
the human emotions, feelings and mental ability in different situations. To detecting the
emotions, the well-defined automatic facial emotion recognition system has been developed
in the recent years because [7] it has several attraction in the mental ability detection process.
from the literature of the different authors opinions, facial epression related images [8] are
used to detect their emotions with accurate manner. Even though the facial expression images
are consume low time, cost, minimize the complexity, some times of the facial points are
difficult to detect with accurate manner. In addition to this, the detected facial points are fails
to classify the exact emotions using the traditional classification techniques. Hence earlier
classification techniques only uses the particular features and develop a classifier for emotion
recognition process. Therefore a new novel combination of techniques has been proposed to
remove the noise, affected region segmentation, feature extraction and the good classification
techniques which improve the classification performance. Here the major objective is to
recognize the facial expression using the effective classification techniques.
2. PROPOSED WORK
In this Proposed work, Jaffe and Cohn Kanade database is used to recognize the facial
expression related emotion using different classifiers. Both the dataset consists of collections
of images which was captured in various emtoions that used to detect the emotions in
automatic way. The database images are used to for both training and testing process which is
done by utilizing the different image processing techniques. Then the sample captured
database image is shown in the figure 2.
Figure 1 Sample Database Facial Expression image
By using the above images the facial expressions has been classified by applying the
proposed methods which is shown in the figure 3.
• Initially the face expression images has been captured and the facial point is detected
with the help of the geometric method and the unwanted noise is removed by using the
Non local median filter.
• Then the feature points are extracted by using the local binary pattern and progression
invariant sub space learning method.
V. Sathya and T.Chakravarthy
http://www.iaeme.com/IJMET/index.asp 128 editor@iaeme.com
• Optimal features are selected using the the Particle Swarm Optimization Process
(PSO)
• Extracted features are trained by back propagation neural networks (BPNN).
• Finally the emotion recognition is done with the help of the different classifiers like
Hidden Markov Model (HMM), Support Vector Machine (SVM) and Back
Propagation Neural Networks (BPNN)
2.1 Noise Removal
Initially the facial expression images are captured by digital camera, the geometric facial
points [9] are detected as follows,
�� = (ln �) (1)
In the above eqn (1), �� �� ℎ� ��������� �� �����. The facial points are detected with
minimum delay,. After detecting the facial points, the color of the images are changed, if the
captured image is color image. The color transformations is done by as follows,
�� = 0.2989 ∗ ! �!�� "(�) + 0.58701 ∗ ! �!�� " (�) + 0.1140 ∗ ! �!�� " (() (2)
The color transmitted images may contains several noise that reduces the emotion
recognition system. So, the noise present in the image is eliminated using the non-local
median filter [10]. First intensity of the images are estimated as follows,
)(�) = *(�) + !(�) (3)
Where)(�) is defined as the observed value from the given image, *(�) is defined as the
“true” value and !(�) is defined as the noise agitation at a pixel �. Then the noise influenced
by the images are examined, then the images are mostly affected by Gaussian noise that is
eliminated with the help of the following assumptions, !(�) areindependent, identically-
distributed Gaussian values with variance +2and zero mean. Based on the assumptions, the
neighborhood pixel value is estimated with the help of the weighted values
,(�, .1)�!� ,(�, .2).According to the above process, each pixel,�is investigated and the
non-local median filter is estimated as follows,
�/(0)(�) = ∑ ,(�, .)0(.)234 (4)
Where 0 is defined as the noisy image, and weights ,(�, .) meet the subsequent
conditions 0 ≤ ,(�, .) ≤ 1 and∑ ,(�, .) = 12 . After estimating the non-local value, the
similarity value between the neighborhood values is calculated as follows,
�(�, .) = ‖0(�7) − 0(�2)‖,9 [1,2] (5)
Where < is defined as the neighborhood filter employed to the neighborhood’s squared
difference. The weights is defined as follows
,(�, .) = �=(7) �>?@A (BC>CDC(E,F)
G (6)
Where+ is as defined as the standard deviation of the noise and 2+ are set to 1.
Where H(�) is defined as the normalizing constant is defined as follows
H(�) = ∑ � IJ(7,2)K2 [1,2] (7)
Where ℎ is defined as the weight-decay control parameter. As earlier mentioned, < is
known as the neighborhood filter with LMNO. The weights of < are computed is as follows
< = �PQRS
∑ 1/(2 ≠ �|1)PQRSNWO (8)
Where � is defined as the distance the weight is from the neighborhood filter’s center.
This process is repeated until to eliminate the noise from the image with effective manner.
Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques
http://www.iaeme.com/IJMET/index.asp 129 editor@iaeme.com
After detecting the face using the geometric approach, different features are derived this is
done with the help of local binary pattern and progression invariant sub space learning
method.
Figure 3 Proposed System Architecture
2.2 Feature Extraction
The next step is feature extraction which is done by using the local binary pattern and
progression invariant sub space learning method [11]. First the local binary pattern process is
applied to the image, that analyze each and every pixel present in the image and the particular
operator is assigned to each pixel. After that the threshold value is examined by using the 3*3
neiughboring pixel image. According to the process the general local binary pattern
representation is shown in the following figure 3.
Figure 3 Local Binary Pattern Representation
From the detected pixels and threshold value, the image has been represented using the
circle of center. Based on the above neighboring pixel representation, the facial texture
features are derived by combining the local descriptors with global descriptors because it
manages the variations and illuminations also it effectively examines the ordinary features
with effective manner. The method examines the features in different directions, rotations and
relationship between the pixels or key points by dividing the captured face image as follows,
V. Sathya and T.Chakravarthy
http://www.iaeme.com/IJMET/index.asp 130 editor@iaeme.com
Figure 4 Different Divisions of Face Images
After segmenting the different regions, the features has been estimated using the
maximum and minimum ccorner information which is done as follows,
X(�, ", +) = /(�, ", YN+) − /(�, ", YZ+) (9)
Where X(�, ", +)the difference of the Gaussian image is, /(�, ", Y+) is the convolution
value of the image, (�, ") is the Gaussian blur value,
/(�, ", Y+) = �(�, ", �+) ∗ (�, ") (10)
According to the above process, the facial key point features have been detected using the
Taylor series as follows,
X(�) = X + [\][^ � + �
�_ [C\[^C � (11)
Then the orientation has been assigned as follows, which is used to identify the direction
of the particular key point is measured by the magnitude and orientation estimation.
�(�, ") = `a/(� + 1, ") − /(� − 1, ")b + a/(�, " + 1) − /(�, " − 1)b (12)
c(�, ") = � �!2a/(�, " + 1) − /(�, " − 1)b, a/(� + 1, ") − /(� − 1, ")b (13)
Where,
�(�, ") = ���!� *�� �d ℎ� ��" �����,
c(�, ") = ����! � ��! ℎ� ��" ���! �����
Based on the above process, the key point features are derived in different orientation
using the 4*4 histogram orientation process that consists of 16*16 region of the key point
which has 8 bins and 28 elements. The extracted elements are normalized with the help of the
threshold value 0.2. According to the histogram and threshold value, different features such as
nose, mouse, eyes and eye brows are derived from face images. The extracted features consist
of lot of information which is difficult to process, so, the optimized features are selected for
making the system so effective.
2.3 Feature Selection
The next step is feature selection which is done by using the particle swarm optimization
method. The PSO [12] method analyze extracted features and the best solution has been
detected which is relevant to the human emotions. In the search space, each feature treated as
the particle and the position, velocity of the feature is estimated because the features are
moved in the searching space while examining the optimal features. Based on the above
process, global features are selected from the feature space that helps to detect the facial
emotions with effective manner.
2.4 Feature Training and classification
The extracted features are trained by using the back propagation neural networks (BPNN)
because it effectively train the features that helps to detect the new facial expression related
Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques
http://www.iaeme.com/IJMET/index.asp 131 editor@iaeme.com
facial points. The BPNN network is one of the supervised learning method but in this work, it
treated as the unsupervised learning concept which computes the activation value of each
selected features. The networks has three layers namely, input, hidden and output layer each
layer has particular weights and bias value. During the training process the network use 70
input nodes,1 hidden nodes and 70 output node with mean square error function as the
training function. Initially the activation value of the layer has been computed as follows,
ef �)� ��! )�g*� = ∑ hN ∗ ,NZ (14)
In eqn (14), hN �� ℎ� �!�* )�g*� �d ℎ� !�*��!
,NZ �� ℎ� ,���ℎ �� )�g*� �d ℎ� !���
By using the activation value, the minimum activation is saved as the index pair and the
output value 1 is assigned to the maximum activation value else the output is assigned as 0.
Thus the output value of each neuron weighted value is estimated as follows,
,NZ (!�,) = ,NZ(�g�) + f i�N + ,NZ(�g�)j kZ (15)
In eqn (15), ,NZ (!�,) �� *��� �� ,���ℎ �� )�g*�
,NZ(�g�) �g� ,���ℎ �� )�g*�
These weight updating process helps to minimize the error value while train the features.
The trained features are stored as template in the database for further emotion recognition
process in the testing stage. These trained features are classified by using different classifiers
like Hidden Markov Model (HMM), Support Vector Machine (SVM) and Back Propagation
Neural Networks (BPNN)
Hidden Markov Model (HMM)
The first classifier is Hidden Markov Model (HMM), the method uses the tested features
which is driven from the image preprocessing, feature extraction and feature selection process
that is discussed in the section 2.1 to 2.4. The tested features are compared with the trained
features which is discussed in the section 2.5. This model works according to the statistical
Bayesian network approach which utilizes the probability value. Then the probability value of
each feature is computed as follows,
l(k) = ∑ l(k|h)l(h)^ (16)
In the above eqn (16), the P(Y) is the probability value of the testing feature sequence
which is compared with the trained features. Based on the comparison process the human
emotions are effectively recognized.
Support Vector Machine (SVM)
The tested features are classified using the support vector machine (SVM). This classifier
is statistical method which classifies the features using the hyper plane that reduces the mis-
classification data. Let x and y are the input and related output class, then the hyper plane has
been chosen to divide the output class labels y{1,-1} and the hyper plane is,
,. � + ( = 0 (17)
Then,
"N(,. � + () ≥ 1 ,ℎ��� � = 1,2,3, … . � (18)
The hyper plane should be separable the data and minimize the difference between data.
Then the difference between the hyper plane is calculated as follows,
�p + �I = ‖q‖ (19)
V. Sathya and T.Chakravarthy
http://www.iaeme.com/IJMET/index.asp 132 editor@iaeme.com
Based on the train data, the new entered features are matched with the template present in
the hyper plane as follows,
X�� �!f� = �r ∑ hN ⊕ kNrNW� (20)
Where hN is the given iris template and kN is the stored template in the database. Based on
the distance the templates are classified into the emotions.
Back Propagation Neural Networks (BPNN)
The last classifier is back propagation neural networks (BPNN) which is one of supervised
neural network. The network has three layers such as input, hidden and output layer. Each
layer uses the tested features and it has been passed to the hidden layers and the output is
estimated as follows,
�� �* �* = ∑ �N ∗ ,NrNW� + ( (21)
During the output estimation process, the network uses the radial basis activation function
that reduces the error rate while computing the facial emotions. In addition to this the weights
and bias values are continoulsy updated for minimizing the mis-classification data. Thus the
mentioned classification methods such as Hidden Markov Model (HMM), Support Vector
Machine (SVM) and Back Propagation Neural Networks (BPNN) successfully recognize the
facial emotions. Then the efficiency of the system is analyzed using the experimental results
3. EXPERIMENTAL RESULTS AND DISCUSSION
This section describes the performance evaluation of the methods described in the proposed
system. During the efficiency estimation process, the automatic expression system uses two
databases such as Jaffe and Cohn Kanade, the noise present in the images are eliminated and
different local binary features are extracted which helps to detect the feature points such as
eyes, eye brows, mouth and nose. Then the detected features are shown in the figure 5.
Figure 5 Face Expression Steps and Relevant outputs
Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques
http://www.iaeme.com/IJMET/index.asp 133 editor@iaeme.com
The assessment of these methods is done in terms of accuracy, specificity and the
sensitivity. These three assessment terms are specified in the following forms.
eff*��f" = (tl + t�) (tl + t� + <l + <�)⁄ ∗ 100%
���f�d�f� " = t� / (t� + <�) ∗ 100%
��!�� �)� " = tl / (tl + <�) ∗ 100%
Where,
TP (True Positives) = correctly classified positive cases, FP (False Positives) = incorrectly
classified negative cases, TN (True Negative) = correctly classified negative cases, FN (False
Negative) = incorrectly classified positive cases. The face expression has been recognized by
HMM, SVM and BPNN which reduces the reduces the error rate while classify the face
exprressions. The minimize error rate is increase the classification accuracy. Then the
performance of the proposed system error rate is shown in the figure 6.
Figure 6 Performance of Means Sqaure Error Rate
The reduced error rate leads to increases the overall efficiency of the system which is
examined by using different emotions such as happy, sad and anger from differnet regions and
the obtained results are shown in the following figure 7.
Figure 7 Accuracy of Different Classifiers
The above figure 7 shows that the three classifers successfully classifies the extracted
features with highest accuracy. Overall the obtained accuaracy value is shown in the figure 8.
V. Sathya and T.Chakravarthy
http://www.iaeme.com/IJMET/index.asp 134 editor@iaeme.com
Figure 8 Overall Accuracy of the classifier
From the above discussions the face expression has been classified using Hidden Markov
Model (HMM) with 98.3%, the Support vector Machine (SVM) classifies with 99.64% and
Back propogation Neural Networks ensures 99.87% when compared to other existing
methods. This makes the additional advantage to the proposed system and acts as a medical
image analysis device for the medical experts to classify the emotions witheffective manner.
4. CONCLUSION
This paper examining the effectiveness of the proposed back propagation neural networks
with hidden markov model based face emotion recognition process using the different face
database such as Jaffe and Cohn Kanade. thesis, facial expression related emotions
recognition system is done by using the Hidden Markov Model (HMM), Support Vector
Machine (SVM) and Back Propagation Neural Networks (BPNN). The captured face digital
images, geometric facial points are detected and the noise present in the images areeliminatd
by using the non-local median fitler. From the noise free image, facial features are extracted
by segmenting the images into the local binary patterns and the key points are detected in
differnet directions and locations using progression invariant sub space learning method.
From the extracted features, optimized global features are selected using the particle swarm
optimization method. The extracted features are trained by back propagation neural networks
and the classification is done by proposed classifiers. The performance of the proposed
system is analyzed by using the Jaffe and Cohn KanadeDataset which consumes the minimum
error rate. These reduced error rates increase the classification accuracy when compared to
previous work and shows that the proposed classification method brings the result with more
sensitivity and accuracy.
REFERENCE
[1] ShrutiBansal, Pravin Nagar, Emotion Recognition From FacialExpression Based On
Bezier Curve, International Journal of Advanced Information Technology, volume5,
number 3.
[2] SpirosV.Ioannou, .Amaryllis T.Raouzaiou, VasilisA.Tzouvaras, TheofilosP.Maili, .Kostas
C.Karpouzis, StefanosD.Kollias, Emotion recognition through facial expression analysis
based on a neurofuzzy network, Neural Networks, Volume 18, Issue 4, May 2005, Pages
423-435.
[3] Ekman, P., Friesen, W. V: Unmasking the Face: A Guide to Recognizing Emotions from
Facial Clues. Prentice-Hall, New Jersey (1975)
[4] ArunaChakrabortyAmitKonar, Fuzzy Models for Facial Expression-Based Emotion
Recognition and Control, Emotional Intelligence in elesviewerpp 133-173
Automatic Facial Expression Related Emotion Recognition Using Machine Learning Techniques
http://www.iaeme.com/IJMET/index.asp 135 editor@iaeme.com
[5] Ludmila I Kuncheva and William J Faithfull, PCA Feature Extraction for Change
Detection in Multidimensional Unlabelled Streaming Data, International Conference on
Pattern Recognition (ICPR 2012), November 11-15, 2012.
[6] Jia-FengYu,Yue-Dong Yang, Xiao Sun, and Ji-Hua Wang, Sequence and Structure
Analysis of Biological Molecules Based on Computational Methods, BioMed Research
International, Volume 2015, 2015.
[7] Tallapragada, Rajan, Improved kernel-based IRIS recognition system in the framework of
support vector machine and hidden markov model, IET Image Processsing in IEEE,
volume 6, 2012.
[8] Tallapragada, Rajan, Improved kernel-based IRIS recognition system in the framework of
support vector machine and hidden markov model, IET Image Processsing in IEEE,
volume 6, 2012.
[9] Hai Nguyen, KatrinFranke, and Slobodan Petrovic, Optimizing a class of feature selection
measures, Proceedings of the NIPS 2009 Workshop on Discrete Optimization in Machine
Learning: Submodularity, Sparsity &Polyhedra (DISCML), Vancouver, Canada,
December 2009.
[10] Hongjun Li, Ching Y. Suen, A novel Non-local means image denoising method based on
grey theory, JournalPattern Recognition in ACM, 2015.
[11] Ying-li Tian, Takeo Kanade, and Jeffrey F. Cohn, RecognizingAction Units for Facial
Expression Analysis, IEEETransactions on Pattern Analysis and Machine
Intelligence,23(2), 2001, 97-113.
[12] Tallapragada, Rajan, Improved kernel-based IRIS recognition system in the framework of
support vector machine and hidden markov model, IET Image Processsing in IEEE,
volume 6, 2012.