Automatic Emotion Recognizer for Low Functioning Persons...

International Journal of Science, Engineering and Technology Research (IJSETR), Volume 3, Issue 5, May 2014

1434

ISSN: 2278 – 7798 All Rights Reserved © 2014 IJSETR

Abstract— Autism spectrum disorders are a group of

developmental disabilities can cause significant social

communication and behavioral challenges over lifetime. The

theory of causation of autism and the other autism spectrum

disorder is incomplete but many number of persons suffering

from this disorder. People with autism have issues in non-verbal

communication particularly low functioning individual with

autism. They might be completely non-verbal and cognitively

impaired, lack of understanding and responding to non-verbal

cues and communication. Several augmentative systems were

developed for low functioning people but they act only as a

learning environment and often failed to operate in conjunction

with real world situations. This system must also require a

manual intervention and assessment reports of an individual

autistic person. So we proposed a new therapeutic system called

automatic emotion recognizer that recognizes Virtual Reality

based facial expressions there by finding different kinds of

emotions i.e. neutral, smile, anger, sad, surprise of an autistic

person.

Index Terms— autism spectrum disorder, facial expression, low

functioning people, virtual reality.

I. INTRODUCTION

Autism Spectrum Disorder is highly variable neuro

developmental disorder which is characterized by great difficulty in communicating with others, abnormal behavior

patterns and forming relationships. People with ASD have

deficits in three major areas: verbal and non-verbal

communication, social awareness and interaction,

imaginative play and cognitive inflexibility. Diagnosis of

autism is based on mainly behavioral assessment, parental

report and clinical report. In some cases, Diagnostician uses

tools such as Childhood Autism Rating Scale (CARS) and

Autism Diagnostic Observation Scale (ADOS) to measure

the severity range of an individual autistic person. . Most of

the autistic people face difficulties in communicating with

others and also lacking in understanding social cues and convention. They are unable to properly express non-verbal

communication and body language. Low Functioning People

with Autism might be completely non-verbal, cognitively

impaired; they does not communicate even with closest of the

family members .These inabilities hinder them from

understanding verbal and non-verbal communications, as

well as reading human facial expressions effectively. The

ability to identify and determine one’s emotions can serve as

an empowerment for the field of artificial intelligence and

gave rise to smarter, more powerful machines that

understands the intention of users. An intelligent machine with emotional awareness can achieve the shortcomings of

autistic people. With that emotional awareness, the machine

is capable of teaching and guiding autistic people on how to

respond appropriately when the person that he or she is

communicating with is expressing various emotions [1]. We

developed an automatic emotion recognizer for low

functioning persons with autism using Desktop VR tool. This

tool breaks the dependency of an autistic person as such it

does not need any manual intervention and also operate

flexibly in conjunction with real world scenarios.

II. EARLY INTERVENTON SYSTEM

An innovative VR-based facial emotional expression

presentation system was developed that allows monitoring of

eye gaze and physiological signals related to emotion

identification to explore new efficient therapeutic paradigms.

The eye tracking and physiological data were analyzed to

determine intergroup and intergroup variations of gaze and

physiological patterns. Performance data, physiological

signals and eye tracking indices indicated that there were

differences in the way adolescents with ASD process and recognize emotional faces compared to their typically

developing peers. However, VR systems applied in the

context of autism therapy focus on performance or explicit

user feedback as primary means of evaluation and thus lack

adaptability Traditional behavioral intervention is not

accessible to the vast majority of ASD population due to lack

of trained therapists as well as intervention costs and often

failed in identifying complex expressions as well as required

more prompts and more time to facial emotional expression

understanding tasks. It uses immersive head mounted

displays (HMD) as VR tool that rated as heavy and discomfort. The main problem found in our existing system

as, classification systems designed to output one emotion

label per input utterance may perform poorly if the

expressions cannot be well captured by a single emotional

label and multiple algorithm need for finding the

Human-emotion. Among the fundamental social impairments

in the ASD population are challenges in appropriately

recognizing and responding to facial expressions. Innovative

technology promises alternative paradigm in increasing

intervention accessibility that recognizes VR-based facial

expressions recognition system in a synchronous manner. Innovative technology promises alternative or assistive

therapeutic paradigms in increasing decreasing assessment

efforts, increasing intervention accessibility, reducing the

cost of treatment, promoting intervention and ultimately skill

generalization. We believe that such ability will provide

insight to the emotion recognition process of the children

with ASD and eventually help in designing new intervention

paradigms to address the emotion recognition vulnerabilities.

III. DIAGNOSING TOOL: ADOS

The Autism Diagnostic Observation Scale (ADOS) is a

standardized , semi-structured which is a combination of the

Automatic Emotion Recognizer for Low

Functioning Persons with Autism in Virtual

Reality Environment G.Aparna1, S.Srinivasan2

1Department of Electronics and Communication, P.B. College of Engineering, Chennai. 2Professor, Dept of Information and Technology, P.B. College of Engineering, Chennai.


1435


two earlier instruments called ICD-10 and DSM-IV

diagnostic criteria used by professional for diagnosing

autism. It assesses communication, social interaction and

imaginative use of materials for individuals who may have an

autism spectrum disorder. The ADOS consist of four

modules, each based on language and age

I- Preverbal / Single Words

II- Phase Speech

III- Fluent Speech, Child/Adolescent

IV- Fluent Speech, Adolescent/ Adult

Each module contains standard activities and materials that

are presented by examiners in order to elicit behaviors that

have been used for identifying and also diagnosis of an ASD

at that age and language level i.e. eye contact, conversation,

use of speech and language, unusual sensory interests and

shared enjoyment. The aim of the activities is to structure the interactions so that the child (or) adult being assessed is

sufficiently interacted those subjects want to participate

socially. In 2002 lord, rutter, dilavore & risi reported ADOS

manual, intraclasss correlations are as follows: test-retest

reliability from .59 to .73 and interrater reliability from .82 to

.93.cronbach’s alphas for internal consistency were

consistently highest for the communication-social interaction

total score from .91 to .94 and lowest for stereotyped

behaviors and restricted interests score (.63 to .65 for

modules 2 &1 and .47 to .56 for modules 4 and 3). Each

module can be administrated in 30-40 minutes and notes are taken by the examiner during its administration. The overall

ratings are completed immediately after the administration,

which are then used to formulate a diagnosis through the use

of the diagnostic algorithm provided for each module.

Scoring made at the end of the module is similar across

modules with some identical items, but others are relevant

only for the module being used. In this study, their results

indicated substantial interrater and test-retest reliability for

individual items, excellent interrater reliability within

domains and external internal consistency. We considered

only two autistic samples under the age group of 31 and 34

with moderate autism.

IV. SYSTEM DESIGN

A. Non-Verbal Communication

Low functioning individual with autism are completely

non-verbal; they express their feelings and share their own

emotions via facial expression, body postures, gestures and eye gaze. The purpose of this study was to conduct a

systematic examination of nonverbal communicative

behaviors using multiple perspectives (both trained observers

and instructors familiar with children's idiosyncratic

communication) and to explore instructors' responsively to

these subtle but important cues. More specifically, the

objectives in this study were as follows:

Fig .1 Autistic Persons with different kinds of Facial Expression

To compare the frequency of potentially communicative

behaviors’ displayed by a group of students with autism, as

identified by trained observers and familiar instructors; To

identify the communicative functions perceived by familiar

instructors; To determine the proportion of communicative

behaviors to which instructors responded and

determine/identify different kinds of facial expressions and

emotions of an individual who is suffered from ASD.Further,

a more elaborate approach to responsively coding, including a broader range of instructors' responses and sequential

analysis, would provide a richer understanding of social

interactions.

B. Experimental Setup

An experiment was performed in a laboratory with two

rooms separated by one-way glass windows for caregiver

observation. In the inner room, the subject sat in front of the

task computer. The caregivers sat in the outside room. A

therapist was present in the inner room to monitor the

process. The task computer monitor was also routed to the

outer room for caregiver observation during training session.

The session was video recorded for the whole duration of

participation.

C. Architectural Design

Initially an image is captured by web camera; face detection

is done using skin tone detection by finding skin-colored

pixels and regions in an image then converting the image into

the HSV color space and classifies to either skin or non-skin

those results in much stretched skin color cluster image. The

face clipped image takes as input for feature extraction that

reduces dimensionality as well as extracts both transient and intransient features i.e. eyes, eyebrows and mouth to perform

the desired task.


1436


Fig. 2 VR-based Automatic Emotion Recognizer System

In this module Local Binary Pattern (LBP) ) exploits

textual nature of human face as well as relationship between

component features via eyes, nose and mouth to detect face

patterns and reduce the dimension of the feature space.. After

applying a textual descriptor, facial features become darker.

For expression analysis, intransient features are selected for

the final emotion-specific feature set. This feature set was reduced using information gain on per emotion class basics

and permit as ranking of the features was implemented using

modified fuzzy C means clustering. Then filtering has been

applied to find the edges of intransient features with the help

Canny’s edge detector. Filtered feature set is analyzed and

classified to maintain trained dataset .Finally, VR-based

facial expressions are recognized there by emotions are

identified for accessing and controlling in the modern

environment.

V. SYSTEM IMPLEMENTATION

When user login has been done in face emotion detection

system, an image of autistic person is taken as input by

clicking browse button

Fig. 3 Human Detection

After that by clicking skin button , a skin detector typically

transforms image in to skin or non- skin pixel and then skin classifier is used to label the pixels for decision boundary to

detect whether it is an human or not.

Fig. 4 Skin Tone Analysis

After recognizing a given image is human then connected button is clicked to apply RGB color space which

is used to isolate the presence of arms, face and gestures as

well as eliminate the illumination condition to best extent.

Though it is a face emotion detection system, remove regions

that are unlikely to represent faces and then face acquisition

is done by examining common features on face.

Fig. 5 Applying RGB color space

Apply Local Binary Pattern by clicking binary button.

Face image can be viewed as texture pattern it provides powerful image representation and feature recognition for

further image processing

Fig.6 Feature Extraction

By clicking a face button, the end result of extraction

task is shown called feature vector that contains both

transient and intransient features.

Camera Skin

Tone Detection

Feature Extraction

Feature Selection

Feature

Filtering

Feature Set

Analysis

Face

Expression

Accessing &

Controlling Process


1437


Fig.7 Facial Feature Analysis

After feature extraction, selection helps to reduce the

feature vector by removing irrevalant, redundant and noisy

features. It selects only intransient features by clicking

eye-lip button..

Fig. 8 Feature Selection using Clustering Algorithm

Canny’s filtering is applied to detect edges of intransient

features .these edges are to be maintained in an edge pixel database for further processing.

Fig. 9 Feature filtering using Canny’s edge detector

After feature filtering, trained dataset has been created

for each autistic person according to 5 different kinds of

emotions. In the "Person” table, it stores the name of autistic

people and their index of 5 kinds of emotion. In the “Position” table, for each index, there are 6 control points for

lip curve, 5 control points for left eye curve, 6 control points

for right Bezier curve.

Fig.10 Trained Dataset

By clicking emotion button, Automatic emotion

recognizer system automatically detects frontal faces from

the captured image and codes them with respect to 5

dimensions in real time: neutral, anger, smile, sadness,

surprise. Emotions are analyzed from training dataset based

on classification technique called decision tree.

Fig. 11 Emotion Detection

VI. CONCLUSION

We have developed a VR-based automatic emotion

recognizer system that was able to collect facial features data

while the subjects were involved in an emotion recognition

tasks. Specifically, we developed controllable levels of facial

expressions of emotion based on long standing research

documenting certain universal expression patterns as well as

a desktop virtual reality presentation to avoid issues and sensitivities individuals with ASD.This system must be more

useful for low functioning individual with autism and

flexibly operate in real world scenarios. Mainly, it reduces

assessment efforts such as clinical reports, parental feedback,

psychological testing etc.

ACKNOWLEDGMENT

We thank “Aadhuraa Special School, Kanchipuram” for providing support for this work


1438


REFERENCES

[1] Emily mower, “ A framework for automatic human emotion

classification using emotion profiles”,IEEE transactions on audio, speech

and language processing, vol-19, pp 1057-1071, July 2011.

[2] H.C.Vijaylakshmi and S.patil kulkurni, “ face detection for skin tone

images using wavelet and texture features”, TECHNIA – international

journal of computing science and technologies , pp 646-650,jan 2011.

[3] Ihsan ullah,Muhammad hussain,ghulam Muhammad,hatim Aboalsamh ,

George Bebis , and Anwar M. Mirza , “ gender recognition from facial

images with local WLD descriptor” , national plans for science and

technology , pp 417-420 ,2012.

[4] Krishna Mohan Kediri, Abas said, and M.Yuntis, “emotion detection

using sub-image based features through human facial expressions”,

international conference on computer and information science, pp 332-335,

2012.

[5] Maringanti hima bindu, Priya gupta, and U.S.Tiwary, “cognitive model

based emotion recognition from facial expressions for live human computer

interaction”, proceedings of the IEEE symposium on computational in

image and signal processing, pp 351-356, 2007

[6] Ming-chuan hung and don-ling yang, “An efficient fuzzy c means

clustering algorithm” , information science , pp 225-232 ,2001

[7] Ryszard S. choras, “image feature extraction techniques and their

applications for CBIR and biometrics system”, international journal of

biology and biomedical engineering, pp 6-16, 2007

[8] Teik toe, shi-min, siu-yeung and yok-yen, “emotional advisor to help

children with autism in social communication”, intelligent decision

technologies, vol-3, pp 278-283, 2012.

[9] Vasantha.M, DR. V. subbiah bharathi, R.dhamodharan, “medical image

feature extraction, selection and classification”, international journal of

engineering science and technology, vol – 2(6), pp 2071-2076, 2010.

[10] Vineet saini, Rajnish garg, “A comparative analysis on edge detection

techniques used in image processing”, IOSR journal of electronics and

communication engineering (IOSRJECE), vol 1, pp 56-59, June 2012.

.

Automatic Emotion Recognizer for Low Functioning Persons...

Documents

Transcript of Automatic Emotion Recognizer for Low Functioning Persons...