Emotional Arousal Detection - RUG Math & CS: …aiellom/tesi/zuidhof.pdfan ambulatory real-time...

Emotional Arousal

Detection

Master's Thesis

November 2013

Student: Ing. H.J. Zuidhof

Primary supervisor: Prof.Dr.Ir. M.Aiello

Secondary supervisor: Dr. H. Riese

University of Groningen

Emotional Arousal Detection

Master’s thesis

Written for theUniversity of Groningen

Faculty of mathematics and natural scienceand UMCG for the department of psychiatry

Supervised and coordinated byProf. Dr. Ir. M. Aiello (University of Groningen),

Dr. H. Riese (University Medical Center Groningen)

By

Ing. H.J. Zuidhof

born the 11th of January 1988

Groningen, The Netherlands

A B S T R A C T

During the day everyone experiences all kinds of emotions. Knowinga person’s current emotion is very useful in medical and psychologyresearch. Detecting these emotions is not a trivial problem. Even thehuman brain has trouble detecting emotional states. Technology canhelp gaining more information about the emotional state of a humansubject. As the emotional state of a human subject changes an increasein heart rate can occur. Previous studies show that a combination ofheart rate and physical movement can be combined to form an indexfor additional heart rate caused by emotional arousal. In this study,an ambulatory real-time system is developed based on the additionalheart rate index to detect emotional arousal of a human subject. Thesystem is evaluated by exposing ten female participants to emotionalstimuli. Subjective rating methods performed on the data acquired bythese experiments show that the moments of additional heart rate areindeed caused by emotional arousal during laboratory experiments.Ambulatory experiments show a less significant distinction betweenadditional heart rate and emotional arousal.

Keywords: Additional heart rate, emotional arousal, emotions

iv

A C K N O W L E D G M E N T S

This report is the result of the master thesis project carried out by H.J.Zuidhof at Department of Psychiatry University Medical Center ofGroningen (UMCG) and is the final product of the master Comput-ing Science - Software Engineering & Distributed Systems at the Uni-versity of Groningen (RUG). I thank all people from the UMCG whohelped me achieve this result. Especially I would like to thank Harri-ette Riese for the coordination and motivation. I want to thank MarcoAiello from the University of Groningen for the help and feedbackduring the project. Without their help I would not have succeeded infinishing this thesis.

v

C O N T E N T S

1 introduction 1

1.1 Emotional arousal detection . . . . . . . . . . . . . . . . 1

1.2 Research hypothesis . . . . . . . . . . . . . . . . . . . . 2

1.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 background information and related work 5

2.1 Background information . . . . . . . . . . . . . . . . . . 5

2.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 Heart rate analysis . . . . . . . . . . . . . . . . . 8

2.2.2 Feedback . . . . . . . . . . . . . . . . . . . . . . . 14

3 hardware 21

3.1 Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.3 Future hardware . . . . . . . . . . . . . . . . . . . . . . 23

3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4 development 27

4.1 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1.1 QRS complex . . . . . . . . . . . . . . . . . . . . 27

4.1.2 R top triggering . . . . . . . . . . . . . . . . . . . 28

4.1.3 Filtering . . . . . . . . . . . . . . . . . . . . . . . 28

4.1.4 Emotional arousal detection . . . . . . . . . . . . 29

4.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.2.1 Class diagram . . . . . . . . . . . . . . . . . . . . 33

4.2.2 EmozionActivity . . . . . . . . . . . . . . . . . . 33

4.2.3 MeasurementService . . . . . . . . . . . . . . . . 33

4.2.4 Parcels . . . . . . . . . . . . . . . . . . . . . . . . 33

4.3 Questionnaires . . . . . . . . . . . . . . . . . . . . . . . . 34

5 evaluation of a real-time ambulatory system for

the detection of emotional arousal 37

5.1 Validity of additional heart rate as an indicator of emo-tional arousal . . . . . . . . . . . . . . . . . . . . . . . . 38

5.2 Laboratory and field study . . . . . . . . . . . . . . . . 39

5.3 Aims of the present study . . . . . . . . . . . . . . . . . 39

5.4 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.4.1 Participants . . . . . . . . . . . . . . . . . . . . . 40

5.5 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.5.1 Questionnaires . . . . . . . . . . . . . . . . . . . 42

5.6 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.6.1 Intake . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.6.2 Experiment . . . . . . . . . . . . . . . . . . . . . 44

5.6.3 Finish . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.7 Data assessment . . . . . . . . . . . . . . . . . . . . . . . 44

5.7.1 Subjective rating by testleader . . . . . . . . . . 45

vii

viii contents

5.7.2 Subjective rating by participants . . . . . . . . . 45

5.8 Statistical analysis . . . . . . . . . . . . . . . . . . . . . . 46

5.8.1 During the laboratory experiments I expect truefeedbacks at predefined emotionally intense mo-ments . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.8.2 During the laboratory experiments low acceler-ation and higher subjective affective scores andheart rate will be found at the true feedback mo-ments compared to random feedback moments 47

5.8.3 During ambulatory assessment the differencesin heart rate, acceleration and subjective affec-tive score between true and random feedbackmoments are less/not distinguishable . . . . . . 47

5.9 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.9.1 Descriptives . . . . . . . . . . . . . . . . . . . . . 47

6 discussion and conclusion 55

6.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 56

a emotional arousal algorithm (myrtek and foer-ster , 2001) 59

b sensors 61

L I S T O F F I G U R E S

Figure 1 Heart system . . . . . . . . . . . . . . . . . . . . 9

Figure 2 Heart rate variability . . . . . . . . . . . . . . . 12

Figure 3 ANI . . . . . . . . . . . . . . . . . . . . . . . . . 14

Figure 4 Example of an android implementation of aslidebar . . . . . . . . . . . . . . . . . . . . . . . 15

Figure 5 Simplified Geneva Emotion Wheel . . . . . . . 16

Figure 6 Affect grid . . . . . . . . . . . . . . . . . . . . . 17

Figure 7 Manikin . . . . . . . . . . . . . . . . . . . . . . . 18

Figure 8 Visualization of the system . . . . . . . . . . . . 23

Figure 9 Flexible hardware attached to the skin . . . . . 24

Figure 10 System during use . . . . . . . . . . . . . . . . 25

Figure 11 QRS Complex . . . . . . . . . . . . . . . . . . . 27

Figure 12 Bandpass filtering . . . . . . . . . . . . . . . . . 29

Figure 13 Disturbed ECG . . . . . . . . . . . . . . . . . . 29

Figure 14 Killing 1 . . . . . . . . . . . . . . . . . . . . . . 31

Figure 15 Killing 2 . . . . . . . . . . . . . . . . . . . . . . 32

Figure 16 Class Diagram . . . . . . . . . . . . . . . . . . . 34

Figure 17 Mood scale . . . . . . . . . . . . . . . . . . . . . 35

Figure 18 Illustration of position of the BioHarness dur-ing use . . . . . . . . . . . . . . . . . . . . . . . 41

Figure 19 Affect grid . . . . . . . . . . . . . . . . . . . . . 43

Figure 20 Trainspotting intensity . . . . . . . . . . . . . . 45

Figure 21 Questionnaire mood . . . . . . . . . . . . . . . 46

Figure 22 Trainspotting emotion intensity . . . . . . . . . 52

L I S T O F TA B L E S

Table 1 Physiological symptoms and motor expressions 7

Table 2 Sensor Summary . . . . . . . . . . . . . . . . . 22

Table 3 Information regarding the participants of theexperiment . . . . . . . . . . . . . . . . . . . . . 41

Table 4 Specifications of the HTC One S . . . . . . . . . 42

Table 5 Specification of the Huawei Ascend Y300 . . . 42

Table 6 participant data . . . . . . . . . . . . . . . . . . 49

Table 7 Feedback data . . . . . . . . . . . . . . . . . . . 53

ix

1I N T R O D U C T I O N

Computers and mobile devices have become everyday products whichare carried around by a vast majority of people. Where in the earlydays computers were mainly focused on just brute force operations.Nowadays computers have become part of our daily life. For exam-ple while traveling or just to keep up to date with everything that’snew. Besides their use in comfort they can help to learn more aboutourselves, they can assist us in making decisions in certain situations.For the computer to make decisions it needs information, this infor-mation can be provided by numerous amounts of sensors. Improve-ments in technology causes sensors to keep getting smaller and moreadvanced. This allows whole systems to become smaller, more com-fortable and smarter by combining data provided by different sensors.During this project research is done in the field of medical and techni-cal practices. By combining these fields the aim is to find a solution formedical and psychology researchers to help in the emotional arousaldetection of a person. Emotional arousal can be detected in many dif-ferent ways, as reviewed in the first part of this thesis. This chaptermainly focusses on introducing the problems that needs to be solvedduring this study and to give a global outline of this thesis.

1.1 emotional arousal detection

Detecting emotional arousal of a human subject is not a trivial prob-lem. Emotions are influenced by many external stimuli like other per-sons or stress and happens everywhere: at home, at school/work orwhen traveling from one location to the other. Knowing these statesat any moment of the day can help researchers with understandinghuman subjects or to assist doctors and psychiatrists to help theirpatients. Numerous studies [30, 31, 25, 38, 11, 15, 5, 8, 42] tried todescribe, analyze and evaluate various aspects of affective states andsome proposed technological solutions for detecting the emotionalarousal of a human subject. Instruments used to measure emotionsdepend on the emotional theory involved and include, among others,self-reports of feelings, methods of behavioral rating, physiologicalparameters, and examination of facial and vocal expressions [38, 42].This study focusses on the detection of emotional arousal by analyz-ing heart rate and physical movement and self-reports of feelings.The main goal of this project is to develop a system with the abilityto detect the amount of emotional arousal of a person.

1

2 introduction

emotional arousal Every human subject experiences emotions,where one emotion can have more impact on your feelings then theother, this impact is the emotional arousal. Before detecting the emo-tional arousal of a person, a definition of emotions is made to helpunderstand what this project is about. This is not as easy as it mayseem. The dictionary states a large number of definitions for emotion,for example:

1. the affective aspect of consciousness

2. a state of feeling

3. a conscious mental reaction (as anger or fear) subjectively ex-perienced as strong feeling usually directed toward a specificobject and typically accompanied by physiological and behav-ioral changes in the body.

These definitions still leave things open for interpretation and isnot strict enough for this study. Previous studies have tried to comeup with answers for this problem. As Scherer et al (2005) put it, thequestion ”What is an emotion?” rarely returns the same answer fromdifferent individuals, scientists or non-professional [39]. Numerousscientists proposed definitions but they do not seem to generate onefinal answer. Kleinginna et al (1981) listed 92 of the proposed scientificdefinitions and merged them to one final definition which is used inthis paper: [18]

Emotion is a complex set of interactions among subjective and ob-jective factors, mediated by neural/hormonal systems, which can

1. give rise to affective experiences such as feelings of arousal,pleasure/displeasure;

2. generate cognitive processes such as emotionally relevant per-ceptual effects, appraisals, labeling processes;

3. activate widespread physiological adjustments to the arousingconditions; and

4. lead to behavior that is often, but not always, expressive, goal-directed, and adaptive

These changes in physiological properties, stated in point 3, can beused to measure emotional arousal. Examples of measurable proper-ties are breathing rate, speech recognition or heart rate. In chapter 2

is elaborated on the properties and the techniques to analyze them.

1.2 research hypothesis

Although numerous systems are already available to detect emotionalarousal [31, 38, 15, 5, 42], most systems require carrying around mul-tiple pieces of hardware or hardware interrupting with normal daily

1.3 contribution 3

activities. For this study I hypothesize that, based on previous stud-ies, an ambulatory real-time analyzing tool can be developed that isable to detect the increase of emotional arousal of a human subjectby analyzing changes in heart rate. The goal is to achieve the sameoutcome as previous studies but with more comfortable and ambu-latory hardware. To test this hypothesis a system is develop and anexperiment is set up where human subjects are exposed to a video toinduce emotional arousal, followed by ambulatory monitoring in ev-eryday life. At moments of emotional arousal detection feedback fromthe human subject is needed for evaluation and future purposes. Inthis document the current state of the art and the steps and methodsused to develop a system are discussed.

1.3 contribution

With this system researchers can monitor and evaluate patients byretrieving the emotional arousal throughout the day. With the tech-nology of today a system can be developed which is much more com-fortable to use for longer periods of time. Data from patients canbe much easier extracted from the hardware by simply connecting asmartphone to a computer.

outline

chapter 2 : In this chapter the current state of the art of physio-logical emotional arousal detection is reviewed and discussed.Research in both medical and technological studies and theirdifferent techniques and methods are evaluated.

chapter 3 : Background information of the heart itself is providedin this chapter. After this brief introduction to the heart the mainrequirements for the system are discussed.

chapter 4 : In this chapter the architecture of the system developedduring this study is discussed. All parts of the system are de-scribed and put together into one overview.

chapter 5 : During the implementation and testing of the systemdifferent challenges were encountered. In this chapter these chal-lenges are discussed and elaborates on the solutions and choicesmade are given.

chapter 6 : Validation of the system is done in three experiments.The set up and evaluation of these experiments is described inthis chapter. For the final experiment ten women participatedby watching a movie and by wearing the system in daily lifesituations.

4 introduction

chapter 7 : A look back is taken on the work conducted during thisresearch project and draw conclusions of the evaluation andfindings of the whole project. Whether or not the system and/or method is feasible for detecting emotional arousal. Finallyin this chapter suggestions for future research directions andfurther implementation of the system are discussed.

2B A C K G R O U N D I N F O R M AT I O N A N D R E L AT E DW O R K

Before diving into the development of the system an overview of thecurrent literature and methods for the detection of emotional arousalis provided. Besides heart rate there are other bodily symptoms thatcan be measured for the detection of emotional arousal. This chapterdiscusses the heart rate to be the best physiological symptom to de-tect the changes in emotional states in ambulatory situations. Startingwith discussing background information about the bodily symptomswhich may change during emotional arousal. Previous work is dis-cussed in emotional arousal detection by physiological properties af-ter which will be elaborated on the work performed in heart rate anal-ysis. Besides the physiological and technological methods and tech-niques research is done in subjective emotional scoring. On the eventof detection of emotional arousal, feedback needs to be acquired fromthe human subject. Section 2.2.2 elaborates on these methods.

2.1 background information

Scherer et al (2005) proposed the distinguishment between emotionsand other affective phenomena such as feelings, moods or attitudes.In his study he discussed ways of measuring emotions. Because emo-tions have such a high impact on physiological properties the per-fect solution for detecting emotional arousal is to analyze all of theseproperties. In Scherer’s words; ’in an ideal world of science, we wouldneed to measure (1) the continuous changes in appraisal processes atall levels of central nervous system processing, (2) the response pat-terns generated in the neuroendocrine, autonomic, and somatic ner-vous systems, (3) the motivational changes produced by the appraisalresults, in particular action tendencies, (4) the patterns of facial andvocal expression as well as body movements, and (5) the nature of thesubjectively experienced feeling state that reflects all of these compo-nent changes [39]. Such measurements have never been performedtogether and is unlikely to become standard procedure in the nearfuture. However, there have been major advances in recent years withrespect to measuring individual components. In this chapter the dif-ferent individual components which can be measured during emo-tional arousal are investigated. K.R.Scherer set up the list providedin table 1 which includes the physiological symptoms and motor ex-pressions which can change or occur during an emotion. Using thisinformation a system can be developed which analyzes one or more

5

6 background information and related work

of these properties to make a suggestion of the emotional arousal ofa human object. Chapter 2.2.1 discusses previous studies where sucha system is build and elaborates on the choses made for the systembuild during my research project.

2.2 related work

The research for this work is focused on the crossroad of medical andtechnological studies. Previous studies show that emotional arousaldetection by analyzing physiological properties or motor expressionsis feasible. E. Leon et al (year) [11] discussed and developed a speechrecognition implementation to detect the emotional arousal of a hu-man subject by classifying different features of the voice like pitchcontour, intonation and speech style. Other motor expressive analyz-ing is performed by Sun et al [42]. By detecting facial expressionsof a human subject an estimation of the emotional state of the per-son is possible. Studies about the correlation of physiological prop-erties and emotional arousal are just as popular, if not more popular.Brain wave analyzes is one example. Technology has come very farin using the electroencephalogram (EEG) to assist people. Analysisof brain waves are mainly used for humans who are physically dis-abled by, such as paralysis, brain stroke and other kind of brain disor-ders to communicate with the real world application like controllingof house hold equipment, playing games, making or receiving callsfrom phone, and using the computer [5]. A.S. AlMejrad et al (2010)discussed the use of EEG signals for use in emotion recognition. An-other possibility of emotional arousal detection is with the use of biogases emanated from the human body: e.g., breath, oral cavity, skingas and body odor. Breath gas is easy to collect in bio gases and itincludes more than 200 kinds of biochemical feature. Breath gas in-formation can be considered to have a relationship with mental statessuch as emotions and stress, since feeling emotions affect the auto-nomic nervous system and the autonomic nervous system controlsphysiological conditions [21]. Besides facial expression, brain waveanalyzes and breathing (gas) analyzes the heart has shown to be agood indicator of emotional arousal. Previous studies show systemsdetecting the changes in heart rate of a human subject and couplethem to emotional arousal [30, 31, 25, 40, 15]. The studies have shownto successively detect emotional arousal. For this study the chose ismade to continue with this method, but with new hardware and tech-niques to make a system which is comfortable and easy to use indaily life.

cardiovascular system Before diving into previous studieswe take a step back to elaborate on the heart itself, what it is and whyit is good for the detection of emotional arousal. The primary func-

2.2 related work 7

physiological symptoms motor expressions

Feeling cold shivers (neck,chest) Smiling

Weak limbs Mouth opening

Getting pale Mouth closing

Lump in throat Mouth tensing

Stomach troubles Frown

Heart beat slowing down Eyes closing

Heart beat getting faster Eyes opening

Muscles relaxing, restful (whole body) Tears

Muscles tensing, trembling (whole body) Other changes in face

Breathing slowing down Voice volume increasing

Breathing getting faster Voice volume decreasing

Feeling warm, pleasant (whole body) Voice trembling

Perspiring, moist hands Voice being assertive

Sweating (whole body) Other changes in voice

Feeling hot, puff of heat (cheeks, chest) Abrupt bodily movements

Blushing Moving towards people or things

Sweating Withdrawing from people or things

Moving against people or things

Other changes in gesture

Silence

Short utterance

Long utterance

Speech melody change

Speech disturbance

Speech tempo changes

Table 1: Physiological symptoms and motor expressions [39].


tion of the heart, a muscular pump, is to supply the energy requiredfor the circulation of blood in the cardiovascular system. The cardio-vascular system (figure 1) is comprised of the heart and vasculature.The vasculature serves as the distribution, exchange and collectionchannels by which the blood, pumped by the heart, travels to everyliving tissues in the body and is subsequently returned to the heart[17].

The heart is comprised of four chambers that function as two pumpsin series. The right-side pump (right atrium and right ventricle) pro-vides the energy necessary to move blood through the pulmonary cir-culation. The left-side pump (left atrium and left ventricle) providesthe flow through the systemic circulation. As a result of a commonpacemaker and the nature of cardiac muscle fibers [17], both atriacontract rhythmically and in synchrony, thereby causing these cham-bers to periodically empty their contents and to fill the ventricularchambers. A fraction of a second later the blood-filled ventricles alsocontract in synchrony into the vasculature. This process is one heart-beat. This beating of the heart is an electromechanical event. Electricalimpulses generated by specialized cells within the heart, referred toas pacemaker cells, initiate the mechanical contraction of heart mus-cle [24, 17]. Using these techniques an electrocardiogram (ECG) canbe extracted tracking the muscle contractions of the human heart.

2.2.1 Heart rate analysis

Emotional arousal detection by heart rate analyzes can be executedwith multiple methods. The most popular methods are additionalheart rate (AHR) and heart rate variability (HRV) analysis. Additionalheart rate is a method of detecting increases in heart rate that are notcorrelated to changes in physical activity. This method was proposedby Myrtek in 1996, in this chapter this method is introduced anddiscussed.

2.2.1.1 Myrtek

In 1996 Myrtek et al [30] proposed an AHR algorithm to detect changesin emotional arousal. In 2005 Myrtek published a new study with animproved algorithm and new research material [31]. The complete al-gorithm is available in appendix A. This study and algorithm is thebases of numerous subsequent studies performed by other scientists,discussed later on in this chapter. Myrtek developed a system to an-alyze human subject’s heart rate changes and their physical activityin ambulatory settings [31]. Emotional arousal is suspected by link-ing increases in heart rate to the physical activity of a human subject.When an increases in heart rate is detected without an increase inphysical movement a feedback signal is given to the user to fill outan (external) questionnaire.

2.2 related work 9

Figure 1: Human ciculatory system. Top figure presents detailed schematicof circulation while bottom figure highlights functional divisionsof circulatory system [17].

algorithm The AHR signal is a continuous estimation of arousalfrom heart rate and physical activity. The default process of addi-tional heart rate computation is explained below. (See appendix A fordetails of the algorithm) For the current emotional arousal state the


current time window is compared to the mean of the previous 3 timewindows. The length of the time window is 60 seconds on default,more detailed information about the time window and the length isfound in chapter 4.

HRC[i] = HR[i] − ((HR[i− 3] +HR[i− 2] +HR[i− 1])/3) (1)

HRC is the current change in heart rate compared to the previous3 time windows (1). When this value reaches HRPLUS (threshold, 3

heart beats on default), an AHR signal is generated. HRPLUS is in-creased by adding the mean acceleration of the current time window.For additional heart rate a score is formed defined as the factor, bywhich the current heart rate change (HRC) exceeds the minimal heartrate change (HRPLUS)

AHR = HRC/HRPLUS (2)

Requirements for a feedback moment (additional heart rate de-tected) are as follows:

1. HRC >= HRPLUS

2. AHR >= 1

3. Acceleration must not exceed the mean of the previous 3 timewindows of acceleration

4. The current heart rate must be higher than its predecessor

random feedback For reasons of comparison, but also to avoidpossible conditioning, random feedbacks are given with no indicationof an emotional event after three event-related feedbacks. In chapter 5

the random and true feedbacks are compared to evaluate the softwareand help prove the hypothesis discussed in chapter 1. Random feed-backs are given randomly after 3 true feedbacks have been generated.For every generated true feedback the change of a random feedbackincreases. The change of random feedback is reset after a randomfeedback is generated. If in the minute for random feedback the crite-ria for a true feedback are fulfilled, then the next following minute isused for random feedback. If the ratio between event-related and ran-dom feedbacks is smaller than 5:2, then no further random feedbacksare given with the exception that within the interval of 20 minutes noevent-related feedback is possible. In this case a random feedback isgiven in minute 20 [31].

2.2 related work 11

myrtek continued Kusserow [27] is one of the researchers con-tinuing Myrtek’s study to do stress arousal detection. In this studymultiple scenarios are analyzed; monitoring a public speaker, mon-itoring a musician on stage, monitoring an Olympic champion andmonitoring in free-living daily activities. Two methods were usedto measure the emotional states of the participants, AHR and HRV.The focus in this chapter is on the AHR method, in the next chap-ter the HRV method is discussed. The experiments during free-livingdaily activities were performed with the AHR method. In his studyKusserow improved the algorithm proposed by Myrtek. Myrtek’s al-gorithm does not address the physiological changes caused by bodymotion transition, e.g., transitions from sitting to standing. Kusserowaddressed this limitation by using multiple accelerometers to indicatethe changes in body transitions. A four participant’s data set of 182

h during natural daily activities was used to test the stress arousaldetection and estimation procedure. The wearable system accordingto the AHR algorithm sensor setup comprised two 3-axis wirelessaccelerometers (BodyANT) placed at the chest and right thigh, anda wireless heart rate monitor chest belt. Data samples were storedby the Q-belt integrated computer. To facilitate interpretation of esti-mated stress arousal phases, participants kept an activity diary andcompleted mood state questionnaires throughout the recordings.

2.2.1.2 Heart rate variability

Besides analyzing increases in heart rate as used with the AHR method,the changes between every heartbeat can be used to make an indica-tion of the state of a human subject. The studies discussed in thischapter use this concept for emotional state and stress detection. Thecentral nervous system (CNS) is involved in the pathways linkingemotions to ANS reactivity. The activation of the autonomic nervoussystem (ANS) can be evaluated by HRV and can therefore help inmeasuring the emotional arousal of a human subject. The changes be-tween heartbeats are computed by taking the R-R intervals. An R-Rinterval is the time between two subsequent heartbeats, figure 2 illus-trates an ECG where the heartbeats are marked. HRV is computed bysumming the difference in time between two subsequent R-R inter-vals.

An example HRV index is the RMSSD; Root Mean Square of theSuccessive Differences. Higher levels of HRV indicate appropriatefluctuations in ANS activation. The higher the HRV, the more capablethe heart is of adapting to stressors, such as increases in physical ormental activity, and reductions in activity. Lower HRV may indicatea decreased ability for the heart to compensate or react appropriatelyduring stressful or otherwise demanding situations [23]. Appelhanset al (2006) reviewed the use of HRV as an index of emotional re-sponding. Their research and other theory support the use of HRV


Figure 2: Heart rate variability. R-tops (heart beats) are marked in red, thetime between two R-tops is represented by the numbers in between(in milliseconds).

as an objective index of emotional responses [8]. Examples of otherstudies and their methods and findings are discussed in this chapter.

kusserow In the same study Kusserow discussed the AHR methodhe worked on stress arousal detection by analyzing HRV. Experimentsare performed with a public speaker, an activity with little physi-cal movement. Kusserow investigated seven features of heart activ-ity from time- and frequency-domain that were reported to indicatebody stress in laboratory investigations. The investigations showedthat time-domain features, in particular heart period, can provide ro-bust information for the talk situation.

analgesia nociception index De Jonckheere et al (2012) hy-pothesized that the Analgesia nociception Index (ANI), an index basedon HRV analysis, can be used as a tool to investigate the processesof emotional regulation of a human subject. The ECG signal is ac-quired using classical electrodes. The algorithm used for this studyis explained below. They concluded that the ANI is indeed capableof measuring changes in emotional state [15]. Tests were performedby inducing a negative emotional stimulus through the projection ofan 80 seconds video sample of the movie ’American History X’. Thissample has been characterized by experts as an outlier in negativeemotions [6].

ANI is computed from the ECG signal as described below (summa-rized). The ECG signal is acquired using classical electrodes and dig-itized at a sampling rate of 250 Hz. The RR series is then built as thetime evolution of the time intervals between two successive R wavesof the ECG (RR value). R tops are detected using a R wave detectionalgorithm [35]. However, RR series is usually disturbed by differentkinds of interferences (ectopic beats, electrode motion...) involving anerroneous analysis of HRV. Therefore, RR intervals series is filteredin real time using an original non linear filtering algorithm [36]. Thisfiltering algorithm, which is based on a morphological analysis of the

2.2 related work 13

RR series, is able to detect each disturbed area and to replace theerroneous samples with the most probable ones.

Filtered RR series are then re-sampled at 8 Hz using a linear inter-polation algorithm. RRi samples are isolated into a 64 seconds movingwindow for analysis and the series is mean centered and normalizedfor inter patients comparability.

In a first step, the mean value (M) is computed as:

M =1

N

N∑i=1

(RRi); (3)

Where RRi represents the RR samples values and N the number ofsamples in the window. Then M is subtracted from each sample of thewindow as: RRi = (RRi - M). The norm values (S) is then computedas:

S =

√√√√ N∑i=1

(RRi)2; (4)

and each RRi is divided by S: RR’i = RRi / S.Since the method is based on the analysis of HF changes, the RR’i

series is then band pass filtered between [0.15-0.5 Hz], which leads toRR’HF [37].

HF changes are mainly modulated by respiration. Indeed, breath-ing influences the way the ANS regulates heart rate. Inhalation tem-porarily inhibits the influence of the parasympathetic nervous systemand increases heart rate, while exhalation stimulates the parasympa-thetic nervous system and decreases heart rate. These rhythmic oscil-lations, which are caused by breathing, are called respiratory sinusarrhythmia (RSA).

We observed that, in case of well-being, RSA modulation on HRis important and causes large magnitude on the RR’HF series (figure3, upper panel), while in case of pain, stress or anxiety, the influenceof each respiratory cycle is more chaotic and the series magnitudedecreases (figure 3, lower panel).

In order to transform this qualitative observation into quantitativeinformation, we designed a graphical index by computing the areaunder the RR’HF series curve as shown in figure 3. Local minima andmaxima are detected and the upper and lower envelopes are plottedby connecting the local maxima together and the local minima aswell. This envelope estimation allows to obtain an index which notdepends on respiratory frequency changes [26].

In order to improve the time sensitivity of the method, we thendivide the 64 second moving window into four subwindows of 16

second. The areas between the lower and upper envelopes are then


measured in the four sub-windows. We defined AUCmin as the small-est of these sub-areas.

ANI is then computed in order to express a fraction of the totalwindow surface, leading to a measure between 0 and 100:

ANI = 100 ∗ [a ∗AUCmin+ b]/12.8; (5)

Where a = 5.1 and b= 1.2 have been empirically determined in ageneral anesthesia data set of more than 100 patients in order to keepthe coherence between the visual effect of respiratory influence on RRseries and the quantitative measurement of ANI.

Figure 3: normalised, mean centered and band pass filtered RR series in twodifferent states of analgesia / nocicetion during general anaesthe-sia with controlled ventilation: adequate analgesia (upper window)and in the case of surgical stimulus, few minutes before haemody-namic reactivity (lower window).

2.2.1.3 Discussion

Studies show that both methods, HRV and AHR, can be used foremotional arousal detection. Previous studies showed succesive re-sults with both methods. The AHR method takes physical activityinto account. Therefor this method is chosen for the system devel-oped during this research study.

2.2.2 Feedback

’Emotions are what people say they are’ [34]. Although different stud-ies doubt this phrase, for our study it is used to get subjective feed-back from the human subjects. At the moment of emotional arousaldetection, feedback is needed for subjective evaluation of the system.Numerous methods of feedback extraction are proposed in previousstudies [22, 41, 16]. In this chapter these methods are introduced anddiscussed.

2.2 related work 15

2.2.2.1 Scale

One of the simplest solutions is to ask for an emotion and let the userchoose a value on a certain scale. Figure 9 shows an example of theimplementation on an android phone. Although simple, this does notprovide flexible measures. More flexible methods are discussed in thesections below.

Figure 4: Example of an android implementation of a slidebar. This partic-ular example shows a scale from positive to negative where themiddle is a neutral state.

2.2.2.2 Geneva emotion wheel

The Geneva Emotion Wheel (GEW) is an example of a self-report mea-sure. The design of the GEW has elements of a free response format,a discrete emotion response format, and a dimensional approach toemotions. The GEW consists of discrete emotion terms correspondingto emotion families that are systematically aligned in a circle. Under-lying the alignment of the emotion terms are the two dimensionsvalence (negative to positive) and control (low to high), separatingthe emotions in four quadrants: Negative/low control, negative/highcontrol, positive/low control, and positive/high control. An exampleis visualized in figure 5.


Figure 5: Geneva Emotion Wheel [41], divided in two axes: control and va-lence. Users select one place of the wheel of which they think re-flects their current emotional state the best.

2.2.2.3 Affect grid

The affect grid (Figure 6) is designed as a quick means of assess-ing affect along the dimensions of pleasure-displeasure and arousal-sleepiness. Previous work showed that the scale is suitable for anystudy that requires judgments about affect of either a descriptive or asubjective kind [16].

Before a participant fills out this grid the testleader explains whatthe different boxes mean by explaining the means of the X and Yaxes. The description below is an example of how this affect grid canbe explained.

2.2 related work 17

Figure 6: Affect grid [16]

’Let’s practice the grid by describing how you were feeling this morningwhen you woke up. There are basically two axes to think about. For thetime being, you can ignore the words in the corners of the grid. We’ll talkabout these two main directions first. The horizontal direction ranges fromunpleasant feelings on the left to pleasant feelings on the right. So the moreunpleasant or pleasant you were feeling this morning, the more to the left orright you’d mark your answer. If you felt more neutral, you would place amark somewhere near the center of the grid. Once you’ve decided how far tothe left or right your mark needs to be, you need to decide how high or lowto put it. The vertical direction of the grid ranges from high arousal at thetop of the grid to sleepiness at the bottom. So now place your mark higheror lower on the grid depending on how aroused or alert / energetic, versussleepy you were. Please note that people’s emotions can be described by anycombination of these four basic states. Now look at the words in the cornersof the grid. These adjectives describe feelings that fall into that section of thegrid. For example, you can see that feeling very pleasant and highly arousedmay be called excitement. This would probably occur for example if you’dwin a large sum of money in the lottery. However, there are other examplesof pleasant experiences that don’t necessarily involve high levels of arousal.Pleasant feelings may also be associated with low arousal or sleepiness. Thinkfor example of taking a long bath or receiving a massage. In the bottom right-hand corner of the grid, you can see that pleasant sleepy feelings may bedescribed as a state of relaxation. Let’s look at the left side of the grid now. Thetop left-hand corner indicates that unpleasant emotions coupled with higharousal may be described as stress. The bottom left-hand corner describesa state of unpleasant feelings and low arousal. This may be described asdepression. Finally, if you feel very neutral you’d describe this with a markplaced in the center of the grid. So for this grid, we ask you to place an X


in just one of the boxes to describe how you were feeling. Try thinking firstabout how pleasant versus unpleasant you felt, and select the column whichbest describes this. Then think about how aroused versus sleepy you felt, andmove either up or down the column accordingly. Finally, take a look at thedescriptive word nearest the square you have chosen, and see if your feelingswere closer to that word than to any of the other descriptive words. Give ita try, and let me know if you have any questions. Let me know when you’refinished. I’d like to see your response and then I’ll try to describe the emotionbased on your answer, just to check if I’ve explained the grid well enough.’

2.2.2.4 Manikin

The Self-Assessment Manikin (SAM) is a non-verbal pictorial assess-ment technique that directly measures the pleasure, arousal, and dom-inance associated with a person’s affective reaction to a wide vari-ety of stimuli. SAM ranges from a smiling, happy figure to a frown-ing, unhappy figure when representing the pleasure dimension, andranges from an excited, wide-eyed figure to a relaxed, sleepy fig-ure for the arousal dimension. The dominance dimension representschanges in control with changes in the size of SAM: a large figureindicates maximum control in the situation. In this version of SAM,the subject can place an ’x’ over any of the five figures in each scale,or between any two figures, which results in a 9-point rating scale foreach dimension [22].

Figure 7: The Self-Assessment Manikin (SAM) used to rate the affective di-mensions of valence (top panel), arousal (middle panel), and dom-inance (bottom panel) [22]

2.2 related work 19

2.2.2.5 Discussion

Means of retrieving affective states can be performed in several ways,this chapter discussed a few of them. Depending on the experiments/hy-potheses a chose needs to be made for the one fitting the requiredresults. In later chapters the choices for different questionnaires areexplained.

3H A R D WA R E

A device for the extraction of the heart rate is needed which does notonly send average heart rates every second, but a device that sendsthe complete ECG of a person. Having access to the ECG afterwardsallows researchers to analyze the data more closely when needed. An-other reason to analyze the ECG instead of using the average heartrate provided by the device is to have more control over the situa-tion. Future implementations may use different sensors which mayprovide different heart rate data. An accelerometer is used to mea-sure the physical movement. The processing of both the ECG andthe acceleration data is done on an external processing unit. Becausethe system will be used in ambulatory situations/in everyday life,the processing is done on a smartphone, this allows the users to usetheir own phone instead of carrying extra hardware with them. Thesoftware for this study is developed for Android phones, future im-plementations can be developed for iPhones or Windows phones.

3.1 sensor

In this section the various heartbeat sensors found are discussed,each with their pros and cons. Note that this section contains sen-sors specifically designed to work on mobile (Android) phones viaBluetooth, as well as other, less specific sensors. For this researchproject a system must be developed to give a comfortable feeling tothe people wearing it. Therefor sensors are required which do not in-terfere with daily life activities of a human subject. Although none ofthe suggested sensors are eliminated because of the need of comfort,some provide more comfort then the other. Another requirement isthe availability of wireless communication to other devices for real-time analyzing of the gathered data. Table 2 lists possible candidatesensors and their details.

21

22 hardware

de

vic

ec

om

fo

rt

(1-1

0)

wir

el

es

sa

nd

ro

id

sd

ke

cg

da

ta

nu

mb

er

of

el

ec

tr

od

es

ac

ce

le

ro

me

te

r

Pola

rW

earl

ink

6X

X2

Zep

hyr

HxM

BT6

X2

X

Chr

onos

EZ430

/EZ

430-R

F2560

5X

X

Smar

tHR

M6

X

Zep

hyr

BioH

arne

ssBT

6X

XX

2X

VU

AM

S4

X5

Smar

tshi

rt7

Tabl

e2

:Lis

tan

dde

tails

ofca

ndid

ate

sens

ors

3.2 architecture 23

3.2 architecture

Architecture can be defined as the framework supporting each andevery component used in, but also used by the appliance. The archi-tecture is therefore a vital part of every software and hardware project.One of the main aspects of this research is ambulatory monitoring inreal-life situations. This means that the research is not performed onstationary, but on people freely moving around. To enhance this in theresearch our idea is to fit the test subjects with their own ’computer’to perform measurements. This computer found its implementationas an Android smartphone. This device connects to the monitoringsensor and processes the data. For this project all data is stored onthe Android device and the heart rate monitor itself. Figure 8 visu-alizes the architecture of the system. More detailed information isavailable in appendix B

Figure 8: Visualization of the system.

3.3 future hardware

As mentioned in the introduction of this document sensors are be-coming better and smaller every year. This means that future imple-mentations of the system proposed during this study can be easilyimproved. During the current study, after the decision for the BioHar-ness was made, several other devices came to our attention. Some of

24 hardware

these devices fit the profile for acceptable sensors that can be usedfor this project. Some of these devices have advantages compared tothe BioHarness. Future studies can take these devices into accountfor their own research projects.

vuams At the moment of writing this document the sensor doesnot include all the features needed to be integrated into our system,namely a Bluetooth connection. Contact with the Vrije Universiteitvan Amsterdam taught us that they are currently working on inte-grating a Bluetooth connection. The VUAMS uses a larger numberof electrodes to extract the ECG and is therefore much more reliableduring physical activity compared to the BioHarness which only usestwo electrodes.

equivital eq02 This sensor seems to have the same functional-ities as the BioHarness, except that the hardware itself can be wornmore comfortable without much sliding and is therefore maybe agood alternative to the BioHarness used during this study.

tattoo sensors In today’s research flexible sensors are devel-oped that can be attached to the skin, like a tattoo. This gives themthe advantage of not being able to slide, compared to the BioHarness.Although not yet available for today’s research this can be a goodalternative for future implementations [4].

Figure 9: Flexible hardware attached to the skin [4].

3.4 discussion

None of the sensors stand out from the others. Based on the pro-vided information we have chosen to use the Zephyr BioHarness BT.Its comfort is sufficient, it provides wireless communication via Blue-tooth and an SDK is available. Besides this the raw ECG data is avail-able. The final system will exists of two devices, the BioHarness anda smartphone, figure 10 illustrates what this will look like in real life.

3.4 discussion 25

Data from the BioHarness is retrieved by a smartphone. During thetests and experiments during this study the android devices usedwere the android emulator provided by the SDK the HTC One S andthe Huawei Ascend Y300 phone.

Figure 10: The system during use. On the left the BioHarness and on theright an Android phone receiving ECG data via Bluetooth.

battery life Battery life is a problem for most smartphones nowa-days. Most smartphones will last a day or maybe two. Running anextra application does not contribute to this problem. The battery lifeduring one hour of running the application is compared to an hourof idle state. During these tests both Wi-Fi and mobile network aredisabled and Bluetooth is enabled. During a period of 5.5 hours thebattery consumption of the smartphone is evaluated. Tests show thatduring this period the battery power of the Huawei Ascend Y300 islowered to 86% when the app is not running. When the app is run-ning during the 5.5 hours the battery power is lowered to 80%.

4D E V E L O P M E N T

During the implementation of the architecture described in chapter 3

challenges are encountered. This chapter describes these challengesand their solutions.

4.1 algorithms

The software on the smartphone uses several algorithms, for exam-ple the detection of heart beats. In this chapter the most importantalgorithms are discussed.

4.1.1 QRS complex

One of the biggest challenge is to retrieve the R tops from the ECGdata. The R tops are used to calculate the heartbeat per minute (bpm).The QRS complex is a name for the combination of three momentsduring one heartbeat on an electrocardiogram. Figure 11 shows anexample of how such a QRS complex looks like.

Figure 11: QRS complex visualizing the activity during one heart beat [3]

The image is a representation of the muscle contractions of theheart during one heartbeat. The one we are most interested in is theR top. This is the peak with the highest amplitude and therefore the

27

28 development

best signal to detect. Once we have two subsequent R tops the bpmcan be calculated by formula 6.

bpm = 60000/R− Rinterval (6)

Because it is not reliable to only use the last known R-R intervalthe bpm is calculated by taking the mean of the R-R intervals duringthe last known 10 seconds of the ECG data, which is the length ofstandard electrocardiograms [40].

4.1.2 R top triggering

The RR series can be detected in multiple ways. The software usedat the psychiatry department of the UMCG (PreCar) uses a thresh-old algorithm. Every peak above a certain threshold is signed as a Rtop. This software is used for analyzing data after all data is received.The user can change the threshold value to check for better results.Because our system analyses the data in real-time this is not a possi-ble solution. As discussed in a previous study of Blaauw & Zuidhof(2012) an algorithm is developed. Here we compare this algorithmwith BandPass filtering.

bandpass filtering BandPass filtering uses the frequencies inthe data signal to filter the signal to only let certain frequencies pass.By using a combination of low- and high pass filtering the R tops canbe triggered. Figure 12 shows an ECG with the R tops triggered usingthis filter.

When the signal gets disturbed the band pass algorithm has troublefinding R tops. This is the main disadvantage of this method and willbe taken into account when deciding the R top triggering method.

peak detection Because the ECG data exists only of peaks (pos-itive/negative), a peak detection algorithm is developed. The algo-rithm detects peaks by building a peak while the continuous data val-ues are going constantly in the same direction (up or down). Whenthe next point is going in the other direction a new peak is started andthe current peak has reached its top. When the peak has a max widthof 20 (default) and a minimum height of 20 (default), it is signedas an R top. The two values max width and minimum height canbe changed in the settings menu. Figure 13 visualizes the results re-trieved with this method.

4.1.3 Filtering

The data retrieved from the BioHarness are not always reliable. Dueto physical movement the heart rate monitor band can slide which

4.1 algorithms 29

Figure 12: Bandpass filtering example of four heart beats. A decision ismade according to the frequencies passing the filters. The orangeline illustrates the decision threshold which is used to mark thesignal as an R-top.

Figure 13: Disturbed ECG with R tops marked in blue.

causes a very disturbed signal. Other reasons of signal disturbancecan be caused by other physical movements, the pulses generatedwhen contracting other muscles can be intercepted by the sensor.

Although the algorithm does still detect R tops when there is suchdisturbance it is not always safe to use them. Therefore the data isfiltered. One of the options is to ignore the R tops and, while thereis disturbance in the time window, the last known heart rate is taken.Because this solution ignores all of the data, including correct data,another solution is chosen. When disturbance is detected by the ECGanalyzer, the disturbed parts are ignored and heart rate is calculatedover the correct data.

4.1.4 Emotional arousal detection

For emotional arousal detection the algorithm from Myrtek et al (2005)is used. In chapter 2 this algorithm is discussed. This algorithm is re-peated below.

30 development

algorithm The AHR signal is a continuous estimation of arousalfrom heart rate and physical activity. The default process of addi-tional heart rate computation is explained below. (See appendix A fordetails of the algorithm) For the current emotional arousal state thecurrent time window is compared to the mean of the previous 3 timewindows. The length of the time window is 60 seconds on default,more detailed information about the time window and the length isfound in chapter 4.

HRC[i] = HR[i] − ((HR[i− 3] +HR[i− 2] +HR[i− 1])/3) (7)

HRC is the current change in heart rate compared to the previous3 time windows (7). When this value reaches HRPLUS (threshold, 3

heart beats on default), an AHR signal is generated. HRPLUS is in-creased by adding the mean acceleration of the current time window.For additional heart rate a score is formed defined as the factor, bywhich the current heart rate change (HRC) exceeds the minimal heartrate change (HRPLUS)

AHR = HRC/HRPLUS (8)

Requirements for a feedback moment (additional heart rate de-tected) are as follows:

1. HRC >= HRPLUS

2. AHR >= 1

3. Acceleration must not exceed the mean of the previous 3 timewindows of acceleration

4. The current heart rate must be higher than its predecessor

random feedback For reasons of comparison, but also to avoidpossible conditioning, random feedbacks are given with no indicationof an emotional event after three event-related feedbacks. In chapter 5

the random and true feedbacks are compared to evaluate the softwareand help prove the hypothesis discussed in chapter 1. Random feed-backs are given randomly after 3 true feedbacks have been generated.For every generated true feedback the change of a random feedbackincreases. The change of random feedback is reset after a randomfeedback is generated. If in the minute for random feedback the crite-ria for a true feedback are fulfilled, then the next following minute isused for random feedback. If the ratio between event-related and ran-dom feedbacks is smaller than 5:2, then no further random feedbacksare given with the exception that within the interval of 20 minutes noevent-related feedback is possible. In this case a random feedback isgiven in minute 20 [31].

4.1 algorithms 31

time window The algorithm uses time windows to compare heartrate means with each other. In his papers Myrtek uses time windowsof 60 seconds without any clear reasoning why this length has beenchosen. This section elaborates on the best length to use for the timewindows by storing ECG data retrieved from the system during differ-ent episodes from the TV show ’The killing’ watched by the testleader.Afterwards the algorithm is run over the data with each time differenttime windows. A time window is needed which detects increases inheart rate that are not caused by reflexes or physical activity. To testthis two different time windows are used, 30 and 60 seconds. Datais logged from the BioHarness while the testleader watches episodesfrom two episodes of the TV show ’The Killing’. During the episodesthe testleader is sitting still in a stationary position.

Figures 14 and 15 visualize the heart rate of the testleader and themoments of feedback.

Figure 14: Heart rate during an episode of the Killing. The red dots show thefeedback moments when a time window of 60 seconds is used.The green dots show the feedbacks when a time window of 30

seconds is used.

The figures show no significant differences in moments of feed-back. The 30 second time windows show a few extra feedbacks atmoments where the heart rate increases for just a small time. Theseshort peaks in heart rate is not what needs to be detected with thissystem, therefor the time window of 60 seconds used in Myrtek’sexperiments stays. The time window can be changed depending onwhat kind of behavior has to be detected. As the results show a lowtime window (10-30 seconds) can be used to detect short, high in-creases in heart rate. These increases can be caused by reflexes andemotions. As emotions imply massive response mobilization and syn-chronization as part of specific action tendencies, their duration mustbe relatively short in order not to tax the resources of the organismand to allow behavioral flexibility [39]. Therefore a time window of

32 development

Figure 15: Visualization of episode X from the Killing. The red dots show thefeedback moments when a time window of 60 seconds is used.The green dots show the feedbacks when a time window of 30

seconds is used.

30-90 seconds is best when trying to detect emotional arousal. Timewindows higher than 90 seconds can be used to detect moods.

acceleration Physical activities of the human body cause anincrease in heart rate. Because an increase in heart rate is exactlywhat the algorithm is supposed to detect the thresholds for heart rateare dynamically changed according to the acceleration data retrievedfrom the accelerometer of the BioHarness. The acceleration is com-puted by using the standard deviation over the same time windowsdefined for the heart rate computation.

rmssd The Root Mean Square of the Successive Differences (RMSSD)is an index of heart rate variability, discussed in chapter 2. Althoughnot used in this study the computation of the RMSSD is implementedto be logged at moments of additional heart rate for future purposes.

4.2 software

The software developed in this research project is based on the soft-ware developed during an earlier study by Blaauw & Zuidhof [12].The software running on Android consists of multiple components. Inthis section only the main flow of information, and the main compo-nents are described. This section was designed for future developersand can therefore contain some jargon.

4.2 software 33

4.2.1 Class diagram

In software engineering, a class diagram in the Unified Modeling Lan-guage (UML) is a type of static structure diagram that describes thestructure of a system by showing the system’s classes and the rela-tionships among objects. Figure 16 shows a simplified class diagramof the software developed during this research project to give an im-pression of the system.

4.2.2 EmozionActivity

When developing an Android application, one will most certainly useAndroid’s Activities at some point in the code. An activity is a single,focused thing that the user can do. Almost all activities interact withthe user, so the Activity class takes care of creating a window foryou in which the user interface can be placed. In this application ev-erything starts at the EmozionActivity. This Activity creates a simpleGUI for the user to interact with. This allows the user to connect to,e.g., the BioHarness or the simulator. More importantly, this Activitystarts the MeasurementService, which keeps the application runningwhen the GUI does not have focus, or is closed.

4.2.3 MeasurementService

The measurement service is the heart of the application. In contraryto the EmozionActivity, this is an Android service. A service is anapplication component representing either an application’s desire toperform a longer-running operation while not interacting with theuser or to supply functionality for other applications to use. As said,this service is the heart of the application. From this service the dif-ferent detecting components are started and connections to the datalisteners are made (such as the BioHarness or a simulator). Whenthe GUI of the application closes, this service will continue capturingmeasurements.

4.2.4 Parcels

Communication between the data listeners and the emotion detectioncomponents flows using parcels and handlers, provided by the An-droid’s SDK. Parcels in this case are containers which can be used tomarshall and unmarshall data, for sending it between classes. Usingparcels can be compared with using Java’s serializable.

34 development

Figure 16: Simplified class diagram of the software developed during thisstudy.

4.3 questionnaires

Feedback from the user is needed to measure the amount of emo-tional arousal at the moment of additional heart rate. The software isbuilt to propose single or multiple emotional states. Figure 17 shows

4.3 questionnaires 35

two example. Figure A shows an example of just one mood, the userwill not have to select a state since there only is one. Another exampleis to ask for one out of a list of emotions, for example angry, happy,disgust, anxious, scared, and other (Figure 17 B). The user is asked toselect the emotional state he or she feels is most present at that mo-ment. The slide bar is used to give a weight to the selected emotionalstate. For the final experiment only one question is asked: Mood. Thescale is set from negative to positive where the middle is neutral.

Figure 17: Example of the implementation for the extraction of the affectivestate. In this case only one state can be selected, mood. The slide-bar gives the possibility to select a value between a negative andpositive mood, where the middle is a neutral state.

5E VA L U AT I O N O F A R E A L - T I M E A M B U L AT O RYS Y S T E M F O R T H E D E T E C T I O N O F E M O T I O N A LA R O U S A L

Emotional arousal studies in the past have showed that emotionalarousal detection is not a trivial task. Several issues arise when detect-ing, eliciting and questioning the emotional arousal from participantsduring experiments in laboratory and real-life environments. In pre-vious laboratory experiments different emotional states are elicited bythe testleader. Among others, dependent variables are activities of theautonomic nervous system (heart rate or respiration; [7, 13]) and self-reports of the emotional state. As opposed to field studies, subjects intheir laboratory experiments know that they will perceive special situ-ations which may enhance the correlations between physiological ac-tivation and self-reported emotional states [28]. In his previous workMyrtek and colleagues performed experiments on emotional arousal.Emotional events were defined by an increase in heart rate withoutan accompanying increase in physical activity (additional heart rate).The conclusion of these ambulatory monitoring studies on the percep-tion of emotions in everyday life [30] was that the detection of addi-tional heart rate indicating emotional arousal was quite different fromthe results suggested by laboratory experiments. Obviously, in every-day life the identification of emotional arousal is much more difficultthan suggested by laboratory studies because we have to account forsubjective hypotheses and schemata as proposed by several authors[19, 9]. In these studies with nearly 500 subjects, physiological param-eters (heart rate, physical activity, additional heart rate) and psycho-logical parameters (excitement, enjoyment) were assessed simultane-ously throughout the day. Based on additional heart rate a feedbacksignal was given which requested subjects to answer predefined ques-tions (excitement, enjoyment). Besides these true feedbacks, randomfeedbacks were generated with no indication of additional heart rate.Subjects in these studies were unaware whether the feedback signalwas triggered by their own heart rate or not. Despite great mean dif-ferences in heart rate (about 7 bpm) with no differences in physical ac-tivity, true and random feedbacks did not show significant differencesin excitement and enjoyment ratings. The above ambulatory studiesdid not ask the subjects for their current emotional state during theexperiments. Myrtek continued with these studies but added directfeedback for both feedback moments (true and random feedbacks). Incase of a feedback moment participants were asked to disclose theircurrent emotional status. The following emotions were listed on a

37

38 evaluation of a real-time ambulatory system for the detection of emotional arousal

display of a device the participants had with them: happiness, anger,anxiety/fear, sadness, surprise, and disgust [31].

5.1 validity of additional heart rate as an indicator

of emotional arousal

The method used in the present study is based on the decomposi-tion of heart rate, which reflects both the physical activity and theemotional activation. Emotional arousal was operationalized in thepresent study by the simultaneous comparison of heart rate and phys-ical activity. An accelerometer (motion detector) is used in order todifferentiate metabolic and non-metabolic (additional) heart rate. Be-cause heart rate is dominated by the physical activity, we may expectstrong correlations between physical activity measured by motion de-tectors and heart rate, regardless of whether emotional events arepresent or not. The strong correlation between physical activity andheart rate is only a prerequisite for the detection of emotional arousal.Other proofs are necessary to endorse the assumption that additionalheart rate taps emotional arousal. Myrtek performed two laboratoryexperiments with male students, where he was able to show thatthe frequency of additional heart rates was significantly higher whenwatching an erotic film as compared to a comedy. This finding holdsfor the student sample watching the films under resting conditions aswell as for the second sample watching the films during light but var-ied physical activity [30]. An ambulatory monitoring study with traindrivers showed that the frequency of additional heart rate was higherfor the driving modes "start" and "braking to stop" as compared totrain speeds between 0 and 200 km/h [32]. The heightened risk ofaccidents while driving into and leaving the station may be the rea-son for the increase in emotional arousal. A third proof stems from astudy with schoolboys [30, 29]. In this study, boys with light TV con-sumption were compared to boys with heavy TV consumption. Boyswith heavy TV consumption displayed a significantly lower frequencyof additional heart rate while watching TV than boys with light con-sumption. In another study with preschool children Myrtek showedtwo cartoons. This study showed that a cartoon with many actionscenes caused a higher frequency of additional heart rate comparedto more neutral cartoons. A movie with more predefined emotionallyintense moments causes more additional heart rates. Further argu-ments supporting the view that additional heart rate is an indicatorof emotional arousal derive from special social situations. With stu-dent samples, Myrtek showed that social contacts with peers revealeda lower frequency of additional heart rate than social contacts withstrangers [30]. Other ambulatory monitoring studies with samples ofcardiac or rheumatic patients showed that the frequency of additionalheart rate was higher during contacts with doctors or nurses as com-

5.2 laboratory and field study 39

pared to contacts with other patients or being alone. In a study withstudents, it was shown that additional heart rate was much higherduring driving a car than during usual daily activities [25, 31].

5.2 laboratory and field study

In the present study ten female participants use a system capable todetect additional heart rate for 4 hours. The experiment consists oftwo parts of both 2 hours starting with a controlled, static environ-ment followed by an ambulatory, open environment. The first part isthe same for every participant, the participant is asked to watch themovie ’Trainspotting’. This decision is made because an activity isrequired that causes minimal physical activity and a way to provokeemotional arousal in a human subject. To provoke these emotions re-search is done on emotional arousing movies. According to Myrtek’spaper negative emotions (anger and anxiety/fear) result in higher in-creases in heart rate compared to a positive emotion (happiness). Ina paper of Schaefer et al [6] research is done in movies that excel in10 emotional categories. The movie that excelled in negative affectwas the movie ’Trainspotting’. The second part of the experiment isfocused on the reliability of the BioHarness signal. Because the Bio-Harness is a band and does not include electrodes placed on the body,the band can slide during physical activity which may result in an un-reliable signal. This part is different for everyone; when the movie isfinished the participant is asked to continue with their normal dailyactivities. For this study only female participants were selected sincefemale subjects have a higher emotional responsiveness compared tomale subjects [10].

5.3 aims of the present study

In the current study a replication of Myrtek’s experiments are per-formed, both in a laboratory and ambulatory setting while usingstate-of-the-art hard- and software. Recent technological developmentsgive the possibilities for conducting these measurements in a way thatmay be more convenient for participants. Therefore, in the presentstudy is decided to use the same method of feedback moments asused in Myrtek’s experiment. Participants are asked to indicate theiremotional state on both true and random feedback moments on anapp running on their smartphone. The experiment is set up to eval-uate the system on accuracy and reliability by analyzing data frommultiple participants in two different settings inspired by previousstudies described earlier in this introduction [30, 31]. In his paperMyrtek has shown that lower mean heart rates were found duringrandom feedback moments compared to true feedback moments, asis expected since true feedback moments are triggered when an in-


crease in heart rate is detected. This is what is expected in this study[30]. In the same paper Myrtek indicates that the expected differ-ences between excitement and enjoyment during true and randomfeedback moments were not found during ambulatory measurements.This was in contradiction with his previous work in laboratory ex-periments. When human subjects had minimal physical activity theadditional heart rate could be addressed to emotional arousal [30, 31].Based on this information I hypothesize that in the present study:

1. During the laboratory experiments I expect to detect additionalheart rates triggering true feedback moments at predefined emo-tionally intense moments in the movie

2. During the laboratory experiments low acceleration and highersubjective affective scores and heart rate will be found at thetrue feedback moments compared to random feedback moments

3. During ambulatory assessment the differences in heart rate, ac-celeration and subjective affective score between true and ran-dom feedback moments are less/not distinguishable

5.4 methods

This chapter elaborates on the participants, methods, procedures andresults used during this experiment and will conclude with answersto the hypotheses defined above.

5.4.1 Participants

Ten female Dutch students (mean age 22.7, range 19-26) were re-cruited by posters at and around the university. Women were eligi-ble for participations if they fulfilled the following criteria; aged be-tween 18 and 26 years, good general health, no known cardiovascularproblems, Dutch as the native language, and preferably possessing anAndroid phone. The complete experiment was set up in accordancewith the Ethical Committee. Before the participants started with theexperiment they gave written informed consent. Participants were re-warded with a chocolate heart of chocolate and had a chance to wina gift certificate of 50 Euros.

5.5 hardware

During the experiment two devices were used for measuring the par-ticipant’s heart rate, acceleration (x, y, z axes) and mood. For the re-trieving of the heart rate and acceleration the BioHarness was used.This device was worn around the chest as figure 18 illustrates, theshoulder strap can be used when the BioHarness is too loose around

5.5 hardware 41

pid age seen ’trainspotting’ good general health

001 23 Yes, Over a year ago Yes


003 25 No Yes

004 22 No Yes

005 25 No Yes

006 23 No Yes

007 19 No Yes

008 26 No Yes

009 20 No Yes


Table 3: Information regarding the participants of the experiment.

the chest. During the experiments it is used by one participant dur-ing both parts of the experiment. Using a Bluetooth connection anAndroid smartphone with our software receives the data from theBioHarness every 250 milliseconds.

Figure 18: Illustration of position of the BioHarness during use [1]

The app on the smartphone analyses this data and shows a ques-tionnaire to the participant according to the algorithm discussed inchapter refch:development. More elaborated information about thequestionnaire is given later in chapter 4.3. Two different smartphoneswere used during the experiments both provided by the testleader,the HTC One S (Table 4) and the Huawei Ascend Y300 (table 5). Since


the BioHarness has a limited Bluetooth range, the participant wereinstructed to always keep the Smartphone within a range of a fewmeters (max 5 meters) of themselves.

htc one s

Operating System Android 4.1.1

Processor Qualcomm MSM8260

Dual Core, 1,7GHz

Memory 1GB

Capacity 16GB

Battery 1.650mAh

Table 4: Specifications of the HTC One S.

huawei ascend y300

Operating System Android 4.1

Processor Qualcomm Snapdragon S4 Play (MSM8225)

Dual Core, 1GHz

Memory 512MB

Capacity 4GB

Battery 1.730mAh

Table 5: Specification of the Huawei Ascend Y300.

laptop In the first part of the experiment a movie is watched bythe participant. This movie is shown on a Dell laptop (Screen size:17", Dutch subtitles enabled).

5.5.1 Questionnaires

An important measure during the experiment is the personal andaffective state of the participant. For the affective state two types ofquestionnaires are used, the slidebar and the affect grid as introducedin chapter 2. For evaluation a personality questionnaire is used toassist in cases of deviated results during analyses.

slidebar A slidebar is used to measure the subjective affectivescore of the participant. This slidebar has a range from negative (low,0) to positive (high, 100) where the middle (50) represents a neutralmood.

5.6 procedure 43

affect grid To measure the affective state besides the slidebar,an affect grid is used during the experiment [16]. The affect grid con-sists of 2 axes, horizontal and vertical. The horizontal direction rangesfrom unpleasant feelings on the left to pleasant feelings on the right.The vertical direction of the grid ranges from high arousal at the topof the grid to sleepiness at the bottom. The participant is asked to se-lect one box that represents their current affective state according tothese axes. This is done four times during the movie, section 5.6 dis-cusses the exact moments. Because only native Dutch speakers canparticipate in the experiment a Dutch version of the affect grid isused during the experiment. The result of one filled out affect grid isa combination of the X and Y axes. Figure 19 illustrates a filled outexample, this example results in X = 11 and Y = 3 which results inthe final value 11 for pleasantness and 3 arousal.

Figure 19: Affect grid with an example of how the grid should be filled outwhen the participant feels pleasant and is positively aroused.

5.6 procedure

5.6.1 Intake

When a participant contacted the testleader she was asked severalquestions, including: Have you seen the movie Trainspotting? If yes,how long ago was this? Is it clear that you cannot do any heavy physi-cal activities during the experiment? After answered all the questionsan appointment was made for the experiment and the participant wasasked for any other remarks.


5.6.2 Experiment

Only one session was held per participant. All sessions started in themorning (at 10:10 on average, range 10:00-11:00 hours) with a 2 hourlaboratory session, followed by a 2 hour ambulatory measurement inreal-life. The laboratory session is performed in a room of the iLabat the UMCG [2]. At the start of the experiment the participant isinformed about the procedure and were asked to put on the BioHar-ness. The developed emotional arousal detection app was installedon the participant’s smartphone by the testleader. In case of a smart-phone not suitable for usage during the experiment (iPhone, Win-dows phone or to low specifications) the smartphone of the testleaderis used. The participant is asked to answer a few questions; Feelinghealthy? No medications? Is it clear you cannot eat or drink duringthe movie? Is it clear what is expected of you? When everything isclear the participant takes a seat in front of the laptop. The procedureof what to do when the smartphone asks for feedback is explainedand the participant is instructed how to fill out the questionnaire onthe smartphone and the affect grid. The testleader provides and ex-plains the affect grids and instructs the participant to fill out the firstaffect grid. After the affect grid is filled out by the participant themovie ’Trainspotting’ is started and the testleader leaves the room.The testleader returns to the room every 30 minutes to provide a newaffect grid (two during the movie, one when it is finished). When themovie is finished a meeting is arranged to take place in two hours.During this meeting the devices are returned and the data from thesmartphone is retrieved. The participant is rewarded with a choco-late heart and informed that she will be notified if she won the giftcertificate.

5.6.3 Finish

After two hours of daily activities the participant returns to the testleader.The BioHarness is returned, the data on the smartphone is stored ona computer and the software is removed from the participants’ smart-phone. The participant is debriefed and rewarded.

5.7 data assessment

Two subjective rating methods are used during analyzes, subjectiverating by the testleader and subjective rating by the participants. Bothmethods having a different purpose. The subjective rating method bythe testleader is used to evaluate whether or not the true feedbackmoments are generated at moments where emotional arousal is ex-pected, and thus validating the reliability of the algorithm. The sub-

5.7 data assessment 45

jective rating by the participants is used to validate the true feedbackmoments on true emotional arousal.

The algorithm running on the smartphone uses both heart rate(expressed in beats per minute) and physical movement (expressedin acceleration) data to indicate whether or not emotional arousal ispresent. A time-window of 60 seconds is used to compare the currenttime-window with the mean of the previous three time-windows. Ondefault, when no physical activity is present, the difference with themean of the previous three time-windows must be above 3 bpm tobe classified as emotional arousal. When physical activity is presentthe threshold is increased according to a specified process (see ap-pendix A). According to the algorithm this is a moment of emotionalarousal called additional heart rate (AHR), which is expressed in unit-s/minute.

5.7.1 Subjective rating by testleader

Subjective rating is used to evaluate the moments where true feed-back moments are generated. By scoring the movie for intense mo-ments (violence, erotic, suspense) a timeline is generated with a valuebetween 0 and 8 for every minute. 0 indicates a minute of low inten-sive moments, 8 indicates a minute of high intensive moments andtherefor a high expectancy of AHR. Figure 20 visualizes the timelinethat will be used during the analyses. All true feedback moments willbe plotted against this graph of expected AHR. The expectancy is tosee a high frequency of true feedback moments at minutes of highexpected AHR.

Figure 20: Timeline of the movie ’Trainspotting’ with a scored value for in-tense moments for every minute on a score from 0 - 8. 0 indicatinga minute of low intensive moments, 8 indicating a minute of highintensive moments.

5.7.2 Subjective rating by participants

When AHR is detected the smartphone alarms the participant by vi-brating and beeping. When the smartphone is turned on the question-naire is directly visible on the screen, ready to be filled out. Figure 21

shows this questionnaire. The participant indicate their current mood


mood status by sliding the pointer up or down. The slidebar has ascale from 0 to 100, this is not visible for the participant. By clickingon the send button the mood is saved and the smartphone can beturned off, waiting for the next AHR.

Figure 21: Slidebar as displayed on the smartphone when additional heartrate (AHR) is detected.

The following parameters were considered: acceleration (units/minute),heart rate (bpm), additional heart rate (bpm) and mood (0-100). An-other variable is derived from this mood score, called affect. Themood variable is 50 (neutral state) on default and therefor the meanvalue over all participants does not give a good indication of arousal.The computation of the variable affect takes this into account by as-signing a value which is in perspective with the neutral state. Affectis computed by taking the absolute value of 50 (neutral) subtractedfrom the mood parameter. Feedback moments are categorized in ’inmovie’ and ’ambulatory’ sections and in random or true feedback mo-ments. The variables assigned to every category were then averagedover all participants.

5.8 statistical analysis

Descriptives and statistical analysis were performed in SPSS [14]. Priorto analysis the distributions of all variables were checked on normal-ity. Below the methods are described which are used to evaluate thehypotheses stated in the introduction.

5.9 results 47

5.8.1 During the laboratory experiments I expect true feedbacks at prede-fined emotionally intense moments

The data derived during the movie is analyzed for testing this hy-pothesis. The emotional intensity of every minute as rated by the re-searcher (visualized in figure 20) are compared to the sum of all truefeedbacks generated during the movie at the corresponding minute.Because the algorithm uses time windows of 1 minute the true feed-back moments are computed for every minute according to the fol-lowing algorithm: minute i = AHRi + AHRi + 1. Evaluations is per-formed by visually checking the correlations between the intensitylevels and the real feedback moments for every minute.

5.8.2 During the laboratory experiments low acceleration and higher sub-jective affective scores and heart rate will be found at the true feedbackmoments compared to random feedback moments

Statistical analysis was applied to the mean heart rate and affectscores. The test used for the comparison of random versus true isperformed by applying a paired-samples T-test contrasting true feed-back moments against random feedback moments.

5.8.3 During ambulatory assessment the differences in heart rate, accelera-tion and subjective affective score between true and random feedbackmoments are less/not distinguishable

Using the same statistical analysis as applied in hypothesis 2, a paired-samples T-test is applied to the values derived during the ambulatoryexperiment. One pair is formed, true and random feedback duringthe real-life activities.

5.9 results

In this section first the descriptives of the data are given then theresults for each of the three hypotheses tested.

5.9.1 Descriptives

In addition to the questionnaires on the smartphone, participantshave filled out an affect grid. Table 6 shows the results. The resultsof the affect grid showed on average a decrease in reported pleasant-ness during the movie, which is expected since the movie is selectedon negativity. The table data also shows a mean increase of arousalduring the movie. Only when the movie was finished (90 minutes)


the arousal went down. This correlates with the trendline in detectedAHR visible in table 22.

During the experiment the participants filled out two different ques-tionnaires, not including feedback moment questionnaires since theyare covered in the next sections. The results of the individual scoreson the affect grid are included in table 6. The results show a decreasein pleasantness on average, which is expected since the movie is se-lected on negativity.

5.9 results 49

tim

e(m

in

ut

es

)0

03

03

06

06

09

09

0

pid

pl

ea

sa

nt

ne

ss

ar

ou

sa

lp

le

as

an

tn

es

sa

ro

us

al

pl

ea

sa

nt

ne

ss

ar

ou

sa

lp

le

as

an

tn

es

sa

ro

us

al

001

98

98

88

87

002

96

83

88

96

003

45

66

65

65

004

74

78

84

98

005

64

77

58

68

006

10

34

63

86

6

007

85

43

23

24

008

84

56

66

55

009

94

97

98

95

010

86

94

85

94

Ave

rage

7.8

4.9

6.8

5.8

6.3

6.3

6.9

5.8

Tabl

e6

:Dat

ade

rive

dfr

omth

epa

rtic

ipan

tsev

ery

30

min

utes

duri

ngth

em

ovie

.


feedback moments During the total length of the experiment230 (per participant: mean 23, range 17-32) true feedbacks and a totalof 28 (mean 2.8, range 0-9) random feedbacks were generated. Sub-jective mood ratings were mainly positive, 67.7% compared to 24.9%negative. 7.4% neutral scores were selected. On average the subjectivemood ratings had a value of 62.25. The mean affect score has a valueof 19. Since mood can be positive and negative the affect score is usedin the next sections, it gives a better representation on the averageemotional arousal. Most of the feedbacks were generated during thedaily activities (part 2 of the experiment) with 53.1% of the feedbacks.Table 7 shows the frequencies and results of the feedbacks during themovie and the daily activities.

hypothesis 1 - during the laboratory experiments i ex-pect to detect additional heart rates triggering true

feedback moments at predefined emotionally intense mo-ments in the movie The first subjective rating method is basedon the number of true feedbacks generated during the movie. Forevery minute the amount of true feedbacks are summed. Figure 22

illustrates the subjective rating of the researcher (see method sectionfor details) of emotional arousal versus the total number of true gen-erated feedback moments over all participants. As expected, the fig-ure shows correlations between emotionally intense moments and ex-pected AHR. Peaks in AHR in minutes not correlating to the intensitylevel of that minute can be caused due to the subjective rating of theintensity level by the testleader. Different people react differently tothe same situations.

5.9 results 51

hypothesis 2 - during the laboratory experiments low

acceleration and higher subjective affective scores and

heart rate will be found at the true feedback moments

compared to random feedback moments During the moviea total of 121 feedbacks were generated of which 106 were true feed-backs and 15 random. The data in table 7 shows a significant differ-ence (t(8) = -2.409, p = 0.043) in affect score between true feedbackswith a mean of 15.915 compared to 9.066 during random feedbacks.The mean heart rate however does not show differences in true andrandom feedback moments (t(6) = -.064 6, p = 0.951). Because thenumber of random feedbacks is low, outliers can have a big influ-ence in mean heart rate. During the movie one outlier in heart rateis found (mean = 76.1 bpm, outlier = 97 bpm). When this outlier isremoved a significant difference in heart rate is found between trueand random feedbacks during the movie (t(5) = -4.050, p = 0.010). Ac-celeration data shows no significant differences between random andtrue feedbacks (t(5) = -1.925, p =.112) during the movie. In 17 of therandom feedback moments AHR values have the expected direction,e.g. higher AHR during true feedback moments compared to randomfeedback moments. However due to lack of power the effect were notsignificant (t(5)=-1.04, p=0.22).

hypothesis 3 - during ambulatory assessment the differ-ences in heart rate , acceleration and subjective affec-tive score between true and random feedback moments

are less/not distinguishable During the ambulatory assess-ment 137 feedbacks were generated of which 124 were true feedbacksand 13 random. The data in table 7 does not show significant differ-ences in affect scores, the range of affect of true and random feed-backs is just above 1. This has not led to a significance of (t(5) = 0.556,p = 0.602) during the ambulary assessment. The data does show sig-nificant differences in heart rate during the ambulatory part (t(4) =-2.721, p = 0.053) of the experiment. Acceleration data shows a sig-nificant difference between true and random feedbacks (t(4) = -5.826,p = 0.004) during the ambulatory part of the experiment. AHR val-ues show expected results with higher AHR during true feedbackmoments compared to random feedback moments as can be seen intable 7. Due to the lack of power however, the effects were not signif-icant (t(3)=-1.84, p=0.16).


Figure 22: Emotional intensity versus the total AHR detected per minuteduring the movie. Emotional intensity is rated by the testleaderon a scale from 1-8. The detected AHR represent the total numberof true feedbacks detected during the movie for all participants.The trendline shows the increase of the frequentie of detectedAHR.

5.9 results 53

Mov

ieM

ovie

Am

bula

ntA

mbu

lant

True

Ran

dom

True

Ran

dom

N106

15

124

13

Hea

rtra

te(b

pm)

Mea

n(SD

)76.8

15

(9.1

11)1

73.4

39

(7.5

94)

2,7

84.6

29

(17.7

27)3

72.3

25

(14.6

35)4

Ran

ge(m

in-m

ax)

40.3

81

(56.9

67

-97.3

48)

37.8

69

(59.4

91

-97.3

61)

99.1

62

(55.5

09

-154.6

71)

39.8

19

(48.9

91

-88.8

11)

Acc

eler

atio

n(U

nits

/min

ute)

Mea

n(SD

)1

.242

(0.9

83)1

1.1

55

(0.8

75)

2,7

2.6

15

(1.9

91)3

1.5

20

(0.6

99)4

Ran

ge(m

in-m

ax)

5.4

70

(0.3

20

-5

.790)

4.1

30

(0.5

00

-4

.630)

9.1

10

(0.4

50

-9.5

60)

2.1

60

(0.6

20

-2.7

80)

Moo

d(0

-100)

Mea

n(SD

)55.2

2(1

8.2

36)

50.9

3(1

2.1

04)

68.4

7(1

7.2

86)

73.6

9(1

4.1

62)

Ran

ge(m

in-m

ax)

71

(16

-87)

43

(31-7

4)

87

(13

-100)

49

(50-9

9)

Aff

ect

(0-5

0)

Mea

n(SD

)15.9

15

(10.2

14)

9.0

66

(7.7

04)

22.3

577

(11

.783)

23.6

923

(14.1

62)

Ran

ge(m

in-m

ax)

37

(0-

37)

24

(0-

24)

50

(0-

50)

49

(0-

49)

Add

itio

nalh

eart

rate

Mea

n(SD

)6

.05

(4.0

5)

2.7

6(2

.28)5

7.8

9(9

.42)

2.3

0(5

.11)6

Ran

ge(m

in-m

ax)

21.5

1(-

2.6

6-

18.8

5)

5.8

4(-

0.4

2-

5.4

2)

95.3

1(-

3.0

6-

92.2

4)

10.7

2(-

1.7

6-

8.9

6)

Tabl

e7:M

ean,

rang

e,st

anda

rdde

viat

ion

oftr

uean

dra

ndom

feed

back

mom

ents

duri

ngth

em

ovie

and

ambu

lato

ryex

peri

men

t.7

Val

ues

give

nin

the

tabl

ear

ew

itho

uton

eou

tlie

r.O

rigi

nalv

alue

sin

clud

ing

this

outl

ier

are:

Hea

rtra

te+

Mea

n(SD

)=

76.0

97

(10.6

79

),A

ccel

erat

ion

+M

ean(

SD)

=1.5

41

(1.4

18

).1N

=8

8.2

N=

9.3

N=

89.4

N=

8,l

ower

sam

ple

size

sw

ere

due

toin

com

plet

enes

sof

the

soft

war

ew

hich

wer

eup

date

ddu

ring

the

expe

rim

entp

erio

d.5

&6

N=

6&

4re

spec

tive

ly,l

ower

sam

ple

size

sw

ere

due

toa

shor

tcom

ing

inth

eso

ftw

are

whi

chw

asno

trec

ogni

zed

and

coul

dth

usno

tbe

reso

lved

duri

ngth

eex

peri

men

tpe

riod

6D I S C U S S I O N A N D C O N C L U S I O N

6.1 discussion

The subjective affect scores derived during a laboratory setting (themovie) from true feedback moments (when AHR was detected) showeda significant higher mean value compared to random feedback mo-ments. During ambulatory assessment the differences between trueand random feedback were not visible in the affect scores. This isin accordance with several authors, including Myrtek, discussing theuse of additional heart rate for emotional arousal detection [31, 9, 19].In accordance with Myrtek’s experiment a significant difference inheart rate is found between true and random feedback moments dur-ing the laboratory settings. Differences in mean heart rate betweenrandom and true feedback moments are found in the ambulatory partof the experiment, which is not in accordance with Myrtek’s results.In his studies no significant differences in heart rate were found be-tween random and true feedbacks during ambulatory settings. As in-dicated in the results section of the experiment the number of randomfeedback moments is low for both parts of the experiments. Outlierstherefor have large influence on the mean values. Without removingthem from the dataset no significant differences are found during thelaboratory setting of the experiment. To solve this problem in futurestudies a larger group of participants is recommended.

Compared to Myrtek’s assessments methods our system for de-tecting addition heart rate has significantly improved. Technologiesin both hard- and software have improved which has resulted in amore convenient system for everyday use. The smartphone is a muchlighter and comfortable device to carry around compared to the sev-eral devices needed in Myrtek’s experiment. It also gives possibilitiesfor future implementations. Data can be stored online which givesresearchers the possibility of monitoring users remotely. The advan-tage of Myrtek’s system is the use of three electrodes put directlyon the human body for ECG extraction. The BioHarness used in thecurrent study gives an easy to use and comfortable sensor, but canslide during physical activity which causes disturbance in the ECGsignal, particular when participants are involved in more physicalactive activities. For follow-up experiments an alternative sensor isrecommended (see conclusion). For the set up of the experiment it-self the same set up is recommended. Starting with a part withoutany physical activity, still by watching one of the emotional excellingmovies selected by Schaefer to provoke emotional arousal. A second

55

56 discussion and conclusion

part with physical activity, but in a controlled manner comparable toprevious experiments of Myrtek and Brugner [30]. Participants canbe asked to play a game on the smartphone or again by watchinga movie while doing simple activities like walking or changing posi-tions (sitting-standing-sitting). This helps the evaluation of the impactof acceleration on heart rate and affect.

6.2 conclusion

The aim of this study was to develop a new system to detect emo-tional arousal. Prior research on physiological symptoms as markersof emotional arousal like brain wave [5] analysis were not performedin ambulatory situations because of the lack of ambulatory hardwareavailable. However, previous studies focusing on heart rate analysisdid provide ambulatory situations which gave a good base to con-tinue the current study with [30, 31, 25, 15]. An additional heart rate(AHR) algorithm proposed by Myrtek et al [33] is used in the cur-rent study. His system is replicated with technologies of today (bothhard- and software) which are more comfortable to wear for the par-ticipants and easier to use for the researchers. The results of the eval-uation show that this was a good choice for the situations in whichthere was no physical activity present. Although Myrtek’s AHR algo-rithm uses physical activity as input the experiments have shown that,when physical activity is present, the algorithm is currently still notable to make a clear distinction between AHR caused by emotionalarousal or by physical activity. Future research will tell whether ornot they can be used to solve the problems like filtering increases inheart rate caused by physical activity.

The device used for analysis can be the users own smartphonewhich most of the people carry around in everyday situations. Theonly extra device is the BioHarness that has to be worn around thechest. The BioHarness is chosen for its comfort and ease of use. Ithas shown its advantages and disadvantages. The main advantageof the BioHarness is the comfort and ease of use for everyday life.The band strapped around the chest is easy to use but brings the dis-advantage of moving when physical activities are performed. Someother devices discussed do not have this problem (3), an example isthe VUAMS using spot-electrodes which are put directly on the hu-man body. When the VUAMS features a Bluetooth connection this isa very good alternative for the BioHarness.

Besides a different sensor a combination of sensors could be usedto combine different physiological symptoms for emotional arousaldetection analysis. Examples are physiological symptoms like breath-ing, brain wave or facial expressions [20]. As technical innovationsgo fast nowadays sensors and software enable data being send to the

6.2 conclusion 57

smartphone and could be merged with the additional heart rate algo-rithm.

With the fast growing knowledge of both technology and physiol-ogy, systems become more advanced and help to develop even moreadvanced systems. The system developed during this research projectis a perfect example of what technology in combination with physi-ology can do to support researchers in gaining more and more infor-mation about human subjects.

AE M O T I O N A L A R O U S A L A L G O R I T H M ( M Y RT E KA N D F O E R S T E R , 2 0 0 1 )

59

60 emotional arousal algorithm (myrtek and foerster , 2001)

BS E N S O R S

polar wearlink+ transmitter with bluetooth One ofthe most used HR monitors are the ones from Polar. Applicationslike ’Endomondo’ (sports monitor) support this device, showing thatit is possible for use on Android devices. They provide a variety ofdevices, for different sorts of appliances. Although there are so many,only one is supported natively by android phones, the ’Polar Wear-link+ transmitter with Bluetooth’. They do have another heart ratemonitor supporting Bluetooth, the Polar H7, but after contacting Po-lar, it seems that these don’t provide support for (all) existing An-droid smartphones. After some research it seems hard to get the SDK.This might have something to do with licensing, or the kit is sup-ported by the Android SDK standard. The licensing problem wouldmean that a license needs to be purchased before the SDK can be usedon the monitor. The second option, that the Android SDK providessupport for these devices already, would be a very good thing. Price:e69,90 (without SDK)

zephyr hxmtm

bt software developer kit The second man-ufacturer of (Bluetooth) heart rate monitors is Zephyr. This companysells several fitness and health concerned devices. One of the devicesbeing sold is the ’HxMTM BT Software Developer Kit’. This kit con-tains a wireless Bluetooth heart rate monitor and moreover a Soft-ware Development Kit for Android phones. In their documentationthey state that the device has ’real-time feedback including heart rate,speed, distance, cadence and R-R telemetry, combined with a focuson a low-hassle, intuitive user interfaces providing a motivating andentertaining experience’. Price: e70,00

SDK: https://www.box.com/shared/c169gssedk2nrgu4t41fZephyr Open: http://code.google.com/p/zephyropen/

chronos ez430 The Chronos EZ430 is a device developed byTexas Instrument. The device is a watch which contains the TI CC430F6137

MCU. This allows developers to develop specific applications for thisdevice. The watch contains several sensors, such as a 3-axis accelerom-eter, pressure/altitude meter and a temperature sensor. These sensorscan be used independently in homemade applications. The device isnot standard equipped with a heartbeat sensor, but these are soldseparately. The heartbeat sensor does however, provide the exact in-formation needed for this project, that is all information about theheart rate (e.g. r-peaks and heart rate intervals). The disadvantage of

61

62 sensors

this device is that it does not (natively) connect to an Android phone,which means a pc must be used to capture the measurements. Alsoprogramming for the Chronos is done in low level C, which mightslow down the development process. One of the advantages of thisdevice is that it is already available for the research; it does not needto be bought. Price: e38,-

ez430-rf2560 The EZ430 is not really a heartbeat monitor, morelike a device to connect the hardware in the Chronos watch to. Thisdevice allows the EZ430 to connect to Bluetooth devices. In that case,the heart rate monitor will connect to the Chronos watch. This watchwill then connect to the RF2560 unit, which will send the measure-ments over Bluetooth to the connected devices. There was some in-formation written about using this device in combination with heartrate monitors, however this was not very elaborate Price: e120,-

smarthrm SmartHRM is another heart rate monitor. There doesnot seem to be much information about this HRM. There exist someapplications for iPhone, but it is not clear whether developers canwrite their own applications using their SDK. Actually, the HRM isonly supported by the iPhone, and is not useable on Android devices.Price: e95,-

zephyr bioharness bt The Zephyr BioHarness BT is the moreelaborated version of previously described Zephyr HxM. This HRMdelivers not only the R-R intervals, but can provide, according toZephyr, a ’medical grade ECG’, which is useful for this project. Be-sides measuring heart rate, it can also be used to measure the breath-ing patterns of a person, using the built in breathing sensor. The de-vice also provides an internal storage, so measurements can be loggedand stored on the device itself. Measurements are sent via a Bluetoothconnection to an Android phone or computer. The SDK is availableand included in the price. It consists of the SDK, example projectsand documentation of the device itself. Price: e500,-

vuams The VU University Ambulatory Monitoring System (VU-AMS) is used for measuring heart rate. The VU-AMS has been de-veloped by the Vrije Universiteit Amsterdam to assist many researchgroups to study stress and emotion in both laboratory and naturalis-tic settings. Fig x demonstrates what the VU-AMS looks like duringuse. The current version (VU-AMS5fs) can be used to measure:

• Heart Rate / Inter beat Interval (IBI)

• Heart Rate Variability (SDNN, RMSSD, IBI power spectrum: HF,LF)

• Respiratory Sinus Arrhythmia (RSA)

sensors 63

• Pre-Ejection Period (PEP)

• Left Ventricular Ejection Time (LVET)

• Respiration Rate (RR)

• Stroke Volume (SV) and Cardiac Output (CO)

• Skin Conductance Level (SCL) and Skin Conductance Responses(SCRs)

• Tri-Axial Accelerometry (Body Movement)

Because it uses multiple electrodes to capture the ECG the signalis very reliable, this in contrast to other devices using a maximum of2 electrodes. This is all very promising, the only disadvantage of thissystem is that it currently does not allow Bluetooth communicationwhich makes it not the right hardware at this moment. According tothe VU Amsterdam Bluetooth will be available in future versions ofthe VU-AMS.

smart shirt Sensors come in many forms and shapes, the smartshirt is an example of the possibilities of today’s technologies. E-textiles, a smart fabric, enables digital components and electronicsto be imbedded in them.

B I B L I O G R A P H Y

[1] Data logger and telemetry physiology monitoringsystem - bioharness. URL http://www.biopac.com/

bioharness-data-logger-telemetry-system-acqknowledge.

[2] ilab. URL http://www.ilab-psychiatrie.nl.

[3] Qrs complex - figure. URL http://en.wikipedia.org/wiki/

QRS_complex.

[4] Flexible sensors. URL http://rogers.matse.illinois.edu/

index.php.

[5] A.S AlMejrad. Human emotions detection using brain wave sig-nals: A challenging. 2010.

[6] X. Sanchezc P.Philippotb A.Schaefera, F.Nilsb. Assessing the ef-fectiveness of a large database of emotion-eliciting films: A newtool for emotion researchers. 2010.

[7] K.Hugdahl B.H.Johnsen, J.F.Thayer. Affective judgment of theekman faces: A dimensional approach. 1995.

[8] Linda J. Luecken Bradley M. Appelhans. Heart rate variabilityas an index of regulated emotional responding. 2006.

[9] D.Cisamolo B.Rime, P.Philippot. Social schemata of peripheralchanges in emotion. 1990.

[10] Yulia E. Chentsova-Dutton. Gender differences in emotional re-sponse among european americans and hmong americans. 2007.

[11] Graham Clarke Enrique Leon and Victor Callaghan. Towards arobust real-time emotion detection system for intelligent build-ings. 2005.

[12] R.Zuidhof F. Blaauw. Detection of emotional arousal. 2012.

[13] G.Stemmler. The autonomic differentiation of emotions revisited:Convergent and discriminant validation. 1989.

[14] SPSS inc. Spss for windows, 2011.

[15] JL. Nandrino M. Jeanne R. Logier. J. De jonckheere, D. Rommel.Heart rate variability analysis as an index of emotion regulationprocesses: Interest of the analgesia nociception index (ani). 2012.

[16] G.A.Mendelsohn J.A.Russell, A.Weiss. Affect grid: A single-itemscale of pleasure and arousal. 1989.

65

http://www.biopac.com/bioharness-data-logger-telemetry-system-acqknowledge

http://www.biopac.com/bioharness-data-logger-telemetry-system-acqknowledge

http://www.ilab-psychiatrie.nl

http://en.wikipedia.org/wiki/QRS_complex

http://en.wikipedia.org/wiki/QRS_complex

http://rogers.matse.illinois.edu/index.php

http://rogers.matse.illinois.edu/index.php

66 Bibliography

[17] J.Papillo and D. Shapiro. Principles of psychophysiology. Syndicateof the University of Cambridge, Cambridge, UK, 1990.

[18] P.R.Kleinginna Jr and A.M.Kleinginna. A categorized list of emo-tion definitions, with suggestions for a consensual definition.1981.

[19] J.W.Pennebaker. The psychology of physical symptoms. 1982.

[20] Huang X Li X Laukka S PietikÃ€inen M SeppÃ€nen T Korte-lainen J, Tiinanen S. Multimodal emotion recognition by combin-ing physiological signals and facial expressions: a preliminarystudy. 2012.

[21] I.Sugimoto K.Takahashi. Remarks on emotion recognition frombreath gas information. 2009.

[22] Peter J. Lang Margaret M. Bradley. Measuring emotion: the self-assessment manikin and the semantic differential. 1994.

[23] M.E.Newel. The connection between emotion, brain lateraliza-tion, and heart rate variability. 2005.

[24] Wilhelm FH Zhou E Conrad A Ritz T Roth WT Meuret AE,Rosenfield D. Do unexpected panic attacks occur spontaneously?2011.

[25] Friedrich Foerster Michael Myrtek. On-line measurement of ad-ditional heart rate. methodology and applications.

[26] J.De Jonckheere B.Tavernier M.Jeanne, R.Logier. Validation of agraphic measurement of heart rate variability to assess analgesi-a/nociception balance during general anesthesia. 2009.

[27] M.Kusserow. Stress arousal monitoring in natural environments.2012.

[28] M.Myrtek. Gesunde kranke - gesunde. 1998.

[29] C.Scharff M.Myrtek. Fernsehen, schule und verhalten. unter-suchungen zur emotionalen beanspruchung von schulern. 2000.

[30] G.Brugner M.Myrtek. Perception of emotions in everyday life:studies with patients and normals. 1996.

[31] G.Brugner M.Myrtek, E.Aschenbrenner. Emotions in everydaylife: an ambulatory monitoring study with female students. 2005.

[32] H.Strohmaier W.Zimmermann S.Lawerenz G.Brugner W.MullerM.Myrtek, E.Deutschmann-Janicke. Physical, mental, emotional,and subjective workload components in engine drivers. 1994.

Bibliography 67

[33] Foerster F. Myrtek, M. On-line measurement of additional heartrate: methodology and applications. 2001.

[34] K.Sato R.Wiers N.H.Frijda, S.Markam. Everyday conceptions ofemotion: An introduction to the psychology, anthropology andlinguistics of emotion. 1995.

[35] M.Bayard M.Couvreur M.Staroswieck M.Delecroix BertrandR. Logier, A.L.Gehin. Detection of transient myocardial ischemiaby numerical processing of the electrocardiographique signal.1993.

[36] A.Dassonneville R.Logier, J.De Jonckheere. An efficient algo-rithm for r-r intervals series filtering. 2004.

[37] J.De jonckheere A.Dassonneville M.Delecroix B.TavernierR.Logier, M.Jeanne. Physiodoloris: a monitoring device foranalgesia / nociception balance evaluation using heart ratevariability analysis. 2011.

[38] Sungbok Lee Samuel Kim, Panayiotis G. Georgiou and ShrikanthNarayanan. Real-time emotion detection system using speech:Multi-modal fusion of different timescale features.

[39] Klaus R. Scherer. What are emotions? and how can they be mea-sured? 2005.

[40] James McNames Mateo Aboy Brahm Goldstein Tran Thong, Ke-hai Li. Accuracy of ultra-short heart rate variability measures.2003.

[41] K.R.Scherer V.Sacharin, K.Schlegel. Geneva emotion wheel rat-ing study. 2012.

[42] Michael S. Lew Theo Gevers Yafei Sun, Nicu Sebe. Authenticemotion detection in real-time video.

Emotional Arousal Detection - RUG Math & CS: …aiellom/tesi/zuidhof.pdfan ambulatory real-time...

Documents

Transcript of Emotional Arousal Detection - RUG Math & CS: …aiellom/tesi/zuidhof.pdfan ambulatory real-time...