open questions

Eva Hudlicka *

Psychometrix Associates, Inc., 180

Received 18 March 2013; received

Computational models of cognitive and affective processes provide a unique and powerfulmeans of refining psychological theories, and can help elucidate the mechanisms that mediate

logically-inspired cognitive architectures. These include the following: the extent to which

processes, and the computational tasks necessary for their implementation; improved under-

The challenges associated with data availability and model validation are also discussed. 2013 Elsevier B.V. All rights reserved.

In spite of the progress in emotion research over the past

some underlying circuitry has been identified for some as-pects of affective processing (e.g., amygdala-mediated pro-cessing of threatening stimuli, the role of orbitofrontal

remainsputation

of cognitive and affective processes provide a unique and

for helping to identify the underlying mechanisms thatmediate affective phenomena.

The development of affective computational modelswithin agent architectures has two additional benefits: (1)developing emotion models within an integrated agentarchitecture provides more realistic constraints on emotion

2212-683X/$ - see front matter 2013 Elsevier B.V. All rights reserved.http://dx.doi.org/10.1016/j.bica.2013.11.002

* Tel.: +1 413 207 7066.E-mail address: hudlicka@ieee.org.

Biologically Inspired Cognitive Architectures (2014) 7, 98125

Avai lab le at www.sc iencedi rect .com

ScienceDirect

.e20 years, emotions remain an elusive phenomenon. While powerful method for refining psychological theories, andIntroduction cortex in emotion regulation), muchabout the mechanisms of emotions. Comunknownal modelsstanding of affective dynamics and development of more accurate models of these phenomena;and understanding the alternative means of integrating emotions within agent architectures.mechanisms in biological affective agents should be simulated or emulated in affective BICAs;importance of more precise, design-based terminology; identification of fundamental affectiveaffective phenomena. This paper outlines a number of open questions and challenges associ-ated with developing computational models of emotion, and with their integration within bio-KEYWORDSEmotion modeling;Emotion theories;Cognitiveaffectivearchitecturest5 Azalea Dr., Blacksburg, VA 24060, USA

in revised form 4 November 2013; accepted 4 November 2013

Abstract

In spite of the progress in emotion research over the past 20 years, emotions remain an elusivephenomenon. While some underlying circuitry has been identified for some aspects of affectiveprocessing (e.g., amygdala-mediated processing of threatening stimuli, the role of orbitofron-al cortex in emotion regulation), much remains unknown about the mechanisms of emotions.REVIEW ARTICLE

Affective BICA: Challenges

journa l homepage: wwwand

l sev ier .com/ locate /b ica

modeling than stand-alone emotion models, and (2) theability of the associated agent to interact with its environ-

9. Validating emotion models and cognitiveaffectivearchitectures. (A distinction is made here between

Affective BICA: Challenges and open questions 99ment (real or simulated, physical and/or social) providesrich opportunities for exploring the benefits of affectiveprocesses for adaptive behavior, as well as the potentialproblems associated with inappropriate or unregulatedaffective states. The development of affective BICAs thushas the potential to both help characterize the mechanismsmediating affective processing in biological agents, and toexplore the benefits, and the potential drawbacks, of emo-tions in synthetic agents.

The aim of this paper is to discuss some of the openquestions and challenges associated with developing com-putational models of emotion, and with their integrationwithin biologically-inspired cognitive architectures. Thepaper is organized as follows. First, I consider the broadissue of whether emotions improve an agents effective-ness. In this context I first distinguish between researchand applied models, and then discuss the roles and func-tions of emotions in biological agents, and functions ofaffect-like processes in synthetic agents. The next sec-tion, titled Open Questions and Challenges, formsthe core of this paper and addresses the issues listed be-low, as they relate to the development of affectiveBICAs:

1. Which features of affective processing in biologicalagents should be adopted by the designers of affectiveBICAs? (A number of specific topics are addressed here,including: multiple modalities of emotions; role of per-ception-based mechanisms in mediating affective pro-cesses; varying time scales and resolution levels ofaffective processes; dedicated circuitry for specifictypes of stimuli (e.g., self-relevant threats); and thedegree to which affective memory facilitates or hindersadaptation.)

2. Establishment of a more constructive, design-based, tax-onomy of affective terms.

3. Identification of fundamental affective processes andstructures. (In this context, the dominant theoreticalperspectives on emotions are briefly summarized, and aset of generic computational tasks mediating emotiongeneration and emotion effects is proposed.)

4. Modeling of complex emotions and affective phenomena,that go beyond the cognitive appraisal processing andaffective expression synthesis currently addressed bymost models.

5. Modeling affective dynamics and addressing the chal-lenges associated with modeling multiple, interactingemotions and affective states.

6. Integrating emotions into agent architectures. (This sec-tion also addresses the important issue of an affectivedesign space and existing architecture templates, mostnotably the 3-level architecture model proposed bynumerous researchers, and provides examples of existingcognitiveaffective architectures.)

7. Constructing architecture structures and memories, viaknowledge engineering and learning approaches.

8. Identifying the empirical data necessary to construct bio-logically-faithful models.model validation (with respect to a biological mecha-nism) and system evaluation (with respect to particularsystem-specific performance criteria).)

The paper then concludes with a summary and conclu-sions, the latter further underscoring the important rolethat affective BICAs promise to play in our improved under-standing of affective mechanisms, in terms of their compu-tational characteristics and requirements.

Terminological notes: the terms agent and agent archi-tectures are used interchangeably; the term agent can referboth to a robotic agent interacting with a physical environ-ment and other agents (robots or humans), and to a syn-thetic virtual agent interacting with a simulatedenvironment. The term cognitiveaffective architecture isthe preferred term over affective architecture, since cogni-tion is necessary for many affective processes (e.g., cogni-tive appraisal), and since the term underscores the factthat cognitive and affective processes should not be consid-ered as distinct. (The term cognitive appraisal refers tothe interpretive processes involved in emotion generation,which take into account an agents goals and beliefs, andtheir influence on the subjective interpretation of particularaffect-triggering stimuli.)

Do emotions improve an agents effectiveness?

This fundamental question is often posed regarding affec-tive architectures, especially by affective skeptics. How-ever, when framed at such a generic level, it is impossibleto provide a meaningful answer. Clearly, the answer de-pends on many specific factors regarding the emotionsthemselves (which emotions, at what intensity, and underwhat circumstances), the context within which the agentoperates, and the agents ultimate purpose. Whether emo-tions should be integrated within an agent architecture,which emotions (or other affective states), and how theintegration ought to be implemented, therefore dependson the architectures (and the agents) objectives. Below Idistinguish between two broad categories of models andarchitectures, as a function of their primary objective.

Research vs. applied models

In considering the utility of integrating emotions in agentarchitectures a fundamental distinction should first be madebetween architectures and models built to characterize themechanisms mediating affective phenomena in biologicalagents, and those built to enhance a particular aspect ofagents functioning; e.g., more effective behavior in anuncertain environment, more believable behavior in itsinteractions with humans. I refer to the former as researchmodels, and the latter as applied models.

In the former case emotions are, by definition, essential,and the aim ought to be to emulate their roles and mecha-nisms as closely as possible, to maintain fidelity with theirbiological analogs. In the latter case there is more freedom

regarding the integration of emotions, and fewer con- and dangerous, although one rarely hears the cognitive ana-

100 E. Hudlickastraints about which aspects of emotions are modeled,and how, as long as the particular agent objectives aremet; i.e., the affective mechanisms make the agent moreeffective across a broader range of environments, morebelievable to human users, etc. While research modelsaim to emulate at least some aspects of affective process-ing in biological agents, to clarify theories and attempt tocharacterize the underlying biological mechanisms, appliedmodels do not have this constraint, and a variety of methodscan be used to simulate the affective mechanisms necessaryfor the agents specific objectives.

Emotions in biological agents

In more complex biological agents emotions are indispens-able, by definition, since the affective neurophysiologicalcircuitry evolved in conjunction with, or, in some cases,prior to, the cognitive circuitry, and the two systems cannottherefore be easily de-coupled. Both systems evolved tofacilitate adaptive behavior. This is not to say that emotionsare always adaptive (see discussion below regarding patho-logical or maladaptive manifestations of emotions, as wellas a thoughtful treatment of this issue by Fellous (2004)).

Emotions in biological agents mediate a number of criti-cal roles, both intrapsychic and interpersonal (e.g., Leven-son, 1994; Oatley & Johnson-Laird, 1987; Rolls, 2007;Simon, 1967). The intrapsychic roles include goal manage-ment and goal selection, resource allocation (e.g., assigningattentional priorities), global alarm mechanism, subsystemcoordination, and resolution of control dilemmas in systemsthat are sufficiently complex to experience them (Gray,Schaefer, Braver, & Most, 2005). Interpersonal roles includecommunication and coordination, as well as attachmentbehavior, mediated by emotion expression, primarily viathe face and voice qualities, but also via body posture andgestures, the quality of movements, and the rapid, accu-rate, and often unconscious perception of these states. Aninteresting phenomenon that has recently received muchattention is emotion contagion (Hatfield, Cacioppo, & Rap-son, 1993), whereby an emotion felt by one agent can berapidly spread and felt by other agents, via the activationof affective circuitry in the perceiver, in response to an ob-served affective expression. (Emotion is of course also com-municated by specific selected actions (e.g., flee, freeze,fight).)

The neural circuitry mediating affective processing isclosely integrated with the circuitry mediating cognitiveprocessing. Historically, affective and cognitive processeshave been considered as distinct, spurring the now obsoletedebates regarding which is more or less critical for a partic-ular function. However, recent research in the affectiveneurosciences is forcing us to reconsider this distinction(Phelps & LeDoux, 2005), as data are emerging regardingthe close coupling of these systems in the brain, and theuse of shared neural circuitry for both cognitive and affec-tive functions (Gray et al., 2005).

In spite of the critical survival roles mediated by emo-tions, emotions in biological agents can, and often do, causeseverely maladaptive processing and behavior. (Note, how-ever, that cognitive processing can be just as maladaptivelog of the familiar dismissive putdown Dont be so emo-tional!). The most obvious of these are affect-induceddysregulation and maladaptive behavior seen in the overtlymentally ill. Less dramatic and obvious, but potentiallymore dangerous, are maladaptive emotion-induced effectson cognition and behavior. These can cause both intrapsy-chic and interpersonal distress, can contribute to a rangeof perceptual and cognitive distortions and biases, andcan result in a variety of social catastrophes; e.g., persecu-tions, ethnic violence, and manipulations of entire nationsinto distorted patterns of thought and behavior. Histori-cally, these maladaptive affect-induced or affect-manifest-ing behaviors have contributed to the, unfortunately, stillpersistent dichotomy describing human behavior as emo-tional vs. rational.

Affective processing in synthetic agents

The frequently asked question regarding whether machinesneed emotions may not be the most useful means of explor-ing this complex issue. It is unlikely that a question posed atsuch a high level of abstraction will generate useful an-swers. Sloman and colleagues have repeatedly pointed outthis problem (Scheutz, Sloman, & Logan, 2000) and haveemphasized the need to refine our questions regarding therole and nature of emotions in agents, biological or syn-thetic, and ask more specific questions, posed in terms ofinformation processing requirements and architecture com-ponents. The question of whether an organism or a robotneeds emotions or needs emotions of a certain type reducesto the question of what sort of information-processingarchitecture it has and what needs arise within such anarchitecture (Sloman, Chrisley, & Scheutz, 2005). This inturn depends on the types of environments within whichthe agent functions, and on the agents objectives.

In contrast to their integral roles in biological agents,emotions are not essential in synthetic agents. However,to behave effectively in complex and uncertain environ-ments, and to interact effectively with humans, syntheticagents need to implement at least some of the roles thatemotions play in biological agents. (Whether or not suchmechanisms should be referred to as affective in non-bio-logical agents is a deeper philosophical issue.) To be believ-able and effective in their interactions with humans,synthetic agents may also need to be able to display statesthat humans can recognize as particular emotions. (Notethat no claim is being made here that synthetic agents actu-ally feel emotions in a way that is similar to felt states inbiological agents.)

Below I provide several examples of circumstances wheresynthetic agents might benefit from affective, or affect-like, mechanisms, such as those that mediate adaptivebehavior in biological agents. I also briefly discuss the po-tential benefits of modeling maladaptive affective states.

Intrapsychic roles of emotions

A number of researchers from different disciplines have ob-served that an adaptive agent operating with limited re-sources in a complex, uncertain and dynamic environmentwill require mechanisms to implement many of the intrapsy-

chic roles of emotions: resource allocation in situations that 2009) suggest that in biological agents the same neural cir-

Affective BICA: Challenges and open questions 101necessitate triaging of available resources; alarm mecha-nisms to signal a need for a change in focus: interruptionor termination of a task and initiation of another task, andgoal re-prioritization; rapid, undifferentiated processing toaddress potential emergencies vs. slower, more complexprocessing that can be brought on-line later, once the emer-gency has passed; and mechanisms that rapidly coordinatemultiple subsystems (e.g., subsystems mediating percep-tion, decision-making and planning, action selection, andbehavior monitoring) (Damasio, 1994; Frijda, 1986; Oatley& Johnson-Laird, 1987; Ortony, Norman, & Revelle, 2005).

Agent architectures have been developed to explorealternative means of implementing the roles outlinedabove, and to evaluate the resulting effects on theagents performance. For example, Scheutz comparedthe use of an affective state-based control of behaviorwith non-affective control in agents attempting to navi-gate hostile environments, and determined that the pres-ence of even simple affective states improvesperformance and enhances survivability in certain environ-mental contexts (Scheutz, 2000).

While not cast as an affective mechanism, researchers inmulti-agent systems and distributed AI have also addressedthe issue of control and conflict resolution (e.g., Raja & Les-ser, 2007). Although this work focuses on the role of meta-cognitive processes, rather than affective processes per se,it addresses similar control dilemmas as those addressed byaffective systems, and also raises the important issueregarding when such control mechanisms may be needed,in terms of the characteristics of the environment (com-plexity, uncertainty, rate of temporal change), and as suchindirectly addresses the issue of when affective mechanismsmay be needed.

Interpersonal roles of emotions

To interact effectively with humans, synthetic agents willneed to implement some of the interpersonal roles of emo-tions; e.g., displaying states that are recognized as emo-tions by humans, and recognizing human emotions. Asrobots and synthetic agents begin to play more complexand important roles in our lives, for example as coachesand tutors, healthcare assistants, and perhaps even socialcompanions (Payr, 2011), demands will increase for addi-tional interpersonal roles, such as empathy and the abilityto form attachments. These agents are referred to as socialagents (or social robots) and increasingly the term rela-tional agents is being used for agents capable of empathi-cally engaging with their human users over longer periodsof time (Bickmore, 2003).

Clearly, both the expression of affect-like states, andtheir recognition, can be implemented via a range of shal-low, black-box models. An interesting aspect of consider-ing these roles in affective BICAs is the degree to whichimplementing such mechanisms requires, or benefits from,deeper models of affective processing. Recent resurgenceof interest in the embodied aspects of emotion has trig-gered an exploration of some candidates for deeper mecha-nisms, integrated within an agent architecture, which maymediate some of the interpersonal roles of emotions. Forexample, Niedenthal and colleagues (Niedenthal & Marcus,cuitry mediates both the experience and recognition of theagents own emotion, and the recognition of anotheragents emotion.

Modeling maladaptive affective processing

Aristotle made the following observation regarding anger:Anyone can become angry, that is easy, but to be angrywith the right person, at the right time, for the right rea-son and in the right way, that is not within everyonespower and that is not easy. The same observation appliesto emotions in general. The challenge associated with aug-menting cognitive architectures with emotion is to achieveAristotles aim; unless, of course, we are aiming to modelthe maladaptive aspects of affective processing, to betterunderstand the etiology, maintenance and treatment ofaffective disorders, and maladaptive affect-induced behav-ior. Models that explicitly focus on modeling the maladap-tive aspects of affective processing can enhance ourunderstanding of psychopathology in terms of dysregulationof the underlying affective circuitry. For example, the MA-MID cognitiveaffective architecture (Hudlicka, 2003,2007, 2008) was used to model multiple types of anxietystates, ranging from an adaptive, protective vigilance to adebilitating panic attack, in terms of the underlying pro-cesses mediating affective biases on cognition. The compu-tational model was also able to generate alternativehypotheses regarding the specific mechanisms mediatingthe observed effects.

But are they really emotions?

Whether or not the roles of emotions outlined above shouldbe implemented by mechanisms that resemble emotions inbiological agents, and whether the processes implementingthese functions in synthetic agents ought to be referred toas emotions, remain open questions. Importantly, we shouldalso be wary of the often unstated assumption that emo-tional states themselves play causal roles: Sloman (2004)points out that saying that states of type X can occur asa side-effect of the operation of some mechanism M thatis required for intelligence does not imply that that statesof type X are themselves required for intelligence.

The development and evaluation of affective BICAs, forboth research and applied purposes, situated in a varietyof environments, both physical and virtual, will help addressthese important questions regarding emotions in a more sys-tematic manner.

Open questions and challenges

Below I discuss some of the open questions and challengesregarding emotion modeling as it relates to the integrationof emotion models within agent architectures.

On borrowing from biological cognitiveaffectivearchitectures

Adopting and adapting computational strategies from bio-logical agent architectures is the central theme of BICAs.

and that this physiological substrate is vastly different from

102 E. Hudlickathe silicon substrate of BICAs. The consequence of this fun-damental difference is that when we consider whether ornot to adopt a particular biological cognitiveaffectivemechanism, we must attempt to determine to what extentthe nature of that mechanism is a function of the neuro-physiological substrate, and to what extent its associatedrole, or the computation it implements, is necessary forthe synthetic agent. (This issue relates to the varying levelsof analysis of an information processing system, such as thelevels proposed by Marr (1982), and is briefly discussedlater.)

Another consideration regards the distinction betweenresearch and applied models pointed out earlier, with re-search models requiring higher degrees of fidelity with re-spect to biological architectures. To put this in terms ofan overused but apt cliche: while applied models dontneed to flap their wings, the research models cannotignore the mechanisms used to accomplish specific emo-tion roles in biological agents if their aim is to help elu-cidate these mechanisms. The challenge in adaptingbiological affective mechanism to applied affective BICAsis to avoid those mechanisms that may be involved inaffective processing purely due to evolutionary accidents,rather than computational necessity.

Finally, it is essential that affective BICA designers befamiliar with the emerging data from affective neurosci-ence regarding affective mechanisms in the brain. Empiricalresearch demonstrates that there are no dedicated emotioncenters or circuits in the brain (Fellous, 2004). While manylocations in the brain have been identified that processaffective stimuli and mediate affective responses (e.g.,the amygdala), these circuits are also involved in manynon-affective processes. Fellous highlights this in his obser-vation that there is no emotional homunculus in thebrain, and points out the persistent, but false, beliefregarding the location of emotions in various emotion cen-ters of the brain (e.g., left brain, limbic system and amyg-dala) (Fellous, 2004).

Below I discuss examples of biological affective mecha-nisms that can serve as candidates for inclusion in affectiveBICAs. The aim is to cover a broad range of affective pro-cessing, relevant for multiple affective roles and mecha-nisms, including emotion generation, the consequences ofemotions, and emotion recognition in the self and others.

Emotions as multimodal phenomena

Affective processes in biological agents occur across severalmodalities. The following four are typically delineated:physiological (neuroendocrine and muscular; ANS andCNS); expressive and behavioral; cognitive; and subjec-tive/conscious (e.g., Scherer, 1984). Emotions in affectiveBICAs clearly occur within two of these: cognitive (i.e., rep-However, this process should not be applied blindly acrossthe board, with the assumption that anything biologicalis necessarily better or more appropriate for addressing aparticular functional or computational problem. The issueof what we can learn about emotion modeling from biolog-ical agents is made more complex by the fact that affectiveprocessing in animals (including humans) is closely linked tothe physiological substrate within which it is implemented,resentation and reasoning) and, expressive/behavioral(assuming a physical or virtual representation of the associ-ated agent).

The nature of the subjective/conscious modality in syn-thetic agents is problematic. Aside from the open issue ofwhether or not machines can manifest consciousness, andwhat the teleological status of this state would be, the cau-sal status of consciousness in affective processing remainsunclear. Until there is more clarity regarding these statesI believe that attempting to implement this modality ofemotion in affective BICAs is premature, and likely to pro-duce superficial treatment of these complex phenomenathat are unlikely to enhance our understanding (Hudlicka,2009). (Note, however, that machine consciousnessresearchers are beginning to identify system characteristicsthat may be required for some aspects of consciousness toemerge (Tononi, 2008) and there is growing interest in the-oretical explorations of the relationship between emotionand consciousness (e.g., Barrett, Niedenthal, & Winkiel-man, 2005).)

The physiological modality is potentially a more produc-tive source of ideas for modeling, particularly in roboticaffective BICAs, where the robots physical infrastructuremay serve as a direct analog to the physiological modalityin biological agents. For example, arousal, (roughly reflect-ing the degree of activation of the autonomic nervous sys-tem and one of the defining dimensions of affective statesin the dimensional models of emotion), might be modeledin affective BICAs in terms of the robots speed of process-ing and movement. A robot capable of responding more rap-idly due to heightened arousal might then escape dangermore quickly, and help communicate potential danger moreeffectively to other robots engaged in a particular task, forexample, search and rescue.

Research-focused affective BICAs aiming to clarify theinteraction among the physiological and the other modali-ties would require an explicit model of the physiologicalmodality. A more interesting question is whether appliedarchitectures would also benefit from an explicit represen-tation of this modality. An explicit model of arousal in ap-plied affective BICAs could provide a central mechanismwhereby a variety of processes across multiple subsystemscould be rapidly and efficiently coordinated, to ensure thata particular task was performed in an optimal manner. Infact, such global, system-wide coordination is one of theroles of emotion in biological systems, implemented withinthe central nervous system by a variety of neuromodulatorytransmitters, with systemic effects across a number of dis-tributed neural circuits.

A particularly intriguing aspect of modeling the physio-logical modality relates to the notion of embodied emo-tions. There has been a resurgence of interest in theembodied aspects of emotion and an exploration of candi-dates for embodied affective mechanisms that may medi-ate some of the interpersonal roles of emotions (e.g.,Barrett, 2005; Niedenthal & Marcus, 2009). Some research-ers suggest that in biological agents the same neural cir-cuitry mediates the experience and recognition of anagents own emotion, the recognition of another agentsemotion, and, in more complex agents, also reasoning aboutemotion, and that these processes are closely linked to thesomato-sensory and motor representations associated with

the embodied aspects of emotions; e.g., facial expressions,posture (Niedenthal & Marcus, 2009). It is intriguing to spec-ulate how these types of mechanisms might be implementedin affective BICAs, which would then provide a context with-in which hypotheses such as those above, proposed by Nie-denthal, Feldman-Barrett and others, could be explored.

Interestingly, since the experience of the physical body

nisms and causal dependencies. This approach is advocated

Affective BICA: Challenges and open questions 103in biological agents appears to be linked to the consciousexperience of emotion,1 exploration of the physiologicalmodality models in affective BICAs could potentially alsoyield interesting results in modeling the elusive subjec-tive/conscious modality of emotion.

The embodied emotion perspective shares a number offeatures with an analogous trend in thinking about cognitiveprocessing. Researchers attempting to characterize the nat-ure of cognition have, until recently, tended to view cogni-tive processing as the dis-embodied manipulation ofsymbols. This brain-in-a-vat view, emphasizing abstract,amodal processing, decoupled from peripheral sensory in-puts and motor outputs, is now being challenged by theembodied cognition perspective. Embodied cognition theo-ries emphasize the critical role that sensorimotor processingand representations play in higher-level cognitive process-ing (Barsalau, 2008; Wilson, 2002). Associated with this viewis the increased emphasis on the role of modality-specificprocessing (vs. processing of amodal symbolic information),and the view that (aspects of) cognition may be mediated byre-enactment of modality specific memories, rather thanby the manipulations of amodal, abstract, symbolic repre-sentations that have been extracted (transduced) from thelow-level, modality-specific representations.

The notion of embodied emotions represents an analo-gous shift in emotion theories, from an emphasis on the cog-nitive, interpretive modality represented by the cognitiveappraisal theories to an emphasis on the somatic/physiolog-ical modality, represented by embodied emotion theories.The former emphasize the critical role of cognitive evalua-tion of the eliciting stimuli in emotion generation (e.g., OCC(Ortony, Clore, & Collins, 1988; Reisenzein, 2001; Roseman,2001; Scherer, 2001a, 2001b). The latter emphasize thecritical role of the somato-sensory and motor representa-tions associated with emotion; specifically, they suggestthat it is the perception of these states that defines theaffective experience, and also mediates emotion recogni-tion in social contexts, as well as reasoning about emotions(Atkinson & Adolphs, 2005).

Broadening our perspective on emotion to include modal-ities other than cognition is of course a welcome develop-ment. It would, however, be a mistake to replace thepreviously dominant role of cognition with a dominant roleof perception of embodied affective states (Hudlicka,2009). Rather, we should strive for an integrated view,where the roles of all of the multiple modalities comprisingemotion are considered, and the relationships among themexplored, with an emphasis on identifying feedback mecha-

1 There is empirical evidence that neural structures that mediatebody perception are active during emotional episodes (e.g.,anterior cingulate cortex, insula and somatosensory cortices)(Niedenthal & Marcus, 2009), although the causal status of theseactivations with respect to a conscious, felt affective state remainsto be determined.in the componential models of emotion (e.g., Scherer).

A single perception-based mechanism mediating multipleaffective processes?

Related to the ideas of embodied cognition and emotion,and the hypothesis that representations of bodily statesmediate the recognition of own and others emotions, isthe possibility of using a single, perception-based mecha-nisms to mediate multiple affective functions. In biologicalagents the perception of the agents physiological state,and its expressive and behavior activities, appear to play arole in the recognition of ones own emotion. The formeris related to the James-Lange feeling theories of emotion,which reverse the intuitively accepted causal sequence offeeling and behavior during an emotional episode, and sug-gest that we feel fear because we run. The latter is re-lated to the facial feedback theories of emotion (Ekman,2007; Laird & Strout, 2007), which suggest that we canexperience, or at least enhance, an emotion simply by con-sciously configuring our facial muscles into expressions char-acteristic of that emotion. (Feeling theories have acontroversial status in emotion research, having been dis-missed due to Cannons criticism, but appear to be return-ing in the guise of embodied emotion theories.)

In affective BICAs such a perception-based mechanismscould be implemented by providing proprioceptive sensorswithin the robot architecture, and mapping specific config-urations of their values onto distinct emotional states. Themechanism could potentially be extended to recognizeother agents emotions, possibly via some analog of themimicry functions implemented by mirror neurons.

Varying time scales of affective processes

Also related to the multi-modal nature of emotion are thevarying time scales and rates of change of the distinctcomponents of an affective process. In biological agents,processing across the distinct modalities occurs at differ-ent speeds (Scherer, 2000). Perhaps the most dramaticand familiar example of this phenomenon is the experi-ence of feeling our sympathetic systems activation (rapidbreathing and heart rate, increased perspiration) longafter some real or imagined danger has passed. Once acti-vated, the neuroendocrine components of the sympa-thetic system require a certain amount of time toreturn to a baseline level of activation, because the medi-ating elements (e.g., hormones) need time to dissipate.The critical question regarding the modeling of thesevarying time scales is whether this aspect of affectiveprocessing is a bug or a feature in biologicalagents. Implementation of these processes in affective BI-CAs would support a systematic exploration of the draw-backs and benefits of these multiple time scales of thedistinct components of affective processes.

Varying resolution of processing

Related to the multiple times scales is the possibility ofimplementing a range of processes with varying levels ofthe speed vs. accuracy tradeoff. In biological agents affec-tive phenomena are mediated by multiple neural pathways,with distinct characteristics. For example, the dual-pro-

circuits could be developed, each implementing the rapid

104 E. Hudlickacessing theory of emotions emphasizes a distinction be-tween subcortical and cortical pathways mediating differ-ent aspects of affective processing (LeDoux, 1996).Subcortical pathways mediate the fast, unconscious, high-capacity processing, associated with rapid, often hard-wired responses, but also capable of classical conditioning(Rolls, 2007). Cortical pathways mediate the slower, moredeliberate analysis of the triggering stimuli, and more com-plex, flexible behavioral responses. The two pathways func-tion in parallel, with complex interactions and feedbackconnections, with the unconscious components of emotionrepresenting the bulk of the iceberg of affective process-ing (Scherer, 2005).

The rapid processing within the subcortical pathways ismediated by structures residing in the limbic system,including the amygdala, as well as nuclei in the brain-stem, and is closely linked to the activation of the auto-nomic nervous system, which controls arousal (Lewis &Todd, 2005; Panskepp, 1998; Phelps & LeDoux, 2005).Much of the processing along this pathway is thus auto-matic, often relying on hardwired circuitry that imple-ments the detection of stimuli critical for survival, andassociated adaptive behavioral responses that have provedto be beneficial for survival. The innate circuitry performsboth the evaluation of the current situation, and thepreparation for, and coordinated execution of, appropri-ate behavior. These pathways are associated with severalcharacteristic features of emotions, including a sense of alack of voluntary control, a rapid onset, a characteristicdistinct subjective feeling, and the urge to act (Ekman,1994).

The cortical pathways involve additional processing with-in several cortical regions, including the orbitofrontal cor-tex and the anterior cingulate cortex, and link thethalamus with the sensory and prefrontal cortices, whichproject back to the amygdala and to the motor cortex (Le-wis & Todd, 2005). The involvement of the pre-frontal cor-tex in these pathways is associated with the use of workingmemory and its slower, limited-capacity processing. Thecortical pathways are also associated with conscious aware-ness of the emotion, and a more elaborate analysis of thetriggering stimuli, consequences for well-being, deliberateplans for action, possible re-evaluation and re-appraisal, ex-plicit representation of the self, and a higher degree of flex-ibility in general.

It is readily apparent that a dual processingmodel, such as the one outlined above, would have ben-efits for affective BICAs, enabling the agent to be moresuccessful in a dynamic and dangerous environment, per-haps at the expense of behaving in an overly risk-aversemanner; i.e., reacting to false positives and therebyexpending resources for non-essential activities. Imple-mentation of multiple parallel pathways would also re-quire a need for conflict resolution mechanisms insituations where the conclusions of the two systems werecontradictory.

It is interesting to speculate about possible extensions ofthe dual processing model to multiple processes, located atvarious points on the time vs. accuracy tradeoff continuum,along multiple dimensions of the feature space of theagents environment. Affective BICAs provide a computa-tional context within which such explorations are possible.processing of particular types of cues, including internalcues, and possibly complex internal states; e.g., specificcontrol states or conflict states, defined as a function ofthe type of tasks and environments within which the agentwas operating, as well as the possible states of the agentsarchitecture.

Remembrance of things past

An interesting aspect of amygdala-mediated processing ofthreats, and its response to a dangerous experience, is thatthe synaptic structure of the amygdala appears to changepermanently as a result of a processed traumatic experi-ence (LeDoux, Romanski, & Xagoraris, 1991). This is clearlybeneficial for survival, at least in some types of environ-ments, but it can have unintended and unfortunate conse-quences in biological agents, as evidenced by the long-term problems experienced by trauma survivors. Interest-ingly, curing post-traumatic stress disorders does not ap-pear to eradicate the memory traces in the amygdala.Instead, it appears to involve the development of new path-ways, originating in the cortex, that modulate the amygdalaactivation and thereby reduce its trauma-enhanced reactiv-ity (Shin, Rauch, & Pitman, 2006).

If affective BICAs were to follow the amygdala model ofprocessing an important design decision for the modelerwould regard the establishment of the threshold for eradi-cating a danger-induced memory vs. maintaining it to facil-itate more rapid processing next time the same danger wasencountered. Such thresholds would necessarily need to beempirically determined, and would depend on the nature ofthe agents tasks and environment. In an environmentwhere a particular dangerous event is unlikely to occuragain, permanent storage of its detection and processingDedicated circuitry mediating detection of self-relevantthreats

The rapid processing described above is possible in biologi-cal agents because of dedicated circuitry that mediates spe-cific functions; e.g., the amygdala nuclei, located withinthe subcortical limbic system, mediate the processing ofthreatening stimuli, help direct attention and perceptionto self-relevant stimuli, particularly those associated witha potential threat, and play a role in modulating the encod-ing and retention of emotional events.

In biological agents there are many examples of neuralcircuitry dedicated to specific functions required for affec-tive processing. At a level of organization below that of spe-cific brain structures (e.g., the amygdala) there arefeature detectors dedicated to fast perception of spe-cific types of cues expressing emotions in other agents.For example, there appear to exist both conscious andunconscious processes that mediate the perception of emo-tion from visual cues, primarily facial expressions, mediatedin part by neural circuitry that specializes in the detectionof dynamic stimuli, to facilitate the detection of affectiveexpression of others, via both facial expressions and bodymovements (Atkinson & Adolphs, 2005).

In adapting these biological strategies to affective BICAswe would not need to be limited to a single dedicated cir-cuit processing a potentially life-threatening event.Depending on the agents environment, a number of such

tion mixes up motivations, attitudes, moods, and other

Affective BICA: Challenges and open questions 105affective states and processes, and is therefore is too dee-ply flawed to be a useful component of scientific questionsand theories (Sloman, 2004, p. 1) and emphasizes theneed for defining mental states, processes and functions[] within a generative theory of types of information pro-would seem to be counterproductive, leading to the types offalse-positive reactions characterizing post-traumatic stressdisorders. Again, affective BICAs provide a computationalcontext within which the possible design spaces implement-ing such alternatives can be explored.

Establishing a more constructive terminology

The term emotion is often used rather loosely in the AI andcognitive science community, to refer to a wide range ofaffective states, characterized by varying degrees of com-plexity, temporal patterns, and modalities (Hudlicka, 2009).

Contributing to the terminological confusion is the broadapplication by some researchers of the terms emotion oraffective to a range of mental and physiological statesthat are either mixed affectivecognitive states (e.g., con-fusion), or even states that are not affective at all (e.g., fa-tigue.) A number of emotion researchers have addressed theproblems associated with imprecise affective terminology.Averill warns against the use of the term feeling as synon-ymous with emotion, pointing out that feelings are nei-ther necessary nor sufficient conditions for being in anemotional state (Averill, 1994)), since we can feel hungry,tired or confused none of which refer to an emotionalstate. Sloman cautions against overly-inclusive definitionsof particular emotions: We should not put both a generalpreference for saving effort, and fear produced by a stam-peding herd, in the same conceptual basket when they haveso very many differences, including the (relative) perma-nence of the first and the transience of the second (Slo-man, 2004).

Among researchers focusing on affective phenomenathere appears to be at least some terminological agree-ment; e.g., the term emotion is typically reserved for briefaffective episodes, lasting between seconds and minutes,characterized by a coherent expressive and behavioral dis-play associated with the specific emotion. In contrast, theterm mood refers to an affective tone that lasts for muchlonger periods (hours to days to months). However, beyondthis high level consensus there remains a considerable lackof clarity regarding what the term emotion actually refersto in biological organisms, and whether such a highly-aggre-gated construct is useful. Many emotion researchers pointout that the term does not refer to a uniform entity; e.g.,Emotion is too broad a class of events to be a single scien-tific category (Russell & Barrett, 1999, p. 805). Taking aneven more extreme position, some emotion researcherseven question whether emotions are natural kinds (Bar-rett, 2005).

One of the most articulate critics of this terminologicalproblem is Sloman, who has been advocating a computation-ally driven deconstruction of highly-aggregated constructssuch as emotion, and promoting design-based definitionsof mental states in general (Sloman & Croucher, 1981; Slo-man, 2004). He notes that our everyday concept of emo-cessing architectures for many types of organisms or ma-chines (Sloman, 2012).

Generative theories should provide fundamental, well-defined and computationally grounded concepts that canthen be used to construct more complex phenomena. Thismeans defining emotions and affective processes in termsof specific architecture modules, structures and processes,and the information flow among them, both data and con-trol. Sloman (2012) uses the periodic table of elements asan example of this approach in a different discipline, andlaconically comments on the current level of understandingreflected in our affective terminology by observing thatemotion, mood, desire, belief, anger, etc. are onlyslightly better than earth, air, fire, and water.

A key challenge in understanding emotions, and deter-mining how and when to include them in agent architec-tures, is therefore the development of a more precise,consistent, and computationally-grounded vocabulary,where the distinct terms are linked to specific architecturalelements and processes, and the relationships among them.Ideally, this endeavor would involve cross-disciplinary col-laborations among computer scientists, psychologists andneuroscientists. The next section discusses some prelimin-ary attempts in this direction.

Identifying fundamental affective processes andstructures

In-depth understanding of a phenomenon involves an identi-fication of underlying principles or components; e.g., taxo-nomies of organisms and fundamental cellular processessuch as DNA replication or protein construction in the bio-logical sciences; fundamental particles in physics, and theperiodic table in chemistry. It is therefore natural to askwhether analogous underlying fundamental affective mech-anisms or structures can be identified in biological cogni-tiveaffective architectures. These could then be used tosupport more systematic development of emotion modelsand affective BICAs. Several preliminary candidates of suchfundamental processes and structures are described below.

Theoretical perspectives on emotion

Three theoretical perspectives on emotions developed inpsychology are most directly relevant for computationalaffective modeling. Each perspective defines its own seman-tic primitives, and, to a greater or lesser extent, definesspecific processes required to model a subset of affectivephenomena. Fig. 1 illustrates the differences among thesetheoretical perspectives in the context of emotiongeneration.

Discrete theories of emotions emphasize a small set ofdiscrete or fundamental emotions. The underlying assump-tion of this approach is that these fundamental, discreteemotions are mediated by associated neural circuitry, witha large innate, hardwired component. Different emotionsare then characterized by stable patterns of triggers, behav-ioral expression, and associated distinct subjective experi-ences. The emotions addressed by these theories aretypically the basic emotions; joy, sadness, fear, anger,and disgust. Because of its emphasis on discrete categoriesof states, this approach is also termed the categorical ap-

ers

106 E. Hudlickaproach (Panskepp, 1998). For modeling purposes, thesemantic primitives representing emotions in affectivemodels are the basic emotions themselves.

An alternative method of characterizing affective statesis in terms of a small set of underlying dimensions that de-fine a space within which distinct emotions can be located.This dimensional perspective describes emotions in terms oftwo- or three-dimensions. The most frequent dimensionalcharacterization of emotions uses two dimensions: valenceand arousal (Russell, 2003; Russell & Barrett, 1999; Russell& Mehrabian, 1977). Valence reflects a positive or negativeevaluation, and the associated felt state of pleasure (vs.displeasure), as outlined in the context of undifferentiated

Fig. 1 Alternative theoretical paffect above. Arousal reflects a general degree of intensityor activation of the organism. The degree of arousal reflectsa general readiness to act: low arousal is associated withless energy, high arousal with more energy. Since this 2-dimensional space cannot easily differentiate among emo-tions that share the same values of arousal and valence,e.g., anger and fear, both characterized by high arousaland negative valence, a third dimension is often added. Thisis variously termed dominance or stance. The resulting 3-dimensional space is often referred to as the PAD space(Mehrabian, 1995) (pleasure (synonymous with valence),arousal, dominance). The representational semantic primi-tives within this theoretical perspective are thus these 2or 3 dimensions.

The third view emphasizes the distinct components ofemotions, and is often termed the componential view(Leventhal & Scherer, 1987). The components referredto in this view are both the distinct modalities of emotions(e.g., cognitive, physiological, behavioral, subjective) andalso the components of the cognitive appraisal process.These are referred to as appraisal dimensions or appraisalvariables, and include novelty, valence, goal relevance,goal congruence, and coping abilities. A stimulus, whetherreal or imagined, is analyzed in terms of its meaning andconsequences for the agent, to determine the affectivereaction. The analysis involves assigning specific values tothe appraisal variables. Once the appraisal variable valuesare determined by the organisms evaluative processes theresulting vector is mapped onto a particular emotion, withinthe n-dimensional space defined by the n appraisal vari-ables. The semantic primitives for representing emotionswithin this model are thus these individual appraisalvariables.

It must be emphasized that these theoretical perspec-tives should not be viewed as competing for a singleground truth, but rather as distinct perspectives, eacharising from a particular research tradition (e.g., biologi-

pectives on emotion generation.cal vs. social psychology), focusing on different sets ofaffective phenomena, considering distinct levels of resolu-tion and fundamental components (e.g., emotions vs. ap-praisal variables as the distinct primitives), and usingdifferent experimental methods (e.g., factor analysis ofself-report data vs. neuroanatomical evidence for distinctprocessing pathways), and providing different degrees ofconceptual and empirical support for modeling a particu-lar aspect of affective processing; e.g., the componentialtheories provide extensive details about cognitiveappraisal.

Core affective processes and generic computational tasks

Given the multiple-modalities of emotion, the complexityof the cross-modal interactions, and the fact that affec-tive processes exist at multiple levels of aggregation, itmay seem futile, at best, to speak of fundamental pro-cesses of emotions. Nevertheless, for purposes of devel-oping symbolic models of emotions, and modelingemotions in symbolic agent architectures, it is useful tocast the emotion modeling problem in terms of two broadcategories of processes (Hudlicka, 2008, 2012). Thoseresponsible for the generation of emotions, and those

which then mediate the effects of the activated emotionson cognition, expressive behavior (e.g., facial expressions,speech) and action selection.

This temporally-based categorization (before and afterthe felt emotion) provides a useful perspective for compu-tational affective modeling, and helps manage the complex-ity of the modeling effort, by supporting a systematicdeconstruction of these high-level processes into theirunderlying computational tasks, as discussed below. (Thereare of course many complex interactions among the individ-ual subprocesses within and across these broad categories ofprocesses.)

Identifying the fundamental processes required to modelemotions is only the first step. For this perspective to beuseful, we must deconstruct these high level processes,and identify the individual computational tasks required toimplement emotion generation and emotion effects. Theobjective here is to move beyond the approach where indi-

Affective BICA: Challenges and open questions 107vidual models are the organizing dimension, as is typicallythe case in existing literature on affective modeling, and to-wards a more general approach, organized in terms of thedistinct computational tasks, and the associated represen-tational and reasoning requirements. The generic computa-tional tasks required to implement emotion generation andemotion effects discussed below provide a basis for manag-ing the complexity of affective modeling, and provide foun-dations for more concrete guidelines for model design. Theyrepresent generic computational building blocks, fromwhich specific models can then be constructed. Fig. 2 pro-vides a schematic illustration of the relationship betweenthe basic computational tasks, the core processes of emo-tion and the distinct emotion roles.

It is not suggested that these processes correspond todistinct, discrete neural processing mechanisms, but ratherthat they represent a useful means for managing the com-plexity associated with symbolic affective modeling. Specif-ically, the core process/generic tasks view provides aframework that helps organize existing theories and data,as well as a natural hierarchical structure that links thehigh-level emotions roles to the underlying computationaltasks necessary to implement them.

The distinct computational tasks necessary to implementemotion generation via cognitive appraisal are as follows(Hudlicka, 2012) (refer to Fig. 3):

Fig. 2 Relationship between emotion roles, the core pro-cesses of emotion and the computational tasks necessary toimplement these processes. Define the emotion_elicitortoemotion mapping(depending on the theoretical perspective adopted, thismay involve additional subtasks that map the emotionelicitor onto the intermediate representation (PADdimensions or appraisal variable vectors), and the subse-quent mapping of these onto the final emotion(s)).

Calculate the intensity of the resulting emotion. Calculate the decay of the emotion over time. Integrate multiple emotions, if multiple emotions weregenerated.

Integrate the newly-generated emotion with existingemotion(s) or moods.

While models of emotion generation typically focus ononly one modality (the cognitive modality and cognitive ap-praisal), models of emotion effects cannot as easily ignorethe multi-modal nature of emotion. This is particularly thecase in models implemented in the context of embodiedagents that need to manifest emotions not only via behav-ioral choices, but also via expressive manifestations withinthe channels available in their particular embodiment(e.g., facial expressions, gestures, posture, etc.)

The multi-modal nature of emotion effects increasesboth the number and the type of computational tasks neces-sary to model emotion effects. The abstract computationaltasks required to model the effects of emotions across mul-tiple modalities are as follows (Hudlicka, 2012) (refer toFig. 4):

Define and implement the emotion/moodtoeffectsmappings, for the modalities included in the model(e.g., cognitive, expressive, behavioral, neurophysiologi-cal). Depending on the theoretical perspective adopted,this may involve additional subtasks that implement anyintermediate steps, and are defined in terms of moreabstract semantic primitives provided by the theory(e.g., dimensions, appraisal variables).

Determine the magnitude of the resulting effect(s) as afunction of the emotion or mood intensities.

Determine the changes in these effects as the emotion ormood intensity decays over time.

Integrate effects of multiple emotions, moods, or someemotion and mood combinations, if multiple emotionsand moods were generated, at the appropriate stage ofprocessing.

Integrate the effects of the newly-generated emotionwith any residual, on-going effects, to ensure believabletransitions among states over time.

Account for variability in the above by both the intensityof the affective state, and by the specific personality ofthe modeled agent.

Coordinate the visible manifestations of emotion effectsacross multiple channels and modalities within a singletime frame, to ensure believable manifestations.

The generic tasks outlined above implicitly define repre-sentational constructs, which need to be represented withinthe architecture. These include, in addition to the represen-tations of the emotions themselves, various types of exter-nal and internal emotion elicitors (e.g., cues from theenvironment representing aspects of current situations orother agents, aspects of the self); representations of situa-

ia cs),

108 E. HudlickaFig. 3 A computational perspective on emotion generation vprocess, showing the inputs (triggering stimuli), output (emotionstimulus-to-emotion mapping.tions, events, other agents and the self; internal structuressuch as goals, beliefs, plans; and representations of alterna-tive actions and their expected effects. These constructscan be usefully organized into several domains, again, tohelp control the modeling complexity and to provide a foun-dation for more systematic design guidelines. The next sec-tion discusses a number of domains that have beenidentified which are necessary to implement emotion gener-ation and emotion effects.

Distinct domains required for modeling emotions

Broekens and colleagues developed a generic set-theoreticformalism, and an abstract framework, for representing,and comparing, appraisal theories. Building on the work ofReisenzein (Reisenzein, 2001), Broekens and colleagues(Broekens, DeGroot, & Kosters, 2008) offer a high-level,set-theoretic formalism that depicts the abstract structureof the appraisal process, and represents both the processesinvolved, and the data manipulated. We have augmentedtheir original framework to also represent modeling of emo-tion effects.

Fig. 4 Computational tasks necessary to model emotion andmood effects.The framework illustrates the distinct processes involvedin emotion generation and emotion effects modeling, andthe data manipulated by these processes (e.g., perception(evaluative processes produce a series of mental objects),appraisal (processes that extract the appraisal variable val-ues from the mental objects), and mediation (processesthat map the appraisal values onto the resulting emo-tion(s)). The distinct processes operate on different typesof data: their associated domains. Fig. 5 provides a sche-matic view of these distinct domains, and relates them tothe core affective processes discussed earlier.

This framework complements the computational taskbased perspective with a set of domains required to imple-ment both emotion generation and emotion effects, andhelps define the constituent elements of these domains.These definitions then form a basis for defining the map-pings among these domains that are necessary to implementemotion generation and emotion effects.

Together, the core affective processes, the generic com-putational tasks and the abstract domains provide a basisfor a more systematic approach to the design of emotionmodels. However, the usefulness of these representational

ognitive appraisal. High-level view of the emotion generationand the distinct computational tasks required to implement theand reasoning remains to be established. Much work remainsto be done to determine whether these specific processesand structures are in fact useful building blocks and abstrac-tions, to further refine these building blocks, and to developsystematic modeling guidelines. This represents anothermajor challenge for emotion modelers.

Modeling complex emotions and multi-modalaffective phenomena: beyond appraisal

The majority of existing architectures model simple (ba-sic) emotions, do not address the complexities associatedwith modeling affective states of varying durations, and fo-cus on a small number of modalities: the cognitive modalityin emotion generation, and the behavioral and/or expres-sive modality in emotion effects models. Below I discusssome of the challenges associated with increasing the com-plexity of affective states, addressing their varying dura-tion, and modeling multiple modalities.

and perspiration;

nvossumemem

Affective BICA: Challenges and open questions 109Complexity

The majority of existing efforts in affective modeling andaffective architectures focus on the so-called basic emo-tions. While the term is somewhat controversial, andshould not be understood in terms of algebraic bases, it isnevertheless useful for referring to a small set of fundamen-tal emotions, characterized by a relatively stable set of trig-gers and expressive and behavioral manifestations. Theemotions typically included in this set are: joy, fear, anger,sadness and disgust. However, a human with only these ba-sic emotions would have a rather limited affective, not tomention social, existence. Humans, and other biologicalagents, have a much richer affective repertoire, which in-cludes more complex social emotions, such as pride, guilt,shame, and love. Affective modeling researchers are begin-ning to model these emotions, but these efforts are still intheir infancy. Increasing efforts to model these more com-

Fig. 5 Abstract framework representing the distinct domains iproposed by Broekens et al. (2008)). Note that this figure arepresenting variables that mediate both emotion generation andan abstract domain. The solid arrows indicate paths mediatingeffects.plex states will not only contribute to more socially sophis-ticated and believable social agents, but will also contributeto our understanding of the nature and roles of these morecomplex affective states, both intrapsychically andinterpersonally.

Multiple modalities

Early symbolic affective models focused almost exclu-sively on the cognitive modality, and on models of emo-tion generation via cognitive appraisal. Cognitiveappraisal refers to a cognitive assessment of the agentscurrent situation with respect to its goals and beliefs. Itessentially reflects a goodness of fit for the agents ac-tive goals. One of the first computationally-friendly treat-ments of emotions was the now classic CognitiveStructure of Emotions by Ortony, Clore and Collins(OCC) (Ortony et al., 1988). Until recently, the cognitiveappraisal approach proposed by OCC was the dominantmeans of modeling emotion generation. This focus onthe cognitive modality is not surprising, given the cogni-tive revolution in psychology and the focus on cognitionemphasized in cognitive science, as well as the focus ofAI on symbolic problem solving and the suitability of sym-bolic processing for modeling cognitive phenomena.

However, the almost exclusive focus on cognition inemotion generation clearly provides a narrow and limitedperspective on emotions, which are inherently multi-modal.A full-fledged emotional episode includes:

cognition (both during the generation of emotions andreflecting the impact of the generated emotion on theorganism, e.g., obsessive worry associated with anxiety,increased attentional focus associated with fear);

physiology (aside from the essential involvement of theneural circuitry of the central nervous system, and theneuroendocrine systems mediating affective manifesta-tions, emotions involve the autonomic nervous systemand are associated with a number of specific markersreflecting arousal, such as rapid heart rate and breathing

lved in emotion modeling (augmented version of an frameworkes the existence of some intermediate, abstract structuresotion effects. Not all affective models necessarily require suchotion generation; the dashed arrows paths mediating emotion expression and behavior (different emotions are associ-ated with distinct expressive and behavioral patterns;e.g., fighting/freezing/fleeing); and

subjective conscious experience; e.g., the unique, expe-rienced, felt sense of associated with distinct emotions.

Emotion modelers are increasingly becoming interestedin representing additional modalities. This interest isfueled by two other trends: the renewed interest inembodied emotion, and the rapid development of roboticarchitectures that offer, at least in theory, multiplemodalities. Incorporating non-cognitive modalities in emo-tion models is challenging however. While it is relativelyeasy to represent some neurophysiological element by adedicated variable in a computational model (e.g., levelof a particular neurotransmitter, degree of physiologicalarousal), it is doubtful that such highly-abstracted modelswill enhance our understanding of the underlying affectivemechanisms.

In spite of much recent progress in affective neurosci-ence regarding the neural structures and circuitry mediating

emotions, existing neuroscience data do not yet provide suf- dynamics remains one of the major challenges in affective

110 E. Hudlickaficient basis for constructing multi-modal models of emo-tion generation, as this process involves multiple circuitsand currently intractably complex interaction of many neu-rotransmitter and hormonal systems. While subsymbolicmodels of single isolated circuits or phenomena are beingdeveloped (e.g., conditioning in amygdala), the integrationof these models into an integrated symbolic architecture,capable of displaying complex affective behavior, remainsan open problem.

The challenge associated with multi-modal models ofemotions thus brings into focus the broader problem of cre-ating multi-resolution models and integrating high-level,highly-aggregated symbolic models with subsymbolic mod-els of isolated, simpler phenomena.

Another challenge associated with multi-modal affectivemodels is the relative lack of empirical data regarding thecausal relationships among the distinct modalities. Emotionresearchers emphasize the complex feedback relationshipsamong the distinct modalities, and some emotion theoriesexplicitly emphasize multiple modalities; e.g., Scherersand others componential theories of emotions (Scherer,2000, 2001a). Many emotion researchers consider the mul-ti-modal nature of emotions their essential aspect. Indeed,some researchers define emotions in terms of the simulta-neous involvement of these multiple modalities and syn-chronized activation of the associated multiplesubsystems; that is, e.g., cognitive (evaluation), physiolog-ical (preparation for response), behavioral (execution of re-sponse) (Scherer, 2000).

Concrete examples of cross-modal interactions includethe facial feedback hypothesis (Ekman, 2007; Laird &Strout, 2007), which suggests that emotions can be gener-ated or enhanced by activating the expressive patterns asso-ciated with their experience (e.g., if we smile, we willbecome happy, or happier), and which appears to be sup-ported by empirical evidence.

The primary challenges associated with developing mul-ti-modal models of emotions thus include lack of the neces-sary empirical data, challenges in developing multi-resolution models of the required complexity, and difficul-ties associated with modeling interacting processes operat-ing at multiple time-scales. Newells argument for thenecessity of integrated symbolic architectures to study hu-man reasoning (Newell, 1990) applies to the study of emo-tions as well. Autonomous agent architectures,particularly robotic architectures that must physically inter-act with their environment, provide a suitable context with-in which multi-modal models of affective processes can bedeveloped and evaluated.

Modeling temporal fidelity and affective dynamics

Directly related to the challenge of modeling multiplemodalities of emotions is the modeling of multiple pro-cesses with different temporal characteristics, and model-ing realistic affective dynamics. Although we speak ofaffective states, emotions are in fact intrinsically dynam-ical phenomena of widely different time constants (Fel-lous, 2004). Yet our understanding of affective dynamics iscurrently rather limited. Developing models of affectivemodeling.

Temporal fidelity

We are probably all familiar with the situation where weexperience a sudden fright, for example, having our car skidinto a ditch on an icy road. The physiological components ofthe fear episode last longer than the associated subjectiveexperience of the fear emotion; our heart continues to racerelatively long after we consciously realize that the dangerhas passed. This simple episode illustrates the challenge ofmodeling multiple processes with different rates of change(introduced in Section 3.1.3 above), in the context of multi-ple time scales of the distinct emotion modalities. Modelingmultiple affective processes operating on distinct timescales, and the interactions among them, remains one ofthe greatest challenges in affective modeling.

Affective dynamics

Affective dynamics refers to both the intensity of an emo-tion over time, including its specific onset and decay pat-terns, and to the combination of multiple emotions, andintegration of their effects on the agents internal process-ing, expressive behaviors and action choices. This aspect ofaffective modeling is not as well developed as emotion gen-eration via cognitive appraisal, and even the seemingly sim-ple problem of calculating emotion intensity can pose achallenge, especially in situations where multiple modali-ties need to be taken into consideration. Calculation ofintensity requires first the identification of the factors thatinfluence intensity, and the specification of the formulaethat combine these to produce a single intensity value. Dif-ferent weights may need to be associated with differentfactors (e.g., existing external stimulus vs. recalled mem-ory; self- vs. other-relevant cue). The intensity of a newlyderived emotion must then be combined with the intensityof any existing emotions or moods, to ensure smooth andappropriate transitions among distinct states. These calcu-lations must take into account any differences in the decayrates of different emotions, which are subject to a varietyof influences that have not yet been identified or quantifiedto the degree required for computational modeling.

Most existing models of appraisal use relatively simpleformulae for calculating emotion intensity, typically focus-ing on desirability and likelihood; e.g., [desirability \ likeli-hood] (Gratch & Marsella, 2004), [desirability \ (change in)likelihood] (Reilly, 2006). A number of complexities are typ-ically not addressed. For example, Reilly (2006) points outthe need for representing asymmetry of success vs. failure;in other words, for different types of individuals (and differ-ent goals) success may be less (or more) important than fail-ure; e.g., extraversion is associated with reward-seekingwhereas neuroticism is associated with punishment-avoid-ance. Modeling of these phenomena requires distinct vari-ables for success (desirability of an event, situation orworld state) vs. failure (undesirability of the same).

Directly related to the intensity calculation is the calcula-tion of the emotion onset and decay rates, which brings up aquestion regarding the extent to which emotions representself-sustaining processes, that must run their course. Reillysummarized current approaches to decay calculation as being

linear, logarithmic, exponential, or some arbitrary mono-tonically decreasing function over time (Reilly, 2006).

Unfortunately for modelers, emotion dynamics are notwell understood, and the data for precise calculations ofintensities and onset and decay rates are not available.Existing empirical studies provide qualitative data at best.Variability of these processes across emotions and individu-als, while documented, has also not been quantified; e.g.,high neuroticism rate predisposes individuals towards fasterand more intense negative emotions; anger appears to de-cay more slowly than other emotions (Lerner & Tiedens,2006). Even more importantly, some researchers point outthat the appraisal dimensions identified for emotion differ-entiation may not be the same as those that allow predic-tion of duration and intensity, and that the current set ofappraisal dimensions may be incomplete (Scherer, 2001a,p. 375).

Combining multiple emotions

Emotions rarely occur in a pure form. Multiple emotions maybe generated by the appraisal processes and existing emo-tion(s) must be combined with newly-generated emotion(s).At their maximum intensity, we may feel, and express, a

much help. Should opposing emotions cancel each otherout? (Are we likely to feel calm and neutral if our houseburns down but we have just won the lottery?) Is it evenappropriate to think of emotions in pairs of opposites? Canwe assume that the strongest emotion is the right one,as some models do (e.g., Hudlickas MAMID (Hudlicka,2004, 2007)? At what stage of processing should emotionsbe combined and any contradictions resolved? Should con-flicting emotions be resolved at the appraisal stage, to avoidthe problem entirely? At the cognitive effects stage, e.g.,during goal selection? Or at the behavior selection stage?The latter being potentially the most problematic; and yetit is apparent that this phenomenon occurs in biologicalagents. One only needs to witness the scrambling of a fright-ened squirrel as a car approaches to see a dramatic conse-quence of the failure to resolve contradictory behavioraltendencies. Fig. 6 illustrates several alternative points inthe emotion effects modeling sequence where multipleemotions may be integrated.

Developing models that address the full complexity ofaffective dynamics represents another major challenge inemotion modeling, and one that is likely to require theadoption and refinement of non-symbolic formalisms (e.g.,

roc

Affective BICA: Challenges and open questions 111single emotion. However, more typically, multiple emotionsinteract to form the subjective feeling experience andinfluence cognitive processing and behavior selection.These phenomena are not well understood, let alone quan-tified, to the degree required for computational modeling.

Reilly analyzed several existing approaches to combiningsimilar emotions and highlighted their drawbacks and bene-fits, as follows. Simple addition of intensities can lead to toomuch intensity (e.g., few low intensity emotions lead to ahigh intensity reaction). Averaging the intensities may re-sult in a final intensity that is lower than one of the constit-uent intensities: an unlikely situation in biological agents.Max (or winner-take-all) approach ignores the cumulativeeffects of multiple emotions.

No analogous analysis exists for combining opposing emo-tions. Nor do existing theories and empirical data provide

Fig. 6 Multiple points (marked by the circles) within the pintegrated.dynamical systems), and more refined empirical methodsto obtain data about affective dynamics in biologicalagents. (Attempts to use dynamical systems to model emo-tions are beginning to emerge (e.g., Treur, 2013).)

Integrating emotions into agent architectures

Above I highlighted some of the challenges associated withmodeling a variety of individual affective processes andphenomena. In this section I address the central tasks indeveloping affective BICAs:

how should affective processing be integrated within theagent architecture, and

what types of architecture organizations are suitable forsuch integration.

essing sequence where multiple emotions and moods can be

ctial.

112 E. HudlickaTable 1 A detailed summary of the characteristics of the readeliberative and meta-management layers (see also Ortony et

Reactive

Sensor-to-effectorpaths

Fixed, direct

Intermediate statescan exist

No

Cognitive processing No

Memory Fixed only;implicit ininnate wiring

Representation of time (past, future) NoExplicit representation of self &reasoning about self

No

Representation of internal states &(some) ability to monitor &

NoSloman has written extensively about this issue in abroader context (not limited to emotion modeling) and itis important to keep in mind his observation regarding agentarchitectures.

Different architectures will support different classesof possible states and processes. If the architectures ofhuman infants, toddlers and adults are different, thendifferent sets of concepts may be required for describingthem. Even more different will be the range of possibleaffective states in insects and other animals very differ-ent from humans. Neither human infants or insects havethe ability to be obsessively politically ambitious: thearchitectural requirements, the conceptual require-ments, and the knowledge of the world required for sucha state are beyond their reach.(Sloman, 2004)

Developers of affective BICA therefore need to carefullyconsider both the types of emotions the agent will need,and the types of environments within which they will oper-ate. Distinct types of architectures may need to be con-

control themAble to represent possible orimagined states (of the world, self)

No

Able to generate alternativepossiblefuture states (what-if reasoning)

No

Able to represent and comparealternative plans

No

Able to representmultiple/conflicting goals

No

Able to represent and react tochanging goals

No (fixed goals, implicitin hardwired S-R pattern

Able to generate alternativepast states

No

Learning possible NoPotential for neuromodulation Lowve,, 2005).

Deliberative Meta-management

Flexible, indirect(intermediate states)

Flexible, indirect(intermediate states)

Yes Yes

Less complex(associative memory,routineprocessing, rule-basedreasoning)

More complex (complexproblemsolving, ability to thinkabout newproblem in new ways)

Can encode new informationin memory

Yes YesNo Yes

? Yesstructed to model different emotions, and, more broadly,to model the full range of what Sloman refers to as de-sire-like states (contrasted with belief-like states), whichinclude emotions but also moods, preferences, attitudesand other mixed cognitiveaffective states.

Emotion modelers have hypothesized some of the struc-tures and processes thought to mediate different types ofemotions and other affective states (see discussion on tri-une architectures below and Table 1). However, much workremains to be done before we can develop a principled ap-proach for designing architectures capable of generating abroad range of affective states and more realistic affectivedynamics. In fact, it would seem that reaching such a levelof understanding would require that we actually understandhow emotions work in biological agents. At this point in timewe are quite far from this goal, and this objective repre-sents the most significant challenge for affective BICAdesigners.

In addition the issues discussed above, designers ofaffective BICAs also need to consider the following:

Yes world Yes world & self

Yes (if can do relief) Yes

Yes (simple) Yes (complex)

Yes/? Yes/Yes

s)Yes (simple) Yes (complex)

Yes (if can do regret) Yes

Yes YesHigher High

Is the objective of the architecture to characterize affec- Specific capabilities of the architecture

Affective BICA: Challenges and open questions 113tive mechanisms in biological agents, or to produce moreadaptive and believable behavior in the associated syn-thetic agent? In other words, are we aiming to developresearch or applied models of emotion within the archi-tecture? This has implications for the degree of biologicalfidelity necessary in the architecture: does it need toemulate or simulate biological affective processing?

Which of the roles that emotions perform in biologicalagents does the architecture need to implement?

Which specific emotions should be modeled in thearchitecture?

Affective BICA design space

Considerations of these high-level design requirements thenguide the specific design choices regarding the following as-pects of the architecture (see also Sloman & Scheutz, 2002):

Modules

Architecture modules included and the location of spe-cific processes within particular modules.

Types of memory included, differentiated by contenttype and encoding methods (declarative, episodic, pro-cedural) and by stability (long-term vs. short-term).

Representations

Types of mental constructs represented explicitly (e.g.,goals, preferences, beliefs, situations, expectations,plans, procedure hierarchies, self representations).

Degree to which time is represented explicitly and tem-poral representations can be manipulated (e.g., hopeneeds a representation of possible future events; regretneeds a representation of past events and counterfactualpossibilities).

Degree to which architecture states and processes arerepresented explicitly to enable meta-cognitive process-ing (monitoring and control).

Architecture organization

Type of vertical organization (how many layers, types ofstructures and processes at each layer) (e.g., reactive,deliberative, meta-management).

Type of horizontal organization (degree of complexityacross the perceptual, cognitive and motor components;which modules are included in the architecture, such asattention, expectation generation and planner).

Communication among modules and between the archi-tecture and the environment

Data and control paths among layers and components;which processes in which layers or modules can monitorand influence other processes, in the same or differentlayers.

Connections to the environment: which layers are con-nected directly to sensors and effectors, and which pro-cess only internal data. Degree and type of parallelism possible, and feedbackand coordination among multiple parallel processes.

Specific reasoning capabilities (e.g., deductive, induc-tive, what-if and counterfactual reasoning, causal andabductive reasoning);

Degree of neuromodulation possible, at different layersand for different processes;

Affective modalities implemented; Aspects of affective dynamics implemented; Degree, and mechanisms, of learning possible, and the

architecture components that can be modified by thelearning processes.

The designer also needs to consider which theoreticalperspective is best suited to accomplish the architectureobjectives (e.g., discrete, dimensional, componential, orsome combination); where the emotions, and affective pro-cessing in general, will be localized within the architecture;whether processing will occur sequentially, in a parallel-dis-tributed fashion, or some combination of both; and whethersymbolic or subsymbolic representations and reasoning willbe used, or both.

A systematic approach to making the design choices out-lined above has yet to be developed. However, recent the-oretical work in cognitiveaffective agent architecturesprovides some basis for matching elements of the designspace outlined above with the specific affective require-ments for a given architecture. Below I summarize currentthinking about the possible structure of cognitiveaffectivearchitectures, and highlight some of the open questionsregarding the integration of emotion into agentarchitectures.

Location of emotion within the architecture

Before addressing the architecture structure proper it isimportant to consider where emotion and affective process-ing will be localized within the architecture. A designer ofan affective BICA needs to decide whether the affectiveprocesses will be localized in, and confined to, particularmodules, corresponding to a specific functionality, distrib-uted among multiple modules, and/or implemented viaparameter-controlled systemic effects throughout thearchitecture, or which particular combination of the above.The organization of the architecture in terms of both thevertical layers (e.g., reactive through deliberative tometa-management) and horizontal processing stages(see-think/feel-do) will clearly play a key role in the distri-bution of affective processes within the architecture.

For some affective processes this localization is rela-tively straightforward. For example, emotion generationvia cognitive appraisal can be localized within an emotiongeneration module of the architecture. For other affectiveprocesses, for example, effects of emotions on cognition,such localization is not meaningful, since many emotion ef-fects can more appropriately be implemented as systemicchanges across multiple cognitive processing. For example,the MAMID architecture (Hudlicka, 2007) represents a rangeof affective biases in terms of parameters that control pro-

cessing across multiple the architecture modules thatimplement a see-think/feel-do processing sequence.

Architecture templates and the triune cognitiveaffective architecture

A range of generic layered agent architecture templates canbe defined, with a varying number of processing layers, anddistinct patterns of mapping the perceptual inputs onto ac-tion outputs. Dix, Kraus, and Subrahmanian (2002) distin-guishes among horizontal layering, where each layer isassociated with its own input and output; one-pass controlvertical layering, where perceptual input enters the lowestlayer and is gradually mapped through the succeeding layersinto an action output; and a two-pass control vertical layer-ing, where perceptual input enters the lowest layer, istransformed by increasingly complex perceptual processingto the highest layer, and then mapped down to the effectorsresiding in the lowest layer. These generic structures offer arange of specific configurations of the see-think-do se-quence, in terms of the layers involved and the distributionof specific processes across these layers, and the configura-tions of input/output paths within the architecture.

114 E. HudlickaTriune architecture. Within this broad framework of lay-ered hierarchical architectures, a number of researchershave proposed a three layer architecture to develop agentscapable of complex adaptive behavior, and implementingboth the cognitive and affective processing necessary togenerate a broad range of affective states (Leventhal &Scherer, 1987; Ortony et al., 2005; Sloman, 2003; Slomanet al., 2005). While gold standard architecture templateshave not yet been established in cognitiveaffective agentresearch, to support a systematic mapping of the require-ments to specific architecture structures and processing,the triune hierarchical architecture template representsa convergence of ideas by a number of emotion and agentarchitecture researchers, and is outlined in more detailbelow.

The triune architecture framework implements the see-think/feel-do (perception central processing motor

Fig. 7 The triune architecture schematic (reflectingarchitectures proposed by Sloman; Leventhal & Scherer; Ortonyet al.; Arbib & Fellous).control) processing sequences at three levels of complexity,which are termed (from least to most complex): reactive(also sensorimotor), deliberative (also routine or sche-matic), and meta-management (also reflective or concep-tual). Processing occurs in parallel at all three layers, withcomplex feedback mechanisms among the layers coordinat-ing the independent processes and influencing the final out-come. (Refer to Fig. 7.)

The lowest reactive layer implements simple stimulusresponse mappings, without cognitive processing, and in-nate, hardwired seedo sequences. It is stateless, has nomemory, beyond that which is implicit in the hardwiredmappings mediating the stimulusresponse behavior, andwhich represents the result of evolutionary learning. It hasno representation of the past or the future, and reacts in-stead only to its immediate input. Due to the direct sen-sor-t