Norm internalization in artificial societies

AI Communications 23 (2010) 325–339 325DOI 10.3233/AIC-2010-0477IOS Press

Norm internalization in artificial societies

Giulia Andrighetto a,∗, Daniel Villatoro b and Rosaria Conte a

a LABSS, Institute of Cognitive Science and Technologies, CNR, Rome, Italyb Artificial Intelligence Research Institute (IIIA), Spanish National Research Council, Spain

Abstract. Internalization is at study in social-behavioural sciences and moral philosophy since long; of late, the debate wasrevamped within the rationality approach to the study of cooperation and compliance since internalization is a less costly and morereliable enforcement system than social control. But how does it work? So far, poor attention was paid to the mental underpinningsof internalization. This paper advocates a rich cognitive model of different types, degrees and factors of internalization. In orderto check the individual and social effect of internalization, we have adapted an existing agent architecture, EMIL-A, providingit with internalization capabilities, turning it into EMIL-I-A. Experiments have proven satisfactory results with respect to themaintenance of cooperation in a proof-of-concept simulation.

Keywords: Rich cognitive modeling, norms, internalization, agent architecture, autonomous agents, agent based social simulation

1. Introduction

The problem social scientists still revolve aroundis how autonomous systems, like living beings, per-form positive behaviors toward one another and com-ply with existing norms, especially since self-regardingagents are much better-off than other-regarding agentsat within-group competition. Since Durkheim, the keyto solving the puzzle is found in the theory of internal-ization of norms [30,32,34,39,43]. One plausible ex-planation of voluntary non-self-interested compliancewith social norms is that norms have been internalized.

Internalization occurs when

a norm’s maintenance has become independent of externaloutcomes – that is, to the extent that its reinforcing con-sequences are internally mediated, without the support ofexternal events such as rewards or punishment [4, p. 18].

Agents conform to an internal norm because so do-ing is an end in itself, and not merely because of exter-nal sanctions, such as material rewards or punishment.

Norm internalization is one of the common themesrunning across all of the social-behavioral disciplinesand there are not many. Not only sociologists, butalso developmental, social and cognitive psychologistshave perceived its crucial role in socialization [6,38,44]. Drawing on the early work by Vygotzky (pub-lished in the US as late as 1978 [53]), Piaget [44] andKohlberg [38], and several other psychologists showedthat a parental attitude oriented to elicit norm inter-

*Corresponding author. E-mail: [email protected].

nalization predicts children’s later well being and eventheir inclination to other-regarding behavior [25].1

Despite these important contributions, however, thecommunity’s scientific definition and understanding ofthe process of norm internalization is still fragmentaryand insufficient.

The main purpose of this paper is to argue for thenecessity of a rich cognitive model of norm internal-ization in order to (a) provide a unifying view of thephenomenon, accounting for the features it shares withrelated phenomena (e.g., robust conformity as in auto-matic behavior) and the specific properties that keep itdistinct from them (autonomy); (b) model the processof internalization, i.e., its proximate causes (as com-pared to the distal, evolutionary, ones; see [29,30]);(c) characterize it as a progressive process, occurringat various levels of depth and giving rise to more orless robust compliance (see [25] for a fine grained dis-tinction of different degrees of norm internalization);and finally (d) allow for flexible conformity, enablingagents to retrieve full control [9] over those normswhich have been converted into automatic behavioralresponses [28].

Thanks to such a model of norm internalization, itwill be possible to adapt existing agent architectures(see EMIL-A, [2,15]) and to design a simulation plat-form to test and answer a number of hypotheses andquestions such as: which types of mental propertiesand ingredients ought individuals to possess in order to

1See [40], for an overview of the research on Norm Internaliza-tion.

0921-7126/10/$27.50 © 2010 – IOS Press and the authors. All rights reserved

326 G. Andrighetto et al. / Norm internalization in artificial societies

exhibit different forms of compliance? How sensitiveeach modality is to external sanctions? What are themost effective norm enforcement mechanisms? Howmany people have to internalize a norm in order for it tospread and remain stable? What are the different impli-cations for society and governance of different modal-ities of norm compliance?

Throughout the paper, the process of norm internal-ization will be meant as a mental process that takes (so-cial) norms as inputs and gives new terminal goals ofthe internalizing agent (from now on, the internalizer)as outputs.

Emotions, playing a significant role in this process,will not be investigated at this stage.

2. Related work

Norms frequently become internalized [46]. Norminternalization is mainly favoured by socialization in-stitutions, such as family and school, working hard tomake internalize a wide variety of norms, especially inyoung people, and by informal organizations of friendsand peers.2

Contributions to explain internalization are some-times based on reinforcement learning theory. Scott[46], for example, theorized that norm internalizationleads to robust compliance, provided the external sanc-tioning system is never completely abandoned. Un-fortunately, this explanation is incompatible not onlywith the view that social norms can get internalizedto the extent that they do not need social enforce-ment [10, p. 2], but also with experimental evidence.For example, subjects playing ultimatum games arefound to follow fairness considerations even when un-observed [11].

In the last few years, a strong renewal of interestaround the notion of norm internalization appeared inthe evolutionary game theoretic study of cooperationand prosocial behaviour. Gintis [29,30] argued that theincrease in social complexity of early human societyproduced a rapidly changing environment, which inturn posed an adaptation problem to the genetic mech-anisms for altering goals. Internalization of norms isadaptive because it facilitates the transformation ofdrives, needs, desires and pleasures into forms that aremore closely aligned with fitness maximization, whilea purely genetic adaptive process would have taken or-

2Gintis [29] claims that these formal and informal institutionsfavour vertical, oblique and horizontal transmission.

ders of magnitude longer in time [30, p. 62]. Thoughextremely interesting, this theory is focused on the dis-tal (social and evolutionary) causes of norm internal-ization while leaving aside the proximate (internal andcognitive) causes.

Some authors (see [28]) conceive norm internaliza-tion as a process leading to a sort of automatic orthoughtless conformity. Epstein [28] writes: “whenI’d my coffee this morning and I went upstairs toget dressed, I never considered being a nudist for theday”. People, observes Epstein [28], blindly conformto the norm: they observe what the majority does andthey act accordingly. Then the more they have doneso in the past, the more they will redo it in the future.Agents learn not only which norms to conform to, butalso how much they should think about them. In theauthor’s view, internalization is learning not to thinkabout norms.

Does this mean internalized norms are thoughtlesslycomplied with? What about the difference between anaction done out of a ‘sense of duty’ and a habit? Howcan the internalizer gain control again over an auto-mated action and refrain from applying a given rou-tine? How can it move on when the traffic light is redbut the policeman invites it to proceed? Even if anotherroutine is activated by the new event (policeman invita-tion to proceed), how and why is one routine (stop) in-terrupted and fired the complementary one (move on)?How is the conflict solved?

The crucial necessity, here, is to provide a commonground, an agent model that can exhibit different formsof norm internalizations, and, what is more difficult,can shift form one to the other. We need to account forreversible routines, or, which is the same, for flexibleconformity. In principle, a modular normative architec-ture nicely fits flexible automaticity and as it will be de-scribed in Section 6 EMIL-A seems a good candidatefor this undertake.

3. Objectives

This work is aimed to propose agent based mod-eling, and in particular rich cognitive modeling, as aframework for casting a theory of the cognitive under-pinnings of internalization and to characterize norm in-ternalization as a progressive, multi-step process, lead-ing from externally-enforced norms to norm-corres-ponding goals, and actions pursued for their own sake.

The normative agents we model are provided with anew measure, the salience of the norm, allowing them

G. Andrighetto et al. / Norm internalization in artificial societies 327

to detect the degree of activity of the social normswithin their social setting. As we will show, this fea-ture provides the agents with several advantages suchas an additional criterion to better evaluate whether toobserve the norm or not. The normative salience allowsagents to leverage their normative social informationwhen effectively addressing the norm internalizationprocess.

We also provide a proof-of-concept simulationaimed to test the normative architectures discussed.The objective of the simulation is threefold:

1. Check the emergence and stability of cooper-ation in a simplified scenario. Agents performdyadic interactions playing a classic Prisoner’sDilemma. Some agents are initialized with a Co-operation norm, and they will also defend thisnorm through the application of sanctions.

2. Observe run-time the different phases an agentgoes through when the internalization process istaking place.

3. Analyze the interaction dynamics and their effecton different normative architectures. By allowingdifferent types of agents to interact in a normativecontext, we can observe as system designer, theconsequent effects of their behavior in the soci-ety. This knowledge would be helpful for policy-makers in open multi-agent societies.

The rest of this article is organized as follows. First,we will introduce a theory of mental dynamics ofnorms, in order to provide some basic concepts (seeSection 4). Second, norm internalization will be an-alyzed, focusing on different types and levels of thisprocess (see Section 4). We will provide some prelimi-nary hypotheses about factors affecting internalizationfurther on.

In Section 6, we will present a normative agentarchitecture, EMIL-I-A, able to account for differentforms of norm internalization. In Section 7, we willpresent a simulation model aimed to test our normativearchitecture in a simplified scenario. In Section 7.4 re-sults will be discussed. Finally, conclusions and futurework will follow.

4. The cognitive dynamics of norms

In order to understand the process of norm inter-nalization, some preliminary notions should be clar-ified. Building on Ullman-Margalit’s definition of anorm [50] as a prescribed guide for conduct which

is generally complied with by the members of soci-ety, we [2,15] define a norm as a behavior that spreadsthrough a given society to the extent that the corre-sponding prescription spreads as well, giving rise to ashared set of normative beliefs and goals. A normativebelief is a mental representation, held to be true in theworld, that a given action is either obligatory, forbid-den or permitted for a given set of individuals in a givencontext. On the other hand, a normative goal is an in-ternal goal3 relativized 4 to a normative belief: it is thewill to perform an action because and to the extent thatthis is believed to be prescribed by a norm.

There are at least three main types of normative be-liefs:

– The main normative belief, stating that: there is anorm prohibiting, prescribing, permitting that . . .[22,23,37,52].

– The normative belief of pertinence, indicating theset of agents on which the norm is impinging.

– The norm enforcement belief, indicating that apositive sanction is consequent to norm obedienceand a negative sanction is consequent to norm vi-olation.

In order to be compliant with the norm, the firsttwo normative beliefs are necessary conditions: agentsshould recognize that there is a norm and that it ap-plies to them. We claim that when individuals do nothave them in mind, norms exert no effect on the behav-ior [2,15]. Furthermore, the more these normative men-tal representation are salient, the more they will elicit anormative behavior [11,16,55]. Norm compliance andnorm salience are strongly intertwined: findings frompsychology [7,17] and behavioral economics [12,54]have pointed out that drawing people’s attention on asocial norm and making it salient elicits an appropri-ate behavior. Making a norm salient typically meansproviding people with information about the behaviorand beliefs of the other individuals [13, p. 4] (see Sec-tion 5), for a detailed description of the dynamics ofsalience.

3From a cognitive point of view, goals are internal representationstriggering-and-guiding action at once: they represent the state of theworld that agents want to reach by means of action and that theymonitor while executing the action [20].

4A goal is relativized when it is held because and to the extent thata given world-state or event is held to be true or is expected [18]. Anexample is the following: tomorrow, I want to go sunbathing to thebeach (relativized goal) because and to the extent that I believe to-morrow it will be sunny (expected event). The precise instant I ceaseto believe that tomorrow it will be sunny, I will drop any will to goto the beach.


Despite the main normative belief and the belief ofpertinence, the norm enforcing belief is not a definingelement of the norm, it simply enforces it. A normativecommand is a special command that is intended to beadopted by its addressees because it is normative andnorms must be obeyed [52]. Of course this motive canbe absent or weak in the minds of people, depending onthe socialization and education process and the creditobtained by current institutions. Sub-ideally, norms areoften complied with because they are enforced by asystem of sanctions. But ideally, they are meant to beobserved because are norms and should be compliedwith for their own sake.

However, a belief is not yet a decided action. Nor-mative beliefs are necessary but insufficient conditionsfor norms to be complied with. What leads agents en-dowed with one or more normative beliefs to executethem, especially since, by definition, norms prescribecostly behaviours? How can norms generate goals?

Usually normative beliefs generate normative goalsby reference to an external enforcement5 (sanctions,approval, etc.). The agent calculates the costs and ben-efits of complying with or violating the norm and thendecides how to behave. If no such a goal is generated,the norm will be violated. On the other hand, a norm isinternalized when the norm addressee complies with itindependent of external sanctions and rewards. In sucha case, the normative goal is no more relativized to anexpected sanction, but only to the main normative be-lief.

There are also other types and levels of internaliza-tion, so for example under some circumstances (seeSection 5), the decision-making is avoided and the out-put is a conditioned action in the agent’s repertoirefired by a perceived event. In the traffic light example,this consists of the sequence of movements necessaryto activate the breaks of the car, a behavioral responseso deeply internalized that one can hardly make it ex-plicit.6

Norm compliance is expected to be more robust ifnorms are internalized then is the case when conductsare ruled only by external sanctions. If everybody in-ternalizes a given norm, there is no incentive to defect

5See [21,23] for a fine grained analysis of different reasons behindnorm compliance.

6Interestingly, however, under the effect of other perceived events,conditioned actions may be blocked for the time interval required toprocess a disturbing or interfering event and restored later [9] in asemi-conscious fashion. Here, the behavioral response is automatic.For example, rather than stopping at the traffic light, go ahead to letan ambulance overtake.

and the norm remains stable [5, p. 1104]. Gintis arguesthat

where people internalize a norm, the frequency of its oc-currence in the population will be higher than if peoplefollow the norm only instrumentally (i.e., when they per-ceive to be in their interest to do so) [29, p. 61].

Since it is very difficult that everybody in the popula-tion internalizes the norm, an interesting question formodeling, also suggested by Axelrod [5], is how manypeople have to internalize a norm in order for it to re-main stable. This and other questions on norm inter-nalization will be tested with a simulation model andsome results will be shown in Section 7.

Moreover, in many circumstances, an agent that hasinternalized the norm will exercise a special form ofsocial control, getting others to comply with the norm,reproaching transgressors and reminding would-be vi-olators that they are doing something wrong. Norm de-fence is extremely important in the spreading and sta-bilization of norms over a population of autonomousagents [24]. As Axelrod [5] suggests, lowering thetemptation to violate the norm might be not enough.Even in groups in which most people comply with thenorm, if no one has an incentive to punish the remain-ing violators, the norm could still collapse. This meansthat internalizers are agents that not only comply withnorms but also have an incentive to punish anyone elsewho does not comply with the norm (see [3], for a the-oretical and simulative model of this phenomenon). Ifagents are driven to honour norms, they are likely todefend it, directly or indirectly.

5. Factors affecting internalization. Somepreliminary hypotheses

Why do agents observe a norm irrespective of ex-ternal enforcement? Far from providing a complete listof the factors favouring norm internalization, in thepresent work we will focus on some of the elementsplaying a key role in this process, such as:

– consistency;– self-enhancing effect;– urgency;– calculation cost saving;– norm salience.

Let us start with consistency. This mechanism op-erates at two stages: first by selecting which norm tointernalize and later by enforcing it (self-enforcement)


and controlling that one’s behavior corresponds to it(self-control). Consistency of new norms with one’sbeliefs, goals and previously internalized norms playsa crucial role in the selection process. Successfuleducational strategies favor internalization processes,often by linking new inputs with previously internal-ized norms. Analogous considerations apply to poli-cymaking. Consider the antismoking legislation: theefficacy of antismoking campaigns based on frighten-ing announcements and warning labels (e.g., sentenceslike ‘Smoking kills’ on cigarette packages, see [31])is still controversial. One of the factors reducing theefficacy is the effect known as hyperbolic discount-ing ([14]; see also [45]), a psychological mechanismthat leads to invest in goal-pursuit a measure of ef-fort that is a hyperbolically decreasing function of thetime–distance from goal-attainment and leads peopleto procrastinate energy-consuming work until the verylast moment. Due to hyperbolic discounting, people,especially young people, are unable to act under therepresentation of delayed consequences of current ac-tions. On the other hand, much more efficacious anti-smoking campaigns are those playing on previouslyemerged and diffuse set of social norms, such as thelive-healthy precepts, highly consistent with the mes-sage they want to transmit.

The second factor playing a role in norm internaliza-tion is the self-enhancing effect of norm compliance:the norm addressee realizes that it achieves one of itsgoals by observing a given norm. Suppose I succeed inrefraining from smoking and that after a few days, I re-alize an advantage that I had not perceived before: foodstarts to taste again. This discovery generates a goal(quit smoking to enjoy good food), not relativized tothe norm but supporting it: I have converted the norminto an ordinary goal. Whether this goal will be strongenough to out-compete addiction is another matter.

Third, we focus on urgency (see, for example, [56]).In particular, one can argue that the more a given normallows to answer problems frequently encountered un-der conditions of urgency, when time for decision-making is none or scanty, the more likely that normwill be internalized.

Fourth, we claim that agents are parsimonious cal-culators: under certain conditions, they internalizenorms in order to save calculation and execution time.Imagine a driver’s decision to stop at the traffic lightwhen it turns red. Each time our driver approachesa red traffic light, it calculates the costs and benefits ofcomplying with the norm: e.g., it predicts that if it doesnot stop, it will gain time, but that with a certain proba-

bility a fine will follow to its violation. It then chooseswhat is best for itself. After a certain amount of cal-cula, always giving the same output (e.g., the driveralways decides to stop in order to avoid punishment),it will abstain from calculating: it will stop when thetraffic light is red without thinking anymore on whatto do. The agent will save calculus time, thus acting ina more effective way. Thinking declines when normsgain force and gradually stops once they are internal-ized.

This last point is strictly intertwined with anotherimportant factor favoring norm internalization: i.e.,norm salience.

Norm salience (see [2,15]) is defined as the degreeof activity and importance of a norm within a socialgroup and a given context. The more salient the norm,the more likely it will be internalized as a conditionedaction, a routine activated under specified conditions.Moreover, the higher the salience of the norm, the moredeeply it will be internalized.7

The salience of a normative belief can vary depend-ing on several social and individual factors. On onehand, the actions of others provide information aboutthe importance of a norm within a group [11,17,27,33].

For example, the surveillance rate (frequency andintensity of punishment), the quality of normative ser-vices (e.g., if the road signs are well marked), theamount of compliance and the costs and efforts sparedto educate the population to a certain norm are all cuessignaling us the relevance of a norm within a group.On the other hand, norm salience is also affected bythe individual sphere: it is dependent on how much thatnorm is entrenched with beliefs, goals, values and pre-viously internalized norms of the agent [19,49].

6. Internalizer: The EMIL-I-A architecture

In order to account for the different forms, levelsand processes of internalization, a rich cognitive plat-form, namely a BDI-type architecture is required andEMIL-A [2,15] seems a good candidate.

This normative architecture consists of mechanismsand mental representations allowing norms to affectthe behavior of autonomous intelligent agents. Asany BDI-type (Belief-Desire and Intention) architec-

7It has to be pointed out that the norm salience can also graduallydecrease: for example, it happens when the agent realizes that normviolations do not receive any punishment or if the normative beliefsstay inactive for a certain amount of time, this meaning that the normis not very active in the population anymore.


ture EMIL-A operates through modules for differ-ent sub-tasks (recognition, adoption, decision-making)and acts on mental representations for goals and beliefsin a non-rigid sequence.

For further references on how the norm recogni-tion module works, we refer the reader to [15]. Afterrecognition, a norm activates the three types of norma-tive beliefs described in Section 4, that stored in thenormative board.8 Once generated or activated, nor-mative beliefs will be inputed to the norm-adoptionmodule: a normative goal – relativized to the expectedenforcement – will be generated. In this condition thenormative agent adopts the norm, because it wants toavoid punishment. The normative goal is then inputedto the decision-maker and compared with other goalspossibly active in the system. The decision-maker willchoose which one to execute and will convert it into anormative intention (i.e., an executable goal). Once ex-ecuted, this normative goal will give rise to norm com-pliance and/or norm defense and/or norm transmissionthrough communication. Otherwise, it will eventuallybe abandoned, solution that brings again to norm vio-lation.

A crucial aspect of EMIL-A is the possibility toaccount for the occurrence of interruptions, modifica-tions and deviations from the processes described sofar: a norm can be internalized and even become ahabit, a (semi) automatism, a routine behavior. In thiswork we have endowed EMIL-A with an internaliza-tion module, thus creating EMIL-I-A (EMIL Internal-izer Agent).

EMIL-I-A internalizes a norm when two conditionsapply: (a) the norm salience and (b) the cost-benefitcomputation time exceed a certain threshold. The in-ternalizer is endowed with a normative thermometer,signaling him the social and individual salience of acertain norm. If the norm is highly salient, the agentwill internalize it. The internalizer is also a computa-tion costs optimizer. After having weighted for a cer-tain number of times, the costs and benefits of com-plying or not with a certain norm (and having reachedeverytime exactly the same decision), the agent stopscalculating and consider it the best choice. Once in-ternalized, EMIL-I-A stops the normative deliberationand complies with the norm.

Norm salience is an important feature of our agents,improving their performance in several ways. It allowsinternalizers to observe norms in a flexible and auto-

8The normative board is a portion of the long-term memory wherenormative beliefs are stored, ordered by salience.

matic way. Salience enables the agents to dynamicallymonitor if the normative scene is changing and to adaptto it.9 For example, in an unstable social environment,if the norm enforcement suddenly decreases, agentshaving highly salient norms are less inclined to violatethem. A highly salient norm is a reason for which anagent continues to comply with it even in the absenceof punishment. It guarantees a sort of inertia, makingagents less prompt to change their strategy to a morefavorable one. Vice versa, if a specific norm decays,internalizers are able to detect this change, ceasing tocomply with it and adapting to the new state of affairs.Finally, if an agent faces an emergency or a normativeconflict, norm salience allows him to decide which ac-tion to perform providing him with a criterion to com-pare the norms applicable to the context.

This is possible because our normative agents areas autonomous as socially responsive. They are au-tonomous in that they act on their own beliefs and goals(on the basis of their salience). However, they are alsoresponsive to their environment and to the inputs theyreceive from it, especially to social inputs.

7. The model

In order to test the theory presented in this workwe have provided our agent architecture with the nec-essary capabilities to internalize norms in a proof-of-concept multi-agent based simulation. Before explain-ing the simulation, we define the three types of agentswhose performance we compare: strategic, normativeand internalizers.

The decision-making process of a normative agentworks in the following way. A normative agent(Fig. 1(b)) is a BDI agent which has (a) the normativebelief of the existence of a norm (i.e., the main nor-mative belief) and (b) the normative belief of a con-sequent sanction if the norm is not observed (i.e., thenorm enforcement belief) and the goal of maximizingits utility. We hypothesize that normative agents ob-serve norms because of the existence of this sanction(as it reduces its utility), i.e., if a norm is intensivelydefended through the application of sanctions, the nor-mative agent will consequently observe it; on the otherhand, when the norm is not defended, the normativeagent will probably not observe the norm.

9It is interesting to notice that this mechanism allows agents torecord the social and normative information, without necessarilyproactively exploring the world (e.g., with a trial and error proce-dure).


(a)

(b)

(c)

Fig. 1. Agent architectures. (a) Strategic, (b) normative, (c) internal-izer.

In our implementation, a normative agent knowsthat a norm exists and is able to predict that violatorsare punished. Both beliefs are presented as normativebeliefs. However, agents do not know beforehand whatthe surveillance rates of the norm are. During the sim-ulation, agents update (with their own direct experi-ence and observed normative social information) theperceived probability of being punished. The decision-making of a normative agent is also sensible to a risktolerance rate: when the perceived punishment proba-bility is below the risk tolerance threshold, agents willdecide to violate the norm; they will observe the normsotherwise. Although this process might provide agentswith a maximum benefit, it yields the computational

cost of evaluating each of the options at everytime stepand a cost to the society as norm abiding agents willonly behave normatively in the presence of punish-ment.

As we explained in Section 6, an internalizer agentis basically a normative agent who is able to inter-nalize norms (same normative structure plus an in-ternalization module). An internalizer (Fig. 1(c)) ini-tially behaves as a normative agent. Meanwhile, thesalience mechanism also works in the agents minds.This salience, as specified in Section 5, provides agentswith a measure of importance of a norm within thepopulation, and is constantly updated with the nor-mative social and individual information available toagents. Norm observance, violations, norm defense,explicit deontic messages are some of the parametersthat affect the norm salience.

From the technical point of view, an agent will in-ternalize a norm when both following conditions arefulfilled:

1. The norm salience is above a certain threshold,indicating the norm is important within the soci-ety.

2. The agent rationale specifies that it has done thebenefit–cost calculation for all the possible ac-tions (as all normative agents do so during thedecision-making process) for a certain number oftimes and that this cost of calculation times isnow above its tolerance threshold.

Once a norm is internalized, internalizers do notmake the benefit–cost calculation anymore and theyobserve the norm as an automatism. Nevertheless, thesalience mechanism is still active and is still contin-uously being updated. In this way agents are able tounblock the automatism of a norm and return to thebenefit–cost calculation stage.

The third types of agents are strategic ones. Theydo not know about the existence of the norms and willalways choose the action that has provided them themaximum benefit in the past. This agents have beenimplemented as Q-learning agents as in [47,51].

7.1. Simulation model

In order to observe the dynamics of agents behav-ior, we have designed a multi-agent based simulationwhere the different types of agents can interact to per-form the same task. The simulation is structured inthe following way (see Fig. 2). Agents are connectedwith a social network which specifies their interaction


Fig. 2. Simulation process.

topology. Each timestep, they will repeatedly inter-act in randomly formed couples of neighboring agentsand play a Prisoner’s Dilemma (PD) (first stage). Af-ter the game, all agents can punish their opponents incase they defected (second stage). In order to provide amore realistic model of punishment than those presentin [26], we include two different types of punishment:strategic punishment and normative punishment. Bothof them yield a cost to the punisher, affecting also theutility of the punished defector and producing a de-terrent effect. However, the normative punishment isalso accompanied with a deontic message or a nor-mative evaluation making explicit the norm existence.Both normative and internalizers inflict strategic pun-ishment while the norm’s salience of the agents is be-low a certain threshold (agents do not yet believe in thenorm); once this threshold is reached, agents will con-tinue punishing normatively, obtaining the same deter-rent effect but also adding an educative perspective tothe sanction. We consider that punishing normativelyis also a meta-observance of the norm.

For the proposed agent architecture, agents need toprocess the normative social information available intheir environment. This social information affects di-rectly the norm salience mechanism which orchestratesthe internalization process. In order to obtain this in-formation we have added one more phase to the game:after all agents have chosen their first stage action (co-operate or defect) and their second stage action (punishor not), they can draw the following social normativeinformation within their neighbours:10

– First stage cooperators: neighboring agents whochose cooperate in the PD.

10The amount of neighbours is defined by the structure of the so-cial network in which agents interact.

– First stage defectors: neighboring agents whochose defect in the PD.

– Non-punished defectors: amount of neighboringagents that defected in the first stage and were notpunished in the second stage by any other agent.

– Consistent strategic punishments observed:neighboring consistent agents11 who have appliedstrategic punishment, or, neighboring agents whohave received a strategic punishment from a con-sistent agent.

– Consistent normative punishments observed:neighboring consistent agents10 who have applieda normative punishment, or, neighboring agentswho have received a normative punishment froma consistent agent.

– Consistent educative messages observed: neigh-boring consistent agents10 who have sent an ed-ucative message, or, neighboring agents who havereceived an educative message from a consistentagent.

– Consistent strategic punishments received:amount of strategic punishments a certain agentreceives from a consistent agent.

– Consistent normative punishments received:amount of normative punishments a certain agentreceives from a consistent agent.

– Consistent educative messages received:12

amount of educative messages a certain agent re-ceives from a consistent agent.

An aggregation of this social information providesagents with an estimate measure of the salience degreeof a norm within the society and from their subjectivepoint of view. The weights’ values used in the aggre-gation calculation (interpreted from [17]13) are shownin Table 1.

The resulting value from the aggregation functionsis normalized between 0 and 1 and added accumu-latively to the norm salience. We have not permit-ted salience to go above 1 or below 0. Normativeagents update their probability of being punished (non-punished defectors divided by number of defectors)

11An agent is consistent if when it chooses to punish, has alsocooperated in the PD.

12These normative messages are sent by educational agents.These types of agents are agents which communicate norms. Therole of educative can be assumed by a normative agent or by an in-ternalizer.

13The intuitive justification for the usage of these values is that ofgiving a higher weight to those social cues that are highly related tonormative motivations and lower weights to those which would haveselfish/utilitarian motivations.


Table 1

Normative social information weights for salience aggregationfunction

Social cue Weight

Self first stage cooperation 0.99

Self first stage defection −0.99

Self second stage strategic punishment 0

Self second stage normative punishment 0.99

Observed first stage cooperators 0.33 × n

Non-punished defectors −0.66 × n

Consistent strategic punishments observed 0.33 × n

Consistent normative punishments observed 0.99 × n

Consistent educative messages observed 0.99 × n

Consistent strategic punishments received 0.33 × n

Consistent normative punishments received 0.99 × n

Consistent educative messages received 0.99 × n

each timestep with the same information by which theyupdate their norm salience.

7.2. Experimental design

In order to study the behavior of the 3 types ofagents, normatives, internalizers and strategics, someparameters are fixed for all the simulations performed.Agents are located in a fully connected network, allow-ing agents to potentially interact with all other agentspresent in the simulation.

As the objective of this work is to observe the be-havior of the internalizers, we will not focus on pun-ishment dynamics (for further information we refer thereader to [3]); the cost of punishment will be fixed to1 unit for punishers reducing violators utility in 4 units(1:4 punishment technology is used because it has beenshown [42] to be more effective in promoting coop-eration). The decision-making of both normative andinternalizers is not affected by the cost of applying apunishment; however, the cost of being punished af-fects their first stage decision-making. All the simula-tions are populated with a fixed amount of 10 educa-tors,14 and a total population of 100 agents, varyingthe proportion of pure strategic and internalizers (e.g.,in a population where there are the fixed 10 educators,and 20 internalizers, the other 70 agents are strategicagents). We remind the reader that an internalizer is anormative agent that will eventually internalize a be-havior and that is the reason why we do not explicitlyinclude normative agents in our experiments: by hav-

14Educators are agents hold the norm since their creation and alsosend educative messages.

ing internalizers, we already represent the dynamics ofthese normative agents.

Our hypotheses to be proven through experimenta-tion are:

– Strategic and Normative agents do behave in aselfish efficient strategy, provoking a collapse ofcooperation when punishment rates are low.

– Internalizer agents are able to maintain the socialorder imposed by social norms even if those arenot defended (temporarily, allowing the systemto recover from possible failures; permanently,salience will indicate the state of the norm, allow-ing internalizers to unblock the automatism gen-erated by the norm and start the benefit–cost cal-culation process again).

– Internalizer agents are able to unblock normativeautomated actions when this norm disappears.

The results presented in this section are the averageresults of 25 simulations. All non-strategic agents areinitialized with a constant propensity to violate normsif the perceived probability of being punished is equalto or lower than 30%. They are also given a constantexploration rate of 0.5% allowing them to take a firststage random action. To study the effect of punishmenton norm internalization, we have designed several pun-ishment probability distributions:

1. Constant: agents have a constant probability ofbeing punished.

2. Linearly increasing: the probability continuouslyincreases as the simulation runs.

3. Linearly decreasing: the probability continuouslydecreases as the simulation runs.

4. Step down: at a certain moment of the simulation,the probability of being punished drops from 1 to0, i.e., from total punishments to no punishment.

5. Step up: at a certain moment of the simulation,the probability of being punished raises from 0to 1, i.e., from no punishment to total punish-ments.

These different punishment probability distributionsallow us to observe the dynamics of cooperation withthe different types of agents. On the other hand,these punishment probabilities are completely unre-lated with the second stage mixed strategy decision.Once agents decide whether to or not to punish a defec-tor, then, this decision might be unachievable becauseof environmental conditions (simulated with these pun-ishment probability distributions).


7.3. Experimental results

This first experiment has the main objective ofshowing the norm internalizers dynamics when withina society formed by pure strategic agents and otherinternalizers. Information such as the salience of thenorm and the number of internalizers in the automa-tism phase (when norms are internalized) are those thatwe need to observe carefully.

The experimental results in Figs 3 and 4 show thedynamics of the internalizers.15 In this experiment, thepunishment probability distribution decreases linearly.These results are obtained from simulations with afixed amount of agents (=100), and changing the dis-tribution of internalizers and strategic agents.

The x-axis specifies the timesteps of the simula-tion, the y-axis specifies the number of internalizers.In Fig. 3(a) the z-axis specifies the cooperation rates;the salience in Fig. 3(b) and the internalization rates inFig. 3(c). We can observe (in Fig. 3(a)) that the amountof internalizers is directly proportional to the stabil-ity of the cooperation rates: the more the internalizers,the longer the cooperation. The explanation of the phe-nomenon is found in the dynamics of the internaliz-ers: they start behaving as normative agents, and as thepunishment probability is above their risk tolerance,they comply with norms. At a certain moment, thosethat are able to internalize, do so (as it can be seenin Fig. 3(c)). However, when the punishment rates de-crease, strategic agents start defecting. As specified be-fore, the fewer the internalizers in the population, themore the strategic agents. When the perceived punish-ment probability decreases, strategic agents start de-fecting. With a higher number of strategic agents, theamount of defections is also higher, affecting saliencein a more radical way (as can be seen in Fig. 3(b)).

The conclusions drawn from this first experiment areconfirmed by the following one (in Fig. 4). The simula-tion conditions are the same except for the punishmentdistribution, which in this case is a Step down distribu-tion: agents will be able to punish up to timestep 600,after that, they will not be able to punish anymore.

In Figure 4(a), we can observe that in populationswith a larger amount of internalizers, cooperation re-mains stable unlike in those with fewer internalizers.By observing the results in Fig. 4(c), we can analyzethe dynamics of internalizers: internalizers take around

15Internalizers are normative agents (that already hold the norm)and do have the capability to internalize the norms.

600 timesteps to internalize the norm; after punishmentdisappears (in timestep 600), internalizers remain inthe internalization phase for a longer time when thenumber of them is higher.

In order to observe the dynamics of internalizerswith respect to the amount of strategic and normativeagents, we run an exhaustive experiment of the searchspace with all the punishment distributions presented.However, due to space constraints, we show the resultsrelated to a low amount (10) of internalizers (in Fig. 5).Obviously, by introducing a higher amount of internal-izers, cooperation rates stabilize in a more robust waythan with fewer internalizers.

(a)

(b)

(c)

Fig. 3. Internalizer dynamics with a linearly decreasing punishmentdistribution. Internalizers are normative agents able to internalize.(a) Cooperation rates, (b) salience, (c) internalization rates.


(a)

(b)

(c)

Fig. 4. Internalizer dynamics with punishment distribution step downat 600. Internalizers are normative agents able to internalize. (a) Co-operation rates, (b) salience, (c) internalization rates.

The experiment with a low amount of internaliz-ers (Fig. 5) shows that cooperation rates are more un-stable with a higher number of strategic agents. Thisphenomenon is caused by the natural propensity ofstrategic agents to exploration and utility maximiza-tion; when punishment rates decrease, they start de-fecting, thus leading to unblock the generated automa-tism. In the same figure, we can also observe how nor-mative agents are sensitive to punishment probabilitydistributions: when the perceived probability is below0.3 (their risk tolerance rate), defection is the dominantstrategy.

One important remark about the effects of internal-ization concerns the costs of punishments for society.

In Fig. 6, we can observe how in populations with ahigher amount of internalizers (and lower amount ofstrategic agents), the amount of punishments inflictedis lower than in those with fewer internalizers. A re-duction of the amount of costly punishments would im-ply a significant reduction in agents’ expenditures tomaintain cooperation.

7.4. Discussion

The proof-of-concept simulation model has con-firmed our initial hypotheses. Allowing different typesof normative agents to interact provides system policy-makers with a tool that can help them predict the dy-namics of prosocial behavior.

Internalizers are endowed with a rich cognitive ar-chitecture allowing them to maintain high cooperationrates even when punishment rates are low. This phe-nomenon is not observed when dealing with popula-tions of strategic agents, whose ultimate intention is tomaximize their utility, leading to a general collapse ofcooperation. We have observed an interesting phenom-enon with normative agents, which will maintain coop-eration when the punishment rates are above their risktolerance threshold.

These results would lead us to think that a com-plete population of internalizers would be the best so-lution in terms of system performance; unfortunately,this is not true. Internalizers do need a certain amountof strategic or normative agents to unblock the nor-mative automated actions when necessary. The normsalience mechanism allows the system to recover froma possible failure in the sanctioning structures. How-ever, when the norm disappears, agents will eventuallyunblock the automatism generated by the internaliza-tion process and start the whole process of norm recog-nition and internalization again.

We have also observed that a significant number ofinternalizers is convenient for the society in general, asthey keep the cost of punishment low.

8. Advantages and disadvantages of norminternalization

Hypotheses concerning the effect of internalizationfollow from the properties of internalization analyzedso far. To some extent the advantages of internalizednorms are easily identifiable: norm compliance is ex-pected to be more robust if norms are internalized thanis the case when conducts are ruled only by external


(a) (b)

(c) (d)

(e) (f)

Fig. 5. Cooperation rate with different punishment probability distributions and with 10 internalizers. (a) Pr = 0.2, (b) Pr = 0.4, (c) Pr = 1,(d) linearly decreasing, (e) step down at 600, (f) step up at 600.

Fig. 6. Punishments applied in a linearly increasing probability dis-tribution.

sanctions because they emancipate the norm addressesfrom external, sanctioning entities.

However, what should we expect from the compar-ison between internalized and fully endogenous men-

tal states? Internalized goals are here hypothesized tobe more persistent and lead to a more vigorous goal-attainment [8] than originally inner goals. The argu-ment is based on prospect theory [36] (for a recent con-tribution, see [1]), which assumes loss aversion, i.e.,people tendency to strongly prefer avoiding losses toacquiring gains, as a prominent feature of human be-ings. Internalized goals are already formed in the mind:unlike fully endogenous goals, internalized ones areselected among goals initially acquired under the ef-fect of external influence. The more effort is investedin the attainment of these goals, the lesser they will beabandoned later, the more vigorously will they be at-tained.

A specific hypothesis is based on the confirmationbias (according to which people are likely to accept


inputs that confirm their beliefs and to reject discon-firming ones [41]; for a recent contribution, see [48]).Based on it, the internalizer is expected to show higherintolerance with regard to norm violation than boththose who follow the norm under external enforcementand those who are spontaneously motivated to behaveaccordingly. Violation is a disturbing factor for the in-ternalizer, which might lead it to weaken and even re-vise the commitment made. Hence, norm internalizersare expected to be more consistent and compliant thanexternally-enforced norm observers and endogenouslymotivated agents. A further consequence of the theoryis that agents are much better at defending the internal-ized norms than externally-enforced observers. A con-sequence of the latter prediction is that norm internal-ization is decisive, if not indispensable, for distributedsocial control. Internalization is probably not only onemechanism of private compliance, but also a factor ofsocial enforcement.

In short, internalization is a good predictor of com-pliance and second order cooperation (i.e., urging oth-ers to comply with the norms [35]). But what are, ifany, the disadvantages of internalization? Again, thetheory leads to formulate some hypotheses. First, in-ternalization takes long and it is not necessarily suc-cessful: self-training may be too hard and it oftenrequires several trials before getting through. Peoplealmost never quit bad habits and antisocial conducts onthe very first try. Second, failures may have counter-effects: after a number of unsuccessful attempts, lossof self-esteem and feelings of helplessness may ren-der too weak and fragile future private commitments,and jeopardize internalization. The question of courseis what are the factors that may favor its success onthe first try. Third, internalization emphasizes selec-tion of inputs, autonomy. Moral autonomy is often en-couraged at least in western societies, but it may havecounter-effects as well. For example, it may lead to ex-cessive variance in compliance. Fourth, how does inter-nalization perform in societies characterized by a highdegree of norms, possibly in sharp conflict with one an-other? One might expect that internalization is incom-patible with perceived norm and value conflicts. Couldit be that the future of societies is characterized by frag-ile and variable internalization? Another question forinvestigation.

9. Conclusions and future works

When Vygotsky first formulated his theory of in-ternalization, he noted that only “the barest outline of

this process is known” [53, p. 57]. We do not know,yet, how people manage to internalize beliefs and pre-cepts with a reasonably adequate success, partly be-cause we still do not agree about what to investigate orwhat we mean by this notion. Consequently, no usefulnotion and model is available for applications, despiteits wide and profound implications. Questions suchas how norm internalization unfolds, which factorselicit it, which are its effects, obstacles and counter-indications, are issues of concern for all of the behav-ioral sciences. The internalization of social inputs is in-dispensable for the study and management of a broadspectrum of phenomena, from the development of a ro-bust moral autonomy to the investigation and enforce-ment of distributed social control; from the solutionto the puzzle of cooperation, to fostering security andfighting criminality, etc. After a cognitive definition ofthe subject matter, the paper presents and discusses thebuilding blocks of a rich cognitive model of internal-ization as a multi-step process, including several typesand degrees of internalization. Next, factors favoringdifferent types of internalization are discussed. Themodular character of BDI architectures, like EMIL-Ais shown to fit the approach advocated in the paper.

In this work we have also implemented and pre-sented the results of the internalization module thathas been incorporated into the existing platform, cre-ating the new EMIL-I-A (EMIL Internalizer Agent).This new implementation allowed us to perform exper-iments and observe the behavior of internalizers in so-cieties with different types of agents. The results ob-tained from the experiments allowed us to observe howEMIL-I-A indeed goes through all the phases of inter-nalization when a norm is salient and returns to its nor-mative behavior when the norm is no more salient. Inorder to achieve this result, a certain amount of strate-gic agents is needed within the society.

What is the value added of a rich cognitive modelof internalization, as compared to simpler ones (e.g.,reinforcement learning)? There are several competitiveadvantages. First, reinforcement learning does not ac-count for the main intuition shared by different authors,i.e., the idea that internalization makes complianceindependent of external enforcement. Second, a richcognitive model, namely a BDI architecture with itsspecific modules, accounts for different types and de-grees of internalization, bridging the gap between self-enforcement and automatic responses. Third, a BDIarchitecture accounting for different levels of internal-ization allows flexibility to be combined with automa-tism, as well as thoughtless conformity with auton-


omy. A BDI system can host automatism, but a simpleragent architecture does not allow for flexible, innova-tive and autonomous action.

In our agenda for future work there are several pointsto cover. As an immediate work to perform, and af-ter observing the satisfactory results of our proof-of-concept simulation, we plan to apply our EMIL-I-A ar-chitecture to a more realistic scenario. We believe thatthe abstract simulations performed in this work havea significant meaning as they represent a widespreadproblem (Prisoner’s Dilemma). However, bringing ourtheories to a more realistic scenario would allow thereader to better understand the advantages provided bythe internalization process, as well as observe their im-portant utility within the society.

The application of our architecture to a more real-istic scenario will also help us to address the secondproblem we deal with. What happens when an agenthas two or more independent norms conflicting withone another? Bringing back the example posed alongthe paper, what would an agent do when facing a redtraffic light but listening to an ambulance siren? Shouldit wait or move on? Salience is a decisive element forus to solve these kind of problems.

Acknowledgements

This work was supported by the Spanish Educationand Science Ministry [Engineering Self-*Virtually-Embedded Systems (EVE) project, TIN2009-14702-C02-01]; Proyecto Intramural de Frontera MacNorms[PIFCOO-08-00017] and the Generalitat de Catalunya[2009-SGR-1434]; the European Science FoundationEUROCORES Programme TECT, funded by the Ital-ian National Research Council (CNR); the EC SixthFramework Programme and European project COSTAction IC0801 Agreement Technologies. Daniel Villa-toro is supported by a CSIC predoctoral fellowship un-der JAE program. We also thank the CESGA and RedeGalega de Bioinformatica for the technical support.

References

[1] M. Abdellaoui, H. Bleichrodt and C. Paraschiv, Loss aversionunder prospect theory: A parameter-free measurement, Man-agement Science 53(10) (2007), 1659–1674.

[2] G. Andrighetto, M. Campennì, R. Conte and M. Paolucci, Onthe immergence of norms: a normative agent architecture, in:Proceedings of AAAI Symposium, Social and OrganizationalAspects of Intelligence, Washington, DC, 2007.

[3] G. Andrighetto, D. Villatoro, R. Conte and J. Sabater-Mir, Sim-ulating the relation effects of punishment and sanction in theachievement of cooperation, in: Proceedings of the Eight Eu-ropean Workshop on Multi-Agent Systems, Paris, 2010.

[4] J.M. Aronfreed, Conduct and Conscience; The Socializationof Internalized Control Over Behavior, Academic Press, NewYork, 1968.

[5] R. Axelrod, An evolutionary approach to norms, The AmericanPolitical Science Review 4(80) (1986), 1095–1111.

[6] A. Bandura, Social Learning Theory, Prentice Hall, New York,1976.

[7] A. Bandura, Social cognitive theory of moral thought andaction, in: Handbook of Moral Behavior and Development,Vol. 1, W.M. Kurtines and J.L. Gewirtz, eds, Lawrence Erl-bauml, Hillsdale, NJ, 1991, pp. 45–103.

[8] J.A. Bargh, P.M. Gollwitzer, A. Lee-Chai, K. Barndollar andR. Troetschel, The automated will: unconscious activation andpursuit of behavioral goals, Journal of Personality and SocialPsychology 81 (2001), 1004–1027.

[9] J.A. Bargh, P.M. Gollwitzer, A. Lee-Chai, K. Barndollar andR. Trotschel, The automated will: Nonconscious activation andpursuit of behavioral goals, Journal of Personality and SocialPsychology 81(6) (2001), 1014–1027.

[10] K. Basu, Social norms and the law, in: The New Palgrave Dic-tionary of Economics and Law, P. Newman, ed., 1998, avail-able at SSRN: http://ssrn.com/abstract=42840.

[11] C. Bicchieri, The Grammar of Society: The Nature and Dy-namics of Social Norms, Cambridge Univ. Press, New York,2006.

[12] C. Bicchieri and A. Chavez, Behaving as expected: Public in-formation and fairness norms, Journal of Behavioral DecisionMaking 23(2) (2010), 161–178.

[13] C. Bicchieri and X. Erte, Do the right thing: But only if oth-ers do so, MPRA Paper 4609, University Library of Munich,Germany, 2007.

[14] W.K. Bickel and M.W. Johnson, Time and Decision, ChapterDelay discounting: a fundamental behavioral process of drugdependence, Russell Sage Foundation, New York, 2003.

[15] M. Campennì, G. Andrighetto, F. Cecconi and R. Conte, Nor-mal = normative? The role of intelligent agents in norm inno-vation, Mind and Society 8 (2009), 153–172.

[16] R. Cialdini and N. Goldstein, Social influence: Complianceand conformity, Annual Review of Psychology 55 (2004), 591–621.

[17] R.B. Cialdini, R.R. Reno and C.A. Kallgren, A focus theoryof normative conduct: Recycling the concept of norms to re-duce littering in public places, Journal of Personality and So-cial Psychology 58(6) (1990), 1015–1026.

[18] P.R. Cohen and H.J. Levesque, Intention is choice with com-mitment, Artificial Intelligence 42(2,3) (1990), 213–261.

[19] A.M. Collins and M.R. Quillian, Retrieval time from semanticmemory, Journal of Verbal Learning and Verbal Behavior 8(1969), 240–247.

[20] R. Conte, Rational, goal-oriented agents, in: Encyclopedia ofComplexity and Systems Science, R.A. Meyers, ed., Springer,2009, pp. 7533–7548.

[21] R. Conte and C. Castelfranchi, Cognitive and Social Action,University College of London Press, London, 1995.

[22] R. Conte and C. Castelfranchi, From conventions to prescrip-tions. towards a unified theory of norms, AI and Law 7 (1999),323–340.


[23] R. Conte and C. Castelfranchi, The mental path of norms, RatioJuris 19(4) (2006), 501–517.

[24] R. Conte and F. Dignum, From social monitoring to normativeinfluence, Journal of Artificial Societies and Social Simulation4(2) (2001).

[25] E.L. Deci and R.M. Ryan, The “what” and “why” of goal pur-suits: Human needs and the self-determination of behaviour,Psychological Inquiry 11(4) (2000), 227–268.

[26] A. Dreber, D. Rand, D. Fudenberg and M. Nowak, Winnersdon’t punish, Nature 452 (2008), 348–351.

[27] N. Epley and T. Gilovich, Just going along: Nonconsciouspriming and conformity to social pressure, Journal of Experi-mental Social Psychology 35 (1999), 578–589.

[28] J. Epstein, Generative Social Science. Studies in Agent-BasedComputational Modeling, Princeton Univ. Press, Princeton–New York, 2006.

[29] H. Gintis, The hitchhiker’s guide to altruism: Gene-culture co-evolution, and the internalization of norms, Journal of Theo-retical Biology 220(4) (2003), 407–418.

[30] H. Gintis, The genetic side of gene-culture coevolution: inter-nalization of norms and prosocial emotions, Journal of Eco-nomic Behavior and Organization 53 (2004), 57–67.

[31] C.E. Goodall, Modifying smoking behavior through publicservice announcements and cigarette package warning labels:A comparison of Canada and the United States, PhD thesis,Ohio State University, OH, 2005.

[32] J.E. Grusec and L. Kuczynski, Parenting and Children’s In-ternalization of Values: A Handbook of Contemporary Theory,Wiley, New York, 1997.

[33] M.D. Harvey and M.E. Enzle, A cognitive model of so-cial norms for understanding the transgression-helping effect,Journal of Personality and Social Psychology 41 (1981), 866–875.

[34] C. Horne, The internal enforcement of norms, European Soci-ological Review 19(4) (2003), 335–343.

[35] C. Horne, Explaining norm enforcement, Rationality and So-ciety 19(2) (2007), 139–170.

[36] D. Kahneman and A. Tversky, Prospect theory: an analysis ofdecision under risk, Econometrica 47 (1979), 263–291.

[37] H. Kelsen, General Theory of Norms, Oxford Univ. Press,USA, 1979.

[38] L. Kohlberg, The Psychology of Moral Development: The Na-ture and Validity of Moral Stages, 1st edn, Harper & Row, SanFrancisco, 1984.

[39] M. Mead, Cultural Patterns and Technical Change, The NewAmerican Library, New York, 1963.

[40] M. Neumann, Norm internalisation in human and artificial in-

telligence, Journal of Artificial Societies and Social Simulation13(1) (2010), 12.

[41] R.S. Nickerson, Confirmation bias: A ubiquitous phenomenonin many guises, Review of General Psychology 2 (1998), 175–220.

[42] N. Nikiforakis and H.-T. Normann, A comparative staticsanalysis of punishment in public-good experiments, Experi-mental Economics 11(4) (2008), 358–369.

[43] T. Parsons, The Structure of Social Action. A Study in SocialTheory with Special Reference to a Group of Recent EuropeanWriters, Free Press, New York, London, 1937.

[44] J. Piaget, Moral Judgement of the Child, Free Press, New York,1965.

[45] H. Rachlin, The Science of Self-Control, Harvard Univ. Press,Cambridge, London, 2000.

[46] J. Scott, Internalization of Norms: A Sociological Theory ofMoral Commitment, Prentice-Hall, Englewoods Cliffs, NJ,1971.

[47] S. Sen and S. Airiau, Emergence of norms through social learn-ing, in: Proceedings of IJCAI-07, 2007, pp. 1507–1512.

[48] R.J. Sternberg, Critical Thinking in Psychology, Chapter Crit-ical thinking in psychology: It really is critical, CambridgeUniv. Press, Cambridge, 2007.

[49] Y. Sun and B. Wu, Agent hybrid architecture and its decisionprocesses, in: International Conference on Machine Learningand Cybernetics, 2006, pp. 641–644.

[50] E. Ullman-Margalit, The Emergence of Norms, Clarendon, Ox-ford, 1977.

[51] D. Villatoro, S. Sen and J. Sabater, Topology and memory ef-fect on convention emergence, in: Proceedings of the Interna-tional Conference of Intelligent Agent Technology, IEEE Press,2009.

[52] G.H. von Wright, Norm and Action. A Logical Inquiry, Rout-ledge & Kegan Paul, London, 1963.

[53] L.S. Vygotskii and M. Cole, Mind in Society: The Developmentof Higher Psychological Processes, L.S. Vygotsky, M. Cole etal., eds, Harvard Univ. Press, Cambridge, 1978.

[54] E. Xiao and D. Hauser, Avoiding the sharp tongue: Anticipatedwritten messages promote fair economic exchange, Journal ofEconomic Psychology 30(3) (2009), 393–404.

[55] E. Xiao and D. Houser, Emotion expression in human pun-ishment behavior, Proc. Natl. Acad. Sci. USA 102(20) (2005),7398–7401.

[56] H. Zhang and S.Y. Huang, Dynamic control of intention prior-ities of human-like agents, in: Proceeding of the 2006 Confer-ence on ECAI 2006, Amsterdam, The Netherlands, 2006, IOSPress, The Netherlands, pp. 310–314.

Norm internalization in artificial societies

Documents

Transcript of Norm internalization in artificial societies