The Wisdom of Networked and Evolving Agents › ~nama › Top › InvitedTalk › 06-01_it.pdf ·...

0 - Artificial Economics ’06 (Aalborg)

　　　　The Wisdom of Networked and Evolving Agents

Akira NamatameNational　Defense Academy，Japan

E-mail: [email protected]

How mutual beneficial rules can be built and sustained without knowing the motives of others, and without sharing goals.


Social Intelligence

<Intelligence is valuable in social contexts>. Economic theory presupposes rationality that rely on the existence of intelligence, but why would such intelligence has evolved?

Two hypotheses have been considered as explanations for this question.(1) The ecological hypothesis:

The demands of the non-human environment were a key impetus to higher intelligence.

(2) The social intelligence hypothesis:A key factor is interactions with other human beings. :The Red Queen effect (co-evolution) in biology was introduced to explain why species have been evolved to higher intelligence.


•Global stability despite many local interactions“It is remarkable that in almost all cases interactions play a vital role, yet many aggregate outcomes are not changed too seriously by them.”(Philip W. Anderson: in his Nobel Prize lecture, 1977)

(2) Endless and meaningless co-evolution?•The Red Queen principle (arms race)The more things change, the more they stay the same

<van Valen (evolutionary biologists), 1973>

(1) Why do we need to focus on interactions?

Basic Issues [1]


Flocking BehaviorsWhat is the mechanism behind flocking behavior?


Relationship Rather Than Rationality

•BOID Model:

<Frocking Behavior consists of three relational rules>

Craig W. Reynolds (1994)

Cohesion Separation Alignment

•Cohesion: Head for the perceived center of mass of the neighbors. •Separation: Don't get too close to any object.•Alignment: Try to match the speed and direction of nearby neighbors.


Basic Issues [2]Self-interest vs. Mutual Benefit

Self-interested agents are often faced with the dilemma of acting in their own interest or pursuing a more cooperative action in social contexts. The prevailing orthodoxy in the evolutionary theory is that the better individual (or gene) is the “unit of selection.”:Group solidarity would work for a desirable outcome for mutual benefit, but evolutionary dynamics based on natural selection can work against it.A challenging task is to identify conditions under which agents are more coordinated than they are acting in their own interest.


• Social sciences: <Crowd psychology>　　herding, cascade, group think,…

• Game theory: <the problem of a coordination failure>The existence of externalities lead to coordination failure and results in Pareto-inferior outcomes.

• Computer science: <the price of anarchy> Selfish behavior may not achieve full efficiency at the collective level.

Madness of Crowds vs. Wisdom of Crowds (1)

Researches on collectives of agents


The Price of Anarchy (1)

Coordination versus competition Equilibria of selfish agents are not efficientPareto optimal is usually not Nash equilibrium

Price of anarchy: Quantify inefficiency in terms of a global objective

Price of Anarchy objective function value at NEPoptimal objective function value=


The Price of Anarchy (2)

The efficient outcome is (L, L)Pure strategy Nash equilibria include (H,H) and (M, M), but not (L,L)

Agent A Agent B

H M L

H 5,50,0

0,02,2

0,12 10,100,0

L 12,0

0,0

M A modified coordination game

Price of Anarchy=5(or 2)/10=1/2 (or1/5)

--How can we improve the How can we improve the price of anarchy price of anarchy without the without the central authoritycentral authority??

--What mechanisms one can What mechanisms one can use to improve the overall use to improve the overall system performance?system performance?


“The Wisdom Of Crowds, Surowiecki, Random House (2004)”A large collection of people are smarter than an elite few.

(Example: Wikipedia)• Surowiecki suggests new insights regarding how our social and economic activities should be organized.• The wisdom of crowds emerges only under

the right conditions, which are: diversity, : independence, : decentralization, and : aggregation.

Madness of Crowds vs. Wisdom of Crowds (2)

Network topology of interactions also play a vital role.


Concept of Social Games

Networked agents repeated play pair-wise games:

Coordination gameNash demand gameRock-scissors-paper game

Agents evolve their strategiesBest-response strategy learning has been the focus

of much research. However, this approach can yield undesirable outcomes.

We seek a proper learning model leading agents to desirable outcomes.


Payoff: Internal enforcement

Network topology:

External enforcementAGENT

It is important to interact with the right peoples.It is not what you know, it’s who you know that account.

Effects of Network Topology

• Lattice network

• Small-world network：Rewiring with probability p=0.5• Random network:Rewiring with probability p= 1


Fixed Rule vs. Learnable Rule 　

Learning: Memory-based strategy choice : Memory → At (strategy at time t)

Is a short memory enough or a longer memory may be necessary in more complex games?: Reactionary strategy with history one: ( ht-1 ) → At

: Finite history of k: { ( ht-k , … , ht-2 , ht-1 ) } → At

Agents play the game repeatedly by evolving a rule rather than by a fixed rule such as TFT.

Own Opp4 0 0 #5 0 1 #6 1 0 #7 1 1 #

strategy at tbitpast strategy Crossover


• The inferior strategy S2 is likely to be selected. The basin of attraction of the retreat is 9 times as large as that of the attack.

• From the point of view of social welfare this is a shame becauseagents are better off at the joint attack equilibrium.

A Coordinated Attack ProblemAgents need to decide to attack or retreat.• If they both attack, they win. • However, one sided attack would result in losing the battle.• (S1, S1): Pareto-dominates (S2, S2),• (S2, S2): risk-dominates (S1, S1)

Coordination Game

Agent A

S1(attack)

S1(attack)

S2 (retreat)

S2(retreat)

1

-9

0

0

1 -9

0 0

Agent B

S1 θ=2/3 S2

Pareto-dominance risk-dominance

θ=0.1 θ=0.1


generation 200 generation 300 generation 2000

Latticenetwork

Small-world network

Random network

Simulation Results: Coordination Game

average　0.99 average　0.98average　1.0

average　1.0average　1.0 average　1.0

average　0.0 average　0.0 average　0.0


What did Agents Learn through Co-evolution?

・The rules of 2,500 agents are aggregated into a few rules.Array location1: Initial strategy2: Choice when outcome is (0, 0)3: Choice when outcome is (0, 1)4: Choice when outcome is (1, 0)5: Choice when outcome is (1, 1)

S1:　0S2:　1

small-world network

random network

lattice network

Own Opp4 0 0 #5 0 1 #6 1 0 #7 1 1 #

strategy at tbitpast strategy


The Phase Diagram of the Plays of Two Agents

00

10 11

01

converges to “00”: Pareto-optimal outcome

• random network

converges to “11” :risk-dominat outcome

•small-world network•lattice network

Initial strategy


Evolutionary Bargaining: Sharing Game

There are three Nash equilibriums in pure strategy with one equitable (M, M), and two inequitable ones (H,L), (L,H)

Agent A Agent B

H = 7M = 5L = 3

H = 70,00,0

0,05,5

3,7 3,35,3

L = 37,3

3,5

M = 5

The payoff matrix of a sharing game

H

LM

Best reply:L

Best reply:M

Best reply:

H

Belief of agent « i »

(Phan, 2005)


H

LM Within-group

Emergence of Self-reinforcing Classes

Selective interaction: Agents can distinguish each other using tags. Classes formation: Agents behave fair (M) to agents within the same group but behave unfair between groups

<Left-hand side> Both black and grey agents are fair within their group (Play M)<Right-hand side>Black agents have beliefs in behavior of grey agents, and their best

response is to play L (70%). Grey agents have beliefs in behavior of blacks. agents, and their best response is to play H, and therefore unfair outcome (H, L) is realized.

Between-group

H

LM

(Axtell, 2001)

(Phan,2005)


Simulation Results: Sharing Game

: Highest payoff Average Payoff

1.91

)(1 RockS

)(2 MS(i) The average payoff per agent (ii) the ratio of each strategy

All agents become to choose M and realize both efficient andfair outcome

H=7 M=5L= 3

H = 70,00,0

0,05,5

3,7 3,35,3

L = 37,3

3,5

M = 5


Repeated Games and the General Theory of Efficiency

Economics recognizes three general ways to achieve efficiency:

Competition: efficient resources allocation via marketsContracts/mechanism design: as a means to enforce efficient outcomesRepeated games: provide a formal framework to examine why self-interested agents manage to coordinate in a long-term relationship


<The Folk Theorem>In a repeated interaction, any mutually beneficial outcomecan be sustained in an equilibrium.

The Folk Theorem in Repeated Games

<Criticism of The Folk Theorem>

The theory of repeated games does not provide a criterion for equilibrium selection

The majority of works on repeated games are symmetric games. The theory of repeated games is rather silent for asymmetric situations.


• λ=2: Conventional R-S-P game• λ>2: Dispersion games

Generalized Rock-Scissors-Paper Games

• A Nash equilibrium strategy: (S1: 1/3, S2: 1/3, S3: 1/3)• The expected payoff at Nash equilibrium : (λ＋1)/3• Pareto-efficiency (average payoff): λ/2• Inequity at Pareto-efficiency : A lucky agent gets λ, and an unlucky agent gets nothing.


Characteristics of RSP Game•Group solidarity would work for a Pareto-efficient outcome for mutual benefit, but evolutionary dynamics, which is based on natural selection, can work against it. •The RSP game is not stable in the evolutionary dynamics. :However, with slight modification of the payoff of the original RSP game, the Nash equilibrium becomes a stable attractor.


•Previous Strategy

•Own •Opponent

•0 •0 •#

•0 •1 •#

•0 •2 •#

•1 •0 •#

•1 •1 •#

•1 •2 •#

•2 •0 •#

•2 •1 •#

•2 •2 •#

•Next Strategy

0：Rock, 1: Scissor, 2: Paper #: 0, 1 or 2

Coupling Rules with Three Strategies

Learnable coupled rule

Agents update the coupling rule by changing value of # .

Coupled rule

Strategy choice is driven by the outcomes of the joint actions


Simulation Results (1): λ=2

max

averagemin

11 .0

)Scissor(S 2

(i) The average payoff per agent (ii) the ratio of each strategy

max

max

min

1 .0

)(2 ScissorS

)(1 RockS

Error rate: 10%

Error rate: 0%


What Did Agent Learn through Co-evolution?λ＝2 （Error rate: 0%）

The rules of 400 agents are aggregated into 67 types

＃＃＃ 0 １０＃ 0 ＃

00 01 02 10 11 12 20 21 22

The phase diagram of the plays between two agents

Learned coupling rules

Initial strategy

Nash equilibrium: 1Pareto-efficiency: 1


Simulation Results(2): λ=10

Error rate: 0%

Error rate: 10%

: Highest payoff

Average Payoff

: Lowest Payoff

2.8

4.0

1.91

3.99

)(1 RockS

)(3 PaperS

)(2 ScissorS

)(1 RockS

)(2 ScissorS)(3 PaperS

(i) The average payoff per agent(ii) the ratio of each strategy

Nash equilibrium: 3.3Pareto-efficiency: 5

The strategy population is close to Nash equilibrium


2 2 2 0 2 1 1 0 2

2 2 2 0 2 1 1 0 0 76

Rule Type 1 126

Rule Type 2

1 2 2 0 2 1 1 0 2

1 2 2 0 2 1 1 0 0 54

Rule Type 3

Rule Type 4

58

00 01 02 10 11 12 20 21 22

The rules of 400 agents are aggregated into 8 rules with some common values.

40

19

17

10

Rule Type 5

Rule Type 6

Rule Type 7

Rule Type 8

2 2 2 0 0 1 1 0 2

1 2 2 0 0 1 1 0 2

2 2 2 0 0 1 1 0 0

1 2 2 0 0 1 1 0 0

Number of Agents with the Same Rule

What Did Agents Learn through Co-evolution? λ＝10（Error rate: 10%）


00 01 21

11 20 02

22 12 10

draw

win

win

draw

draw

Phase Diagram: Two Agents with the Same Rule

lose

lose

lose

win

If the two agents of having the same rule play, they absorb in the drawing cycle.

Initial State


00 01 21

11 20 02

22 12 10

draw

win

win

draw

draw

lose

lose

lose

win

Initial State

Winning phase Losing phase

Phase Diagram: Agents with the Different Rules

Limiting cycle

If the two agents of having the different rules play, they absorb in the limiting cycle.


Emergence of Turn-taking Behavior

By visiting all efficient outcomes alternatively underturn-taking behavior, the agents could achieve

desirable outcomes in efficiency and fairness.


Concluding Remarks (1): The inverse problem of networked agents

Desired collectives

Emergent behavior

Emergent behaviorInverse problem

Forward problem

Interacting agents with micro-motives

<The forward problem>: Analysis，Explanation, Understanding

<The inverse problem>：Infer micro behavioral rules of agents: Design micro-rules through co-evolutionary learning


Agent Based models (ABM) are widespread and many exciting applications, but lots of hype, not enough understanding of when ABM adds value.Replicator dynamics (DE) is the most thoroughly studied dynamic model for evolution with the more successful strategies being imitated more often.

What are the differences between ABM and Differential Equations (DE) methods?

Concluding Remarks (2):ABM vs. DE: What are the differences?


•Rule

•Meta-strategy

World,

State

Learning Algorithm

strategy

Observations,Sensations

Rewards <Learning relationship> <Learning relationship>

--AgentsAgents should learn a should learn a better way to be coupled better way to be coupled with otherswith others

--ABM is the only way for the ABM is the only way for the study of relationshipstudy of relationship

Concluding Remarks (2):ABM vs. DE: What are the differences?(cont.)Best-response strategy learning has been the focus of much

research. However, this approach can yield undesirable outcomes.•Replicator dynamics converge to Nash equilibria which are not always desirable outcomes.


Social norm

Agent

shared rule preference

Internalizationof social norm

Agent

Concluding Remarks [3]:Collective Norm Development

Social norms are rules that are understood by members of a society, and thatguide and constrain their behavior without the force of laws. Emergence and sustainability of social norms as core issues for social sciences. Social norms emerge out of social interaction and they are internalized in networked agents.


Open Problems：Smart Agents Using ICT

(ICT:Information/Communication Technology)

Congestion control receive much attention

What wisdom for avoiding congestion will be emerged in networked smart agents equipped with ICT?


Open Problems(cont.)：Smart Networks Design by Agents

Smart Business Networks，Ubiquitous Web of Services,…How mutual trust can be built and sustained without knowing the motives of others, and without sharing goals.Smart Networks Design with Genetics Approach


New book: World Scientific, 2006


　　　　　　　Thank you!! and Question time

The Wisdom of Networked and Evolving Agents › ~nama › Top › InvitedTalk › 06-01_it.pdf ·...

Documents

Transcript of The Wisdom of Networked and Evolving Agents › ~nama › Top › InvitedTalk › 06-01_it.pdf ·...