Modeling the effects of International Interventions with Nexus Network Learner Dr. Deborah Duong...

Modeling the effects of International Interventions

with Nexus Network Learner

Dr. Deborah Duong

Agent Based Learning Systems

AAAI12 Spring Symposium, Stanford University

2Social Impact Model

Purpose: To show the benefits of Symbolic Interactionist Simulation for the Simulation of Social Aggregation

Purpose and Agenda

Agenda

•Coevolution and Symbolic Interactionist Simulation

•The Sociological Dynamical System Simulation

•SISTER: Symbolic Interactionist Simulation of Trade and Emergent Roles

•Nexus Network Learner

•Modeling Corruption


Coevolving Agents

• Genetic Algorithms, Neural Networks, or Reinforcement Learning Algorithms in Agents can co-evolve

• Agents learn to optimize a function in an environment composed mostly of agents also learning to optimize a function.

• Moving fitness landscapes– Agents apply selective pressure upon each other.– Selective pressure causes the optimal way to achieve the goal

to change over time.

• Evolutionary Stable Strategies– Maynard-Smith’s theory that a co-evolutionary system

converges when no species (or inducing agent) can make a change that will make it better off (Nash Equilibrium)

• Agents Differentiate into a System– Species (or inducing agents) can be both cooperative and

competitive


Symbolic Interactionist Simulation

• Coevolutionary reinforcement learning algorithm.

• Autonomous agents each have an inductive mechanism.– For example, an entire Genetic Algorithm or Neural

Network.– Agents only experience through their senses (no direct

knowledge transfers from other agent minds).

• Agents choose to interact with other agents based on signs that the other agents display.

– Agents induce both the signs to display and the signs that they read.

• Social system emerges– Signs come to mean behaviors. – Behaviors interlock into a system of expectations.– Value function becomes reward function as society

evolves (Adam Smith’s invisible hand).


The First Symbolic Interactionist Simulation: The Sociological Dynamical System Simulation

• A System of IAC Networks as the Basis for Self Organization in a Sociological Dynamical System Simulation

- Duong, 1991; Duong and Reilly, Behavioral Science 1995.

• Employer agents hire from a pool of employee agents, periodically laying off employees.

• Employees who are less talented are laid off in greater proportions than those who are more talented.

• Employees seek employment.

• Employees can choose to wear three different kinds of signs– One of two signs they can not change. This is their “skin color”.– One of three signs they have to pay for with money from

employment. This is their “suit”.– One of three signs they can change arbitrarily. This is their “fad”.

• Employees induce what sign they should wear based on what has gotten them employed in the past.


The Sociological Dynamical System Simulation (SDSS)

• Employees have a hidden, unchanging, talent level that employers can not see until after the employee is employed.

• Employers seek talented employees.

• Employers hire based on the signs the employees wear, inducing how much talent they have from their signs.

• The two “races” of employees are equally talented, but the employers do not know this.

• Employers and Employee Agents both have the same kind of induction mechanism.

– Each Agent has their own Interactive Activation and Competition (IAC) Neural Network to induce the signs they read and display.


Interactive Activation and Competition

• An Employer’s IAC

• The IAC is a Constraint Satisfaction (Hopfield) Neural Network.

• Nodes within each pool are mutually inhibited.

• A central pool contains memory instances.

• Each instance has positive connections to its features.

• To guess the talent of an applicant, the employer “turns on” the applicants features and sees which talent node turns on.

• Simulates schema, or mental groupings of features that go together


SDSS: Emergent Phenomena

• Status Symbols– Employee agents learn to buy expensive suits and Employer

agents learn to seek expensive suits.– Because the less talented get laid off more, they have less

money, and talented employees learn to differentiate themselves.

• Racism and Social Class– One race gets into a vicious cycle: because of schema, by

accident one race gets associated with less talent (even though they are equally talented).

– Many talented in a race could not afford suits because they were never given opportunity.

– One race would have less money than the others as a result

• Meaning attributed to Fads


SISTER: Symbolic Interactionist Simulation of Trade and Emergent Roles

“SISTER: A Symbolic Interactionist Simulation of Trade and Emergent Roles” - Duong 1995, Duong and Grefenstette, JASSS 1/2005.

• Trade is good for “farmer” agents.– Agents need each of four food groups, and as much as they

can get of each.– Agents can make more food if they concentrate their efforts on

fewer of them.

• Agents have efforts to spend on making or trading food as they wish.

• Agents can trade if they have corresponding trade plans.

• Agents have a sign to display to attract trade.

• To learn to trade, agents induce what signs to display and what signs to trade with based on whatever gets them the most of each food in the four food groups at the end of the day.


Coevolving Genetic Algorithms

• Each agent has an entire genetic algorithm that tells it:

– Where to place efforts– What sign to trade with,

what and how much to trade

– What sign to display


SISTER: Emergent Phenomena

• Division of Labor

• As agents learn to trade, their utility increases, and their sign comes to mean a role

• Price

• Goods become valued at standard ratios

• Money

• In a third of the runs, one good is traded for the purpose of trading again

• Different types of stores

• Central bargain stores and local convenience stores


SISTER: Results


Nexus

• SISTER is a “theoretical” symbolic interactionist simulation

– SISTER embodies the formation of social patterns of behavior

– Its scenarios are not realistic

• Nexus is a “data-driven” symbolic interactionist simulation

– In order to do analysis, we must start from a scenario in the real world.

– Being realistic and theoretically correct at the same time is difficult.

– Nexus attempts to mirror the virtuous and vicious cycles of the real world that created its input data.

14

What is Nexus Network Learner ?

• One of the two Nexus Cognitive Agent models that Debbie Duong wrote at the OSD/CAPE/Simulation Analysis Center.

– Nexus Network Learner models Social Role Networks– Nexus Schema Learner models Cognitive Dissonance

• A Simulation of Social Role Networks in which Agents learn: – To choose role partners to perform transactions with:

- Choice based on signs, social markers and communications on past transaction behaviors.

– Transaction behaviors and signs.- Choice based on signs and social markers.

– Based on Cultural Values.

• Social markers, roles, transaction behaviors, signs, role-based communications and cultural values are all input to the program.

– Population data determines the initial population tendencies.– Utility and motivation determines how they change.

15

How Does Nexus Network Learner Work?

• Artificial Intelligence Technologies represent Cognition.– Rule Based.

- An ontology of roles with crisp rules for roles. – Represents general social structures, that can be used in many

scenarios.– Defines utility of transactions.

– Machine Learning.- Bayesian networks initialize social markers , signs/transaction

behaviors, and role choice behaviors.

- The Bayesian Optimization Algorithm (BOA) changes those behaviors based on the utility of transactions.

– BOA can be seeded with initialization data and injected data.– A form of Evolutionary Computation using reinforcement Learning

optimizes (satisfices) utility.– As conditionals change, the equilibrium point moves (in accord

with the New Institutional Economics).

16

What Happens in Nexus Network Learner?

• Individual Agents Choose Network Partners.– Ontology tells who may choose and how many.

- Example: an “Employer” may choose an average of 5 employees with a standard deviation of three.

– Bayesian network tells how the choices are ranked.

– Passive role may have an option to reject offer.

- Example: an “Employee” may reject an employer because a role relation has told her he steals paychecks.

– Ontology may include a chance occurrence of natural attrition.

• Individual Agents engage in transactions.– Account distributions send funds through networks according to rules in

ontology and transaction behaviors in Bayesian networks.

– Probability of observing, reporting, and knowing about behaviors are role-based.

– Agents may go to jail, and not be allowed to participate in transactions for a time.

• Every N cycles, they judge their learned strategies by utility based on transactions that their valued role partners engaged in.

– Ontology determines culturally valued individuals and transactions.

– After testing all strategies agents recombine them.

17

Performing Tests with Nexus Network Learner

• A wide variety of tests relevant to Irregular Warfare (IW) may be performed.

• For example, new network formations and behaviors may be tested based on many different things…

– The effect of different utility functions.- For example, make agents care only for self rather than larger social

network.

– The effect of different penalties.- For example, a penalty attribute that encodes different jail terms or

different chances of getting caught.

– The effect of different exogenous resources.- For example, test resource rents or foreign aid.

– The effect of different abilities to observe.- For example, the effect of a media agent.

– The effect of removing different agents.- For example, measure how long it takes to replace a terrorist leader

• Monte Carlo methods reveal if new structures are the result of different CONOPS.

– Bayesian Networks make Nexus Stochastic

18

How Nexus Agents Learn

• As each agent learns, all the agents coevolve, making them very adaptive.

– Every agent has its own private learning algorithm.– Their behaviors effect the larger social structure and the larger

social structure effects their behaviors.- Micro-Macro Integration is modeled.

– They can adapt to data from other simulations and to initial country data as well.

• The learning algorithm in each agent makes the adaptation to data flexible.

– BOA (Bayesian Optimization Algorithm) can start learning from initial data.

- In the calibration phase. agents to adapt to initial data, so that they generate it though their perceptions and motivations.

- Thus they “explain” the data, going from correlation to cause.– This greater ability to ingest data also allows them to meld with

other simulations in a composition.- Together, composed simulations create a coherent picture of

the social environment.- Conflicts are resolved through mutual adaptation.


Use Case: Modeling Corruption with Nexus Network Learner

Assumptions

• A role perspective is appropriate for examining corruption– With Nexus, we may explore how the patron-client role relationships in traditional

African societies interact with the bureaucratic relationships made necessary by globalization

• People adjust their behaviors based not only on policies but on other peoples reactions to policies– With Nexus, we can explore how agents adjust their behaviors to meet their cultural

goals, given that other agents are doing the same thing• Corruption is a social process, a vicious cycle

– People typically do not participate because they like it, but because they feel they have to

– People take into account both legal penalties and social penalties in adjusting their behavior

– With Nexus, we can explore the effects of legal penalties on eight specific corrupt behaviors, and how they interact with social penalties

• The ability to hide what you are doing (bounded knowledge) is important– The chances that one person will know about another’s corrupt behaviors is based

on the role relation between the two– The more people know about a corrupt behavior, the more likely the perpetrator is to

get caught– With Nexus, we can explore the effects of transparency programs on the chances of

getting caught for a corrupt behavior, based on social relations

Perspective Orientation

• The foundation for the Nexus Network Learner is built upon a rich literature in social constructivism and social emergence where methodological individualism (only thing that matters is an individual) is rejected.

• Nexus embraces the study of both individuals and institutions: endogenously (within the model) modeling the institution-individual linkage simultaneously with the individual-institution links.

Individuals

Institutions

PressureTransform

[1] For an in-depth view see Bourdieu, P. (1977). Outline of a Theory of Practice. (R Nice, Trans.). Cambridge: Cambridge University Press. (Original work published in 1972) and Giddens, A. (1984). The Constitution of Society. Berkeley: University of California Press. Another potential source is Sawyer, RK. (2005). Social Emergence: Societies as complex systems. Cambridge: Cambridge University Press.

Interpretive Social Science Used in Nexus

• From economics: The New Institutional Economics (NIE) (North)– Institutions (Social and Legal Norms and Rules) underlie economic activity

and constitute economic incentive structures– Institutions come from the efforts of agents to understand their

environment, so as to reduce uncertainty, given their limited perception– When some uncertainties are reduced, others arise, causing economic

change– To find the leverages to corruption, NIE would look at actor’s definition of

their environment, and how this changes incentives and thus institutions

• From sociology: Symbolic Interactionism (Mead)– Roles and Role Relations (such as in trade roles and trade relations) are

learned, created during the display and interpretation of signs (such as gender, ethnicity, and other demographic characteristics)

– Institutions (social and legal norms and rules) are commonly accepted interpretations of symbols, that start out as a subjective perception and engrained in society as an objective rule

[1] See http://coase.org/niereadinglist.htm for an extended reading list[2] See Duong, Deborah Vakas, “The Generative Power of Signs: Tags for Cultural. Reproduction” Handbook of Research on Agent-Based Societies: Social and Cultural Interactions, Goran Trajkovsky and Samuel Collins, eds., 2008. http://www.scs.gmu.edu/~dduong/GenerativePowerOfSigns.pdf and also Blumer, Herbert (1969). Symbolic Interactionism: Perspective and Method. Berkeley: University of California Press.[3] Duong, Deborah Vakas and John Grefenstette. “SISTER: A Symbolic Interactionist Simulation of Trade and Emergent Roles”. Journal of Artificial Societies and Social Simulation, January 2005. http://jasss.soc.surrey.ac.uk/8/1/1.html.

http://coase.org/niereadinglist.htm

http://www.scs.gmu.edu/~dduong/GenerativePowerOfSigns.pdf

http://jasss.soc.surrey.ac.uk/8/1/1.html

Conceptual Model

• Nexus is a model of corruption based on the theory that corruption is a result of globalization.

• Many social scientists assert that corruption is the result of conflict between the roles and role relations of the kin network and the trade and bureaucratic networks, separate social structures with their own institutions forced together because of globalization.

[1] Smith, Daniel Jordan. 2007. A Culture of Corruption. Princeton: Princeton University Press.

Incentives

Trade Network

Bureaucratic Network

KinshipNetwork

Transaction-based Utility

Strategy (Behavior)

Nexus Main Components

• Nexus models individuals and their interactions.

• Individuals have various roles on the three different networks and dynamically interact with other agents through these roles.

– Retailer – Customer– Government Employer – Government

Employee– Head Of Household– Dependent

• Role Networks are Input to Nexus

• Only commodity is Money – Agents pass money to other agents’

accounts– External support is in the form of

injections of funds to certain individuals (that have certain roles)

– Utility of agents (their “happiness”) is raised when they spend the money on things they need

Trade Network


KinshipNetwork

Role Interactions

IndividualDetermined TraitsSituational TraitsBehavioral Traits

Transaction-based UtilityExperienceCognition

InstitutionsEmerged Learned Attributes

External ControlUser-defined Policies

ExternalSupport

Nexus Main Components: Individuals

• They want their kin to be happy, and can think about how to adjust their behaviors towards that goal, based on experience of what met that goal in the past

• They have the demographic characteristics, both determined and situational, of the modeled country

– Determined Traits: Gender, Ethnicity, Age, etc.– Behavioral Traits: Tendency to Steal or Bribe, based on other traits and on

learning during the run– Situational Traits: Employment, Are they under penalty, etc.

• They actively seek role relationships, following socio-cultural rules about who proposes the relationship, what sort of person is chosen

– For example, a husband chooses a wife, or an employer chooses an employee – They judge others based on characteristics they can see or they have heard

rumors about

• There role responsibilities include the distribution of funds to accounts they are responsible for

• They are happy when funds flow through certain accounts, for example, from the household budget to the grocery store income.

• They have differing length legal penalties, as well as social stigma

Nexus Main Components: Role and Role Relations

• There are eight types of corruption relations possible in the three networks (example actions provided):

– Nepotism: Hiring Kin/ Trade Network– Commission for Illicit Services: Bribing/Government Network– Misappropriation: Stealing/ Trade or Gov Network– Rig Election: Elected Officials bribing for Employment– Gratuity: Bribing/ Trade Network– Unwarranted Payment: Accepting Bribes/Government Network– Levy Toll Sidelining: Stealing/Government Network – Scam: Stealing From Customer in Trade Sector.

• There are many other types of role and role relations (64) in the model:– Each role has a corresponding role– Roles are dynamic (such as an agent can move from a government employee to unemployed)


Gov Employee Gov EmployerAttempts to Bribe

Accepts/Rejects Bribe

Observer

Nexus Main Components: Kinship Network

Active Roles Corresponding Passive Role

Father Child

Head Of Household Home Receiver

Husband Wife

Brother

Dependent (Provider)

MaternalAunt

MaternalCousin

MaternalGrandparent

Mother

Parent

PaternalCousin

PaternalGrandparent

PaternalUncle

Sibling

Sister

Spouse

Provider (Dependent)

• Three main active roles, from which 14 more are derived

• Derived Roles are used to model Residence

– Utility (Satisfaction) calculated based on Residence in Anthropology

– Matrilineal, Patrilineal of Neolocal

• Support account goes from Provider to Dependent

Derived Roles

Nexus Main Components: Trade Network

• Derived Roles are ten different income levels

• Accounts include personal salary, employee salaries, money for office purchases


HeadOfCorporation CorporateReceiver

Customer Retailer

Employer Employee

Purchaser Vendor

Service Providee Service Provider

Nexus Main Components: Bureaucratic Network


Taxpayer Taxman

GovernmentEmployer GovernmentEmployee

GovernmentPurchaser GovernmentVendor

HeadOfGovernment GovernmentReceiver

Service Providee Service Provider

• Derived Roles are ten different pay grades

• Accounts include corporate and income taxes, government salaries, government office money, government money for purchases

Lets talk about the Agents

• Agents are able to learn and adapt to new role behaviors through the use of evolutionary computation techniques of artificial intelligence, also known as genetic algorithms. (cognitive agents)

• They learn other behaviors based on utility. – Utility is in the trade interactions (transaction-based utility) of the themselves and the kin that an

agent cares about. – Agents learn how to navigate their environment according to their individual traits and

experience through their own Bayesian Optimization Algorithm. – Agents learn the type of persons to include in their social network, including kinship, ethnicity,

and bribing behavior. – They also learn whether to divert funds across networks through bribing and stealing. Agents

have money in accounts (which is a situational attribute)

• Corruption behavior changes through synchronous individual interaction, driven by new incentive structures created from government policies, agent’s reactions to those policies, and agent interaction.

Determined

Situational

Behavior

Traits

BehaviorCognitive

Mechanism

Utility

Experience

Computational Model

• Nexus was created with the REPAST Simphony, a free and open source agent-based modeling toolkit

• REPAST makes use of the social network software Jung, shown below

– Dots along the circle are agents’– Different colored arrows

represent relations of different networks

- Bureaucratic is Blue, Trade is Green, Kin is Red

• All the analysis functions of Jung and data mining software Weka are available for Nexus

– Weka displays number of agents that did each type

– Jung can describe characteristics of the network like centrality, reach, etc.

- With Jung, you can tell who are the important actors

So what happens first? (Initialization)

• Demographic Data Input– The input to Nexus are the demographic characteristics for an entire population.

- Physical characteristics, social categories, and behavioral traits that are based on these physical characteristics and social categories.

- Variables we are trying to explain are used to calibrate the simulation in the beginning.- Example: We know that a subset of the population in this region who have characteristics

A and B, have a greater propensity of corruption than you would expect by chance.

• Application of Bayesian Network Algorithm (Data Interpretation)

• Initial Population Representation– Describes characteristics that agents cannot change, for example, social markers such as

ethnicity or gender. (Determined Traits)– Describes characteristics that agents can change on an individual basis during the simulation,

for example, behavioral characteristics, such as bribing or stealing, or preferences for choices of others in social networks. (Situational and Behavioral Traits)

– Describes demographic characteristics which individual agents do not learn, but are rather the output of the computations made during the simulation, such as unemployment statistics. (Aggregated)

Demographic DataApplication of Bayesian Network Algorithm

Initial PopulationRepresentation

Application of Bayesian Network Algorithm

• A Bayesian network is a graphical model that encodes probabilistic relationships among variables of interest. When used in conjunction with statistical techniques, the graphical model has several advantages for data analysis:

– Because the model encodes dependencies among all variables, it readily handles situations where some data entries are missing.

– A Bayesian network can be used to learn causal relationships, and hence can be used to gain understanding about a problem domain and to predict the consequences of intervention.

– Because the model has both a causal and probabilistic semantics, it is an ideal representation for combining prior knowledge (which often comes in causal form) and data.

– Bayesian statistical methods in conjunction with Bayesian networks offer an efficient and principled approach for avoiding the over-fitting of data.

– Methods to construct Bayesian networks include using prior knowledge and implementing the Bayesian statistical methods for using data to improve the models. This includes both parameters and the structure of a Bayesian network, and techniques for learning with incomplete data.

Source: Neural Network Learning and Expert Systems, Stephen I Gallant, MIT Press, Cambridge, MA, 1993

Initial Population Representation

Bayesian Logic

Observed CorrelationsExample: We know bribing employers is common practice in Ethnic Group A

Real World DataExample: % of Ethnic Population in a Region

How does the Cognitive Mechanism work?

• In this model, the cognitive mechanism is the Bayesian optimization algorithm (BOA).

• The output of this algorithm is an agent’s strategy (how to distribute funds to maximize utility – network choices and behaviors)

• Recall at model initialization, the raw data is converted by a Bayesian network into model input. Among other things, this process generates the individual agent traits.

– Determined (or fixed traits) are set per agent for the simulation run. – There are also two categories of behavioral traits (or learned behaviors):

- Learned network choice behaviors (Initialized Random) – You choose a network partner when you choose a wife or

employee– You may choose a wife or employee based on ethnicity, – You may choose an employee based on whether he bribes you– You may reject an employer if your kin tells you he steals

- Simple learned behaviors (set by Bayesian network during initialization)

– You may learn not to steal even if you started out that way, if it harms your dependents

So what happens when you hit go? (Simulation)

• Remember that Agents are initialized with money in their accounts. – When the money reaches a certain threshold they will distribute it.

- Distribution includes trade, for example, you distribute your money to a grocery store, and gain utility for yourself in doing so

– Exogenous funds are then pumped in on a regular basis (if there are any like diamond revenue)

- Wages, Taxes, Accounts Receivable are internal

• All those with active roles seek relationships. (an active govt employee will seek a new govt employer)

– Both active and passive agents have thresholds for initiating or accepting a partnership.

– There are distance thresholds associated with the role choice, that make them have to match by a certain amount or no partner will be chosen.

• As required, agents update their strategies (based on experience, traits, and evaluation of past strategies) to maximize utility.


Experiment

37

Experiment: Stuck in Stealing Mode

• Comparison of the evolution of a society which initiates in a strong vicious cycle of stealing to one with more moderate levels of stealing.

• If they started out stealing excessively, they never learned not to, never attempted to find service providers who wouldn’t steal from them

• If the stealing is in more moderate amounts, agents learn to find service providers that do not steal from them within two years

• Agents in a stealing vicious cycle never use bribing to accomplish goals

• Agents with moderate stealing used bribes, but after fifteen years, employers and service providers stopped bribing

• Convergence occurs at the fifteen year mark

• After 15 years in both the excessive stealing scenar-io and the ‘normal’ stealing scenario, we see ho-mogeneous responses across strategies. For in-stance, agents generally provided the same re-sponse for a particular parameter (say ‘Bribe-ForServices’) after 15 years as opposed to more heterogeneous responses for the same parameter after two years.

• Implication: Diversity of Behavior is needed for flexibility


Summary

39

Symbolic Interactionist Simulation

• Symbolic Interactionist Simulation is a form of Reinforcement Learning by Coevolution.

• Agents learn associated rules in the form of actions to take with other agents based on signs displayed and read.

• Symbolic Interactionist Simulation can be Theoretical (SISTER) or Data-Driven (Nexus).

• Symbolic Interactionist Simulation can model the motivation based vicious and virtuous cycles of behavior that determine social structure


Questions and Comments

POC: Deborah Duong [email protected]

Modeling the effects of International Interventions with Nexus Network Learner Dr. Deborah Duong...

Documents

Transcript of Modeling the effects of International Interventions with Nexus Network Learner Dr. Deborah Duong...