Page | 1

Replication of the MacroABM Model

Replication Issues in the Context of Economic Agents

Richard Oliver Legendi AITIA International and Eötvös Loránd University

Budapest, Hungary

rlegendi@aitia.ai

Laszlo Gulyas AITIA International and Eötvös Loránd University

Budapest, Hungary

lgulyas@aitia.ai

Abstract – Agent-based approaches are getting more and more attention

recently. In our current work we replicated the initial model of Domenico Delli

Gatti et al. described in their work entitled Macroeconomics from the Bottom-

up. We address the question of validating and verifying simulations in general,

also in the context of economic modelling, and summarize the lessons we

learnt from the replication of the aforementioned model. The results highlight

the importance of explicit documentation of the actors, timing of the events,

and partial results that replicate the hallmarks of the model which can be

verified independently with a set of simulation runs.

replication, simulation, complex adaptive systems, emergent economics

Introduction Classical economy created well defined approaches and several tools that seemed to serve us for the

last decades. However, recent economic fluctuations emphasise the need of alternative tools,

approaches, frameworks and methodologies to be able to describe such phenomena. Recent works

emphasize their weaknesses [1].

Econometric models are in principal statistical models used to predict the state of a system,

supposing that the trends are generally unchanged. In other words, they work only if the

system stays (more or less) in the same state, but fails in the face of wild oscillations we are

experiencing recently.

Another common criticism stated against widely used tools (specifically: against dynamic

stochastic general equilibrium models) is that actors are homogeneous, the system has an

„optimal” equilibrium state, and assume a perfect world (where great oscillations like crises

are ruled out inherently).

Recent years show substantial growth in interest among scientists for alternative approaches,

especially towards agent-based modelling. The reason is twofold. First, the agent-based framework

allows us to move away from the representative agent paradigm into a model of heterogeneous

actors, agents. Second, the bottom-up approach enables the system to change in a non ad-hoc way.

Page | 2

This approach is popular in various fields like political science, social science and economics as well,

because it allows us to conduct experiments with the modelled system. This computational model

helps us find emergent behaviour that propagates from lower (micro) level interactions to higher

(macro) level regularities in the formalized phenomena.

Replication: Verification, Validation and Certification In the case we consider simulations as experiments, reproducibility is a crucial question. In the

context of models created to describe real-world phenomena (for instance in physics and natural

sciences), emphasis must be put on the reproducibility of experiments to validate the results as a

scientific result. If a model cannot generate the same output for the exactly same conditions, its

scientific value is questioned.

In the context of computer simulations, there is a growing list of publications covering the topics of

docking (the alignment of different models to demonstrate they produce similar results) and model

replications (the reimplementation of an original model, typically in a different environment). An

extensive review of these approaches can be found in [22].

Alignment of computational models, or docking, as referred commonly (in analogy to orbital docking

of dissimilar spacecrafts), is a process where we fit a series of models against each other, and try to

verify if they can produce the same results – which is crucial for equivalence testing of models. The

ability to tell if two different models targeting the same phenomena can (or cannot) generate the

same results is essential. A few examples of successful docking experiments include [6], [18] or [19].

Replication is another critical issue, which is an important method of scientists. These models were

designed to describe, explain or predict different real-world phenomena, and without being able to

replicate artificial models it is impossible to target real-world systems. Unfortunately, most of the

agent-based models were never replicated by anyone but the original author, with the exception of

some „classic” models [15]. Questions like how to replicate a model, and how to evaluate the

replication is often raised at several forums.

One of the major problems is ambiguity: there are several concepts how a model should be defined

in order to minimize uncertainty. Several different approaches exist, from textual frameworks like

the Overview, Design concepts, and Design details (ODD, [20]), extensions on top of standardized

modelling languages like AgentUML [21], but there is still no consensus on using them. Until these

frameworks become widely accepted and applied, replication is an issue of concern.

An example for replication of a classic agent-based model on the platform we are also approaching is

[16], while several notable examples can be found in [22].

The authors of the Bottom-up Adaptive Macroeconomics (BAM) model [5] applied empirical external

validation based on real-world data. In our current work, we validate the simulation through

replication. Our goal is to create an alternative implementation in a different environment and verify

if these models produce the same results – in other words, we test their identicality.In our recent

work, we successfully created an alternative implementation of the original BAM model. The

replicated model resembles the results and hallmarks of the original one. First, we briefly describe

the structure of the original model, and discuss the reasons for replication. Then, we demonstrate an

Page | 3

approach we used to be able to generate comparable simulations, and summarize the problems we

faced and what options we had to solve them regarding to Validation. In the last sections, we present

our results and conclude our findings.

Replication of models helps us get a deeper understanding of the relevant properties, identify the

key issues of the analysed phenomena, and to deploy simulation as a research tool successfully.

Overview of the Original Model The original model considers three different actors, Households, Firms and Banks: Households supply

labour, buy consumption goods and hold deposits, Firms demand labour, produce and sell

consumption goods, give shares to their owner, and Banks receive deposits from households and

extend loans to Firms.

There are three different markets in the model between these participants: the labour market (which

is homogeneous), the consumption market (which is also homogeneous, and contains non-durable

goods) and the credit market. An overview of the model can be seen on Figure 1.

Figure 1. Structure of the model [24]

In the model, the behaviour of the agents is not (necessarily) the outcome of an optimization

process: generally behaviour changes adaptively according to rules of thumb. The markets are fully

decentralized and characterized by a continuous search and matching process.

Households The jth worker (j = 1, 2, ..., J) supplies inelastically one unit of labour. Once employed, the worker

receives a wage for the entire duration of the labour contracts. The individuals receive wage and

dividends as owner of the firms. This income (wage and dividends) is spent in part on consumption

goods, and savings are deposited at the bank (even involuntary savings are added to voluntary ones).

The propensity to consume is a nonlinear decreasing function of accumulated savings (wealth).

The worker visits a finite number z of firms to ascertain the selling prices, and then it buys first from

the lowest price selling firm. If the most preferred firm is in short supply, the household can resort to

Page | 4

the remaining (z – 1) firms. If the jth worker does not succeed in satisfying its consumption plan, it

saves involuntarily.

Firms The ith firm (i=1, 2, ..., I) produces a consumption good using only labour at the tth (t=1, 2, …, T) time

step. Technology is strictly linear (Yit = αitLit, where Yit is the level of production, αit is the labour

productivity, Lit is the labour). The ith firm (i=1, 2..., I) knows the status quo in the tth time step, Ait =

(Pit, Yit). At the price Pit, given the average price Pt, demand Dit for the products of the firm can be

different from production Yit. Hence the output is the inventories: Sit = Yit - Dit.

Based on the status quo, the firm decides to change the price, and decides to update the expectation

of future demand and therefore the desired scale of activity Yit+1d. The decision on the "desired" scale

of production Yitd, determines the demand for labour Lit

d = Yitd / αit

d.

Figure 2. Price and quantity adjustments for the firms [5]

Firms also post labour vacancies if the demand for labour to reach the desired scale of production is

greater than the operating workforce. Moreover, if internal financial resources (net worth: Ait ) are in

short supply with respect to the wage bill witLit - i.e. if there is a financing gap - the firm asks for a

bank loan Bit.

If funds Bits + Ait are still not sufficient to pay for the wage bill (credit rationing), the firm will fire

redundant workers. If the firm ends up with excess supply, it gets rid of the unsold (non-durable)

goods at zero costs.

Banks The bank will extend a total amount of credit to firms equal to a multiple of its equity base: Ct = Et / v,

where 0 < v < 1 can be interpreted as a capital requirement coefficient. Hence 1/v is the maximum

allowable leverage for the bank. Ct represents the amount of “credit vacancies” posted by the bank

on the credit market.

Page | 5

We assume that bank k offers to the ith firm a standard debt contract, which consists of an interest

rate rit and the corresponding repayment schedule – rit is a markup (over the policy rate r), which

increases with the borrower’s leverage lit = Bit/Ait (external finance premium).

Market Processes In each simulation step, the following events take place in order:

1. At time t, each firm computes net worth Ait. If Ait < 0 the firm goes bankrupt and exits. If Ait >

0 the firm survives and proceeds to the next time step. If all the firms goes bankrupt, the

simulation ends.

2. Starting from the status quo, the firm decides (adaptively) the quantity to be produced

(hence the demand for labour) and the price to be charged.

3. Firms post labour demand approximated by excess.

4. The credit market opens:

a. The bank decides credit conditions, and posts credit vacancies (supply of loans) at a

certain offer interest rate.

b. Each firm contact the banks and decide to whether take a loan or not.

5. Firms redefine labour demand, and determine the effective number of vacancies.

6. The job market opens:

a. Firms publish their vacancies:

i. Firms with excess workforce fire random workers at a certain offer wage.

ii. Firms with insufficient workforce hire and try to fill the empty workspaces

with unemployed workers.

In both cases firms and workers randomized so that no one is privileged.

b. Once the matching has been completed, the firm computes the wage bill and the

financing gap, which has to be filled by means of loans.

7. Market for the consumption goods opens:

a. Workers receive their wages for the current period.

b. Consumers compute their consumption budget.

c. Firms post their offer price.

d. Each consumer contacts a given number of randomly chosen firms to buy

consumption goods, starting from the one which offers the lowest price.

e. Once the matching has been completed, the firm assess whether there is sufficient

demand to sell its goods. If not - i.e. in case of excess supply (inventories) - it gets rid

of the unsold goods (at zero costs) because they are non-durable.

f. Unspent money is added to involuntary savings.

g. Production takes the whole period tth time period.

8. Accounting and surviving:

a. At the end of the period, firms collect revenues and calculate profits.

b. If gross profits are big enough, the firm validates financial commitments.

c. Earnings after interest payments and dividends are retained profits, which are used

to update net worth.

9. The cycle starts again with the next simulation step (t+1).

Page | 6

Bankrupcy

If the ith firm goes bankrupt there is a negative demand spillover due to the loss of employment.

Other firms will experience a reduction of demand. Moreover, the bank will record a non performing

loan, which will affects the bank’s net worth and therefore, through capital regulation, also the total

amount of credit extended: Ct = Et / v. Credit extended to other firms will be curtailed.

Reasons for Replication In the introduction of this paper we described that model replication might help us validate the

scientific results of an agent-based model. However, there were some additional issues we

considered.

Efficiency In our work, we tried to focus our efforts not just on creating a validated replication for the Mark 1

model, but tried to make it as efficient as it is possible on the Java platform. The reason why we

chose Java as the basis of our work is its ease of use and support of various simulation tools. That is

also a reason why we chose Mason, “a fast discrete-event multiagent simulation library core in Java,

designed to be the foundation for large custom-purpose Java simulations […]” [17] as our simulation

platform, whose main design goal and philosophy focuses on supporting the creation of efficient

computer simulations.

Tool Support: Model Exploration Module The Model Exploration Module (MEME) is an automated, generic tool (supporting various simulation

platforms like NetLogo, Repast, Fables and Mason) designed to help modellers dealing with

enormous parameter spaces in a reasonable time [13].

One of the approaches to handle such amount of required simulation runs is to exploit a private grid

or distributed cloud computing resources in order to execute large-scale parameter space

explorations or sensitivity analyses [12].

MEME also offers statistical tools to reduce the number of simulation runs needed for collecting

enough information (i.e., samples) from a given parameter space region to judge the behaviour of

the simulation. These statistical methods are based on the Design of Experiments [14] literature that

was developed for the experimental sciences with the goal to reduce the number of costly

experiments without jeopardizing the amount of knowledge that can be collected from the

experiments.

MEME has a growing list of classic designs of experiments, including (fractional) factorial design(s),

the Box-Behnken design, the central composite design and Latin hypercube designs, as well as it

offers a number of more advanced, dynamic exploration strategies, based on heuristic optimization

methods. These latter include Genetic Algorithms and Iterative Interpolation. A good example how

these techniques can be exploited in computer simulations is [11].

Page | 7

Our Chosen Approach During our work, we heavily used the advantages of different software testing tools. These

frameworks are commonly referred when model replication and validation is an issue to solve [2],

and there are even tools specifically designed to help modellers to write test their simulations [23].

First, we identified different parts of the original model which can be isolated and verified separately

(e.g., initialization of the model and the agents, different utility functions, the various markets, etc.).

Then, we carefully examined the parameters and state variables having influence on the behaviour of

the subsystem (input), the state variables that were changed during the activation (output), and

prepared several black-box unit tests for each component: it the same input is given for the

subsystem, when executing it, then the result must contains the exactly same output.

Ensuring Comparable Results We had a few issues to solve, and one of them was the different handling of pseudo-random

generators in Java and Matlab. To be able to generate comparable results, we had to synchronize

them.

We exploited the Matlab-Java interoperability here: with some minor restrictions, Matlab is able to

use any Java component. It was a simple solution to use the random generator in the Java core

libraries from Matlab: this way we were even able to determine the exact number of random

function calls at any point in the original model.

JAVA_COMPATIBLE_RANDOM_GENERATOR = 1;

randCallCtr = 0;

javaRandom = java.util.Random(0);

function randVal = jrand()

randCallCtr = randCallCtr + 1;

if JAVA_COMPATIBLE_RANDOM_GENERATOR

randVal = javaRandom.nextDouble;

else

randVal = rand;

end

end

Code Listing 1. Changing the default random generator in Matlab

There was another issue with the randperm(n) function which was heavily used in the model. The

source code of the function is accessible from the Matlab distribution, so it was easy to change the

new random generator:

function randPerm = jrandperm(n) if JAVA_COMPATIBLE_RANDOM_GENERATOR dummy = zeros(1,n); for act = 1:n dummy(act) = jrand(); end [junk,randPerm] = sort(dummy); else randPerm = randperm(n); end end Code Listing 2. Generating random sequences

Page | 8

Results – Comparison When we finished the replication model, preliminary results showed that it replicates well the

behaviour of the original model. We demonstrate the output of a typical simulation run in Appendix

1: Sample Simulation Run (the parameter vector used for this sample run is shown in Table 1. in the

column entitled Sample). However, a rigorous evaluation is appropriate.

Authors of the original model used two approaches to explore the parameter space because of the

limited computational resources. With the tool support available for the replicated model these

constraints are not entirely removed, but extended considerably.

The model verification procedure consisted of two separate steps, namely internal validity and

sensitivity analysis.

First, a set of runs were executed with exactly the same parameter combination, but with different

seed values. The random seed employed by the pseudo-random generator did not cause large

variability in the aggregated outcome - it was proven that the model is sufficiently accurate.

As a second step, the authors performed a univariate sensitivity analysis. During these experiments,

the model was analysed by changing the value of a specific parameter, leaving the other input values

untouched. For each of these parameters, different scenarios were evaluated.

Authors claim that the parameters having the most significant effect on the simulation results are the

ones related to the duration of labour contracts, to the number of opportunities to locally explore

searches for market transactions, and the total size of the economy.

In our current work, we also concentrate on parameters that prove to be crucial considering the

results. These are the ones influencing the number of trials in the goods market (z), and the number

of households/capitalists in the simulation (I and J). Note that the number of banks did not have any

major effect on the results, so we fixed it on a default value.

Name Parameter Value (Sample)

Value (Validation)

T Number of time periods 100 1000

I Number of workers 650 650/1300/3250/6500

J Number of firms/capitalists 100 100/200/500/1000

Bk Number of banks 1* 1*

div Share of dividends 0.2 0.2

refi General refinancing rate 0.02 0.02

α Labour productivity 0.1 0.1

φ Credit demand contraction 0.8 0.8

τ Rate of debt reimbursement 0.05 0.05

wb Wage rate 1 1

z Number of trials in the goods market 2 2/3/4/5/6

c Propensity to consume 0.8 0.8 Table 1. Parameter values used in simulations

During the simulations, we changed the value of parameter z incrementally on the interval [2,6], and

used different agent numbers by multiplying the total number of agents by 1, 2, 3, 5 and 10 (resulting

Page | 9

in a total of 7500 agents in the latter case). Note that the overall ratio of the agents remained at a

constant level. As a total, we executed 40 different simulation runs for the models.

Benchmarking During the implementation, we paid special attention to efficiency. We tried to use one of the most

efficient modelling libraries available for Java, Mason, and we tried to use the most efficient language

constructs (until it was against readability). Note that this was not a design concept with the original

Matlab model. The execution time of 40 runs is averaged on Figure 3 where each measurement point

shows the average of 5 different runs.

Figure 3. Averaged execution time comparison for both the original and replicated model

As a result we got that the Java model is faster by several magnitudes (usually between a factor of 6

to 10) compared to the original model.

Verification There are several hallmarks of the original model, a number of emergent properties that can be

identified (stylized facts).

First, the economy shows a tendency towards self-organization, i.e. a spontaneous order

("equilibrium") emerges. This order is generally characterized by a persistent involuntary

unemployment and unsold production and/or credit rationing. This order occasionally displays

remarkable and irregular upswings and downswings ("coordination failures") so that a great

depression can occur because of the transmission of an idiosyncratic shock, i.e. in the absence of a

major negative aggregate shock.

In the original model, the authors used 1.000 simulation time steps, and (in order to get rid of

transients), only the last 500 periods have been considered. In our work, we included the initial

phase as well because of two reasons: i) there are heavy oscillations in several statistics during that

period, and ii) allows us evaluate the model on a wider interval. The most relevant results are

summarized on Table 2, and they verify the equivalence of the original and the replicated model.

750 agents 1500 agents 3750 agents 7500 agents

Matlab 39,8406 94,2619 370,6098 1164,0148

Mason 2,4436 8,8554 52,8372 206,8896

0

200

400

600

800

1000

1200

1400

exe

cuti

on

tim

e (

in s

eco

nd

s)

Performance Comparison

Page | 10

Table 2. Summarizing the average results of 40 different simulation runs

Page | 11

Conclusion In our current work, we were able to successfully replicate the results of an agent-based economic

model. We also performed a basic exploration of the model, and got results that are comparable to

the original results.

The original model is roughly about 350 lines of code. On the other hand, our replicated model is

about 2500 lines of code, from which 1500 line is the core model, and an additional 1000 line is

required to use a generic modelling library and a graphical interface. Another interesting fact is that

we created about 6700 lines of test code that ensures completely identical simulation results, which

is more than 4 times longer than the core replicated model.

We also described our simple approach how an existing model might be translated into another

language, and (in the context of Matlab and Java) we documented some issues model replicators

might face.

The benefits we gained by an alternative implementation is at one hand the speedup (the

performance gain is over 5 magnitude), and the window of widely used simulation tools is opened,

allowing us to perform more extensive parameter space explorations by exploiting cloud systems or

by using sophisticated Design of Experiments techniques.

Acknowledgements This work was partially supported by the Hungarian Government (KMOP-1.1.2-08/1-2008-0002), the

European Union Seventh Framework Programme FP7/2007-2013 under grant agreement CRISIS-ICT-

2011-288501 (CRISIS – Complexity Research Initiative for Systemic InstabilitieS) and mOSAIC 2011-

256910 (Open-Source API and Platform for Multiple Clouds). These supports are gratefully

acknowledged.

Rererences [1] Farmer, J. Doyne, and Duncan Foley. “The Economy Needs Agent-based Modelling.” Nature 460,

no. 7256 (August 5, 2009): 685–686.

[2] Gilbert, G N, and K G Troitzsch. Simulation for the Social Scientist. Open University Press, 2005.

[3] Tesfatsion, Leigh. “Agent-Based Computational Economics: Growing Economies From the Bottom

Up.” Artificial Life 8, no. 1 (2002): 55–82.

[4] Tesfatsion, Leigh “Agent-Based Computational Economics: Modeling Economies as Complex

Adaptive Systems.” Information Sciences 149 (2003): 263–269.

[5] Gatti, Domenico Delli, Saul Desiderio, Edoardo Gaffeo, Pasquale Cirillo, and Mauro Gallegati.

Macroeconomics from the Bottom-up. 1st ed. Springer, 2011.

[6] Epstein, Joshua M., Robert Axtell, and 2050 Project. Growing Artificial Societies: Social Science

from the Bottom Up. Brookings Institution Press, 1996.

[7] Moss, Scott, Bruce Edmonds and Centre For Policy Modelling. Sociology and Simulation:

Statistical and Qualitative Cross-Validation. American Journal of Sociology 110 (2005):1095–1131.

[8] Troitzsch, Klaus G. “Validating Simulation Models.” Proceedings of 18th European Simulation

Multiconference on Networked Simulation and Simulation Networks, SCS Publishing House

(2004): 265–270.

Page | 12

[9] Grimm, V. et al. (2006). A standard protocol for describing individual-based and agent-based

models. Ecological Modelling 198(1-2):115-126

[10] Bernhard Bauer, Jörg P. Müller, James Odell. “Agent UML: A Formalism for Specifying Multiagent

Interaction.” Agent-Oriented Software Engineering, Paolo Ciancarini and Michael Wooldridge

eds., Springer-Verlag, Berlin, pp. 91-103, 2001.

[11] Szabó, A., L. Gulyás, I. J. Tóth (2009) "Sensitivity Analysis of a Tax Evasion Model Applying

Automated Design of Experiments" in Seabra Lopes, L.; Lau, N.; Mariano, P.; Rocha, L.M. (Eds.)

Progress in Artificial Intelligence, pp. 572-583., LNAI 5816., ISBN 978-3-642-04685-8.

[12] Máhr, Tamás, Bocsi, Rajmund, Gulyás, László: „Simulation as a Service: The Model Exploration

Service”, 3rd World Congress on Social Simulation, Kassel, Germany, 2010.

[13] Márton Iványi, László Gulyás, Rajmund Bocsi, Gábor Szemes, and Róbert Mészáros: „Model

exporation module”, In Agent 2007: Complex Interaction and Social Emergence Conference,

Evanston, IL, USA, November 2007.

[14] George E. P. Box, J. Stuart Hunter, and William G. Hunter: „Statistics for Experimenters: Design,

Innovation, and Discovery”, 2nd Edition. Wiley-Interscience, July 2005.

[15] Uri Wilensky, and William Rand. “Making Models Match: Replicating an Agent-Based Model.”

Journal of Artificial Societies and Social Simulation vol. 10, no. no. 4 2 (2007).

http://jasss.soc.surrey.ac.uk/10/4/2.html.

[16] Bigbee, Anthony, Claudio Cioffi-Revilla, and Sean Luke. “Replication of Sugarscape Using

MASON.” In Agent-Based Approaches in Economic and Social Complex Systems IV, edited by

Takao Terano et al., 3:183–190. Agent-Based Social Systems. Springer Japan, 2007.

[17] Luke, Sean, Claudio Cioffi-Revilla, Liviu Panait, Keith Sullivan, and Gabriel Balan. “MASON: A

Multiagent Simulation Environment.” Simulation 81, no. 7 (July 1, 2005): 517–527.

[18] Grimm, Volker, Eloy Revilla, Uta Berger, Florian Jeltsch, Wolf M. Mooij, Steven F. Railsback, Hans-

Hermann Thulke, Jacob Weiner, Thorsten Wiegand, and Donald L. DeAngelis. “Pattern-Oriented

Modeling of Agent-Based Complex Systems: Lessons from Ecology.” Science 310, no. 5750

(November 11, 2005): 987–991.

[19] Miodownik, Dan. “Cultural Differences and Economic Incentives: An Agent-Based Study of Their

Impact on the Emergence of Regional Autonomy Movements.”. October 31, 2006.

http://jasss.soc.surrey.ac.uk/9/4/2.html.

[20] Grimm, Volker, Uta Berger, Finn Bastiansen, Sigrunn Eliassen, Vincent Ginot, Jarl Giske, John

Goss-Custard, Tamara Grand, Simone Heinz, and Geir Huse. “A Standard Protocol for Describing

Individual-based and Agent-based Models.” Ecological Modelling 198, no. 1–2 (2006): 115–126.

[21] Bauer, Bernhard, Jörg P. Müller, and James Odell. “Agent UML: A Formalism for Specifying

Multiagent Interaction.” In In: Ciancarini, P.; Wooldridge, M. [eds.], Agent-oriented Software

Engineering, 91–103. Springer, 2001.

[22] Miodownik, Dan, Britt Cartrite, and Ravi Bhavnani. “Between Replication and Docking: Adaptive

Agents, Political Institutions, and Civic Traditions; Revisited.” Journal of Artificial Societies and

Social Simulation 13, no. 3 (2008): 1.

[23] Gurcan, O., O. Dikenelli, and C. Bernon. “Towards a Generic Testing Framework for Agent-based

Simulation Models.” In Computer Science and Information Systems (FedCSIS), 2011 Federated

Conference On, 635 –642, 2011.

[24] Domenico Delli Gatti, personal communications

Page | 13

Appendix

Sample Simulation Run

Page | 14

Page | 15