1
Models, Simulations and Scientific Method
Caroline LyonSchool of Computer ScienceUniversity of Hertfordshire
http://homepages.feis.herts.ac.uk/~comrcml
29.11.2006
2
Overview
• What is “scientific method” ? - traditional approach - problem fitting models and simulations into the framework - necessary conditions for scientific simulations
Examples to illustrate criteria for evaluation of models and simulations• London underground map• Nowak’s model of the evolution of words in human language• Elman’s simulation of the development of grammar
• Conclusions: The difference between necessary and sufficient conditions for scientific method.
3
Scientific method – some historical landmarks
300BC Aristotle and methods of categorising knowledge
c. 1000AD Islamic mathematics and scientific experiments
1600 Bacon – controlled experiments, empirical scientific investigations
1637 Descartes - Discourse on Method, rationalism
1680 Newton – hypotheses that enabled predictions
1920 Fisher – statistical analysis
1934 Popper – falsifiability as a criterion
1962 Kuhn –The Structure of Scientific Revolutions
4
What is a model or simulation?
• Models and simulations contrast with real objects or events• Word model used in differing ways.
One end of a spectrum: an artifact which aims to reproduce the function of a real biological system as closely as possible. E.g. cochlear implant
A contrasting usage: language model, a term which denotes a collection of statistics on word frequencies
• Simulation typically means a model with dynamic elementsIEEE has 2000+ Computer Simulation Standards for scientific processesMDA – Model Driven Architecture
5
Characteristics of a scientific approach
• Objective versus subjective
• Both deductive ( as in mathematics) and empirical knowledge
• Supporting evidence, repeatable experiments– But limitations of an inductive approach e.g. all swans are white
• Production of hypotheses that can be falsified (cf Karl Popper)– Falsification rather than verification– Example of Newtonian mechanics– Contrast with other forms of belief, e.g.
“St Paul’s cathedral is a beautiful building”“It is wrong to eat people”
6
Problems fitting models into this framework
• Since models and simulations abstract certain features and ignore others they can be falsified by examining other features that were not abstracted.
• Example: the map of the London underground Purpose: to show passengers which route to take Functional efficiency: excellent But, not to scale. Relative distances between stations do not match actual
distances.
• Models and simulations cannot be assessed using a naïve falsification test
• Note that underground map is not arbitrary: corresponds topologically to the real system.
7
Map of London Underground
8
Feature abstraction (1) Picasso – portrait of a young girl
9
Feature abstraction (2) Picasso – portrait of a young girl
10
Criteria for evaluating simulations and models
• Consistency between empirical evidence and declared aims– fitness for purpose– external validity: a reality check
• Consistency within the model or simulation
– internal cohesion
• Well founded choice of parameters– arbitrary choices explicit– (caution in use of metaphorical language)
11
Use of simulations in language acquisition and evolution
Research into language acquisition - in evolutionary time- in historical time- in the lifetime of an individual
Simulations needed because• There are few ways in which we can find out about events millions of
years back. We create a virtual laboratory for experiments
• They avoid unethical investigations into the functioning of the brain
12
Example 1“Computational and evolutionary aspects of language” M. A. Nowak et
al. (Nature, Vol 417, 6.6.2002)
Aim: to produce a model, “a theoretical framework explaining how darwinian dynamics lead to fundamental properties of human language”
Process includes assumptions:“that a language can be seen as an infinite binary matrix linking phonetic forms to semantic forms”“ambiguity …. is the loss of communicative capacity that arises if individual sounds are linked to more than one meaning”
Applying a reality check: this is inconsistent with human language. English and other languages have many ambiguous sounds there / their here / hear one / won two / to / too etc. etc.
Disambiguation through context.
13
Example 1 continued
This model also shows how a limited number of phonemes can be combined to produce an indefinite number of words
“the maximum fitness of a language increases exponentially with word length”
Plotkin and Nowak, J. of Theoretical Biology, 2000, vol 205, p158
Lacks external validity
• Not a model for the evolution of human language
• Could be a model for communication between synthetic agents
14
Example 2 “Distributed Representations, Simple Recurrent Networks, and
Grammatical Structure” Elman, Machine Learning, 1991
Purpose of investigation:“How viable are connectionist models for understanding cognition?” (p.220)
“The connectionist model can be seen as a mechanism for gaining new theoretical insight” (p. 197)
Elman’s model claims to represent long distance dependencies, critical in speech and language.
(e.g. pre-planned co-articulation: lip position for “tea” and “two”)
15
Example 2 (cont)
• Recurrent neural net combines input at time t with previously processed input from time t-n
• Supervised training, using back prop
• Prediction task: What word will come next? Is it grammatical? • Lexicon of 23 words
John feeds dogs.*Boys sees JohnBoys who see John feed dogs.
16
Example 2 (cont)Learning Long Term Dependencies with Gradient Descent is Difficult,
Bengio et al.,IEEE Trans. On Neural Networks, 1994
• It is possible to train a recurrent NN on a particular task
• Models with short dependencies are trainable
• Trade off between efficient learning and latching information for longer periods.
• “gradient descent becomes increasingly inefficient when the temporal span of the dependencies increases” (p. 164)
17
Necessary conditions for scientific simulations
• External validity
• Internal cohesion
• Well founded abstractions
• Fitness for purpose – but what is the purpose?
A simulation might meet these conditions but not be scientific. These conditions are not sufficient.
Consider a simulation of bullying among school boys
18
Bullying scenario (1)
Players adopt the persona of Jimmy Hopkins, a 15-year-old thug who has been incarcerated in a boys’ boarding school.
Points can be scored by terrorising other pupils with a range of physical and psychological abuse. Players use their on-screen persona to kick and punch other pupils and even to spit in their food.
They can use weapons such as baseball bats and catapults. …… The game has angered children’s campaigners.
Sunday Times on the game Bully 14.8.2005
19
Bullying scenario (2)Simulation tool to help victims address bullying problems
Synthetic agents representing school children populate a virtual playground.They develop relationships as they interact - talking, playing and sometimesbullying.
The user identifies with one of the bullied agents and can explore this virtual world, seeing how different reactions might reduce or increase bullying.
Scenario 1 is a game.Scenario 2 may be a scientific search for knowledge about ways of teaching
20
Role of unexpected events in scientific method
Discovery by chance – e.g Galvani and the frogs’ legs, Fleming and penicillin
Recent example:
Behaviour of agents or robots that was unexpected (but can be explained post hoc)
E.g. “Exploiting Physical Constraints: Heap Formation through Behavioural Error in a Group of Robots” Maris and te Boekhorst, IEEE Proc IROS 1996
Experiments with robots on object avoidance failed, but instead delivered results oncreation of clusters. Placement of sensors on robots led to blind spots so that objects were sometimes pushed instead of avoided.
21
ConclusionsScientific method has a bundle of characteristics.
They must include:• A search for knowledge • Objective, repeatable experiments• Logical deductions
They may include • Use of models and simulations that meet necessary conditions of external
validity, internal cohesion, well founded abstractions• Production of falsifiable hypotheses
and should include:• An open mind, ready to expect the unexpected“Chance favours the prepared mind” Louis Pasteur
Top Related