Artificial Science - a simulation test-bed for studying the social processes of science, Bruce...

Artificial Science - a simulation test-bed for studying the social processes of science, Bruce Edmonds, ESSA 2004, http://cfpm.org/~bruce slide-1

Artificial Science– a simulation test-bed for studying the social

processes of science

Bruce EdmondsCentre for Policy Modelling

Manchester Metropolitan University


Part 1: Introduction

Science as an object of study


Why study science as a process?

• Its important impacts on society• Improve the “management” of science• Understand its strengths and weaknesses• It is interesting• Science seems to achieve things that have been

found difficult in DAI/MAS, e.g.:– building complex value-added chains– spontaneous autopoesis and self-organisation of fields

and sub-fields– spontaneous specialisation and distribution of skills

across the problem space– reliability of established results in comparison to the

uncertain reliability of individuals' work


Philosophy/Sociology of Science

• Normative philosophy of science (basically until 1960s and Kuhn)

• Then a move towards a more descriptive approach

• Tends to focus on what an individual scientists do• Social processes tend to be introduced either as:

– Part of a critique of the received picture etc.– To support special status of science (e.g. its norms)

• Now some more study/discussion of observed social processes in science (Giere, Longino, Knorr-Cetina, etc.)


Computational models of science

• Aspects of scientific thinking:– Problem-solving by sub-casing (Newell&Simon)– Induction (Holland &al)– Inference of causality (Pearl)

• More integrated models:– BACON (Langley &al)– Meta-Dendral (Buchan&Mitchel)– PI (Thagard)

• Social aspects of science:– Gilbert (1997) A simulation of the structure of academic

science


Part 2: A Test-Bed

How can we simulate the innovative discovery of the

unknown?


Aims of test-bed

• To explore the relationship between the (social) behaviour of individuals and the knowledge that is collectively produced

• In particular:– What behaviours seem to reproduce observed

patterns of discovered knowledge?– What are the effects of certain behaviours on

the structure of discovered knowledge?– What behaviours might be better than others,

wrt target properties of discovered knowledge?


Keys design issues

The test-bed must…

• be a sufficiently challenging problem

• requiring much independent (i.e. parallel) work by individual agents

• but also requiring the combination of knowledge produced by others in chains

• involving the discovery of the unknown

• where some knowledge may be used to transform other knowledge.


Structure of the test-bed

• A formal (i.e. logical or mathematical) system stands in for what is true

• Existing knowledge is represented by the theorems that are in the agents’ individual memories or as published in journals etc.

• Unknown to the agents certain theories are designated as intrinsically important corresponding to important discoveries

• It is very difficult for agents (and probably the simulator) to find out what is true (or how to get there) without painstakingly “working forward”

• Theories may be used as knowledge or as techniques for producing new knowledge


Part 3: An Example Simulation

Exploring the aspect of distributed inference and coordination via

journal publication


Aims of this Simulation

• To illustrate the workings and challenge of this test-bed

• To show how a simulation might be done within it• To explore a particular subset of behaviours,

particularly w.r.t. inferring new knowledge (as in maths) and how this is disseminated via a journal

• Also (not discussed here) to explore the different kinds of behaviour the agents might use and the effect on the effectiveness of collective knowledge development


The test-bed problem

• The theories are those of classical propositional logic with connectives: , , , , , T, F

• formulated as a “Hilbert System” with:– 14 axioms– 1 rule, Modus Ponens (MP) (explained shortly)

• 110 designated “target theories” taken from textbooks

• New theories developed by taking applying MP to to existing theories

• Makes for a fairly tough problem - space of theorems is more than exponential in size


Agent-1

Agent-2

Structure of the Simulation

The Journal

The Axioms

MP

MP


Action of the MP inference rule

yyxx ))((

)()(( aaaa

BA (Major Premise)

A (Minor Premise)

)( aa

B (Inference)


Agents

• Agents have two (limited) stores, for knowledge (minor) and techniques (major)

• Each iteration each agent:1. Decides what new items of knowledge to add to its

private stores from the published set, also which to drop (both major and minor).

2. Decides which major premise and what set of minor premises it will try with the MP rule and add any results to its (minor) store.

3. Decides which of its private knowledge (that is not already public) it will submit to the journal

• Agents may “panic” if they have not discovered anything within a certain number of iterations and replace their knowledge (minor or major)


The Journal (the Journal of Artificial Sentences and Successful Syllogisms)

• The journal is the public repository of knowledge (accessible to all)

• Each iteration the journal:1. Makes a short-list of submissions that meet

basic criteria (e.g. novelty, number of vars.)

2. Ranks the short-list using a weighted score (in this case, shortening, shortness, past success of submitter, number of variables)

3. Chooses from the ranked short-list (e.g. top N, randomly, probabilistically etc.)


Part 4: Some results from the example simulation

Exploring the effects of differing rates of publishing by a journal


The Experiment

• 20 agents over 50 iterations• Each agent stores 4 major and 27 minor

premises as its current knowledge and submit all unpublished formulas they find

• 1 journal, selecting for (in descending order) shortening; shortness; prestige; num vars.

• Vary the number of formula the journal publishes each iteration from 1…10

• Results are averages over 25 independent runs for each setting


Other settingsaddRateMajor = 1 dropRateMajor = 1addRateMinor = 2 dropRateMinor = 2interalScoreDecay = 0.9fertilityDecay = 0.9numToTry = 9panicSoonTime = 4 panicLaterTime = 12

minNumberOfVariablesForPublication = 1shorteningWeight = 0.1 shortnessWeight = 0.01sourceWeight = 0.001 numVariablesWeight = 0.0001

General behaviour: fertilityIsInverseLength matchTargetGenerically publishBest submitAll

Major Behaviour: addFertileProb dropWorst feedbackOnUsedFailure panicLater replaceBestIfBad tryBest

Minor Behaviour: addRandom dropWorstProb feedbackOnFertility panicSoon replaceProbFertileIfBad tryUntried


Example output from a run

…Iteration 3agent 3 found '->' ('->' A ('->' B C)) ('->' B ('->' A C))agent 3 found '->' ('->' A ('->' ('->' B C) D)) ('->' ('->' B C) ('->' A D))agent 6 found '->' ('->' A ('->' B B)) ('->' A ('->' A ('->' B B)))agent 6 found '->' ('->' A ('->' A B)) ('->' A B)agent 6 found '->' ('->' A B) ('->' ('->' B A) ('->' A B))agent 17 found '->' ('->' A ('->' A B)) ('->' A B)agent 19 found '->' ('->' A B) ('->' ('->' C A) ('->' C B))agent 19 found '->' ('->' A B) ('->' ('->' C ('->' ('->' A B) D)) ('->' C D))Iteration 4agent 7 found '->' ('->' A B) ('->' ('->' ('->' A B) C) C)agent 7 found '->' A ('->' ('->' A B) B)agent 13 found '->' ('->' A ('¬' A)) ('¬' A)agent 15 found '->' ('->' A ('¬' A)) ('¬' A)Iteration 5Iteration 6…


Number of formulas in public domain

0

100

200

300

400

500

600

0 10 20 30 40 50iteration

tota

l nu

mb

er

form

ula

fo

un

d

njp=1

njp=2

njp=3

njp=4

njp=5

njp=6

njp=7

njp=8

njp=9

njp=10


Number of targets found

10

10.5

11

11.5

12

0 10 20 30 40 50iteration

tota

l nu

mb

er

targ

ets

fo

un

d

njp=10

njp=9

njp=8

njp=7

njp=6

njp=5

njp=4

njp=3

njp=2

njp=1


Total number found by agents (also num. submitted for publication)

0

20

40

60

80

100

120

140

160

180

0 10 20 30 40 50iteration

tota

l nu

mb

er

form

ula

su

bm

itte

d

njp=1

njp=2

njp=3

njp=4

njp=5

njp=6

njp=7

njp=8

njp=9

njp=10


(Average) spread (the SD) of numbers of formulas found by agents

0

1

2

3

4

0 10 20 30 40 50iteration

sd

of

nu

mb

ers

age

nts

fo

un

d

njp=1

njp=2

njp=3

njp=4

njp=5

njp=6

njp=7

njp=8

njp=9

njp=10


Number found by agents (a single run, njp=2)

0

20

40

60

80

100

120

140

160

0 10 20 30 40 50Iteration

Nu

mb

er

fou

nd

by

ag

en

ts


Number found by agents (a single run, njp=10)

0

20

40

60

80

100

120

140

160

0 10 20 30 40 50Iteration

Nu

mb

er

fou

nd

by

ag

en

ts


Part 5: Concluding Discussion


Limitations of current simulation

• It is not validated, though it could be via citation structure etc. (at least partially)

• Includes many ‘kludges, (e.g. journal’s selection mechanism by weighted rank)

• Focuses on certain inference (more like maths than experimental science)

• Does not manage to sustain the discovery of important theories – this is a challenge!

• Does not scale well (with a single journal)


What has been achieved?

• I’ve had a lot of fun• It shows that (at least some aspects) of the social

processes in science can be simulated• It shows that there are some interesting

unanswered questions to be addressed• It suggests that increasing the publication rate

does not necessarily result in more meaningful knowledge being produced

• The dynamics in even such simplified simulations of science can be quite complex


Extensions (a few of the many possible…)

• Many journals co-evolving in terms of scope, publication and selection policies

• Instead of inference, agents could hypothesise theories and test them experimentally (a line of the truth table)

• Journals could use a peer-review system with agents rating shortlisted submissions

• Informal social networks etc. can be another channel for knowledge

• New student agents inherit knowledge from older agents and sometimes they retire


The connection with distributed theorem-proving

• Single agents in this simulation operate in a similar way to some established forward-chaining theorem provers (e.g. OTTER)

• Not designed to be an efficient (“tuned”) theorem proved

• But lessons learned might be applicable to the design of distributed theorem provers (exploiting parallelism)

• Collections of agents may be able to be “trained” on a logic and then deployed


Some issues that could be investigated in this test-bed include:

• Is any norm on methods for discovering new knowledge is counterproductive? (Feyerabend)

• What is the effect of the framework (within which knowledge is expressed) on the structure of new knowledge? (Kuhn)

• When and how do social processes act to increase the reliability of knowledge collectively produced (or otherwise)? (Merton, Popper)

• Is it helpful to have an inviolate core of knowledge/techniques that is not open to revision? (Lakatos)


The End

Bruce Edmonds

bruce.edmonds.name

Centre for Policy Modelling

cfpm.org

Ad: Socially Inspired Computing

A 2-day workshop 12th&13th April 2005

Part of AISB 2005 convention: Social Intelligence and

Interaction in Animals, Robots and Agents

At Univ. of Hertfordshire, UK (just north of London)

Workshop website: http://cfpm.org/sic

Co-located with 15 other symposia, including:

Emerging Artificial Societies (14th&15th April)

Artificial Science - a simulation test-bed for studying the social processes of science, Bruce...

Documents

Transcript of Artificial Science - a simulation test-bed for studying the social processes of science, Bruce...