Analysis of Complexity in Cognitive Tasks Graeme S. Halford School of Psychology University of...

Analysis of Complexity in Cognitive Tasks Graeme S. HalfordSchool of PsychologyUniversity of [email protected]

Acknowledgments

This work has been developed in collaboration with Bill Wilson, Steve Phillips, Glenda Andrews, John Bain, Andrew Neal, Murray Maybery, Keith Holyoak, Thomas Suddendorf, Paula Irving, Christine Boag, Julie McCredden, Damian Birney, Geoffrey Goodwin, and many other people have contributed in some way. Their contributions are gratefully acknowledged. We also acknowledge the contribution of Air Services Australia to our analyses of air traffic control situations.

The problem of what makes tasks complex occurs in many areas of cognitive psychology:– cognitive development

– mathematics and science education

– animal cognition

– industrial situations (e.g. air traffic control) Comparison of task difficulties depends on

solving this problem

A transitive inference problem such as the following might seem rather simple: – Tom is happier than Mark

– Peter is happier than Tom

– Who is happiest? To answer this we might order the elements:

Tom

Mark

Peter

Tom

Peter

Tom

Mark

Integration of premises produces a measurable processing load effect.

What is the source of this? Answer: We have to consider both premises to

assign an element to a slot.

Tom

Mark

Peter

Tom

Peter

Tom

Mark

How do we quantify this load? Premise integration entails assigning entities to

three slots. – It relates three variables

– The task entails processing a ternary relation. We can quantify complexity of tasks in terms of

the relations they entail. – More complex tasks entail processing more complex

relations. There are three requirements to establish its

validity:

Requirements of relational complexity theory:

1 . Tasks with the same complexity should be similarly difficult.

2 . Tasks with lower levels of relational complexity should be simpler, while tasks with higher levels of relational complexity should be harder, given that materials, procedure and knowledge are constant over levels.

3 . Performance should be predicted by tasks of the same level of relational complexity from different domains.

Task difficulty can vary for many reasons: – declarative and procedural knowledge

– method of presentation, and strategy adopted.

Transitivity is one of a set of tasks that are difficult for young children.

The causes of this have been controversial, but it is fair to say that, when allowance has been made for the many hypotheses in the literature, there is a component of the difficulty that requires to be explained.

Our research suggests that the complexity that is inherent in the task is one source of this difficulty.

Another task that has been similarly difficult is class inclusion.

The classical question is to ask children: are there more blue things or more circles?

Children under about 5 years typically say there are more circles. There are a number of known sources of false negatives, but when these are corrected (e.g. using the procedures of Hodkin, 1987 or Halford & Leitch, 1989) the task is still difficult for young children.

What is the cause of this difficulty?

Bluethings

Circles Triangles

IncludedIn

IncludedIn

ComplementOf

Superordinateclass

Subordinateclasses

•There is a superordinate class (blue), a subordinate class (circles) and a complementary subordinate class (triangles).

•Class inclusion, like transitivity, is a ternary relation.

• The three classes must be assigned to the appropriate slots in the hierarchy.

•Blue is assigned to the superordinate slot because it includes triangles and circles.

•This imposes a processing load, that has been verified by Halford & Leitch (1989).

Transitivity and class inclusion are superficially different, yet both entail ternary relations:

Complement-of

Included-inIncluded-in

B

>

Transitivity

Fruit

CA

> >

ClassInclusion

Non-ApplesApples

• Transitivity and class inclusion were both originally Piagetian tasks, but

• Other paradigms that entail ternary relations and are similarly difficult for young children: E.g. concept of mind.

WhiteBird

BlueBird

Blue Filter

ObjectAttribute“White”

Variables

ViewingConditionBlue Filter/ NoFilter

PerceptSeeBlue/ white

• The the relation between a property of an object, and the person’s percept, is modulated by a third variable, the viewing condition.

The concept of mind task entails relating three variables, and is ternary relational. Consequently a structural complexity effect was predicted ahead of evidence (Halford, Children’s Understanding: The Development of Mental Models, 1993) and has since been confirmed.

Concerning the second requirement for relational complexity theory, if we can reduce the relational complexity of a task, holding materials and procedure constant, then difficulty should be correspondingly reduced.

We have done this successfully in several concept domains.

The procedure for transitive inference is shown in the following Figure:

Transitive Inference

yellowblue red green

purple blue green red

green

red

blue

green

blue

red

Premises

Binary Ternary

The ternary relational task is to first place green and blue. It has to be inferred that green is above red and red is above blue, therefore the order is green, red, blue. This is a ternary relation. Once green and blue are in place, the child is asked to place red. This is very easy if the correct order has already been mentally constructed, and serves as a check on guessing or other lower level strategies.



green

red

blue

green

blue

red

Premises

Binary Ternary

The binary relational task is to first place green and red. This can be inferred from the single premise green above red, and entails processing only a binary relation. Then the child is asked to place blue. This can be done by processing the binary relation, red above blue.



green

red

blue

green

blue

red

Premises

Binary Ternary

Notice that binary and ternary tasks entail placing the same three blocks, using the same set of premises. The percentages of children at each age succeeding at the .05 level are shown in the following figure:



green

red

blue

green

blue

red

Premises

Binary Ternary

Children’s performance: transitive inference

83.393.3 95.8 100 96.7

6.4

46.7

66.7 71.4

86.7

0

20

40

60

80

100

120

4 5 6 7 8

Age (years)

Percent children

succeeding

BinaryTernary

The comparatively small change in procedure produced a dramatic difference in performance, especially for the younger children (Andrews, 1997). Relational complexity produces powerful effects on performance.

Relational complexity can also be manipulated with classification tasks.

The category induction task (Gelman & Markman, 1987) is binary relational, because it entails a category and its complement. If we tell a child that (say) cats climb trees, and ask them whether the attribute tree-climbing applies to instances of a complementary category. The structures of category induction and class inclusion are in this figure:

Structures of category induction and class inclusion

BasicComplementary

Basic

SubordinateComplementary

Subordinate

A

Cold water fish: have bones

Sharks

Salmon: Swimupstream

Trout: Have dark flesh

B

Assessing Category Induction and Class Inclusion To make them comparable, the tasks have to be

assessed by the same method. Both can be assessed by property inference (Johnson, Scott & Mervis, 1997).

We used familiar categories, but unfamiliar attributes (Halford, Andrews & Jensen, 1998).

Within-level questions



Sharks


1. Basic to complementary-basic:

Do all sharks have bones?

2. Basic to basic:

Would this cold-water fish have bones?

4. Subordinate to subordinate:

Would this salmon swim upstream?

3. Subordinate to complementary-subordinate:

Do all trout swim upstream?

Predictions: Relational Complexity Theory

Relational complexity theory predicts that category induction should be easier than class inclusion. To test this hypothesis, the tasks must be assessed by procedures that are strictly comparable. This can be done by property inference:



Sharks


Subordinate to subordinate:

Would this salmon swim upstream?

Subordinate to complementary-subordinate:

Do all trout swim upstream?


than of non-salmon (trout).

Property Inference 1: An attribute should generalise more to instances of the same category than to instances of a complementary category (within level inference): e.g. swimming upstream is more likely to be true of other salmon…….,

Getting both these questions right should indicate understanding the relation between a category and its complement - a binary relation.

Between - level questions


Sharks



1. Basic to subordinate:

Do all salmon have bones?2. Subordinate to basic: Do all coldwater fish swim upstream?

4. Complementary-subordinate to basic:

Do all coldwater fish have dark flesh?

3. Basic to complementary-subordinate:

Do all trout have bones?



Sharks



1. Basic to subordinate:

Do all salmon have bones?2. Subordinate to basic: Do all coldwater fish swim upstream?

Property Inference 2: An attribute should generalise more to instances of a lower level category than to instances of a higher level category: e.g. attributes of coldwater fish should generalise to salmon.

But not necessarily the reverse. Getting both these questions right should indicate understanding of

relations between hierarchical categories. This is a between-level inference, and corresponds to a ternary relation.

The task is designed so questions are matched within and between levels.

The hypothesis that ternary relational reasoning would be harder than binary relational was confirmed in two experiments.

The third requirement for relational complexity theory is that performance should be predicted by tasks of the same relational complexity in other domains.

We have shown this to be true in numerous experiments. In the category experiment just outlined, we found ternary relational performance was predicted by transitivity and class inclusion, both ternary relational tasks, the first from a different domain.

In another study we obtained similar results for concept of mind, which is also ternary relational, as noted above.

In other experiments we have found that relational complexity scores in 6 domains loaded on a single factor that accounted for 80 percent of the age related variance in fluid intelligence (Andrews & Halford, submitted).

Relational Complexity Metric The essence of our theory is that complexity of relations

processed in the same decision (in parallel) is a good way to define cognitive complexity.

Relational complexity refers to the number of entities related. It corresponds to number of slots or arity of relations.

A binary relation has two slots: e.g. Larger-than(_______, _______) has a slot for a larger entity

and one for a smaller entity. Each slot can be filled in a variety of ways:

Larger-than(elephant, mouse)

Larger-than(mountain, molehill)

Larger-than(ocean-liner, rowing-boat)

A slot corresponds to a variable or dimension. An N-ary relation is a set of points in N-dimensional space.

Complexity of relations can be defined by the number of slots:

Unary relations have one slot: e.g. class membership, as in dog(Fido).

Binary relations have two slots: e.g. larger(elephant, mouse).

Ternary relations have three slots: e.g. addition(2,3,5).

Quaternary relations: e.g. proportion (2/3 = 6/9).

The relational complexity metric is very general. It has been applied to

– child and adult cognition

– higher animals such as chimpanzees

– is currently being applied to dolphins.

– industrial contexts, including air traffic control. Relational complexity theory encompasses declarative and procedural

knowledge Processing capacity is an enabling factor, but concept acquisition is

knowledge acquisition, and cognitive development is experience driven.

There is a broad correspondence between levels of relational complexity and the phenomena that Piaget attributed to stages

However, there is no suggestion that concepts of a given level are all acquired synchronously. Concept acquisition is a function of experience, given that the relevant processing capacity is available.

Relational complexity applies to explicitly represented relations. It does not apply to tasks that can be processed by perceptual or associative processes, such as transitivity of choice (Chalmers & McGonigle, 1984; Wynne, 1995).

Capacity limits

A quaternary relation is the most complex that adults can process in parallel– (though a minority of people can probably process

quinary relations under optimal conditions) This is a soft limit

– Processing capacity is not all or none

– Increased complexity produces increased errors and decision times, rather than sudden catastrophic failure.

In order to handle more complex concepts, mechanisms for reducing processing loads are required. These are:

Conceptual chunking involves recoding concepts into less complex relations. However there is a temporary loss of access to chunked relations.– For example, speed = distance/time, is a ternary relation, but

speed can be recoded into a unary relation, speed(60kph) as when speed is indicated by the position of a pointer on a dial.

– However the chunked representation does not permit us to answer questions such as “How does speed change if we cover the same distance in half the time?”

Segmentation entails breaking tasks into less complex steps, which can be processed serially. Strategies and algorithms are common ways of doing this: – e.g. adding one column at a time in multidigit addition.

Chunking and segmentation skills are important components of expertise.

Complexity estimates are based on the mental models used to represent a concept, taking chunking and segmentation into account.

Variables cannot be chunked if relations between them must be considered.

Assumptions about chunking and segmentation are made explicit and applied consistently across domains.

A useful heuristic is that relational complexity cannot be reduced if the variables interact. This is analogous to analysis of variance: – Interacting variables must be interpreted jointly.

– A mathematical procedure for determining effective relational complexity is described by Halford et al. (Behavioral & Brain Sciences, 1998, 21(6), 803-864, Section 3.4.3). If a relation can be decomposed into simpler relations, then recomposed without loss of information, effective complexity is equivalent to the less complex relation.

Applications of relational complexity theory Now we consider some applications of the theory,

beginning with work on air traffic control in collaboration with Christine Boag and Andrew Neal.

Two air traffic control situations are shown. The first has fewer aircraft, but was reported as a heavy load by the controller. The second contains a lot of aircraft, but is a simple situation for a controller.

The first situation entails more complex relations between aircraft.

AERODROME B

AERODROME A

AERODROME D

AERODROME C

TO AERODROME E

TJS

Display A

RDS

TMK

LAO

MEL

KRD

AERODROME B

AERODROME A

AERODROME D

AERODROME C

TO AERODORME E

Display B

MTJ

EWF

SDK

RMS

TJT

TWE

TAKDJT

WEF KAL

LAT

SRD

The theory has also been applied to data from chimpanzees. The relational match-to-sample task requires participants to discriminate the relation same/different between a pair of objects.

Choice of YY is correct. Chimpanzees succeed, if they have been taught symbols for

the relations. – This indicates they can process binary relations.

– We have work in progress with dolphins on this task, in collaboration with Thomas Suddendorf and Paula Irving.

Sample XX

Choices YY WY

Cardinality

Understanding that counting yields the cardinal value of the set requires a mental model that incorporates the principles of counting.

Successor Model In the successor model, numerals are assigned to objects.

next next next next

items

next next next next

numbers one two three four five

(a) Successor Model

Sets Model In the sets model, numerals are assigned to sets,

but it does not distinguish items that have been counted from those that have not.

next next next next

larger

larger

larger

larger

sets


(b) Sets Model

Inclusion Model


next next next next

included in included in included in included in

sets

(c) Inclusion Model

In the inclusion model numbers are assigned to sets, which include (a) an item not previously counted, and (b) the set of all items previously counted (this set is empty on the first item).

The inclusion model enables recognition that the last number represents the cardinal value of the set - the cardinal word principle.

Also, because it distinguishes previously counted objects from the object currently counted, it makes the order in which objects are counted irrelevant.


next next next next

included in included in included in included in

sets

(c) Inclusion Model

Understanding cardinality was assessed using three question types:

How many: Children should say the last number they mentioned in counting S, and should not recount.

Show me X: Children should indicate all the objects in the set, not just the last object.

Reverse count: Asking children how many there would be if counted in the opposite direction.

Fractions: Quaternary Relation

Fractions and proportions cause difficulty for young children – (for reasons that never appear to have been completely

explained) Proportion is a quaternary relation.

Fractions: Quaternary relation

The examples alongside show that comparison of fractions entails working with four dimensions, because a change in either numerator or either denominator can affect the comparison.

Comparison of integers requires working with only two dimensions.

The powerful effect of structural complexity has been largely overlooked.

1 3=

2 6

1 4<

2 6

1 3<

3 6

5 5>

7 8

In order not to be misunderstood:– There is nothing to prevent younger children from

understanding fraction concepts that entail simpler relations. (e.g. half a pie entails only binary relations, and should be understandable by two year olds).

– Notice that the predicted age is actually earlier than appears to have been observed.

A good relational complexity analysis frequently uncovers unrecognised potential; – e.g. it predicts that even two year olds should be able to make

balance scale judgements based on either weight or distance, though not both.

– This is more optimistic than current norms, and has been confirmed empirically (Halford& Dalton, 1995)

Relational Complexity Metric

Piagetian Stage Cognitive Processes Typical Tasks PDP Implementation

Early Sensori-motor No Accessible representation(Elemental association)

Conditioning,linear transitivity of choice

Sensori-motor Computed,Non-structural (configuralassociation)

Conditional discrimination,prototype,circular transitivity

Preconceptual Unary Relations Match-to-sample,identity position integration,category label distinct from category

Intuitive Binary Relations,Univariate functions

Relational match to sample,A not-B,complementary categories

Concrete Operational Ternary Relations,Binary Operations,Bivariate Functions

Transitive inference,hierarchical categories,concept of mind

Formal Quaternary Relations,composition of binary operation

Proportion, Balance scale

Quinary Relations Doubly-centre embedded sentences,Tower of Hanoi

Input Output

Input Hidden Output

R (loves)

Arg1

(j oe)

Arg2

(ja ne)

R(d o g)

A rg ( ta il -w a g g in g)

Re lation

Ar g1

Ar g2

Arg n

The relational complexity metric forms a single scale from the most basic psychological process, elemental association, to the most complex level of which humans are capable.

Sample concepts at each level are shown, together with:– the corresponding Piagetian stage and:

– the neural net architecture that is characteristic of each level.

There is a unique empirical indicator for each level, and each level imposes requirements on neural net models,

over and above requirements of lower levels. This provides converging evidence that the levels are

genuinely distinct.

A complexity metric is essential for orderly interpretations of findings in Psychology. – Without it, questions such as whether infants or animals

have similar cognitive capabilities to adult humans are inherently unanswerable.

To investigate these questions systematically, we need ways of equating, or differentiating between, levels of cognitive functioning.

We suggest that the relational complexity theory makes a start in providing such a metric.

Abstract

Techniques are outlined for analysis of cognitive complexity in general cognition, cognitive development, mathematics education, reasoning tasks, psychometric test items, and industrial decision making, especially command and control. Complexity in cognitive tasks can be assessed by analysing the number of entities that have to be related in a single representation. This corresponds to the number of interacting variables that are processed in parallel. Adult humans are typically limited to processing quaternary relations (I.e. to relating four entities, or four variables) in parallel. More complex concepts are processed by segmentation (breaking a task into components small enough not to exceed capacity, and which are processed serially) or conceptual chunking (collapsing to a smaller number of variables, which reduces processing load, but makes some relations inaccessible). Processing load analyses are based on the principle that interacting variables cannot be processed serially. Neural net models of higher cognitive processes offer explanations for processing loads. Complexity analysis also leads to discovery of new capabilities. A single complexity metric is presented for all levels of cognition.

Analysis of Complexity in Cognitive Tasks Graeme S. Halford School of Psychology University of...

Documents

Transcript of Analysis of Complexity in Cognitive Tasks Graeme S. Halford School of Psychology University of...