Science & Statistics in PsychologyLecture 3

A thousand zeros is still nothingA thousand zeros is still nothing

Dr Caleb Owens

Lecture Plan

1. Theory & Evidence : Science and pseudoscience; the importance of a rationale

2. The power of a name : Measurement and constructs3. A thousand zeros : Types of research design; internal

and external validity4. Predictions : Hypotheses in science; the null hypothesis;

the importance of a disprovable hypothesis5. The coin toss : Understanding variability in sampling

and measurement; probability and the appeal to ignorance

6. Too much of a good thing : Statistical power; Practical significance

Lecture Outline• Research designs

– Anecdotes and case studies– Correlational studies– Quasi-experiments– True experiments

• External validity– Random sampling

• Internal validity– Random allocation

• Blinding

Have you heard ?

• The plant Echinacea was marketed throughout the 1990’s as an ancient herbal remedy which could cure the common cold (not unlike Vitamin C in the 1960’s)

– But in 2003 when the evidence from a large double blind study was released, the alternative-remedy lobby group backed down, saying that it was unwilling to promote questionable cures which may cause serious side-effects

• Or did it ?

What actually happened

• Some alternative health lobby groups strongly disputed that the tests were valid

– “Children have different immune systems!”

Excuses

• The answer is coming

• The tests were flawed

• “This topic is beyond the scope of western science”

• We are convinced by overwhelming anecdotal evidence

Anecdotal evidence

• “The plural of anecdote is not evidence”• Anecdotes are usually worse than useless

– Because they generally support what we believe– The anecdote contains the (proposed) explanation

which cannot be untangled once it is passed on– It cannot be replicated or documented by non-

prejudiced observers– It changes with every retelling and depends on fallible

memory• At most anecdotes provide clues and leads

Case studies

• Significantly more systematic than anecdotes– All details taken and recorded in a scientific manner

• May simply consist of the combination of previously collected information

• Can consist of the systematic collection of longitudinal data– E.g. The famous television series: “7-up”, “49-up”

• Provide important details which can lead to later systematic research and breakthroughs

• Should not be over-interpreted since the sample size is often small and rarely randomly selected– Ideal for investigating rare or highly specific topics

• E.g. prosopagnosia

• Anecdote

• My Aunty Liz got Cholera from the terrible smells and gases coming from the river near her house. She moved house. Whenever she smells something bad from now on she stays away and hasn’t got Cholera since!

• Case Study

• Elizabeth Cromwell contracted Cholera on 12th July 1723 during a particularly hot summer.

• When winds were blowing from the East, her house became filled with noxious gases causing a terrible smell.

• She was an otherwise healthy women whose only other affliction was mild Dysentery some years earlier.

• Her house was located next to the river Thames which at the time contained a large amount of raw sewage.

• She drew her drinking water from a well which was exposed to the river.

Correlational studies• Measure two naturally occurring variables

• Find a relationship between them

• Only a very weak causal inference could be drawn (a suggestion, a lead for more research)

• No independent variables are controlled

e.g. Simply look through lots of data and find relationships

Correlation Coefficients

XY = – 0.7

The sign (+ or – ) indicates the direction of the

relationship (+ : as values on one variable get bigger,

values on the other get bigger; - : as values on one variable get bigger, values on the

other get smaller)

The number indicates the strength of the

relationship(0.1, 0.2, 0.3 = weak ; 0.7, 0.8, 0.9 = strong)

Scatterplots

X

100806040200

Y

600

400

200

0

-200

-400

-600

X

100806040200

Y

120

100

80

60

40

20

0

-20

XY = 0.25

XY = 0.88Positive Correlations

The score on two IQ tests taken on the same day

The relationship between High School grades and University grades

Each person has two scores and these are plotted on separate axes

Scatterplots

X

100806040200

Y

20

0

-20

-40

-60

-80

-100

XY = -0.93

X

100806040200

Y

400

300

200

100

0

-100

-200

-300

-400

XY = -0.16

Negative Correlations

The number of hours of exercise people do and their excess fat in a pinch test

The amount of time people spend on the internet and their university grades

Correlation and Causation

• Just because two variables are correlated with each other does not mean that one causes the other.

• Example: The number of ice-creams consumed on a certain day correlates positively with deaths by drowning (0.8)

Correlation and Causation

• It could be that:– Eating ice-cream causes drowning– Drowning causes ice-cream to be eaten– Amount of ice-cream eaten and number of

drownings are both caused by the temperature on a particular day

• Does the cold of winter cause colds and flu ?

Some plausible correlational findings

• Living close to the beach is correlated with being rich– Causation in the other direction ?

• Religiousness is correlated with being on death row – Causation in the other direction ? Third variable ?

• Being aboriginal is correlated with being in prison– Third variable ?

• Being at University is correlated with being homosexual– Measurement bias ?

• Height is correlated with IQ– Third variable ?

Correlation and Causation

• In general, if A & B are correlated it is possible that:– A causes B– B causes A– Some other variable C causes both A & B

• Correlation Causation!

Independent & Dependent Variables

• Presumed Cause• Antecedent Cause• Stimulus Variable

• Behaviour of Interest• Response Variable

IndependentVariable

DependentVariable

Manipulated byexperimenter

Measured byexperimenter

Correlational studies• Measure two naturally occurring variables

• Find a relationship between them

• Only a very weak causal inference could be drawn (a suggestion, a lead for more research)

• No independent variables are controlled

e.g. Simply look through lots of data and find relationships

True Experiments• Randomly allocate subjects to different levels of the

independent variable• So all independent variables of interest are ‘controlled’• By not letting subjects choose, most confounds

cancel each other out because you have a random amount of each in each group

– E.g. When subjects turn up you toss a coin to see which condition they go into. If the experiment is based on speed, then with enough subjects, naturally ‘fast’ and ‘slow’ people in each condition will cancel each other out

Quasi-experiments• Some attempt at controlling variables

– Matching– Manipulating some IV’s but not others

• Always a study of ‘pre-existing groups’ in some sense• Whenever you have pre-existing groups you have

obvious confounds• Causal inference is weak• Example

– When studying the psychological impact of domestic violence, you find a ‘control’ group which has not experienced violence (and you want to ensure the groups are as similar as possible), and you may give both groups different kinds of therapy

Random allocation of independent variables

• Correlational study : NONE

• Quasi – experiment : At least one (but not others); or none, but some attempt to control one (e.g. matching)

• True Experiment : ALL

Research Designs: Drug use example

• Correlational: simply record the amount of alcohol consumed and University Grades– Weak causal inference– Numerous alternative explanations

• Quasi-experimental: Select a group of people who drink regularly and a group who do not, match them on diet, exercise, age and lifestyle and compare their grades– Eliminates some confounds– Does not control for all variables (e.g. attitude, personality)

• True experiment: Randomly allocate 250 people to drink heavily and 250 to drink a placebo control– Controls for most extraneous variables– Puts 50% at risk !

Research Designs: Self-Esteem Example

• Correlational: Record self-esteem and University Grades and look for a relationship– Weak causal inference– Numerous alternative explanations

• Quasi-experiment: take people who have high self-esteem and low self-esteem; compare their grades and try to improve these with extra classes or peer support – Eliminates some alternative explanations– Self-esteem is still pre-existing and may have been caused

by a third variable• True experiment: Randomly allocate 250 people to

high self-esteem and 250 people to low self-esteem and measure the impact on Uni grades– Controls for most extraneous variables– Impractical, unethical

Controlling variables

• Hard to randomly allocate:– Treatments– Lifestyles – Education– Experience– Diet

• Impossible to randomly allocate– Sex – Brain damage (in humans)*– Psychological disorders– Extreme stress / trauma

/anxiety– Sexuality– Family– Location– Religion– Race– Income

*TMS, transcranial magnetic stimulation, now makes this possible

Sex differences• Males are better than females at spatial tasks

(e.g. mental rotation)• Hypothesize: Maybe this is caused by males

having more experience with ‘spatial’ toys• Match males & females on “Lego experience”

– This is called ‘controlling’ the variable (holding the variable constant: it’s not the same as the control you have in a true experiment)

– ‘Males are better than females at spatial tasks but when we controlled for spatial task experience the difference disappeared’

External validity

• Based on sample size, and the representativeness of the sample and where and how the experiment is conducted

• Depends on random selection of subjects and how closely the setting matches the real world

• How applicable is the data to the real world?– How applicable to the real world are laboratory based

findings?

• How well can you generalize from your results to the real world?– NB: This depends heavily on the topic

Internal validity

• The ability to draw a causal inference

• The extent to which changes in the DV are likely to be due to changes in the IV

• Depends on random allocation of subjects to different levels of the IV and the elimination of alternative explanations and confounding variables

Type of study Internal validity* External validity**

Case studies None Extremely high ?

Survey / Questonnaire

Very poor Good

Quasi-experiment Poor-good Good

True (field) experiment

Good Okay

True (lab) experiment

Excellent Poor

*Decreasing internal validity certainly limits the causal inference you can draw but does not make the study ‘bad’ in any overall sense- there are plenty of well designed and very useful survey studies, and some awful, badly designed studies using true experiments

**External validity depends on many factors unrelated to design (especially sampling), but in general tends to decrease as the experiment becomes more controlled

Blindness• Being ‘blind’ means you do not know what

treatment/condition is being administered

• If the researcher is not blind– May not randomly allocate correctly– May unconsciously change data– May unconsciously affect outcome

• If the subject is not blind– The effect arises because they expect it to (placebo effect)

• If both subject and experimenter don’t know what condition is being administered it is called ‘double-blind’

• Could anecdotal evidence ever be of a randomized double blind nature?

• Unblinded– Participant takes a pill knowing that it is intended to

relieve pain

• Single-blind– Participant is given a pill by a researcher (who knows

what it does) but is not told whether or not it will relieve pain

• Double-blind– Researcher 1 makes the pills Yellow and Blue. She

gives them to Researcher 2 who does not know which one has an active ingredient. Researcher 2 gives the participant one of the pills and records how effective it is. Researcher 1 takes the data which records how effective the Yellow and Blue pills were and makes a conclusion (only they knew that the Blue pill contained an active ingredient)

Clever Hans

-see http://en.wikipedia.org/wiki/Clever_hans

Dowsing: the “ideomotor effect”

Important points • You can have a ‘good’ correlational study and a

‘bad’ true experiment – based on how well they are designed

• Beyond the design, replication is everything in science– Replication means the same findings can be found

again by repeating the study– Beware of people and websites who pick out just a

handful of (often older) articles to support their view, and ignore the fact such things were never found again, or are found very rarely (indicating a result due to chance or bias)

– Beware of groups who insist their results can only be replicated by people who are ‘trained’ in doing what they do

Summary

• Be able to distinguish between and define these research designs:– Case-study– Correlational study– Quasi Experiment– True Experiment

• Understand this important terminology– External Validity– Random Selection– Internal Validity– Random Allocation– Replication