PSBE2-08

Research Methods

Week 2

1Tassos Sarampalis

Null Hypothesis Significance

Testingand

Power

2

Null Hypothesis Significance

Testing

• Goal– determine whether mean differences among groups

in an experiment are greater than differences expected simply because of chance (error variation)

• First step– assume that the groups do not differ (H0)

• = null hypothesis

• assume the independent variable did not have an effect

3

Null Hypothesis Significance

Testing

• Next steps

– Probability theory: estimate likelihood of observed outcome, while assuming null hypothesis is true.

– “statistically significant”

• outcome has small likelihood of occurring under H0

• reject H0

• conclude IV had an effect on DV– difference between means is larger than what would be expected

if error variation alone caused the outcome

4

5

6

pro

ba

bili

ty

0 2 4 6 8 10 12 14 16 18

“heads” count

7

pro

ba

bili

ty

0 2 4 6 8 10 12 14 16 18

“heads” count

Null Hypothesis Significance

Testing

• How small does the likelihood have to be to decide outcome isn’t due to chance?

• scientific consensus: p < .05

• = alpha (α) or level of significance

• What does a statistically significant outcome tell us?– outcome at p ≈ .05 has about a 50/50 chance of being repeated

(at p < .05) in an exact replication

– as probability of outcome decreases (e.g., p = .025, p = .01), likelihood of observing a statistically significant outcome (p < .05) in an exact replication increases

– APA recommends reporting exact probability of outcome

8

9

Null Hypothesis Significance

Testing

• What do we conclude when a finding is notstatistically significant?

– do not reject the null hypothesis of no difference

– don’t accept the null hypothesis

• don’t conclude that the IV didn’t produce an effect

– cannot make a conclusion about the effect of an IV

• some factor in experiment may have prevented us from observing an effect of the IV

• most common factor: too few participants

10

NHST Criticisms

• A difference between populations can almost always be found, given a large enough sample

• A statistically significant finding may not be relevant in practice, whilst a true effect of practical significance may not appear statistically significant if the test lacks the power

• Fairness of exclusion

• Publication bias and the file-drawer problem

11

Experimental Sensitivity and

Power

• Sensitivity

– likelihood an experiment will detect the effect of an IV when in fact, the IV has an effect

• affected by experiment methods and procedures

• sensitivity increases with good research design and methods

– high degree of experimental control

– little opportunity for biases

12

Experimental Sensitivity and

Power

• Power

– likelihood that a statistical test will allow researchers to reject correctly H0

• low statistical power increases Type II errors

• Power = 1 - β

• three factors affect power of statistical tests– level of significance (alpha)

– size of the effect of the IV

– sample size (N)

13

Experimental Sensitivity and

Power

• Prospective Power Analysis• step 1: estimate effect size of IV

– examine previous research involving the IV

• step 2: refer to “Power Tables”– identify sample size needed to observe effect of IV

• step 3: use adequate sample size– most studies in psychology are “underpowered” because of

low sample size

• Retrospective Power Analysis• Determine the power of a study based on the effect

size, sample size, and significance level

14