Statistics II: Measurement & Data Analysis

Brian J. Piper

Goals

• Operationalization• Psychometrics (Reliability & Validity)• Scales of Measurement• Hypothesis Testing• Advanced Topics– Power– Effect Size

Operationalization

• What areas of the brain are important for music appreciation?

• Important = Active– EEG (electrical activity)– PET (sugar use)– fMRI (oxygen use)

Operationalization 1: Spatial Function

• Do men and women differ in their spatial abilities?

• Spatial Function = mental rotation

Operationalization 2: Motor Function

• Do men and women differ in their fine-motor abilities?

• Motor Function = rotary pursuit

Reliability

• Consistency of measurement• Types– Test-re-test reliability

Time 1

Time 2

Split-Half Reliability

• Consistency of measurement on two-halves of test

• Foundations for repeated measurements

Even

Odd

Extension: Short-Form of Wisconsin (Berg) Card Sorting Test (BCST)

Extension: Short-Form of Wisconsin (Berg) Card Sorting Test (BCST)

r(205) = +0.77

Fox et al. (in review). J Biol Biomed Reports.

Validity

• Does a test measure what it claims to?• face “faith” validity: does it seem valid based on

intuition (non-numerical)

Criterion Validity

• Does performance on new measure match with older “gold standard” measure?

• Continuous Performance Tests Example

Reaction Time (Conner’s)

Reac

tion

Tim

e (P

EBL)

Construct Validity

• Does a test measure the construct it claims to?

• Convergent Validity: Does test A correlate (converge) with test B?

• Discriminant Validity: Does test A measure something different (discriminate) than test C?

Measurement Scales (Self-Test)Level Definition

Nominal

Ordinal

Interval

Ratio

Measurement Scales (Self-Test)Level Definition

Nominal categorical, e.g. sex

Ordinal ranking, e.g. Olympic medal

Interval equal spacing, e.g. IQ, ACT, SAT

Ratio true zero, e.g. Reaction Time

Hypothesis Testing

• Null hypothesis (H0): A = B

• Alternative hypothesis (HA): A ≠ B

Alpha

• The cut-off used to decide between H0 and HA

• Probability that finding is not due to chance (p value)

• .05: conventional• .10: liberal (some medical environments)• .01: conservative, large N

Alpha

P value obtained

Decision

.50 H0

.11 H0

.06 H0

.0500000001 H0

.0499999999 HA

Decision Making

HO is True HO is False

Fail to reject H0 Correct decision

Reject H0 Correct decision

Reality

Decision

Decision Making

HO is True HO is False

Fail to reject H0 Correct decision

Reject H0 Type I error Correct decision

Reality

Decision

Type I Error: rejecting H0 when it is true

Decision Making

HO is True HO is False

Fail to reject H0 Correct decision Type II error

Reject H0 Type I error Correct decision

Reality

Decision

Type I Error: rejecting H0 when it is trueType II Error: fail to reject H0 incorrectly

Publication Bias

• H0 results often don’t get shared• Reasons: – Journal prestige– Research ego– Higher standard

• Solution: registry?– Replication?

Solution 1: Effect Size Distribution

• A quantitative index of the magnitude of group difference’s

• Calculated as (Mean1 – Mean2)/SD

# St

udie

s

Solution 1: Effect Size Distribution

• A quantitative index of the magnitude of group difference’s

• Calculated as (Mean1 – Mean2)/SD

# St

udie

s

# St

udie

s

Solution 2: Power Analysis

• Power: the probability that a real effect will be detected

• Probability of Type II error: Beta• Power = 1 - Beta

N Power

50 0.40

100 0.70

500 0.80

1000 0.85

Other Terminology

• Population: all members of identifiable group• Sample: a subset of the population• Confidence Interval: inferential statistic,

contains range of where population mean sits

Margin of Error

• Is accurate if sample is representative of population.

Summary• Operationalization• Reliability & Validity (face, criterion, construct)• Scales of Measurement (nominal, ordinal,

interval, ratio)• Hypothesis Testing: Type I versus Type II error• Advanced Topics– Power– Effect Size