GETTING COMFORTABLE WITH YOUR DATA II

One way to turn your data into knowledge,

and another way that’s probably better

Winter Storm 2010Stats workshop

Dave Kleinschmidt

ANOVAWhat is it, anyway?

WHAT YOU WANT

You’ve designed + run your experiment

It sorts observations into groups

Is there any difference between groups?

YOUR DATA IS NOISY

This could be a big problem for you

What if the noise is too big,

and drowns out the effect of your groups?

More importantly, how can you tell?

STATISTICS TO THE RESCUE

Statistical models quantify noise

ANOVA is one kind of model

Mixed-effects models (MEMs) are another

ANOVA

ANalysis Of VAriance

Tells whether group means are identical

(tests a null hypothesis)

Compare variance between groups (good)

with variance within groups (bad—noise)

ANOVA

Figure from PDQ Statistics, Norman and Streiner

ANOVA

If differences between groups outweigh noise within groups, then you can safely reject the null hypothesis

(which is that your experiment did nothing)

ANOVA—ONE LAST NOTEANOVAs come in different flavors:

• One-way ANOVA tests one grouping

• Factorial ANOVA tests multiple crossed groupings

• Repeated-measures ANOVA tests a design where each subject is exposed to each condition (a within-subjects design)

SO WHAT’S THE PROBLEM?ANOVA’s considered the gold-standard

Especially for factorial designs

However, ANOVA makes assumptions:

• Data is perfectly balanced

• Each group has identical variance

• No systematic variability between subjects or items

MIXED-EFFECTS MODELS TO THE RESCUE!

MEMs can represent nearly any sort of variability between subjects/items.

Balance these differences with the need to draw general conclusions about the average character of the whole population

MIXED-EFFECTS MODELS TO THE RESCUE!

Do other nice things, too

• Far more robust to missing data

• Can model nearly any data distribution (not just normal, like ANOVA)

WHAT IS A MEM?

Combines fixed and random effects:

• Fixed effects are deterministic and common to all subjects/itmes

• Random effects vary from subject-to-subject/item-to-item

`

WHAT IS A MEM?

Fixed effects describe how the experimental manipulations affect the observations

Think of it as the slope of a line:

dataij = fixed * xij

(xij is the condition that dataij comes from)

`

WHAT IS A MEM?

Of course, we have to add noise.

If the noise of each subject/item combination is independent, than we just get

dataij = fixed * xij + noiseij

Where all of the noiseijs are independent and normally distributed (with mean zero)

(this is the essence of an ANOVA)

`

WHAT IS A MEM?

What if some subjects are just faster/better than others?

Then we just add another noise term by subjects:

yij = fixed * xij + noise0j + noiseij

Note that this changes the intercept for the line for each subject, but leaves the slope the same for each

`

WHAT IS A MEM?

In the same way, we can let the slope of the line vary a little by subject, too.

This is equivalent to saying that we believe the experimental manipulation affects some subjects more than others.

`

SO WHY DOESN’T EVERYONE USE MEMs?Soon, everyone will (probably).

No pencil-and-paper solution, unlike ANOVA

(but software is widely available now)

ANOVA is the established standard

(but more and more are using MEMs)

LET’S TRY SOME