Major Points

• Formal Tests of Mean Differences• Review of Concepts: Means,

Standard Deviations, Standard Errors, Type I errors

• New Concepts: One and Two Tailed Tests

• Significance of Differences

Important Concepts

• Concepts critical to hypothesis testing– Decision– Type I error– Type II error– Critical values– One- and two-tailed tests

Decisions• When we test a hypothesis we

draw a conclusion; either correct or incorrect.– Type I error

• Reject the null hypothesis when it is actually correct.

– Type II error• Retain the null hypothesis when it is

actually false.

Possible Scenarios

Null Hypothesis is actuallyResults show TRUE FALSE

TRUE Correct Acceptance Type IIFALSE TypeI Correct Rejection

Type I Errors

• Assume there are no differences (null hypothesis is true)

• Assume our results show that they are not same (we reject null hypothesis)

• This is a Type I error– Probability set at alpha ()

usually at .05

– Therefore, probability of Type I error = .05

Type II Errors

• Assume there are differences (alternative hypothesis is true)

• Assume that we conclude they are the same (we accept null hypothesis)

• This is also an error– Probability denoted beta ()

• We can’t set beta easily.• We’ll talk about this issue later.

• Power = (1 - ) = probability of correctly rejecting false null hypothesis.

Critical Values

• These represent the point at which we decide to reject null hypothesis.

• e.g. We might decide to reject null when (p|null) < .05.– Our test statistic has some value with p

= .05– We reject when we exceed that value.– That value is the critical value.

One- and Two-Tailed Tests

• Two-tailed test rejects null when obtained value too extreme in either direction– Decide on this before collecting data.

• One-tailed test rejects null if obtained value is too low (or too high)– We only set aside one direction for

rejection.

One- & Two-Tailed Example

• One-tailed test– Reject null if number of red in Halloween

candies is higher

• Two-tailed test– Reject null if number of red in Halloween

candies is different (whether higher or lower)

Within subjects t tests

• Related samples

• Difference scores

• t tests on difference scores

• Advantages and disadvantages

Related Samples• The same participant / thing give us data on two

measures– e. g. Before and After treatment

– Usability problems before training on PP and after training

– Darts and Pros during same time period

• With related samples, someone high on one measure probably high on other(individual variability).

Cont.

Related Samples--cont.

• Correlation between before and after scores– Causes a change in the statistic we can use

• Sometimes called matched samples or repeated measures

Difference Scores

• Calculate difference between first and second score– e. g. Difference = Before - After

• Base subsequent analysis on difference scores– Ignoring Before and After data

Difference between Darts and Pros TIME TIMENO PROS DARTS1January-June1990 1.00 12.70 .002February-July1990 2.00 26.40 1.803March-August1990 3.00 2.50 -14.304April-September1990 4.00 -20.00 -7.205May-October1990 5.00 -37.80 -16.306June-November1990 6.00 -33.30 -27.407July-December1990 7.00 -10.20 -22.508August1990-January1991 8.00 -20.30 -37.309September1990-February1991 9.00 38.90 -2.5010October1990-March1991 10.00 20.20 11.2011November1990-April1991 11.00 50.60 72.9012December1990-May1991 12.00 66.90 16.6013January-June1991 13.00 7.50 28.7014February-July1991 14.00 17.50 44.8015March-August1991 15.00 39.60 71.30

Results• Pros got more gains than darts

• Was this enough of a change to be significant?

• If no difference, mean of computed differences should be zero– So, test the obtained mean of difference scores

against = 0.– Use same test as in one sample test

t test

85.62.1

22.8

9

6.322.8

n

sD

tD

D and sD = mean and standard deviation of differences.

df = 100 - 1 = 9 - 1 = 8

Cont.

t test--cont.

• With 99 df, t.01 = +2.62 (Table E.6)

• We calculated t = 2.64

• Since 6.64 > 2.62, reject H0

• Conclude that the Pros did get significantly more than Darts

Advantages of Related Samples

• Eliminate subject-to-subject variability

• Control for extraneous variables

• Need fewer subjects

Disadvantages of Related Samples

• Order effects

• Carry-over effects

• Subjects no longer naïve

• Change may just be a function of time

• Sometimes not logically possible

Between subjects t test

• Distribution of differences between means

• Heterogeneity of Variance

• Nonnormality

Pros during ups and downs in DOW

• Effect of fluctuations in DOW: did it effect Pros– Different question than previously

• Now we have two independent groups of data– Pros during positive DOW, and Pros during

negative DOW– We want to compare means of two groups

Effect of changes in DOW TIME TIMENO PROS DARTS DJIA DowTrend1January-June1990 1.00 12.70 .00 2.50 12February-July1990 2.00 26.40 1.80 11.50 13March-August1990 3.00 2.50 -14.30 -2.30 04April-September1990 4.00 -20.00 -7.20 -9.20 05May-October1990 5.00 -37.80 -16.30 -8.50 06June-November1990 6.00 -33.30 -27.40 -12.80 07July-December1990 7.00 -10.20 -22.50 -9.30 08August1990-January1991 8.00 -20.30 -37.30 -.80 09September1990-February1991 9.00 38.90 -2.50 11.00 110October1990-March1991 10.00 20.20 11.20 15.80 111November1990-April1991 11.00 50.60 72.90 16.20 112December1990-May1991 12.00 66.90 16.60 17.30 113January-June1991 13.00 7.50 28.70 17.70 114February-July1991 14.00 17.50 44.80 7.60 115March-August1991 15.00 39.60 71.30 4.40 116April-September1991 16.00 15.60 2.80 3.40 1

Differences from within subjects test

Cannot compute pairwise differences, since we cannot compare two random data points

We want to test differences between the two sample means (not between a sample and population)

Analysis

• How are sample means distributed if H0 is true?

• Need sampling distribution of differences between means– Same idea as before, except statistic is

(X1 - X2) (mean 1 – mean2)

Sampling Distribution of Mean Differences

• Mean of sampling distribution = 1 - 2

• Standard deviation of sampling distribution (standard error of mean differences) =

2

2

2

1

2

121 n

sns

sXX

Cont.

Sampling Distribution--cont.

• Distribution approaches normal as n increases.

• Later we will modify this to “pool” variances.

Analysis--cont.

• Same basic formula as before, but with accommodation to 2 groups.

• Note parallels with earlier t

2

2

2

1

2

1

2121

21

ns

ns

XXs

XXt

XX

Degrees of Freedom

• Each group has 5 data points.

• Each group has n - 1 = 50 - 1 = 8 df

• Total df = n1 - 1 + n2 - 1 = n1 + n2 - 2

50 + 50 - 2 = 98 df

• t.01(98) = +2.62 (approx.)

Assumptions

• Two major assumptions– Both groups are sampled from populations with

the same variance• “homogeneity of variance”

– Both groups are sampled from normal populations

• Assumption of normality– Frequently violated with little harm.

Heterogeneous Variances

• Refers to case of unequal population variances.

• We don’t pool the sample variances.

• We adjust df and look t up in tables for adjusted df.

• Minimum df = smaller n - 1.– Most software calculates optimal df.