Tests of Significance · Tests of Significance Data from a Hypothetical Cohort Study Dead Alive...

1

1

Tests of Significance

Analysis of

Two-by-Two Table Data:


UI-MEPI-J: Research Design and Methodology Workshop

2


ILL

VANILLA | + - | Total

-----------+-------------+------

+ | 43 11 | 54

- | 3 18 | 21

-----------+-------------+------

Total | 46 29 | 75

Single Table Analysis

Odds ratio 23.45

Cornfield 95% confidence limits for OR 5.07 < OR < 125.19*

RISK RATIO(RR)(Outcome:ILL=+; Exposure:VANILLA=+) 5.57

95% confidence limits for RR 1.94 < RR < 16.03

Ignore risk ratio if case control study

Chi-Squares P-values

----------- --------

Uncorrected: 27.22 0.00000018 <---

Mantel-Haenszel: 26.86 0.00000022 <---

Yates corrected: 24.54 0.00000073 <---

2

3


Objectives

Describe the reason for using statistical tests

Describe what a P-value is

Describe the two main influences on a P-value

for a two-by-two table

Properly interpret the results of chi-square

statistical tests

4


P < 0.05

P = NS

3

5


What is a Statistical Testing?

Also called “hypothesis testing”

… the process of inferring from your data

whether an observed difference is likely to

represent chance variation or a real difference

(Does NOT address bias, confounding, or

investigator error!)

For two-by-two table data, influenced by:

Number of subjects or observations in study

Size of difference in results between groups

6


Data from a Hypothetical Cohort Study

Dead Alive Total % Dead

Diabetic 2 2 4 50.0%

Nondiabetic 1 3 4 25.0%

Total

3 5 8 37.5%

4

7




Diabetic 10 10 20 50.0%


Total

15 25 40 37.5%

8




Diabetic 20 20 40

50.0%


Total

30 50 80 37.5%

5

9




Diabetic 200 200 400 50.0%

Nondiabetic 100 300 400 25.0%

Total

300 500 800 37.5%

10


Steps in Statistical Testing

1. State the null and alternative hypotheses

2. Choose a statistical test for testing the null hypothesis

3. Specify a significance level

4. Perform the statistical test, i.e., calculate probability of obtaining data you got if null hypothesis were true

5. Make a decision about the hypotheses

6

11


Mindset for Statistical Testing

2 groups – diabetics vs. non-diabetics, cases

vs. controls, etc.

Each is a sample from some larger population

Are they likely to be samples from the same

population, or different populations?

12


RR = 1

7

13


0

5

10

15

20

25

30

35

40

45

Relative Risk

RR = 1

14


1. State the Null and Alternative Hypotheses

Null hypothesis

H0: The observed difference is not real, i.e., the

observed difference is the result of chance

Alternative hypothesis

HA: H0 is not true, i.e., the observed difference is

not due to chance

8

15


Investigation: Gastroenteritis after a Wedding

16


1. State H0 and HA – Wedding Cake Study

Study 1: Wedding attendees

Attack rate, cake+ = 254 / 411 = 61.8%

Attack rate, cake− = 33 / 223 = 14.8%

H0: the attack rates in the two groups are the

same (RR=1)

HA: the attack rates in the two groups are not the

same (RR ≠ 1), or

HA: those who ate cake had higher attack rate (RR > 1)

9

17


2. Choosing a Statistical Test

Choice depends on:

study design

measurement scale of the variables

study size

Test for comparison of 2 means:

Test for 2-x-2 table data:

Student t-test

Chi-square test

18


Statistical Tests for a 2-by-2 Table

Fisher Exact Test

– use when any expected value < 5

Chi-square Test

– use when all expected values > 5

– 4 variations

– Uncorrected

– Mantel-Haenszel uncorrected

– Yates corrected

– Mantel-Haenszel corrected

10

19


ILL


-----------+-------------+------

+ | 43 11 | 54

- | 3 18 | 21

-----------+-------------+------

Total | 46 29 | 75


Odds ratio 23.45






----------- --------

Uncorrected: XX.XX 0.XXXXXXXX

Mantel-Haenszel: XX.XX 0.XXXXXXXX

Yates corrected: XX.XX 0.XXXXXXXX

20


Test Statistic (“Uncorrected”)

degrees of freedom = (rows−1) (columns −1)

Chi-square test determines whether the

deviations between observed and expected are

too large to be attributed to chance.

expected

expected) - (observed 2

2

Chi-Square Test for Independence

11

21


Oswego Data

Ill Well Total AR

Ate vanilla

ice cream?

Y 43 11 54 79.6%

N 3 18 21 14.3%

46 29 75

How many degrees of freedom?

What is expected in a 2-by-2 table?

1 d.f.

22


Observed number in cell A = 43

Overall attack rate (AR) = 46 / 75 = _____

Expected AR (each group) under H0 = _____

N who ate vanilla ice cream = _____

Expected # cases among those who ate vanilla

ice cream, under H0 = _______________

So, Expected (a) =

column total x (row total / table total)

What’s Expected in a 2-by-2 Table?

0.613

0.613

54

54 x 0.613 = 33.1

12

23


What’s Expected in a 2-by-2 Table?

Ill Well Total

Ate vanilla

ice cream?

Y H1V1 /T H1V0 /T H1

N H0V1 /T H0V0 /T H0

V1 V0 T

In general, expected value =

row total x column total / table total

24


Oswego: Observed vs. Expected

Observed Expected

Cell a 43

Cell b 11

Cell c 3

Cell d 18

Total 75

13

25



Observed Expected

Cell a 43 33.12

Cell b 11

Cell c 3

Cell d 18

Total 75

20.88

12.88

8.12

75.00

26



Observed Ill Well Total % Ill

Vanilla IC+ 43 11 54 79.6%

Vanilla IC− 3 18 21 14.3%

46 29 75 61.3%

Expected Ill Well Total % Ill

Vanilla IC+ 33.12 20.88 54 79.6%

Vanilla IC− 12.88 8.12 21 14.3%

46 29 75 61.3%

14

27


Chi-Square Tests for 2-by-2 Tables

Uncorrected (Pearson) Chi-square Test

Mantel-Haenszel Chi-square Test

Yates corrected Chi-square Test

0101

22 )(

VVHH

bcadt

0101

2

2 2VVHH

tbcadt

0101

22 ))(1(

VVHH

bcadt

28


Example: Randomized Clinical Trial

Cured Not Total Cure (%)

New Tx 7 1 8 87.5%

Old Tx 2 5 7 28.6%

Total 9 6 15

Can we use chi-square? Calculate expected value for cell d.

7 x 6 / 15 = 42 / 15 = 2.8 Use FET

15

29


3. Specify a Level of Significance

Level of significance = an arbitrary cut-off, a small

probability, for deciding whether to declare the

null hypothesis untenable

Also called alpha level

Commonly, alpha set at 0.05 (5%) or 0.01 (1%)

30


4. Perform the Statistical Test, Compute P -value

Chi-square tests provide chi-square test statistic,

which must be converted to P-value (use

computer or look-up table)

P-value = probability of observing a difference as

great or greater than the observed difference, if

the null hypothesis were true

P-value influenced by:

– size of difference / strength of association

– size of the sample

16

31



Observed Expected

Cell a 43 33.12

Cell b 11 20.88

Cell c 3 12.88

Cell d 18 8.12

Total 75 75.00

2.947

4.675

7.579

12.021

27.222

(O-E)2

E

32


Chi-Square Tests for 2-by-2 Tables

Uncorrected (Pearson) Chi-square Test

0101

22 )(

VVHH

bcadt

29462154

)3111843)(75( 22

222.272

17

33


Converting a X2 to a P-Value

To convert the X2 into a P-value by hand, use a

special X2 table

The bigger the X2, the smaller the P-value

For data with 1 degree of freedom, i.e., data from

a 2x2 table, the X2 value must be ≥ 3.84 to yield

a P-value ≤ 0.05

Alternatively, let the computer do the conversion

34


ILL


-----------+-------------+------

+ | 43 11 | 54

- | 3 18 | 21

-----------+-------------+------

Total | 46 29 | 75


Odds ratio 23.45






----------- --------

Uncorrected: 27.22 0.00000018 <---

Mantel-Haenszel: 26.86 0.00000022 <---

Yates corrected: 24.54 0.00000073 <---

18

35


36


5. Make Decision about Hypothesis

If computed P-value < alpha, reject H0, i.e.,

conclude that difference is unlikely to be due to

chance*

If computed P-value > alpha, do not reject H0,

i.e., conclude that difference could be due to

chance*

* You could be right or you could be wrong!

19

37


Two Types of Possible Errors

In reality, H0 is…

True False

Decision re: H0, based on our data

Accept OK Type II

(β) error

Reject Type I

(α) error OK

Level of significance (α) = probability of

making Type I error

1 – α = Confidence 1 – β = Power

38


What Influences a P-value?

Strength of association / size of

difference

Number of subjects (size of sample)

20

39


P- value and Strength of Association

D+ D- AR RR

E+ 10 10 20 50% 2.0 X2 = 2.67

E- 5 15 20 25% p = 0.10

D+ D- AR RR

E+ 12 8 20 60% 2.4 X2 = 5.01

E- 5 15 20 25% p = 0.03

40


P- value and Size of Study

D+ D- AR RR

E+ 10 10 20 50% 2.0 X2 = 2.67

E- 5 15 20 25% p = 0.10

D+ D- AR RR

E+ 20 20 40 50% 2.0 X2 = 5.33

E- 10 30 40 25% p = 0.02

21

41


Hypothetical Cohort Study


Diabetic 2 2 4 50.0%


Diabetic 10 10 20 50.0%


Diabetic 20 20 40 50.0%


X2 = 0.53

P = 0.47

X2 = 2.67

P = 0.10

X2 = 5.33

P = 0.02

42


Notes on Interpretation of Statistical Tests

Statistical testing does not address bias!

Statistical significance ≠ importance

“A difference, to be a difference, has to make a difference.” – Carl Tyler

Not significant ≠ no association

“Absence of evidence should not be taken as evidence of absence.”

– Sherlock Holmes

Statistical significance ≠ causation

22

43


Interpret the Findings (Studies 1–6)

Study P value Interpretation

1 0.007

2 0.03

3 0.08

4 0.65

5 0.0001

6 8 x 10-11

44


Interpret the Findings (Studies 7–11)

Study P value Interpretation

7 0.060

8 0.052

9 0.048

10 0.00009

11 0.9

23

45


Objectives

Describe the reason for using statistical tests – To evaluate the role of chance as an explanation

of observed differences / associations

Describe what a P-value is – Probability of observing >difference under H0

Describe the two influences on a P-value – size of difference / strength of association

– size of the sample

Properly interpret the results of chi-square test – reject H0 if P < α, but use judgment!

46


Author, Acknowledgements, References

Author

Richard Dicker

Acknowledgement

Virgil Peavy

24

47


Baking the Cake Layers

48


Filling the Cakes

25

49


Filling the Cakes

Raspberry jam

Syrup

White

Chocolate

Mousse Filling

50


The Strawberry Filling

26

51


Strawberry Filling

Strawberries

– Fresh

– Washed in sink

– Sliced in the

bake room

– Hand-spread

onto cake filling

52


Cake Composition

Cake Layer 3

Cake Layer 1

Cake Layer 2

Baked Cake

Filling

27

53


Icing, Assembly, and Decorating

54


Attack Rates by Type of WCM Filling

Type of

Filling # Ate

Attack

rate

(eaters)

Attack rate

(non-eaters) RR P-value

Strawberry 408 62% 15% 4.2 0.0001

Chocolate

and Mocha 36 53% 45% 1.2 0.3

WCM only 9 44% 45% 1.0 1.0

Tests of Significance · Tests of Significance Data from a Hypothetical Cohort Study Dead Alive...

Documents

Transcript of Tests of Significance · Tests of Significance Data from a Hypothetical Cohort Study Dead Alive...