Tests of Significance

Analysis of

Two-by-Two Table Data:

UI-MEPI-J: Research Design and Methodology Workshop

VANILLA | + - | Total

-----------+-------------+------

+ | 43 11 | 54

- | 3 18 | 21

-----------+-------------+------

Total | 46 29 | 75

Single Table Analysis

Odds ratio 23.45

Cornfield 95% confidence limits for OR 5.07 < OR < 125.19*

RISK RATIO(RR)(Outcome:ILL=+; Exposure:VANILLA=+) 5.57

95% confidence limits for RR 1.94 < RR < 16.03

Ignore risk ratio if case control study

Chi-Squares P-values

----------- --------

Uncorrected: 27.22 0.00000018 <---

Mantel-Haenszel: 26.86 0.00000022 <---

Yates corrected: 24.54 0.00000073 <---

Objectives

Describe the reason for using statistical tests

Describe what a P-value is

Describe the two main influences on a P-value

for a two-by-two table

Properly interpret the results of chi-square

statistical tests

P < 0.05

P = NS

What is a Statistical Testing?

Also called “hypothesis testing”

… the process of inferring from your data

whether an observed difference is likely to

represent chance variation or a real difference

(Does NOT address bias, confounding, or

investigator error!)

For two-by-two table data, influenced by:

Number of subjects or observations in study

Size of difference in results between groups

Data from a Hypothetical Cohort Study

Dead Alive Total % Dead

Diabetic 2 2 4 50.0%

Nondiabetic 1 3 4 25.0%

3 5 8 37.5%

Diabetic 10 10 20 50.0%

15 25 40 37.5%

Diabetic 20 20 40

30 50 80 37.5%

Diabetic 200 200 400 50.0%

Nondiabetic 100 300 400 25.0%

300 500 800 37.5%

Steps in Statistical Testing

1. State the null and alternative hypotheses

2. Choose a statistical test for testing the null hypothesis

3. Specify a significance level

4. Perform the statistical test, i.e., calculate probability of obtaining data you got if null hypothesis were true

5. Make a decision about the hypotheses

Mindset for Statistical Testing

2 groups – diabetics vs. non-diabetics, cases

vs. controls, etc.

Each is a sample from some larger population

Are they likely to be samples from the same

population, or different populations?

RR = 1

Relative Risk

RR = 1

1. State the Null and Alternative Hypotheses

Null hypothesis

H0: The observed difference is not real, i.e., the

observed difference is the result of chance

Alternative hypothesis

HA: H0 is not true, i.e., the observed difference is

not due to chance

Investigation: Gastroenteritis after a Wedding

1. State H0 and HA – Wedding Cake Study

Study 1: Wedding attendees

Attack rate, cake+ = 254 / 411 = 61.8%

Attack rate, cake− = 33 / 223 = 14.8%

H0: the attack rates in the two groups are the

same (RR=1)

HA: the attack rates in the two groups are not the

same (RR ≠ 1), or

HA: those who ate cake had higher attack rate (RR > 1)

2. Choosing a Statistical Test

Choice depends on:

study design

measurement scale of the variables

study size

Test for comparison of 2 means:

Test for 2-x-2 table data:

Student t-test

Chi-square test

Statistical Tests for a 2-by-2 Table

Fisher Exact Test

– use when any expected value < 5

Chi-square Test

– use when all expected values > 5

– 4 variations

– Uncorrected

– Mantel-Haenszel uncorrected

– Yates corrected

– Mantel-Haenszel corrected

-----------+-------------+------

+ | 43 11 | 54

- | 3 18 | 21

-----------+-------------+------

Total | 46 29 | 75

Odds ratio 23.45

----------- --------

Uncorrected: XX.XX 0.XXXXXXXX

Mantel-Haenszel: XX.XX 0.XXXXXXXX

Yates corrected: XX.XX 0.XXXXXXXX

Test Statistic (“Uncorrected”)

degrees of freedom = (rows−1) (columns −1)

Chi-square test determines whether the

deviations between observed and expected are

too large to be attributed to chance.

expected

expected) - (observed 2

Chi-Square Test for Independence

Oswego Data

Ill Well Total AR

Ate vanilla

ice cream?

Y 43 11 54 79.6%

N 3 18 21 14.3%

46 29 75

How many degrees of freedom?

What is expected in a 2-by-2 table?

1 d.f.

Observed number in cell A = 43

Overall attack rate (AR) = 46 / 75 = _____

Expected AR (each group) under H0 = _____

N who ate vanilla ice cream = _____

Expected # cases among those who ate vanilla

ice cream, under H0 = _______________

So, Expected (a) =

column total x (row total / table total)

What’s Expected in a 2-by-2 Table?

54 x 0.613 = 33.1

What’s Expected in a 2-by-2 Table?

Ill Well Total

Ate vanilla

ice cream?

Y H1V1 /T H1V0 /T H1

N H0V1 /T H0V0 /T H0

V1 V0 T

In general, expected value =

row total x column total / table total

Oswego: Observed vs. Expected

Observed Expected

Cell a 43

Cell b 11

Cell c 3

Cell d 18

Total 75

Observed Expected

Cell a 43 33.12

Cell b 11

Cell c 3

Cell d 18

Total 75

Observed Ill Well Total % Ill

Vanilla IC+ 43 11 54 79.6%

Vanilla IC− 3 18 21 14.3%

46 29 75 61.3%

Expected Ill Well Total % Ill

Vanilla IC+ 33.12 20.88 54 79.6%

Vanilla IC− 12.88 8.12 21 14.3%

46 29 75 61.3%

Chi-Square Tests for 2-by-2 Tables

Uncorrected (Pearson) Chi-square Test

Mantel-Haenszel Chi-square Test

Yates corrected Chi-square Test

2 2VVHH

tbcadt

22 ))(1(

Example: Randomized Clinical Trial

Cured Not Total Cure (%)

New Tx 7 1 8 87.5%

Old Tx 2 5 7 28.6%

Total 9 6 15

Can we use chi-square? Calculate expected value for cell d.

7 x 6 / 15 = 42 / 15 = 2.8 Use FET

3. Specify a Level of Significance

Level of significance = an arbitrary cut-off, a small

probability, for deciding whether to declare the

null hypothesis untenable

Also called alpha level

Commonly, alpha set at 0.05 (5%) or 0.01 (1%)

4. Perform the Statistical Test, Compute P -value

Chi-square tests provide chi-square test statistic,

which must be converted to P-value (use

computer or look-up table)

P-value = probability of observing a difference as

great or greater than the observed difference, if

the null hypothesis were true

P-value influenced by:

– size of difference / strength of association

– size of the sample

Observed Expected

Cell a 43 33.12

Cell b 11 20.88

Cell c 3 12.88

Cell d 18 8.12

Total 75 75.00

12.021

27.222

(O-E)2

Chi-Square Tests for 2-by-2 Tables

Uncorrected (Pearson) Chi-square Test

29462154

)3111843)(75( 22

222.272

Converting a X2 to a P-Value

To convert the X2 into a P-value by hand, use a

special X2 table

The bigger the X2, the smaller the P-value

For data with 1 degree of freedom, i.e., data from

a 2x2 table, the X2 value must be ≥ 3.84 to yield

a P-value ≤ 0.05

Alternatively, let the computer do the conversion

-----------+-------------+------

+ | 43 11 | 54

- | 3 18 | 21

-----------+-------------+------

Total | 46 29 | 75

Odds ratio 23.45

----------- --------

Uncorrected: 27.22 0.00000018 <---

Mantel-Haenszel: 26.86 0.00000022 <---

Yates corrected: 24.54 0.00000073 <---

5. Make Decision about Hypothesis

If computed P-value < alpha, reject H0, i.e.,

conclude that difference is unlikely to be due to

chance*

If computed P-value > alpha, do not reject H0,

i.e., conclude that difference could be due to

chance*

* You could be right or you could be wrong!

Two Types of Possible Errors

In reality, H0 is…

True False

Decision re: H0, based on our data

Accept OK Type II

(β) error

Reject Type I

(α) error OK

Level of significance (α) = probability of

making Type I error

1 – α = Confidence 1 – β = Power

What Influences a P-value?

Strength of association / size of

difference

Number of subjects (size of sample)

P- value and Strength of Association

D+ D- AR RR

E+ 10 10 20 50% 2.0 X2 = 2.67

E- 5 15 20 25% p = 0.10

D+ D- AR RR

E+ 12 8 20 60% 2.4 X2 = 5.01

E- 5 15 20 25% p = 0.03

P- value and Size of Study

D+ D- AR RR

E+ 10 10 20 50% 2.0 X2 = 2.67

E- 5 15 20 25% p = 0.10

D+ D- AR RR

E+ 20 20 40 50% 2.0 X2 = 5.33

E- 10 30 40 25% p = 0.02

Hypothetical Cohort Study

Diabetic 2 2 4 50.0%

Diabetic 10 10 20 50.0%

Diabetic 20 20 40 50.0%

X2 = 0.53

P = 0.47

X2 = 2.67

P = 0.10

X2 = 5.33

P = 0.02

Notes on Interpretation of Statistical Tests

Statistical testing does not address bias!

Statistical significance ≠ importance

“A difference, to be a difference, has to make a difference.” – Carl Tyler

Not significant ≠ no association

“Absence of evidence should not be taken as evidence of absence.”

– Sherlock Holmes

Statistical significance ≠ causation

Interpret the Findings (Studies 1–6)

Study P value Interpretation

1 0.007

2 0.03

3 0.08

4 0.65

5 0.0001

6 8 x 10-11

Interpret the Findings (Studies 7–11)

Study P value Interpretation

7 0.060

8 0.052

9 0.048

10 0.00009

11 0.9

Objectives

Describe the reason for using statistical tests – To evaluate the role of chance as an explanation

of observed differences / associations

Describe what a P-value is – Probability of observing >difference under H0

Describe the two influences on a P-value – size of difference / strength of association

– size of the sample

Properly interpret the results of chi-square test – reject H0 if P < α, but use judgment!

Author, Acknowledgements, References

Author

Richard Dicker

Acknowledgement

Virgil Peavy

Baking the Cake Layers

Filling the Cakes

Raspberry jam

Chocolate

Mousse Filling

The Strawberry Filling

Strawberry Filling

Strawberries

– Fresh

– Washed in sink

– Sliced in the

bake room

– Hand-spread

onto cake filling

Cake Composition

Cake Layer 3

Cake Layer 1

Cake Layer 2

Baked Cake

Filling

Icing, Assembly, and Decorating

Attack Rates by Type of WCM Filling

Type of

Filling # Ate

Attack

(eaters)

Attack rate

(non-eaters) RR P-value

Strawberry 408 62% 15% 4.2 0.0001

Chocolate

and Mocha 36 53% 45% 1.2 0.3

WCM only 9 44% 45% 1.0 1.0

Tests of Significance · Tests of Significance Data from a Hypothetical Cohort Study Dead Alive...

Documents

Transcript of Tests of Significance · Tests of Significance Data from a Hypothetical Cohort Study Dead Alive...

Significance tests or confidence intervals: which are ...

Significance of Tests for Petroleum Products

Significance, P-value and t-tests

Significance Tests for Proportions Presentation 9.2.

Hypothesis Tests, Significance - Caltech High Energy Physics

Objectives (BPS chapter 15) Tests of significance: the basics The reasoning of tests of significance Stating hypotheses Test statistics P-values.

11.1 – Significance Tests: The basics

Tests of Significance Ssm

Statistical Tests, Tests of Significance, and Tests of a · PDF file · 2017-07-09Statistical Tests, Tests of Significance, and Tests of a Hypothesis Using Excel ... its value no

8. Tests of Significance

FPP 26-27 Significance Tests. Significance tests Question: Given the collected data, is there evidence against a specified hypothesis about the corresponding.

Significance Tests

Review Tests of Significance. Single Proportion.

Tests of Significance - University of West Georgia · Describe the reasoning of tests of significance. Describe the parts of a significance test. State hypotheses. Define P-value

Significance Testing & Univariate Significance Tests Purposes of NHST Process of Null Hypothesis Significance Testing Univariate Significance Tests.

Effect sizes, significance tests, and confidence intervals- Asses.pdf

The use & abuse of tests Statistical significance ≠ practical significance Significance ≠ proof of effect (confounds) Lack of significance ≠ lack of effect.

Hypothesis Testing and Tests of Significance

8 Tests of Significance