Review Tests of Significance. Single Proportion.

60
Review Tests of Significance

Transcript of Review Tests of Significance. Single Proportion.

Page 1: Review Tests of Significance. Single Proportion.

Review

Tests of Significance

Page 2: Review Tests of Significance. Single Proportion.

Single Proportion Null Hypothesis: Buzz will randomly pick

a button. (He chooses the correct button 50% of the time, in the long run.) ()

Alternative Hypothesis: Buzz understands what Doris is communicating to him. (He chooses the correct button more than 50% of the time, in the long run.) ()

Page 3: Review Tests of Significance. Single Proportion.

Single Proportion Buzz got it right 15 out of 16 times ( This is very unlikely (p-value = 0.0005) to

occur by chance.

Page 4: Review Tests of Significance. Single Proportion.

Single Proportion Theory-based works well when number of

successes and failures are at least 10. A normal distribution is used to predict

what the null distribution looks like. (These are centered on the proportion under the null hypothesis.)

Page 5: Review Tests of Significance. Single Proportion.

Comparing Two ProportionsNull hypothesis: Swimming with dolphins has no association on if someone shows substantial improvement

(dolphins = control or dolphins control = 0)

Alternative hypothesis: Swimming with dolphins increases the probability of substantial improvement in depression symptoms

(dolphins > control or dolphins control > 0)

Page 6: Review Tests of Significance. Single Proportion.

Comparing Two Proportions Our statistic is the observed difference in

proportions 0.67 – 0.20 = 0.47.

 Dolphingroup

Control group

Total

Improved 10 (67%) 3 (20%) 13

Did Not Improve 5 12 17

Total 15 15 30

Page 7: Review Tests of Significance. Single Proportion.

Comparing Two Proportions If the null hypothesis is true (dolphin therapy is not

better) we would have 13 improvers and 17 non-improvers regardless of the group they were in.

Any differences we see between groups arise solely from the randomness in the assignment to the groups.

Randomly assign the groups to the improvers and non-improvers and recalculate the statistic many times.

Page 8: Review Tests of Significance. Single Proportion.

Comparing Two Proportions We did 1000 repetitions to develop a null

distribution and found that just 13 out of 1000 results had a difference of 0.47 or higher (p-value = 0.013).

Page 9: Review Tests of Significance. Single Proportion.

Comparing Two Proportions Just like with a single proportion, the

theory-based test works well when number of successes and failures are at least 10 in each group.

Again, a normal distribution is used to predict the shape of the null distribution. (These are always centered at 0.)

Page 10: Review Tests of Significance. Single Proportion.

Comparing Two Means Null hypothesis: There is no association

between which bike is used and commute time Commute time is not affected by which bike is

used. (µcarbon = µsteel OR µcarbon – µsteel = 0)

 Alternative hypothesis: There is an association between which bike is used and commute time Commute time is affected by which bike is

used. (µcarbon ≠ µsteel OR µcarbon – µsteel ≠ 0)

Page 11: Review Tests of Significance. Single Proportion.

Comparing Two Means

Bike type Sample size

Sample mean

Sample SD

Carbon frame 26 108.34 min 6.25 min

Steel frame 30 107.81 min 4.89 min

Our statistic is the observed difference in means 108.34 – 107.81 = 0.53.

Page 12: Review Tests of Significance. Single Proportion.

Comparing Two MeansThe Original Data

Shuffling assumes the null hypothesis that the bike has no effect on commute times.

Calculate the simulated statistic after shuffling. Repeating this many times develops a null distribution

Shuffled Results

Page 13: Review Tests of Significance. Single Proportion.

Comparing Two MeansStrength of Evidence 705 of 1000 repetition are 0.53 or farther

away from 0. p-value = 0.705.

Page 14: Review Tests of Significance. Single Proportion.

Comparing Two Means A theory-based test works well here when

the sample size is at least 20. A t-distribution is used to predict the

shape of the null distribution and it is centered on 0.

Page 15: Review Tests of Significance. Single Proportion.

Matched Pairs H0: µd = 0

On average, the mean of the differences between the running times (narrow – wide) is 0.

Ha: µd 0 On average, the mean of the differences in

running times (narrow – wide) is not 0.

Page 16: Review Tests of Significance. Single Proportion.

Matched Pairs In this type of test the data starts off as

two separate groups. But there is a naturally pairing. In this case the times for the same person running both paths.

So we need to look at the differences.

Page 17: Review Tests of Significance. Single Proportion.

Matched Pairs

Mean difference is d = 0.075 seconds

Subject 1 2 3 4 5 6 7 8 9 10  narrow angle

5.50 5.70 5.6 5.50 5.85 5.55 5.40 5.50 5.15 5.80 …

wide angle

5.55 5.75 5.5 5.40 5.70 5.60 5.35 5.35 5.00 5.70 …

diff -0.05 -0.05 0.1 0.1 0.15 -0.05 0.05 0.15 0.15 0.10 …

Page 18: Review Tests of Significance. Single Proportion.

Matched Pairs The null basically says the running path doesn’t

matter. So we can randomly decide which time goes with the

which path (Notice we don’t break our pairs.) Each time we do this, compute a simulated difference

in means. We repeat this process many times to develop a null

distribution.Subject 1 2 3 4 5 6 7 8 9 10  

narrow angle

5.55 5.70 5.50 5.50 5.70 5.60 5.40 5.50 5.15 5.70 …

wide angle 5.50 5.75 5.60 5.40 5.85 5.55 5.35 5.35 5.00 5.80 …

diff 0.05 -0.05 -0.1 0.1 -0.15 0.05 0.05 0.15 0.15 -0.1 …

Page 19: Review Tests of Significance. Single Proportion.

Matched Pairs Only 2 of the 1000 repetitions of random

swappings gave a value at least as extreme as 0.075

Page 20: Review Tests of Significance. Single Proportion.

Matched Pairs A theory-based test works well when the

sample size is at least 20. Like comparing two means, a t-distribution

is used to predict the null distribution. The data used in this test are the

differences and this is the same test that is used for a single mean. (Except in testing a single mean, we compare data to any number, not typically just 0.)

Page 21: Review Tests of Significance. Single Proportion.

Comparing Multiple Proportions Null hypothesis: There is no association

between the arrival pattern of the vehicle and if it comes to a complete stop.

(Single = Lead = Follow) Alternative hypothesis: There is an association

between the arrival pattern of the vehicle and if it comes to a complete stop. The alternative hypothesis is that (Not all these long-term probabilities are the same OR at least one is different).

Page 22: Review Tests of Significance. Single Proportion.

Comparing Multiple Proportions

  Single Vehicle

Lead Vehicle

Following Vehicle

Total

Complete Stop 151 (85.8%)

38 (90.5%)

76 (77.6%)

265

Not Complete Stop

25 (14.2%)

4 (9.5%)

22 (22.4%)

51

Total 176 42 98 316

MAD (mean absolute difference)

Page 23: Review Tests of Significance. Single Proportion.

Comparing Multiple Proportions If there is no association between arrival

pattern and whether or not a vehicle stops it basically means it doesn’t matter what the arrival pattern is. Some vehicles will stop no matter what the arrival pattern and some vehicles won’t.

We can model this by shuffling either the explanatory or response variables. (The applet will shuffle the response.) and recomputing the MAD statistic many times.

Page 24: Review Tests of Significance. Single Proportion.

Comparing Multiple Proportions Simulated values of the statistic for 1000

shuffles P-value = 0.083

Page 25: Review Tests of Significance. Single Proportion.

Comparing Multiple Proportions Theory-based tests work well for multiple

proportions if the number of successes and failures are at least 10. (Just like with all proportions.)

The MAD statistic is not used in theory-based, but the chi-squared statistic (and hence a chi-squared distribution) is.

This is test is called a chi-squared test of association.

Page 26: Review Tests of Significance. Single Proportion.

Comparing Multiple Means

Null: There is no association between whether and when a picture was shown and comprehension of the passage(µno picture = µpicture before = µpicture after)

Alternative: There is an association between whether and when a picture was shown and comprehension of the passage(At least one of the mean comprehension scores will be different.)

Page 27: Review Tests of Significance. Single Proportion.

Comparing Multiple Means

Means

3.37

3.21

4.95

No_Picture

Picture_After

Picture_Before

Comprehension0 1 2 3 4 5 6 7 8

MAD = (|3.21−4.95|+|3.21−3.37|+|4.95−3.37|)/3= 1.16.

Page 28: Review Tests of Significance. Single Proportion.

Comparing Multiple Means Simulated values of the statistic for 5000

shuffles P-value = 0.0008

Page 29: Review Tests of Significance. Single Proportion.

Comparing Multiple Means Since we have a small p-value we can

conclude at least one of the mean comprehension scores is different.

We can do pairwise confidence intervals to find which means are significantly different than the other means.

Page 30: Review Tests of Significance. Single Proportion.

Comparing Multiple Means Theory-based tests work well when we

have at a sample size of least 20 in each group. (Like all tests with means.)

The MAD statistic is not used, but an F-statistic (and hence an F distribution) is.

Just like the MAD, the larger the F-statistic, the larger the strength of evidence (and hence a smaller p-value).

This test is called Analysis of Variance or ANOVA.

Page 31: Review Tests of Significance. Single Proportion.

Correlation/Regression

Null: There is no association between heart rate and body temperature.

(ρ = 0 or β = 0) Alternative: There is a positive linear

association between heart rate and body temperature.

(ρ > 0 or β > 0)

Page 32: Review Tests of Significance. Single Proportion.

Correlation/RegressionHR 72 69 72 71 80 81 68 82 68 65

Tmp 98.3 98.2 98.7 98.5 97.0 98.8 98.5 98.7 99.3 97.8

HR 71 79 86 82 58 84 73 57 62 89

Tmp 98.2 99.9 98.6 98.6 97.8 98.4 98.7 97.4 96.7 98.0

96.5

97.0

97.5

98.0

98.5

99.0

99.5

100.0

heart_rate

55 60 65 70 75 80 85 90

temp vs heart rate Scatter Plot

r = 0.378

Page 33: Review Tests of Significance. Single Proportion.

Correlation/Regression If there is no association, we can break

apart the temperatures and their corresponding heart rates by scrambling one of the variables. Just like we did in previous tests. We will do this by scrambling one of the variables.

After each scramble, we will compute the appropriate statistic, either correlation or the slope of the regression equation.

Repeat this many times to develop a null distribution.

Page 34: Review Tests of Significance. Single Proportion.

Correlation/Regression We found that

68/1000 times we had a simulated correlation greater than or equal to 0.378.

Page 35: Review Tests of Significance. Single Proportion.

Correlation/Regression Theory-based test work well when the values of

the response variable are normally distributed for each value of the explanatory variable and these normal distributions have similar variability.

We can use either the correlation or the slope of the regression line as the statistic.

A t-distribution, centered at 0, is used.

Page 36: Review Tests of Significance. Single Proportion.

Review

Confidence Intervals

Page 37: Review Tests of Significance. Single Proportion.

Confidence Intervals Tests of significance answer yes/no

questions. Is there strong evidence that Buzz is not just

guessing? Is there strong evidence that swimming with

dolphins help reduce depression symptoms? Sometimes we might just might want an

estimate of a population parameter. E.g. What proportion of the voters will vote in the next election?

Page 38: Review Tests of Significance. Single Proportion.

Confidence Intervals Confidence intervals are interval

estimates of a population parameter. A population parameter is some fixed

measurement for a population such as a proportion (or long-term probability), a difference in two proportions, a mean, a difference in means, or a slope of a regression equation.

These intervals give plausible (believable, credible) values for the parameter.

Page 39: Review Tests of Significance. Single Proportion.

2SD Confidence Intervals The observed statistics we found are used

as the center of these intervals. We used 2 standard deviations of an

appropriate null distribution as our margin of error to give us a 95% confidence interval.

Observed statistic ± 2SD

Remember the observed statistic can be a single mean or proportion, slope of a regression line, or a difference in two means or proportions.

Page 40: Review Tests of Significance. Single Proportion.

Supersize Drinks A survey found 46% of 1093 randomly selected

NYC voters supported the ban on large soft drinks.

What is our estimate of the population proportion that supports the ban?

0.46 ± 2(0.015) or 0.46 ± 0.03 43% to 49%

Page 41: Review Tests of Significance. Single Proportion.

Meaning of a confidence interval What does 95% confidence mean? If we resampled 1093 NYC voters over and

over and each time produced 95% confidence intervals, 95% of the time we would capture the true proportion of all NYC voters that favor the ban.

The interval (like the observed proportion) is random. The population parameter is fixed.

Page 42: Review Tests of Significance. Single Proportion.

Theory-based confidence intervals Using theory-based techniques, confidence

intervals can easily be found and the confidence levels can easily be adjusted.

The same validity conditions we use for tests of significance should also be used for confidence intervals.

Page 43: Review Tests of Significance. Single Proportion.

What effects the width of CI? As the level of confidence increases, the

width of the confidence interval increases. The wider interval, the more confident we

captured the parameter. (The wider the net, the more confident we capture the fish.)

As the sample size increases, the width of the confidence interval decreases. Larger sample sizes give us more information,

thus we can be more accurate.

Page 44: Review Tests of Significance. Single Proportion.

Connecting confidence intervals and tests of significance A small p-value means that the value

under the null will not be contained in the confidence interval.

H0: , p-value = 0.02, CI (0.52, 0.59)

A large p-value means that the value under the null will be contained in the confidence interval.

H0: , p-value = 0.42, CI (-0.28, 0.57)

Page 45: Review Tests of Significance. Single Proportion.

Significance level and confidence level Suppose H0: , and the corresponding two-

sided p-value = 0.03. Will 0.5 be contained in a: 90% confidence interval?

no 95% confidence interval?

no 99% confidence interval?

yes

If a the p-value is large (greater than ), the value under the null will be contained in a ()% confidence interval.

Page 46: Review Tests of Significance. Single Proportion.

Review

Big Ideas

Page 47: Review Tests of Significance. Single Proportion.

Terminology The population is the entire set of

observational units we want to know something about.

The sample is the subgroup of the population on which we actually record data.

A statistic is a number calculated from the observed data.

A parameter is the same type of number as the statistic, but represents the underlying process or the population from which the sample was selected.

Page 48: Review Tests of Significance. Single Proportion.

Terminology Standard deviation (SD) is the most

common measure of variability. We can think of standard deviation as

average distance values are from their mean.

A distribution is skewed to the right if the right side extends much farther than the left side.

Page 49: Review Tests of Significance. Single Proportion.

Hypotheses and Null Distribution The null hypothesis (H0) is the chance

explanation. (=) The alternative hypothesis (Ha) is you

are trying to show is true. (<, >, or ≠) A null distribution is the distribution of

simulated statistics that represent the chance outcome.

Page 50: Review Tests of Significance. Single Proportion.

Significance and p-value Results are statistically significant if

they are unlikely to arise by random chance (or a true null hypothesis).

The p-value is the proportion of the simulated statistics in the null distribution that are at least as extreme as the value of the observed statistic.

The smaller the p-value, the stronger the evidence against the null.

Page 51: Review Tests of Significance. Single Proportion.

Guidelines for evaluating strength of evidence from p-values p-value >0.10, not much evidence against

null hypothesis 0.05 < p-value < 0.10, moderate evidence

against the null hypothesis 0.01 < p-value < 0.05, strong evidence

against the null hypothesis p-value < 0.01, very strong evidence

against the null hypothesis

Page 52: Review Tests of Significance. Single Proportion.

Three S Strategy Statistic: Compute the statistic from the

observed data. Simulate: Identify a model that represents a

chance explanation. Repeatedly simulate values of the statistic that could have happened when the chance model is true and form a distribution. (Null distribution)

Strength of evidence: Consider whether the value of the observed statistic is unlikely to occur when the chance model is true.

(p-value)

Page 53: Review Tests of Significance. Single Proportion.

Standardized statistics and 2-sided tests The standardized statistic is the

number of standard deviations the observed statistic is above (or below) the mean of the null distribution.

Two-sided tests increase the p-value (it about doubles in simulation-based and exactly doubles in theory-based)

Two-sided tests are said to be more conservative. More evidence is needed to conclude alternative.

Page 54: Review Tests of Significance. Single Proportion.

Biased / Simple Random Sampling A sampling method is biased if statistics from

samples consistently over or under-estimate the population parameter.

A simple random sample is the easiest way to insure that your sample is unbiased. Taking SRS’s allow us to infer our results to the population from which is was drawn.

Simple random sampling is a way of selecting members of a population so that every sample of a certain size from a population has the same chance of being chosen.

Page 55: Review Tests of Significance. Single Proportion.

Types of Variables When two variables are involved in a

study, they are often classified as explanatory and response

Explanatory variable (Independent, Predictor) The variable we think is “explaining” the

change in the response variable. Response variable (Dependent)

The variable we think is being impacted or changed by the explanatory variable.

Page 56: Review Tests of Significance. Single Proportion.

Random assignment / Causation

Confounding variables are controlled in experiments due to the random assignment of subjects to treatment groups since this tends to balance out all other variables between the groups.

Thus, cause and effect conclusions are possible in experiments through random assignment. (It must be a well run experiment.)

Page 57: Review Tests of Significance. Single Proportion.

Random vs. Random With observational studies, random

sampling is often done. This allows us to make inferences from the sample to the population where the sample was drawn.

With experiments, random assignment is done. This allows us to conclude causation.

Page 58: Review Tests of Significance. Single Proportion.

Overall Test We used one overall test (chi-squared test

or ANOVA) when comparing more than two proportions or more than two means.

We do these overall tests since performing many individual tests increases the possibility of making a type I error (false-positive or false alarm).

If significance is found in an overall test, then we can follow up with individual tests or confidence intervals.

Page 59: Review Tests of Significance. Single Proportion.

Correlation Correlation measures the strength and

direction of a linear association between two quantitative variables.

Correlation is a number between -1 and 1. With positive correlation one variable increases as

the other increases. With negative correlation one variable decreases as

the other increases. The closer it is to either -1 or 1 the closer the points

fit to a line.

Page 60: Review Tests of Significance. Single Proportion.

The least-squares regression line is the most common way getting a mathematical model (linear equation) for an association between two quantitative variables.

Slope is the predicted change in the response variable for one-unit change in the explanatory variable.

Regression