Intro to StatsSignificance Testing

A statistical method that uses sample data to evaluate a hypothesis about a population

1. State a hypothesis 2. Use the hypothesis to predict the

characteristics the sample should have 3. Obtain a random sample from the

population 4. Compare the obtained sample data with

the prediction made from the hypothesis

Hypothesis Test

Suggests that the difference or relationship is systematic and not due to chance

A significance level gives the risk that the effect is due to chance

At p < .05, there is about a 5% (5/100) chance that an effect just happened in a normal distribution and is not related to any other variable

“Significant”

p < .05 for 2-tailed occurs at z = 1.96

Difference in size between the sample mean and the original population mean

The variability of scores The number of scores in the sample

Qualifying factors

We make certain assumptions about the impact of the methods used to obtain the data sample

Random sampling◦ That the sample represents the population

Independent observations◦ The first measurement has no relationship to the probability

of the second measurement The value of variability is unchanged by the

treatment◦ In most hypothesis-testing situations, we do not have the

“original” population variability A normal distribution

Assumptions Made

Researchers can never be sure that their hypothesis is “true”◦ Sample may not perfectly reflect the population◦ Other influences (confounds) may cause the

results◦ It just might be one of those few chances

These concerns are lessened every time that a finding is replicated

Never Sure

The null is rejected, but there really is no effect (Type I error)

Or fail to reject the null (accept null), but there really is an effect (Type II error)

Can lead to errors

Never know the truth about the null hypothesis

The likelihood of a Type I error is defined by the level of significance◦ p < .05 means there is a 5% chance of rejecting

the null when the null is true (conclude there is a difference when there is none)

Type II error is related to power and sample size

Never Sure 2

Rather than rely on hypothesis testing, it is now typical to include effect sizes◦ Most hypothesis tests simply state that a finding

is UNLIKELY (or not)◦ Significance does not mean it’s important◦ Effect sizes give a way to capture the SIZE of an

effect and make stronger statements about the relationship

◦ This also sidesteps issues of power Power is strongly linked to sample size and therefore

effects may be underestimated in small samples and overestimated in large samples

Effect sizes

Studies may show significant differences but the differences may not be meaningful◦ Small differences in large samples◦ Small differences that come with a large cost

Studies may not show significance but the differences may be meaningful◦ Small differences in small samples◦ Small differences that come with a big benefit and

little cost

Significance vs. Meaning

Inferences made about the population based on a sample

Which test determined by◦ Continuous vs. categorical variables◦ Number of variables◦ Whether variables vary between subjects or

within subjects

*Your book has a useful chart

Inferential Statistics

1. State the null and research hypotheses 2. Set the level of risk (usually .05) 3. Select the appropriate test statistic 4. Compute the test statistic 5. Determine the critical value for rejection of the

null 6. Determine whether the statistic exceeds the

critical value (usually at p < .05) 7&8. If over the critical value, the null hypothesis is

unlikely THEREFORE effect must be due to other variable

If not over the critical value, the null is accepted INTERPRET

Setting the Stage

One-sample z-tests are inferential statistics that allow you to compare a mean from a sample to the average of a population.

One-sample z-tests

1. State hypotheses◦ Null hypothesis: there is no difference between

civic engagement in Texas A&M students and the national average H0: XA&M = μstudents

◦ Research hypothesis: there is a difference between civic engagement at A&M and the national average H1: XA&M ≠ μstudents

Example 1

Z = X – μ SEM X = mean of sample μ = population average SEM = standard error of the mean

SEM = σ/square root of n σ = standard deviation for the population n = size of the sample

Calculating

Civic engagement of A&M students◦ X = 3.42 organizations, n = 200◦ μ = 1.5 organizations, σ = 1.2

Example 1

Civic engagement of A&M students◦ X = 3.42 organizations, n = 200◦ μ = 1.5 organizations, σ = 1.2

◦ SEM = σ/square root of n

Example 1

5. Determine the critical value for rejection of the null

We know that a z of 1.96 or larger is significant at p < .05

Remember this is the smallest value of z that corresponds to a p of .05 or less (falls in the “extreme range”

Bigger zs = more likely to be a significant difference

Example 1

6. Determine whether the statistic exceeds the critical value◦ 24 > 1.96◦ So it does exceed the critical value

◦ THE NULL IS REJECTED IF OUR STATISTIC IS BIGGER THAN THE CRITICAL VALUE – THAT MEANS THE DIFFERENCE IS SIGNIFICANT AT p < .05!!

7. If not over the critical value, fail to reject the null

Example 1

In results◦ Student participants at Texas A&M University

joined more civic organizations per year (M = 3.42) than the national average, z = 24.00, p < .05.

Example 1