Hypothesis Testing

Hypothesis testing

Dr George Varvatsoulias

Generating testable

hypotheses (1)

Generating testable

hypotheses (2)

A hypothesis is a statement that explains what has been observed.

To check that the statement is true, the hypothesis must be testable.

In any scientific research, there is usually a “research problem”

framed as a question which the research aims to answer. For

example, an investigator makes general observations on the thorny

spines of a certain cactus plant, and comes up with a problem

question: “Are spines related to physical defences against

herbivores?”

Generating testable

hypotheses (3)

However, that was a tentative question too broad as a

statement to be tested by any scientific means. To be

able to build an experiment to answer the question, the

question must first be turned into a realistic research

hypothesis that can be tested.

Generating testable

hypotheses (4)

The hypothesis, then, should be a statement of planned explanation for the observed phenomenon. In the cactus example, the research hypothesis could be: “Cactus spines are found in areas where herbivore grazing occurs, but are lacking in areas where grazing is absent.” Because the hypothesis has been clearly described, the investigator can now design an experiment and collect the data for testing (of the hypothesis).

Generating testable

hypotheses (4)

The experiment might be to count the number of cacti with

spines in an area subject to grazing and in an area where

grazing herbivores are absent. Data from the two areas may

then be compared using statistical tests. This is exactly what

Myers & Bazely (1991) did on islands where cattle grazing

were present and absent.

The Scientific method of

inquiry

The series of steps described so far — observation, question,

hypothesis, experiment — are in fact steps of a formal process to

scientific inquiry, called the “scientific method”. Scientists use this

method to explain the observations they have made, and hypothesis

testing is pivotal to providing the explanation.

Testing of Null and Alternative

Hypotheses (1)

There are two types of hypotheses in statistical testing: a “null

hypothesis” that states no difference in what is being compared (or

no pattern or relation), and one or more “alternative hypothesis” that

states a difference or some relation. The “research hypothesis” is in

reality the alternative hypothesis, but it is usually the null hypothesis

that is being tested, because it shows that the investigator is not

biased toward the outcomes of the test.

Testing of Null and Alternative

Hypotheses (2)

In our example, suppose a two-sample test produced a high

probability (P) value, then the null hypothesis (that the percentage of

cacti with spines is similar in the two areas) should be accepted. The

alternative research hypothesis is considered likely to be true only if

the P-value is low (usually <0.05). In this way results of the

hypothesis testing can be used to support or refute the

question/proposal that spines are an evolutionary defence response

to herbivore pressure.

Tips to generating a testable

hypothesis (1)

When making observations of a topic of your interest, write preliminary

notes and read up about the topic or phenomenon.

Turn your curiosity into a question, such as how or why a certain pattern

occurs as observed.

A hypothesis is a proposed explanation for the pattern observed. Write

down all the explanations you can think of.

Choose one (s) that is uncomplicated, where you could make a prediction

on the outcome; reword the statement to make it precise, unambiguous,

and as simple as possible.

Tips to generating a testable

hypothesis (2)

This becomes your research hypothesis; substitute it with a

statement that specifies the null hypothesis.

Design an experiment, including the statistical techniques to be

used, that can test whether this hypothesis statement is true or not.

If you cannot test the hypothesis, then the hypothesis will not work,

and you will need to generate more hypothesis statements until you

find one that is testable.

Sampling methods in

research(1)

Simple random sampling

1. In a simple random sample (SRS) of a given size, all such subsets of the frame are given an equal probability. Furthermore, any given pair of elements has the same chance of selection as any other such pair (and similarly for triples, and so on). This minimises bias and simplifies analysis of results. In particular, the variance between individual results within the sample is a good indicator of variance in the overall population, which makes it relatively easy to estimate the accuracy of results.

Sampling methods in

research(2)

2. SRS can be vulnerable to sampling error because the randomness of the

selection may result in a sample that doesn't reflect the makeup of the

population. For instance, a simple random sample of ten people from a

given country will on average produce five men and five women, but any

given trial is likely to over-represent one sex and under-represent the other.

(Systematic and stratified techniques), attempt to overcome this problem by

"using information about the population" to choose a more "representative"

sample.

Sampling methods in

research(3)

3. SRS may also be cumbersome and tedious when sampling from an unusually

large target population. In some cases, investigators are interested in "research

questions specific" to subgroups of the population. For example, researchers

might be interested in examining whether cognitive ability as a predictor of job

performance is equally applicable across racial groups. SRS cannot

accommodate the needs of researchers in this situation because it does not

provide subsamples of the population. "Stratified sampling“ addresses this

weakness of SRS.

Sampling methods in

research(4)

Systematic sampling

1. Systematic sampling relies on arranging the study population according to

some ordering scheme and then selecting elements at regular intervals through

that ordered list. Systematic sampling involves a random start and then

proceeds with the selection of every kth element from then onwards. In this

case, k=(population size/sample size). It is important that the starting point is not

automatically the first in the list, but is instead randomly chosen from within the

first to the kth element in the list. A simple example would be to select every

10th name from the telephone directory (an 'every 10th' sample, also referred to

as 'sampling with a skip of 10').

Sampling methods in

research(5)

2. As long as the starting point is randomized, systematic sampling is a type of probability

sampling. It is easy to implement and the stratification induced can make it efficient, if the

variable by which the list is ordered is correlated with the variable of interest. 'Every 10th'

sampling is especially useful for efficient sampling from databases.

For example, suppose we wish to sample people from a long street that starts in a poor

area (house No. 1) and ends in an expensive district (house No. 1000). A simple random

selection of addresses from this street could easily end up with too many from the high end

and too few from the low end (or vice versa), leading to an unrepresentative sample.

Selecting (e.g.) every 10th street number along the street ensures that the sample is

spread evenly along the length of the street, representing all of these districts. (Note that if

we always start at house #1 and end at #991, the sample is slightly biased towards the low

end; by randomly selecting the start between #1 and #10, this bias is eliminated.

Sampling methods in

research(6)

Randomised sample example

Sampling methods in

research(6)

3. However, systematic sampling is especially vulnerable to periodicities in the list. If

periodicity is present and the period is a multiple or factor of the interval used, the

sample is especially likely to be unrepresentative of the overall population, making

the scheme less accurate than simple random sampling.

For example, consider a street where the odd-numbered houses are all on the north

(expensive) side of the road, and the even-numbered houses are all on the south

(cheap) side. Under the sampling scheme given above, it is impossible to get a

representative sample; either the houses sampled will all be from the odd-numbered,

expensive side, or they will all be from the even-numbered, cheap side.

Sampling methods in

research(7)

4. Another drawback of systematic sampling is that even in scenarios

where it is more accurate than SRS, its theoretical properties make

it difficult to quantify that accuracy. (In the two examples of

systematic sampling that are given above, much of the potential

sampling error is due to variation between neighbouring houses -

but because this method never selects two neighbouring houses,

the sample will not give us any information on that variation.)

Sampling methods in

research(8)

5. Systematic sampling can be EPS method (equal probability of selection

method), because all elements have the same probability of selection (in the

example given, one in ten). It is not 'simple random sampling' because

different subsets of the same size have different selection probabilities - e.g.

the set {4,14,24,...,994} has a one-in-ten probability of selection, but the set

{4,13,24,34,...} has zero probability of selection

Sampling methods in

research(9)

Stratified sampling

1. Where the population embraces a number of distinct categories, the

frame can be organized by these categories into separate "strata."

Each stratum is then sampled as an independent sub-population,

out of which individual elements can be randomly selected. There

are several potential benefits to stratified sampling.

Sampling methods in

research(10)

2. First, dividing the population into distinct, independent strata can enable researchers to draw inferences about specific subgroups that may be lost in a more generalized random sample.

Second, utilizing a stratified sampling method can lead to more efficient statistical estimates (provided that strata are selected based upon relevance to the criterion in question, instead of availability of the samples). Even if a stratified sampling approach does not lead to increased statistical efficiency, such a tactic will not result in less efficiency than would simple random sampling, provided that each stratum is proportional to the group's size in the population.

Third, it is sometimes the case that data are more readily available for individual, pre-existing strata within a population than for the overall population; in such cases, using a stratified sampling approach may be more convenient than aggregating data across groups (though this may potentially be at odds with the previously noted importance of utilizing criterion-relevant strata).

Finally, since each stratum is treated as an independent population, different sampling approaches can be applied to different strata, potentially enabling researchers to use the approach best suited (or most cost-effective) for each identified subgroup within the population.

Sampling methods in

research(11)

3. There are, however, some potential drawbacks to using stratified sampling.

First, identifying strata and implementing such an approach can increase the

cost and complexity of sample selection, as well as leading to increased

complexity of population estimates. Second, when examining multiple criteria,

stratifying variables may be related to some, but not to others, further

complicating the design, and potentially reducing the utility of the strata. Finally,

in some cases (such as designs with a large number of strata, or those with a

specified minimum sample size per group), stratified sampling can potentially

require a larger sample than would other methods (although in most cases, the

required sample size would be no larger than would be required for simple

random sampling.

Sampling methods in

research(12)

A stratified sampling approach is most effective when

three conditions are met

Variability within strata are minimized

Variability between strata are maximized

The variables upon which the population is stratified are

strongly correlated with the desired dependent variable.

Sampling methods in

research(13)

Advantages over other sampling methods

Focuses on important subpopulations and ignores irrelevant ones.

Allows use of different sampling techniques for different

subpopulations.

Improves the accuracy/efficiency of estimation.

Permits greater balancing of statistical power of tests of differences

between strata by sampling equal numbers from strata varying

widely in size.

Sampling methods in

research(14)

Disadvantages

Requires selection of relevant stratification variables

which can be difficult.

Is not useful when there are no homogeneous

subgroups.

Can be expensive to implement

Reliability and validity

Reliability and validity (1)

We often think of reliability and validity as separate ideas but, in fact, they're related to each other. Here, I want to show you two ways you can think about their relationship.

One of the metaphors for the relationship between reliability is that of the target (see previous slide). Think of the centre of the target as the concept that you are trying to measure. Imagine that for each person you are measuring, you are taking a shot at the target. If you measure the concept perfectly for a person, you are hitting the centre of the target. If you don't, you are missing the centre. The more you are off for that person, the further you are from the centre.


Another way we can think about the relationship between reliability and validity is shown in the figure next slide. Here, we set up a 2x2 table. The columns of the table indicate whether you are trying to measure the same or different concepts. The rows show whether you are using the same or different methods of measurement. Imagine that we have two concepts we would like to measure, student verbal and math ability. Furthermore, imagine that we can measure each of these in two ways. First, we can use a written, paper-and-pencil exam (very much like the SAT or GRE exams). Second, we can ask the student's classroom teacher to give us a rating of the student's ability based on their own classroom observation.


The first cell on the upper left shows the comparison of the verbal written test score with the verbal written test score. But how can we compare the same measure with itself? We could do this by estimating the reliability of the written test through a test-retest correlation, parallel forms, or an internal consistency measure. What we are estimating in this cell is the reliability of the measure.

The cell on the lower left shows a comparison of the verbal written measure with the verbal teacher observation rating. Because we are trying to measure the same concept, we are looking at convergent validity.


The cell on the upper right shows the comparison of the verbal written exam with the math written exam. Here, we are comparing two different concepts (verbal versus math) and so we would expect the relationship to be lower than a comparison of the same concept with itself (e.g., verbal versus verbal or math versus math). Thus, we are trying to discriminate between two concepts and we would consider this discriminant validity.

Finally, we have the cell on the lower right. Here, we are comparing the verbal written exam with the math teacher observation rating. Like the cell on the upper right, we are also trying to compare two different concepts (verbal versus math) and so this is a discriminant validity estimate. But here, we are also trying to compare two different methods of measurement (written exam versus teacher observation rating). So, we'll call this very discriminant to indicate that we would expect the relationship in this cell to be even lower than in the one above it.


The four cells incorporate the different values that we examine in the

multitrait-multimethod approach to estimating construct validity.

When we look at reliability and validity in this way, we see that, rather than

being distinct, they actually form a continuum. On one end is the situation

where the concepts and methods of measurement are the same (reliability)

and on the other is the situation where concepts and methods of

measurement are different (very discriminant validity).

Hypothesis Testing

Documents

Transcript of Hypothesis Testing