[Hide Lesson Outcomes]

Lesson 2: The Statistical Process & Design of Studies

From BYUI Statistics Text

Contents

1 Lesson Outcomes2 Introduction3 The Statistical Process and Daniel's Experiment

3.1 Step 1: Design the Study3.2 Step 2: Collect Data3.3 Step 3: Describe the Data3.4 Step 4: Make Inferences3.5 Step 5: Take Action3.6 Summary of the Statistical Process

4 Design of Studies4.1 Designed Experiments4.2 Observational Studies4.3 Sampling Methods4.4 Types of Variables

5 Making Inferences: Hypothesis Testing6 Summary7 References8 Additional Reading9 Navigation

1 Lesson Outcomes

By the end of this lesson, you should be able to:

State the five steps in the statistical processDiscuss the importance of understanding the background for a statistical analysisIdentify the difference between a categorical and a quantitative variableClassify quantitative random variables as either discrete or continuousDistinguish between an observational study and an experimentDistinguish between a population and a sampleDiscuss the characteristics of the following sampling schemes: simple random sampling (SRS), systematic sampling,cluster sampling, stratified sampling, and convenience samplingDescribe the role of sampling in an auditIdentify specific instances in which each type of sampling scheme should be usedExplain the importance of using a random sampleDiscuss important considerations in designing a sample survey including: bias, undercoverage, nonresponse, voluntaryresponse samples, convenience samples, and poorly worded questions

These optional videos discuss the contents of this lesson.

Identify the following when presented with the description of an experiment: subjects (or experimental units), factors,and treatment groupsCorrectly utilize the basic vocabulary associated with data collectionDiscuss the role of Statistics in society

Design a simple observational study or experiment

2 Introduction

Statistics are used in every aspect of society. Every statistical analysis follows a pattern we will call the Statistical Process. Thisprocess will be introduced in this lesson and will be used throughout the course.

3 The Statistical Process and Daniel's Experiment

Stained-glass depiction of Daniel's deliverance from the lions' den

in the old Dominican priory church at Hawkesyard in Staffordshire, England

(Photo credit: Fr Lawrence Lew, O.P. Used by permission.)

The Old Testament prophet Daniel planned one of the earliest recorded scientific research studies. We will use his example toillustrate the following five steps of The Statistical Process.

Step 1: Design the study

Step 2: Collect data

Step 3: Describe the data

Step 4: Make inferences

Step 5: Take action

The following icons can help you remember these steps. Notice that each icon has a letter and an image to help you remember thefive steps of the Statistical Process.

3.1 Step 1: Design the Study

An important step in scientific inquiry or problem solving can be to state a research question such as:

Will internet advertising increase a company's revenue?Does expressing gratitude increase a person’s satisfaction with life in general?Does a newly developed vaccine prevent the spread of disease?

Researchers also investigate the background of the situation. What have other people discovered about this situation? How can wefind the answer to the research question? What do we need to do? What is the population (or total collection of all individuals)under consideration? What kind of data need to be collected?

Before collecting data, researchers make a hypothesis, or an educated guess about the outcome of their research. A hypothesis is astatement such as the following:

Using internet advertising will increase the company’s sales revenue.People who express gratitude will be more satisfied with life than those who do not.A newly-developed vaccine is effective at preventing tuberculosis.

Daniel’s Experiment

Watch the first 6 minutes and 20 seconds of the following video:

After taking Israel captive, Babylon’s King Nebuchadnezzar wanted Israelites to serve in his palace. He asked his chief officer tobring Israelite children who were “well favoured, and skilful in all wisdom, and cunning in knowledge, and understanding science…to stand in the king’s palaces” (Daniel 1:4). To aid their preparation, Nebuchadnezzar planned to feed them his meat and wine forthree years (Daniel 1:5).

Daniel did not want to defile himself by partaking of the king’s meat and wine. He asked permission to eat pulse[1](http://www.lds.org/scriptures/bd/pulse?lang=eng&query=pulse) and drink water instead. His supervisor, Melzar, was afraid todisplease the king. He thought that after eating pulse and water, the selected Israelites would look worse than their peers, and hewould be punished (Daniel 1:8-10.)

With an understanding of the background of the situation, Daniel proposed an experiment. He said, “Prove thy servants, I beseechthee, ten days; and let them give us pulse to eat, and water to drink. Then let our countenances be looked upon before thee, and thecountenance of the children that eat of the portion of the king’s meat: and as thou seest, deal with thy servants” (Daniel 1:12-13.) In

[Show Solution]

short, Daniel’s implied research question can be stated as: Will those who eat pulse and drink water appear healthier than thosewho eat the king’s meat and drink his wine? Melzar agreed to the experiment.

Answer the following question:

1. What is Daniel’s hypothesis?

3.2 Step 2: Collect Data

When designing a study, much attention is given to the process by which data are observed. When examining data, it is alsoimportant to understand the data collection procedures. A sample is a subset (a portion) of a population. How is this sampleobtained? How are the observations made?

Daniel’s study design required that data be collected at the end of 10 days. Melzar would compare the appearances of two groupsof people: (1) Israelites who ate pulse and drank water versus (2) Israelites who ate the king’s meat and drank his wine.

3.3 Step 3: Describe the Data

When we describe data, we use any tools appropriate to the situation. This can include creating graphs or calculating statistics tohelp understand or visualize the data.

For Daniel’s experiment, the data are described in Daniel 1:15: “And at the end of ten days [the] countenances [of those who atepulse] appeared fairer and fatter in flesh than all the children which did eat the portion of the king’s meat.”

3.4 Step 4: Make Inferences

Inference is the process of using the information contained in a sample from a population to make a general statement (i.e. to infersomething) about the entire population. Later in the course we will learn techniques that make this type of analysis possible.

Melzar made an inference. Based on the results of the sample, he determined that (in general) those who eat pulse and drink waterwill be healthier than those who eat the king’s meat and drink his wine (Daniel 1:15-16.)

3.5 Step 5: Take Action

The goal of a statistical analysis is to determine which action to take in a particular situation. Actions can include many things:launching an internet ad campaign (or not), expressing gratitude (or not), getting vaccinated (or not), etc.

[Show Solution]

Melzar took action as described in Daniel 1:16: “Thus Melzar took away the portion of their meat, and the wine that they shoulddrink; and gave [all the Israelite children] pulse.”

Was the experiment a success? “Now at the end of the days that the king had said he should bring them in…the king communedwith them; and among them all was found none like Daniel, Hananiah, Mishael, and Azariah… And in all matters of wisdom andunderstanding, that the king enquired of them, he found them ten times better than all the magicians and astrologers that were in allhis realm” (Daniel 1:18-20.)

3.6 Summary of the Statistical Process

Summary

Daniel’s experience can also help you learn the Statistical Process. Look at the first letter of each of the steps in the StatisticalProcess. You can use the phrase "Daniel Can Discern More Truth" to help you to help you remember the five steps in the StatisticalProcess.

The Statistical Process

Step 01: Daniel Design the study

Step 02: Can Collect data

Step 03: Discern Describe the data

Step 04: More Make inferences

Step 05: Truth Take action

The Statistical Process will be used throughout the course. Take time to memorize the five steps.

The study designed by the Old Testament prophet Daniel provides an ancient example of a designed experiment. Daniel’sexperiment included two groups of people: those who had the experimental treatment—eating pulse and drinking water (called thetreatment group) and those who ate the standard food—the king’s meat (called the control group.) The treatment group receivesthe experimental procedure. The control group is used for comparison.

Answer the following question:

2. Why was it important that Daniel’s experiment included a control group?

"Piled Higher and Deeper" by Jorge Cham

4 Design of Studies

Most research projects can be classified into one of two basic categories: observational studies or designed experiments. In anexperiment, researchers control (to some extent) the conditions under which measurements are made. In an observational study,researchers simply observe what happens, without controlling the conditions under which measurements are made. Both types ofstudy follow the five steps of the Statistical Process.

4.1 Designed Experiments

In a designed experiment, researchers manipulate the conditions that the participants experience. They often do this by randomlyassigning subjects to one of two groups, a "treatment" group and a "control" group. The experiment is conducted by applying somekind of treatment to the subjects in the treatment group, and observing the effect of the treatment. Those in the control group do notreceive the treatment and are also observed. In this way researchers can determine the effects of the treatment. The followingexample illustrates the use of these two groups.

Jonas Salk’s First Polio Vaccine Trial

The Salk Polio Trials comprised one of the most famous designed experiments in history. Please view the following video (6 min 31sec), but before you do, please review the questions that follow it. Once you have completed the video, please answer thequestions. (Note: You do not need to submit answers to the questions, but do consider them carefully. They are designed to helpyou think about various aspects of the Statistical Process.)

Answer the questions about the video in each of the following steps of the Statistical Process.

Step 1: Design the study.

1. What year did the Salk polio vaccine trials occur?

2. At that time, parents were afraid to let their children…

3. How many children participated in the trial?

4. The children were separated into two groups. What did the video say would practically guarantee that the two groupswould be nearly identical?

5. How was Salk’s vaccine administered?

Step 2: Collect Data

6. What was the primary outcome for the researchers in the Salk trial?

7. How did the researchers collect the data?

Step 3: Describe the Data

8. Complete the quote from the video: “As the Salk trial progressed, it did appear as if the group that got the vaccine had

________ instances of polio.”

Step 4: Make Inferences

9. Complete the quote from the video: “As the researchers tabulated the data, they found that the differences were...

Step 5: Take Action.

10. Which of the following actions is most likely at the conclusion of the Salk trial?

Elizabeth Toy, a former BYU-Idaho teacher, was one of the children in the Salk trials.

Jonas Salk’s vaccine trial was a great example of a designed experiment. However, it did not start out that way. Initially,there were some serious flaws in their design. Other researchers sharply criticized these errors. Almost 1.1 million children

participated in the initial study. Even though the sample size was large, the flawed study design made the data useless!

'Jonas Salk's Second Study'

After fixing the design, Jonas Salk enrolled hundreds of thousands of additional children for the second phase of his study. Considerhow Jonas Salk applied the Statistical Process in this second study.

Step 1: Design the study

Jonas Salk enrolled hundreds of thousands of children in his second study. The participants in a study are commonly called

[Show Solution]

[Show Solution]

subjects. Sometimes subjects are called experimental units or simply units. In the Salk trials, the children who participated

are the subjects.

The subjects were randomly assigned to one of two groups. The first group was given the experimental vaccine, thetreatment. The treatment is the new or experimental condition that is imposed on the subjects. The subjects who receive the

treatment make up the treatment group.

The second group was given a placebo. ''Placebo'' is another name for a control. In this study, the placebo was an injection

that looked just like the vaccine, but contained a harmless saline solution. The placebo group or control group is made upof the subjects assigned to receive the placebo.

This study was double blind. Neither the children's parents nor their doctors knew whether a particular child received thetreatment or the control. Both parties were ``blinded to this information.

Answer the following questions:

3. Some children can be identified as having a higher risk of developing polio. Would it have been better if they were assigned to thetreatment group so they could get the vaccine?

4. Why is it important for the subject and those who assess the health of the subject to be unaware of whether or not that childreceived the vaccine?

The null and alternative hypotheses for this study are:

: The proportion of children who develop polio will be the same for the treatment and control groups.

: The proportion of children in the treatment group who develop polio will be lower than the proportion ofchildren in the control group who develop polio.

Step 2: Collect data.

The researchers followed up with each child to determine if they contracted polio. They recorded the number of children in

each group that developed polio during the study period. Not all of Salk's experiments were double-blind. Here is a summary

of the results from the regions where a double-blind study was conducted (Francis et al., 1955; Brownlee, 1955 ):

Children Who Developed Polio

Yes No Total

Treatment Group 57 200,688 200,745

Placebo Group 142 201,087 201,229

Step 3: Describe the data.

Ho

Ha

[Show Solution]

[Show Solution]

One way to summarize the data is to compute the proportion of children in each group that developed polio. The proportionof children in the treatment group that developed polio during the study period is:

Answer the following questions:

5. Calculate the proportion of children in the placebo group that developed polio during the study period.

6. Compare the two proportions. What do you observe?

Step 4: Make inferences

When the hypothesis test to compare the proportions was conducted, the -value was 0.00000000093. Assuming the nullhypothesis was true (i.e. the vaccine had no effect), the probability is extremely small that Salk would observe a difference in

these proportions that was as extreme as or more extreme than the results he obtained. Because the -value is so small, it ishighly unlikely that these results are due to chance.

Step 5: Take action

Once it was clear that the vaccine was effective, children who were unvaccinated or had received the placebo were givenSalk’s vaccine. Since 1954, there has been a marked decrease in the number of polio cases worldwide (Offit, 2005). Public

health researchers continue to work to eradicate this disease around the world.

4.2 Observational Studies

In an observational study researchers observe the responses of the individuals, without controlling the conditions experienced bythe individuals. Therefore, they do not assign the participants to treatment or control groups.

Observational studies commonly occur in business settings. One example is a financial audit. The purpose of a financial audit is toassess the accuracy of a company’s financial business practices. ImmunAvance Ltd., a non-government health care organization,hired the Accounting Office at Global Optimization Unlimited to perform an independent audit of their financial practices.ImmunAvance provides inoculation and other preventative health care services in rural African communities.

Imagine you are assigned to serve as a member of the team that will perform this audit. The audit illustrates the five steps of theResearch Process.

= 0.000 283 957200745

P

P

Step 1: Design the study

The volume of financial transactions conducted by ImmunAvance makes it impossible to conduct a census or an examination of theentire collection of ImmunAvance’s financial documents. Instead, you will collect a manageable group of items (called the sample)from the entire collection of financial documents (called the population.) A sample is a subset or a portion of a population. Theinformation gained from the sample is used to make an inference (or generalization) about the population.

Auditors typically cannot consider every item in a population, because there are too many. When it is not possible to conduct acensus, auditors face sampling risk. Sampling risk is the risk affiliated with not auditing every item in the population. It is the riskthat the sample may not adequately reflect the population. The only way to eliminate sampling risk is to conduct a census, which isusually not practical. Auditors can reduce sampling risk by obtaining a sample randomly. This is called random selection. Anotherway to reduce sampling risk is to increase the sample size, the number of items sampled.

4.3 Sampling Methods

Step 2: Collect data

There are several procedures that can be used to select a random sample from a population, including: simple randomsampling (SRS), systematic sampling, cluster sampling, stratified sampling, and convenience sampling (or, haphazard

sampling). These are examples of sampling methods.

A simple random sampling (SRS) is the best method for obtaining a random sample. If there is a list of all items in the

population and they are all accessible, a SRS can be collected. For example, suppose there are 2,000 accounts receivableitems in the population. Auditors can use a random number generator to choose values between 1 and 2,000 to identify

which items are to be audited. Software can be used to create a list of random numbers corresponding to items for the audit.

In Excel, the command to obtain a random number between 1 and 2,000 is =RANDBETWEEN(1,2000). Sometimes it isnecessary for auditors to renumber the items (1 to 2000) to help create random sample. A simple random sample can be

obtained any time there is a complete list of the items to be sampled and they are all assessible. All the statistical proceduresin this course assume that simple random sampling has been used.

A systematic sample is where auditors select every item in the population, beginning at a random starting point. As anexample, suppose there are 2,000 accounts receivable items to be audited, and the auditors want to get a sample of 40 items

from this population. The auditors would sample every 2000÷40=50th item. The auditors will use a random starting point in

the first 50 items. So, suppose a random number generator gave 41 as a random number between 1 and 50. The auditorsselect the 41st item. The next item chosen will be item number 41+50=91. After that, the auditors will choose item number

91+50=141, and so on. Systematic sampling works well when the items are in a random, sequential ordering. If the items are

not arranged randomly, a systematic sample can miss important parts of the population.

A cluster sample (sometimes called a block sample) consists of taking all items in one or more randomly selected clusters,

or blocks. For example, all the accounts receivable items in a few randomly selected months (the blocks) could be sampled.When the variation from one block to another is relatively low, compared to the variation within the block, cluster sampling is

a reasonable way to get a sample. However, in many cases, the number of blocks that must be sampled must be large inorder to draw a reasonable audit conclusion. For this reason, a cluster sample is usually not recommended for audits.

A stratified sample is when the items to be sampled are organized in groups of homogeneous (similar) items called strata,

kth

[Show Solution]

[Show Solution]

[Show Solution]

then a simple random sample is drawn from each of these strata. Stratified sampling works well when the items are similar

within each stratum and tend to differ from one stratum to another. In an audit of accounts receivable items, auditors could

define the strata as: (1) accounts under $1,000, (2) accounts between $1,000 and $4,999, (3) accounts between $5,000and $9,999, and (4) accounts of $10,000 or more. A certain number of items could be randomly selected from each of these

strata. Auditors may choose to include in their sample all items from some strata, such as very large accounts ($10,000 ormore), when there are few items in those strata.

A convenience sample involves selecting items that are relatively easy to obtain and does not use random selection tochoose the sample. Even though the sample is chosen by convenience, it should be done without introducing any systematic

bias if possible. As an example of a convenience sample, an auditor could haphazardly select items from a filing cabinet. This

is frequently done when a quick and simple sample is needed, but may not yield a sample that represents the population well.

When possible, convenience samples should be avoided.

4.4 Types of Variables

Auditors meticulously record information on every item audited, thus generating data. There are two basic types of data:quantitative and categorical.

Quantitative variables provide numerical information. They represent things that are numeric in nature; things that are

measured. For a quantitative variable, it makes sense to apply arithmetic operations to the data (such as adding one to a

value, doubling a value, or comparing two values). The net worth of a company or the number of people it employs are bothexamples of quantitative variables: If one company is worth $1,000,000 and a second company is worth $2,000,000 then

the second company really is worth twice the value of the first.

Categorical variables yield data that can only be considered as categories. Categorical variables are labels, and it does not

make sense to do arithmetic with them. The following are examples of categorical variables: gender, nationality, job title,

telephone number, etc. Even though a telephone number is made of digits, it does not make sense to double someone’sphone number, to add one to their number, or to say one phone number is smaller than another. In Unit 3 we will learn more

about Categorical variables and proportions.

The sample accounts receivable record has information on both types of variables.

Answer the following question:

7. For each of the following variables taken from this accounts receivable record, indicate whether the variable is quantitative orcategorical.

a. Terms

b. Account number

c. Invoice amount

Step 3: Describe the data

After auditors collect a sample and compile the data, they review the evidence. Auditors may use graphs or compute

numbers (such as the mean) to summarize the evidence they found.

5 Making Inferences: Hypothesis Testing

Step 4: Make inferences

Auditors use the information drawn from the sample to form an opinion about the population. Whenever sample data is used

to infer a characteristic of a population, it is called making an inference. Inferential statistics represents a collection ofmethods that can be used to make inferences about a population. Based on the documents reviewed, the auditors assess if

the company is conducting its business in a proper manner.

When conducting an audit, the implicit assumption is that transactions have been posted properly. This foundational

assumption is called the null hypothesis. The null hypothesis is a statement about the population that represents the status

quo, convential wisdom, or what is generally accepted as true. In the audit we're considering, the null hypothesis is

: ImmunAvance’s financial statements are materially correct.

Auditors sample the company’s records to collect evidence against the null hypothesis. If there is sufficient evidence, we

reject the null hypothesis. If the null hypothesis is rejected, it is rejected in favor of another statement about the population: the

alternative hypothesis. In the case of an audit, the alternative hypothesis is

: ImmunAvance’s financial statements are not materially correct.

Notice that both the null and alternative hypotheses are statements about the state of all of ImmunAvance’s financial

documents, not just the documents included in the sample.

"Piled Higher and Deeper" by Jorge Cham

There is a formal procedure for testing the null and alternative hypotheses, called a hypothesis test. In a hypothesis test, the

null hypothesis is always assumed to be true. If there is sufficient evidence against the null hypothesis, it is rejected. The

evidence against the null hypothesis is assessed using a number called the -value. The -value is the probability that a

result will be observed that is as extreme or more extreme than the information observed, assuming the null hypothesis is true.We reject the null hypothesis if the -value is small, say less than 0.05. If we assume the company’s financial statements are

materially correct, the -value is the probability of observing errors with a total value that is as large or larger than that which

HO

Ha

P P

PP

was observed in the audit.

For ImmunAvance’s audit, the -value was determined to be 0.79. Assuming the null hypothesis is true, the probability ofobserving inconsistencies that are at least as great as those observed in the audit was 0.79. This is a very large value. So, it is

not surprising to have observed the total value of the errors in this case. The probability that these differences could occur

due to chance is very high. The conclusion is that the null hypothesis should not be rejected.

If the -value is low, the null hypothesis is rejected. If this probability is large, the null hypothesis is not rejected.

Hypothesis tests sometimes lead accidentally to incorrect conclusions because we use data from samples (as opposed todata from entire populations). When random samples are selected, some of them sample will contain disproportionately few

or many documents with errors, just by chance.

Think about drawing marbles from a container in which most of the marbles are white and a few are red. Each marble

represents a document, and the red marbles represent documents with errors.

If you choose a random sample of the marbles in the jar, you might get all the red marbles in your sample, just by chance.

This might lead you to conclude that there are many red marbles in the container, which is false. This is like an auditor

rejecting the null hypothesis when it is true, because their sample contains more documents with errors than it should---just

due to chance.

Likewise, when drawing marbles from your container, you might select none of the red marbles. This may lead you to

conclude that there are no red marbles in the container, or very few, which is false. This is like an auditor failing to reject the

null hypothesis when it is false, because their sample contains fewer documents with errors than it should---again, just due to

chance.

Such errors are no one's fault; they are an inherent part of hypothesis testing. They make it impossible for us to be certain ofthe conclusions we draw using the statistical process. The thing to remember is that if we carry out the process correctly, our

results are correct often enough to be very useful.

Step 5: Take Action

The auditors prepare a report in which they give their opinion on the status of the company’s current operations.

There is not enough evidence to suggest that ImmunAvance’s financial statements are not materially correct. The auditor’s

conclusion is that no adjustment is necessary, since the observed discrepancies could be attributed to chance.

6 Summary

Remember...

The Statistical Process has five steps: Design the study, Collect the data, Describe the data,

Make inferences, Take action.

In a designed experiment, researchers control the conditions of the study. In an observational

study, researchers don't control the conditions but only observe what happens.

There are many sampling methods used to obtain a sample from a population. The most

important is a simple random sample (SRS) which is a random selection taken from a population.

P

P

Quantitative variables represent things that are numeric in nature, such as the value of a car or

the number of students in a classroom. Categorical variables represent nonnumerical data thatcan only be considered as labels, such as colors or brands of shoes.

The null hypothesis ( ) is the foundational assumption about a population and represents the

status quo. The alternative hypothesis ( ) is a different assumption about a population. Using

a hypothesis test, we determine whether it is more likely that the null hypothesis or the alternativehypothesis is true.

7 References

Bible Dictionary, “Pulse” at http://lds.org/scriptures/bd/pulse.

Brownlee, K. A. (1955). Statistics of the 1954 polio vaccine trials. Journal of the American Statistical Association, 50(272), pp.1005-1013.

Francis, T., et. al. (1955). An evaluation of the 1954 poliomyletis vaccine trials. American Journal of Public Health and the Nation'sHealth, 45(5)

Offit, P. A. (2005). Why are pharmaceutical companies gradually abandoning vaccines? Health Affairs, 24(3), 622-630.doi:10.1377/hlthaff.24.3.622

8 Additional Reading

[Sampling Chapter.pdf (http://statistics.byuimath.com/images/9/99/Sampling_Chapter.pdf) ]

9 Navigation

Previous

Reading:

Lesson 01:

CourseIntroduction

This Reading: Lesson 02:

The Statistical Process & Design of

Studies

Next Reading: Lesson 03:

Describing Quantitative Data: Shape, Center &

Spread

Retrieved from "http://statistics.byuimath.com/index.php?

title=Lesson_2:_The_Statistical_Process_%26_Design_of_Studies&oldid=2786"

This page was last modified on 24 April 2013, at 22:51.

This page has been accessed 1,693 times.

Content is available under Creative Commons Attribution Share Alike.

Ho

Ha