1.4 Should you Believe a Statistical Studyhomepage.stat.uiowa.edu/~rdecook/stat1010/notes/...Eight...

1

1.4 Should you Believe a Statistical Study

LEARNING GOAL

Be able to evaluate statistical studies that you hear about in the media, so that you can decide whether the results are meaningful.

2


EXAMPLE 1 Horoscope Accuracy? Consider a possible newspaper report: “Researchers gave 100 participants their individual astrological horoscopes and asked whether the horoscopes were accurate. 85% of the participants said their horoscopes were accurate. The researchers concluded that horoscopes are valid most of the time.”

3


Some comments/concerns on the Study: !  This is an observational study.

!  How did they define ‘accuracy’ on a horoscope? (Hard to measure, subjective)

!  They have no data for comparison on ‘accuracy’ when people are given a fake horoscope (this would have been very useful).

4


Our Horoscope Experiment: !  Two treatment groups (real horoscope

n1=39, and fake horoscope n2=37). !  Students randomly assigned using birth

date. !  Experimental accuracy results:

Real horoscope group accuracy = 15/39 = 0.38 Fake horoscope group accuracy = 19/37 = 0.51

5

Eight Guidelines for Critically Evaluating a Statistical Study

1.  Get a Big Picture View of the Study 2.  Consider the Source 3.  Look for Bias in the Sample 4.  Look for Problems Defining or Measuring

Variables of Interest 5.  Beware of Confounding Variables 6.  Consider the Setting and Wording in Surveys 7.  Check that Results are Presented Fairly 8.  Stand Back and Consider the Conclusions

6


1.  Get a Big Picture View of the Study !  What was the goal of the study? !  Why were the researchers collecting data? !  What were the variables of interest. !  What was the population under study from which

the sample was taken? !  Was the study an observational study or an

experiment?

7


2. Consider the Source !  In an ideal world, studies should be objective, but

researchers may begin from a specific viewpoint. !  Look for a reference or company name.

!  Was the study on ‘best fertilizer’ provided by Pioneer (for example) or consumer reports?

!  Was it provided by the government or private company? !  Look for potential biases from the researchers that

might invalidate the study’s conclusions.

8

2.  Consider the Source

The company Colgate claims that “More than 80% Of Dentists recommend Colgate” or that its brand was “used and recommended by most dentists.”

!  Who funded the research? !  Who was the ‘representative’ sample?

!  Randomly chosen dentists? !  Dentists who volunteered? !  Dentists who were paid to be in the study?

9


3. Look for Bias in the Sample

Population

Sample

A representative sample is a sample in which the relevant characteristics of the sample members are generally the same as the characteristics of the population.

10


3. Look for Bias in the Sample !  Do the researchers state how the data were

collected? What sampling method was used? !  Some sampling methods:

!  Simple Random Sample (SRS) !  Stratified sampling !  Cluster sampling !  Convenience sampling (proceed with caution) !  Systematic sampling

11


3. Look for Bias in the Sample !  A statistical study cannot be valid unless the

sample is representative of the population under study.

!  A biased sample differs in a specific way from the population.

!  e.g. Sampling just the front row of a class for GPA as a representation of the whole class

!  e.g. Using responses from only website visitors to represent all possible visitors to a hotel.

12

3.  Look for Bias in the Sample Example: Self-selected Poll

The television show Nightline conducted a poll in which viewers were asked whether the United Nations headquarters should be kept in the United States. Viewers could respond to the poll by paying 50 cents to call a “900” phone number with their opinions. The poll drew 186,000 responses, of which 67% favored moving the United Nations OUT of the United States. Around the same time, a poll using simple random sampling of 500 people found that 72% wanted the United Nations to STAY in the United States. Which poll is more likely to be representative of the general opinions of Americans?

13


3. Look for Bias in the Sample

Definitions Selection bias (or a selection effect) occurs whenever researchers select their sample in a biased way.

Participation bias occurs any time participation in a study is voluntary.

A self-selected survey (or voluntary response survey) is one in which people decide for themselves whether to be included in the survey.

14


4. Look for Problems Defining or Measuring Variables of Interest

!  Did the researchers clearly define the variables? !  Could the definition be perceived in multiple ways

(vague)? !  e.g. What do we consider ‘resting heart rate’?

!  Is it really hard to measure? !  e.g. Number of illegal immigrants living in the country

undocumented (how can we know this?). !  e.g. Can we define an ‘accurate’ horoscope?

15


5. Beware of Confounding Variables !  In an observational study, researchers may

present an observed association, but we need to think about other possible lurking variables that may actually be the REAL reason behind what we observe.

!  e.g. Fertilizer associated with corn yield (what about the soil!).

B A A A

A A

A

B B

B B

B B

C C

C C C C

C

16


5. Beware of Confounding Variables !  e.g. Researchers found a positive relationship

between children’s shoe size and IQ!!

shoe size

IQ

This relationship may exist, but it’s not the SHOE SIZE that’s causing this relationship… What other variable is VERY relevant?

High fat High sugar High fat Wealthy country (income,expensive product)

17


5. Beware of Confounding Variables !  e.g. Association: Fat Intake and Death rate?

This relationship may exist, but what other variables may be relavant? As fat intake increases, what else might be increasing in these countries?

18


6. Consider the Setting and Wording in Surveys !  Is is a paper survey? Or will you be answering to

a particular interviewer? !  Are questions negatively worded? !  Dishonest responses are particularly likely when

the survey concerns sensitive subjects !  Is it a health question that you feel is very private? !  Is it a politically charged question? !  Do you feel that how you answer will greatly impact

your ‘character’ in the eyes of the interviewer?

19


7. Check that Results are Presented Fairly !  Even when a statistical survey is done well, it

may be misrepresented in graphics or concluding statements.

!  We return to the Colgate claim:” “More than 80% Of Dentists recommend Colgate” or that its brand was “used and recommended by most dentists.”

20

7.  Check that Results are Presented Fairly Example 1: Colgate “More than 80% Of Dentists recommend Colgate” or that its brand was “used and recommended by most dentists.”

In 2007, the Advertising Standards Authority (ASA) in Britain ruled that the oral health manufacturing giant Colgate could not use its claim. These bans were based on the finding that Colgate had used deceptive statistics to derive its numbers.

http://blogs.plos.org/scied/2016/08/18/why-statistics-should-be-mandatory/

21

7.  Check that Results are Presented Fairly Example 1: Colgate When reading the original claim, consumers would likely think that four out of five dentists had recommended Colgate over its competitors. Instead, ASA revealed that dentists in the study were allowed to recommend more than one brand. The numbers were less impressive than Colgate had made them sound.

22

7.  Check that Results are Presented Fairly Example 1: Colgate The ASA explained that “The claim would be understood by readers to mean that 80 per cent of dentists recommend Colgate over and above other brands, and the remaining 20 per cent would recommend different brands. Because another competitor’s brand was recommended almost as much as the Colgate brand by the dentists surveyed, it was concluded that the claim was misleadingly.

23

7.  Check that Results are Presented Fairly Example 2: Being “below grade level” The school board in Boulder, Colorado, created a hubbub when it announced that 28% of Boulder school children were reading “below grade level” and hence concluded that methods of teaching reading needed to be changed. The announcement was based on reading tests on which 28% of Boulder school children scored below the national average for their grade. Do these data support the board’s conclusion?

24

7.  Check that Results are Presented Fairly Example 2: Being “below grade level” We know there is variability in scores, and it’s impossible for everyone to be above the average!! But we have such a negative perception of this language it can cloud our judgment. The fact that 28% of Boulder children scored below the national average for their grade implies that 72% scored at or above the national average!

25

7.  Check that Results are Presented Fairly Example 2: Being “below grade level” And do we really consider “at grade level” being exactly at the national average? Would everyone below the average need remedial work? Would everyone above the average be ‘gifted’? It seems more relevant to make decisions based on the range of reading abilities (i.e. a distribution of reading abilities) rather than a single value like the national average, in this case.

26

7.  Check that Results are Presented Fairly Example 3: Quaker Oats Sometimes researchers can give a misleading perception simply by showing only part of the axes on a graphic. Quaker Oats

claimed that eating oatmeal was associated with lowered cholesterol and showed this graphic from oatmeal consumption.

27

7.  Check that Results are Presented Fairly Example 3: Quaker Oats But here is the full graphic showing the vertical axis starting at “zero”.

Much less convincing.

http://ksrowell.com/blog-visualizing-data/2012/01/03/the-oatmeal-graph/

28


8 . Stand Back and Consider the Conclusions !  Ask yourself questions such as these:

!  Do the conclusions make sense? !  Can you rule out alternative explanations for

the results (lurking variables, confounding)? !  If the conclusions make sense, do they have

any practical significance? !  If a study shows a new self-cleaning shower

product works 2 minutes faster than it’s competitor, does that really matter?

1.4 Should you Believe a Statistical Studyhomepage.stat.uiowa.edu/~rdecook/stat1010/notes/...Eight...

Documents

Transcript of 1.4 Should you Believe a Statistical Studyhomepage.stat.uiowa.edu/~rdecook/stat1010/notes/...Eight...