Understanding Polling with Claudia Deane
-
Upload
national-press-foundation -
Category
Documents
-
view
236 -
download
0
Transcript of Understanding Polling with Claudia Deane
-
8/11/2019 Understanding Polling with Claudia Deane
1/40
Interpreting and Using Polls in
2014
Claudia Deane
Director, Research Practices
-
8/11/2019 Understanding Polling with Claudia Deane
2/40
-
8/11/2019 Understanding Polling with Claudia Deane
3/40
Value of polls in reporting
Most high profile use: Election horserace
Much more important at Pew Research Center:
Scientifically represent public attitudes, values,
experiences
Add scope, meaning to anecdotes
Track societys changing attitudes
Giving voice to those not always heard
-
8/11/2019 Understanding Polling with Claudia Deane
4/40
Journalists play a critical role in vetting,
interpreting survey research
Just like any source, polls require vetting
But no peer review for many polls
Especially now, varying methodologicalapproaches and quality
Journalists continue to be gatekeepers
-
8/11/2019 Understanding Polling with Claudia Deane
5/40
Basic (but challenging) journalistic question
Is pol l rel iable enough to repo rt?
This is getting harder to answer as survey research,
like journalism, is in flux
-
8/11/2019 Understanding Polling with Claudia Deane
6/40
-
8/11/2019 Understanding Polling with Claudia Deane
7/40
Are they still accurate?
NOTE: Candidate error is the difference between final resultSOURCE: National Council on Public Polls
12
6
5
10
9
3
2
5
2.5 2.72
6.3
4.3
3.2
2.2
4.1
2.2
0.9 0.9 1.6
0
2
4
6
8
10
12
14
1936 1940 1944 1948 1952 1956 1960 1964 1968 1972 1976 1980 1984 1988 1992 1996 2000 2004 2008 2012
Average Candidate ErrorPresidential Elections 1936-2012
-
8/11/2019 Understanding Polling with Claudia Deane
8/40
Questions to ask when writing about polls
AAPOR
1. Who paid for the poll and why
2. Who conducted the poll
3. How were interviews conducted
4. Number of interviews/margin of sampling error5. How were people chosen (probability sample?)
6. What population is poll is trying to represent
7. Timing of poll
8. Question wording/order9. Results based on full sample or subset
10. Were the data weighted, and if so, to what
-
8/11/2019 Understanding Polling with Claudia Deane
9/40
To evaluate the quality of a poll,
you need two things:
1. A copy of the topline/trenddocumentThe actual
questionnaire, including
responses
2. The method box or surveymethodology, with details on
how poll was conducted
Practically speaking
-
8/11/2019 Understanding Polling with Claudia Deane
10/40
1. Who paid for the poll and why2. Who conducted the poll
Questions to ask when writing about
polls
-
8/11/2019 Understanding Polling with Claudia Deane
11/40
Usually easy to find
Washington Post-ABC News poll
This poll was conducted for The Washington Post and ABCNews by telephone April 17 to 21, 2013 among a random
national sample of 1,000 adults, including users of bothconventional and cellular phones. The results from the fullsurvey have a margin of sampling error of plus or minus3.5 percentage points. Sampling, data collection andtabulation by SSRS of Media, Pa. Produced for the
Washington Post by Capital Insight.*= less than 0.5 percent
-
8/11/2019 Understanding Polling with Claudia Deane
12/40
SOURCE: http://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.html
http://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.html -
8/11/2019 Understanding Polling with Claudia Deane
13/40
Here, the polling company, Harris Interactive, and the sponsor, the U.S.
Chamber of Commerce, presented the data in a highly misleading way and
then made false claims about the type of poll that had been conducted.
The Chamber has been a fierce opponent of the health-care law, a.k.a.
Obamacare, and we frequently warn readers they should always be skeptical
of polls peddled by partisan organizations. Perhaps it should be no surprise
that this poll was released just as the GOP-led House of Representatives
scheduled a vote to repeal the law.
Given the way the data was presented, Republican lawmakers thought they
had been handed a gift and ended up with eggon their faces.
Kessler, The Washington Post
http://www.harrisinteractive.com/http://www.uschamber.com/http://www.uschamber.com/http://www.politifact.com/truth-o-meter/statements/2013/jul/29/marco-rubio/sen-marco-rubio-says-75-small-businesses-say-theyl/http://www.factcheck.org/2013/07/gop-mistweets-obamacare-survey-results/http://www.factcheck.org/2013/07/gop-mistweets-obamacare-survey-results/http://www.politifact.com/truth-o-meter/statements/2013/jul/29/marco-rubio/sen-marco-rubio-says-75-small-businesses-say-theyl/http://www.uschamber.com/http://www.uschamber.com/http://www.harrisinteractive.com/http://www.harrisinteractive.com/ -
8/11/2019 Understanding Polling with Claudia Deane
14/40
Possible Sponsors
Federal, state and local governments
Media organizations
Academic institutions
Non-profit groups or foundations
Special interest groups
Businesses and corporations
Political campaigns, consultants and candidates
Who paid for the poll and why?
Who conducted the poll?
-
8/11/2019 Understanding Polling with Claudia Deane
15/40
1. Who paid for the poll and why
2. Who conducted the poll
3. How were interviews conducted
4. Number of interviews/margin of sampling error
Questions to ask when writing about
polls
-
8/11/2019 Understanding Polling with Claudia Deane
16/40
Also usually easy to find
Washington Post-ABC News poll
This poll was conducted for The Washington Post and ABCNews by telephone April 17 to 21, 2013 among a random
national sample of 1,000 adults, including users of bothconventional and cellular phones. The results from the fullsurvey have a margin of sampling error of plus or minus 3.5percentage points. Sampling, data collection and tabulation bySSRS of Media, Pa. Produced for the Washington Post by
Capital Insight.*= less than 0.5 percent
-
8/11/2019 Understanding Polling with Claudia Deane
17/40
0%
2%
4%
6%
8%
10%
12%
14%
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
MarginofSamplingError(+/-)
Sample Size
N=50
MOSE= +/- 14 pts
N=1,000
MOSE= +/- 3 pts
N=2,500MOSE= +/- 2 ptsN=10,000
MOSE= +/- 1 pts
Relationship between sample size and
margin of sampling error
-
8/11/2019 Understanding Polling with Claudia Deane
18/40
-
8/11/2019 Understanding Polling with Claudia Deane
19/40
Modes of administration
Telephone
RDD = Random Digit Dial of landline and (often) cell
phones
Voter list-based phone surveysIn-person
Mail
The Internet?!
-
8/11/2019 Understanding Polling with Claudia Deane
20/40
The Internet is the edge
Challenges
Obvious coverage issues
Less obvious, but bigger issue: How to get a random
sample?Wide variety of methodologies being tried
Internet-based samples are often not
probability samples
Suggestion: Requires extra look to understand
and report on what youre dealing with
-
8/11/2019 Understanding Polling with Claudia Deane
21/40
1. Who paid for the poll and why2. Who conducted the poll
3. How were interviews conducted
4. Number of interviews/margin of sampling error
5. How were people chosen (probability sample?)
Questions to ask when writing about
polls
-
8/11/2019 Understanding Polling with Claudia Deane
22/40
Probability or non-probability
Probability sample Non-probability sampleCAN extrapolate from your sample to
a larger population (Most
Americans)
CANNOT extrapolate to a larger
population (yet)
CAN apply a margin of sampling
error
Margin of sampling error cannot be
computed
Ie, a randomly selected sample, a
random sample of the nationalpublic
Ie, Self-selected samples, click-
through polls, most internet panels
-
8/11/2019 Understanding Polling with Claudia Deane
23/40
Yes No
Easy: Online click-in polls
SOURCE: From www.nationalreview.com, accessed 1/10/12.
http://www.nationalreview.com/http://www.nationalreview.com/ -
8/11/2019 Understanding Polling with Claudia Deane
24/40
Trickier: Hybrid Internet panels
The sample itself is random
But the population from which thesample is drawn is made up of people
who have signed up to be members ofthe panel
Question: In what cases can this representthe full U.S. population?
Answer: Unresolved
-
8/11/2019 Understanding Polling with Claudia Deane
25/40
-
8/11/2019 Understanding Polling with Claudia Deane
26/40
9/17/2014 26
-
8/11/2019 Understanding Polling with Claudia Deane
27/40
September 17, 2014 www.pewresearch.org 27
Is Pew Research ever going to use the kind of online non-probability
panel that the Times and CBS are using?
Scott Keeter: Yes, we willbut the real question is what we will use it for. Our
current standards permit the use of non-probability samples for certain
purposes, such as conducting experiments or doing in-depth interviews. In
addition, we have embarked on a program of research to help us better
understand the conditions under which non-probability samples can provide
scientifically valid data. We also are exploring how to utilize non-survey data
sources, which by their very nature tend to come from samples that are not
random. But until we understand the pros and cons of those methods a lot
better, were going to be very cautious about incorporating them into our
research.
Q
-
8/11/2019 Understanding Polling with Claudia Deane
28/40
1. Who paid for the poll and why2. Who conducted the poll
3. How were interviews conducted
4. Number of interviews/margin of sampling error
5. How were people chosen (probability sample?)
6. What population is poll is trying to represent
Questions to ask when writing about
polls
-
8/11/2019 Understanding Polling with Claudia Deane
29/40
What population is poll trying to represent?
All voters?
All conservatives?
All residents of Pakistan?
A thinking question, mainly, requiring 90% common
sense and 10% methodological savvy
-
8/11/2019 Understanding Polling with Claudia Deane
30/40
So if you want to represent young people
Blue= cell
Green =
landline
SOURCE: Washington Post-ABC News analysis; data from 2009-2012
Q ti t k h iti b t
-
8/11/2019 Understanding Polling with Claudia Deane
31/40
1. Who paid for the poll and why2. Who conducted the poll
3. How were interviews conducted
4. Number of interviews/margin of sampling error
5. How were people chosen (probability sample?)
6. What population is poll is trying to represent
7. Timing of poll
8. Question wording/order
Questions to ask when writing about
polls
-
8/11/2019 Understanding Polling with Claudia Deane
32/40
-
8/11/2019 Understanding Polling with Claudia Deane
33/40
Wording DOES Matter: Govt Surveillance
9/17/2014 33
Q ti t k h iti b t
-
8/11/2019 Understanding Polling with Claudia Deane
34/40
1. Who paid for the poll and why2. Who conducted the poll
3. How were interviews conducted
4. Number of interviews/margin of sampling error
5. How were people chosen (probability sample?)
6. What population is poll is trying to represent
7. Timing of poll
8. Question wording/order
9. Results based on full sample or subset
Questions to ask when writing about
polls
-
8/11/2019 Understanding Polling with Claudia Deane
35/40
ASK IF HEARD A LOT OR A LITTLE AND INTERNET USER ((INT1=1 OR
INT2=1 OR INT3=1) AND (Q.HB1=1,2)) [N=897]:
Q.HB2 Do you think your own online personal information was put at
risk by the Heartbleed bug, or do you think your information was not
put at risk?
Apr 23-27
2014 All internet users
45 Own information put at risk 29
47 Own information not put at risk 30
8 Dont know/Refused (VOL.) 5
9/17/2014 35
Q ti t k h iti b t
-
8/11/2019 Understanding Polling with Claudia Deane
36/40
1. Who paid for the poll and why2. Who conducted the poll
3. How were interviews conducted
4. Number of interviews/margin of sampling error
5. How were people chosen (probability sample?)
6. What population is poll is trying to represent
7. Timing of poll
8. Question wording/order9. Results based on full sample or subset
10. Were the data weighted, and if so, to what
Questions to ask when writing about
polls
-
8/11/2019 Understanding Polling with Claudia Deane
37/40
Were the data weighted, and if so, to what?
CBS News/New York Times
The com bined resul ts have been weigh ted to adjust
for var iat ion in the samp le relat ing to geographic
region , sex, race, Hispanic or ig in, age, education and
number of adu l ts in the household . Respondents in the
land l ine samp le were also weighted to take account o f
the number of te lephone lines into the residence.
-
8/11/2019 Understanding Polling with Claudia Deane
38/40
Weighting of poll data
Its necessary in almost all surveys
(Nearly) Everyone does it
It corrects for the problem of not interviewing people in
the sample in correct proportion to their size in thepopulation
-
8/11/2019 Understanding Polling with Claudia Deane
39/40
Good practices when writing about public opinion
Report the topline story Then break it down: identify key subgroups, look for
interesting differences
Are the numbers changing over time?
Be cautious with causality Are the results confirmed by other recent polls? If
not, write through it!
Use poll archives to validate conflicting claims about
public opinion Use personal quotes/interviews to animate raw
numbers
-
8/11/2019 Understanding Polling with Claudia Deane
40/40
Thank you