Questionnaire Questionnaire ConstructionConstruction

January 28 & 31, 2011January 28 & 31, 2011

2

ObjectivesObjectives

By the end of this meeting, participants By the end of this meeting, participants should be able to:should be able to:

• Distinguish open and closed-ended Distinguish open and closed-ended questions.questions.

• Evaluate questions for their validity and Evaluate questions for their validity and reliability, as well as how well they follow reliability, as well as how well they follow basic rules of question wording.basic rules of question wording.

3

Florida Datamar Democratic Florida Datamar Democratic PollingPolling

Likely Democratic VotersLikely Democratic Voters

ClintonClinton 40%40%

ObamaObama 28%28%

EdwardsEdwards 19%19%

KucinichKucinich 2%2%

RichardsonRichardson 2%2%

4

The Vital Information- FLThe Vital Information- FLa) Sample size 541 +/- 4.2 percent sampling errorb) January 5 – 7, 2008c) Methodology: Findings are from a January 5 – 7,

2008 survey using a comprehensive predictive model of “likely” voters, based on election cycles and other factors of voters in Florida. The survey was conducted using an automated telephone dialer and the voice of a professional announcer.

d) The targeted-registration-based-sampling (TRBS)* selection criteria were based on election cycles and other voter factors in Florida. Datamar proprietary algorithms were used to generate random samples from the target group for calling.

e) Where necessary, responses were weighted according to age, gender, and political party.

5

The Vital Information- FLThe Vital Information- FL

Q18. If the primary election were held today, for whom would you vote? The candidates are: Joe Biden, U.S. Senator from Delaware; Chris Dodd, U.S. Senator from Connecticut; Bill Richardson, Governor of New Mexico; Mike Gravel, former U.S. Senator from Virginia; Dennis Kucinich, Congressman from Ohio; John Edwards, former U.S. Senator from North Carolina; Barack Obama, U.S. Senator from Illinois; Hillary Clinton, U.S. Senator from New York.

6

Research 2000 FL-DResearch 2000 FL-D

Democratic Primary VotersDemocratic Primary Voters

Clinton Clinton 50%50%

ObamaObama 28%28%

EdwardsEdwards 13%13%

OthersOthers 4%4%

UndecidedUndecided 5%5%

7

Vital Information- FloridaVital Information- Florida

Research 2000Research 2000

The Research 2000 Florida poll was conducted The Research 2000 Florida poll was conducted from January 14from January 14thth through January 16 through January 16thth, , 2008. There were a total of 500 likely 2008. There were a total of 500 likely Democratic and 500 Republican primary Democratic and 500 Republican primary voters as well as 500 statewide primary voters as well as 500 statewide primary voters interviewed by telephone. The voters interviewed by telephone. The margin of error is 4.5%. margin of error is 4.5%.

If the Democratic Primary for President were If the Democratic Primary for President were held today, which of the following held today, which of the following candidates would you vote for? (Rotated)candidates would you vote for? (Rotated)

8

Types of QuestionsTypes of Questions

a)a) Open-ended questionsOpen-ended questions• What do you like about the Democratic Party?What do you like about the Democratic Party?• What do you like about the Republican Party?What do you like about the Republican Party?

b)b) Closed-ended questionsClosed-ended questions• Multiple choice questionsMultiple choice questions• Rating scale/ Feeling thermometerRating scale/ Feeling thermometer

c)c) Disadvantages/ Advantages of Each Disadvantages/ Advantages of Each ApproachApproach

9

Rating ScalesRating Scales

Common political typesCommon political types

a)a) Rating scales (generally seven or six Rating scales (generally seven or six points from very positive to very points from very positive to very negative)negative)

b)b) Thermometers (generally 1 to 100)Thermometers (generally 1 to 100)

c)c) Semantic differential (very good-very Semantic differential (very good-very bad)bad)

d)d) Seven point scale (two concepts at each Seven point scale (two concepts at each end)end)

10

Good Standard PracticeGood Standard Practice

a)a) Use common questionsUse common questions

b)b) Use short questions with standard Use short questions with standard speechspeech

c)c) Think of questions as a stimulus that Think of questions as a stimulus that the respondent will react tothe respondent will react to

d)d) Use a battery of questions for more Use a battery of questions for more difficult conceptsdifficult concepts

e)e) Tailor question wording to audienceTailor question wording to audience

11

Things to ConsiderThings to Consider

a)a) Branching format (particularly for Branching format (particularly for scales)scales)

b)b) Response choice orderResponse choice order

c)c) Ambiguous wordingAmbiguous wording

d)d) Biased questionsBiased questions

e)e) Social desirability biasSocial desirability bias

f)f) Double barreled questionsDouble barreled questions

g)g) Double negativesDouble negatives

h)h) No opinion/ donNo opinion/ don’’t knowt know

12

Evaluating QuestionsEvaluating Questions

a)a) ReliabilityReliabilityb)b) ValidityValidity• Face validityFace validity• Convergent validity (similar answers)Convergent validity (similar answers)• Divergent validity (different answers)Divergent validity (different answers)• Criterion validity (direct measure)Criterion validity (direct measure)• Content validity (all parts of a concept)Content validity (all parts of a concept)• Construct validity (consistent with Construct validity (consistent with

previous work)previous work)

13

Constructing a QuestionnaireConstructing a Questionnaire

a)a) Topic OrderTopic Orderb)b) Question order/ Consistency biasQuestion order/ Consistency biasc)c) Response choices/ Response setResponse choices/ Response setd)d) Number of questionsNumber of questions• half sampleshalf samples• rotating questionsrotating questions

e)e) Questionnaire layoutQuestionnaire layoutf)f) Pretesting the questionnairePretesting the questionnaire• cognitive think-aloudcognitive think-aloud

14

Example: Sensitive QuestionsExample: Sensitive Questions

a) Suppose UGA wanted to survey student behavior on campus.

b) Specifically, the administration wants an accurate picture of what percent of students partake in unscrupulous activities.

c) How can the university ensure it gets accurate results?

15

Example: Sensitive QuestionsExample: Sensitive Questions

Innocuous Question TechniqueInnocuous Question Techniquea) Flip a coin. If you get heads, answer

question 1. If you get tails, answer question 2. Do not tell me what you get or what question you answer.1) I have had unprotected sex during the last month: Yes / No

2) 2) I got tails (on the coin flip): Yes / No

16

Example: Sensitive QuestionsExample: Sensitive Questions

a) Proportion of “Yes” responses =(Yes to question 1) + (Yes to question2)/Total # of respondents =

b) Yes to question 1/Total # of respondents + Yes to question2/ Total # of respondents=

c) θP + (1 − θ)δ, whereθ=probability of answering question 1P= proportion who had unprotected sex in the past month(1 − θ)= probability of responding to question 2δ= the probability of getting tails (yes: Q.2)

17

Random Half SampleRandom Half Sample

a) Divide survey sample in half (randomly).

b) Half of respondents get column 1 the other half get column 2.

For January 31For January 31

a)a) Read WKB chapter 4.Read WKB chapter 4.b)b) Answer the following questions:Answer the following questions:

a)a) Imagine we conducted a survey of 350 UGA Imagine we conducted a survey of 350 UGA students and found that 93% thought the students and found that 93% thought the Bulldogs would win the SEC championship. Bulldogs would win the SEC championship. What is the margin of error (at 95%)? Based on What is the margin of error (at 95%)? Based on that margin of error, we can be 95% sure that that margin of error, we can be 95% sure that championship predictions lie in what range?championship predictions lie in what range?

b)b) Imagine we conducted a survey of GA voters Imagine we conducted a survey of GA voters and found that Obama had approval of 55%. and found that Obama had approval of 55%. 750 people were surveyed, what is the margin 750 people were surveyed, what is the margin of error (at 95%)? Based on that margin of of error (at 95%)? Based on that margin of error, we can be 95% sure that Obama’s error, we can be 95% sure that Obama’s support in GA lies in what range?support in GA lies in what range?

19

For February 2For February 2

a)a) Read WKB chapter 5Read WKB chapter 5

b)b) Answer questions 1, 2, & 3 on page Answer questions 1, 2, & 3 on page 130.130.