Applied Opinion Research Training Workshop Day 2.
-
Upload
kenneth-mccoy -
Category
Documents
-
view
214 -
download
0
Transcript of Applied Opinion Research Training Workshop Day 2.
Applied Opinion Research Training Workshop
Day 2
Designing Quantitative Instruments
Mary McIntosh
Quantitative Instruments
Designing a Questionnaire – Keep in Mind: Don’t forget the big picture
Follow your objectives and hypotheses to determine the content of your questions
The quality of your research depends on the quality of your questionnaire
Ask others to review and proofread your survey Always pretest, do a trial run Utilize other sources to assist you in designing your
questionnairePrevious studies, expert input, etc.
Make instructions as clear as possible
Quantitative Instruments
Designing a Questionnaire – Step by Step: Screening respondents Wording questions Response options Order of questions Length of questionnaire How the questionnaire will be administered
Quantitative Instruments
Screening Respondents: Sample Parameters First, you want to decide if you will need to specify
certain criteria that your respondents need to satisfy• Age (e.g., over 18, under 65)• Profession (e.g., government official, private sector, etc.)• Status in household (e.g., mother)
These questions should come first
Quantitative Instruments
Wording Questions Second, start writing your questions, keeping in
mind: Questions should be kept short Use wording appropriate to your respondents’ level of
education Make all definitions, qualifiers, and assumptions explicit
(e.g., Do you think the Bank’s program was effective? Which program?)
Quantitative Instruments
Wording Questions Be careful when using abbreviations, acronyms, and
jargon Be careful with word choice Highlight words that require extra emphasis Make sure your questions are technically accurate Make sure questions are relevant and applicable to
your respondents
Quantitative Instruments
Wording Questions The time span for recall in a question should reflect
the saliency of the topic Also be careful about asking questions about future
intentions; keep to the short-term future Time spans, past or future, should be as specific as
possible (e.g., don’t use “lately”)
Quantitative Instruments
Wording Questions Do not use leading questions that suggest a particular answer or
the researcher’s viewpoint (e.g., Why does working in the public sector make people lazy
and careless?) Do not use loaded questions that bias people towards particular
answers (e.g., Do you think it is important to have a strong police force in
this time of crisis?) Be wary of double-barreled questions. Rephrase them into two
questions whenever possible (e.g., Should the government reduce its financial help and its
technical assistance to other countries?)
Quantitative Instruments
Wording Questions Avoid double negatives (e.g., not prohibit), they can be
confusing. If absolutely necessary, emphasize the “not” by underlining it
Make sure your questions are valid. The question must ask for information which the respondent is capable of providing an informed response
Make the question as specific as possible(e.g., instead of asking “Is privatization a good idea?” ask “Is privatization of utilities a good idea?”)
Quantitative Instruments
Wording Questions Explicitly state alternatives
(e.g., instead of asking “To balance the budget, would you be in favor of a higher income tax?” ask “To balance the budget, would you be in favor of a higher income tax, a higher sales tax, or reduced government services?”)
Use face-saving phrasing (e.g., instead of “Did you vote?” ask “Were you able to vote?”)
Be sure the wording of your question matches your response options
Consider “Don’t Know” options
Quantitative Instruments
Response OptionsCommon response options are:
– Likert/Rating scale– Dichotomous choice/Trade-off– Multiple choice– Open-ended– Ranking
Quantitative Instruments
Likert/Rating Scale Shouldn’t use more than 2-3 scales in each survey effortNumbered Scales Asks for responses on a numbered scale with descriptions for what
the numbers represent (even and odd numbered scales) e.g., Please indicate how favorable your impression of the World Bank is
on a scale of 1-10, 1 being very unfavorable, 10 being very favorable Numbered scales work well in written surveys Numbered scales work well with educated population Numbered scales must be appropriate for culture (e.g., 10 point
scale vs. 7 or 8 point scale)
Quantitative Instruments
Likert/Rating ScaleWord Scales Word scales are similar but meet different respondent needs.
e.g., Please indicate how much you agree with this statement, using a scale of “strongly disagree”, “somewhat disagree”, “neither agree nor disagree”, “somewhat agree”, “strongly agree”
What is your opinion of the Bank’s work in the area of infrastructure development in your country. Is it “very good,” “somewhat good,” “somewhat bad,” or “very bad.”
Word scales are usually 3, 4, or 5 point scales Very common, easily used in written form
or verbally Word scales sometimes preferred because the opinion is clearly
defined for respondent
Quantitative Instruments
Likert/Rating Scale Provides a great deal of information, quickly and
easily Can be used to compute summary statistics (e.g.,
frequencies, means) and significance tests Best used to measure opinions, attitudes, extent of
agreement/intention, etc. Not as good for measuring facts
Consider a “Don’t Know” or “Refuse” response
Quantitative Instruments
Likert/Rating Scale Common formats are 4-points, 5-points, 7-points, and 10-points
(will vary based on how much knowledge respondents have; how nuanced their opinions may be. When in doubt, keep it simple (4 point scale.)
The greater the number of points, the greater variation you will be able to measure in respondents’ responses
An odd number of points allows respondents to give a middle or “neutral” response. Even numbered scale forces respondents to commit.
Odd point middle point can have a number of meanings: don’t know; neutral; neither positive nor negative
Quantitative Instruments
Likert/Rating Scale Respondents need to be comfortable with the scales you
utilize If possible, use a balanced scale
• e.g., 1-not at all effective, 5-very effective• If it is not possible to use a balanced scale, make sure your endpoints
are clear antonyms In your number descriptions, 1 should describe the smallest,
most negative response and 4/5/7/10 should describe the largest, most positive response
Quantitative Instruments
Dichotomous Choice/Trade-Off Asks for an either/or response
• e.g., Were you able to vote in the last election? Yes/No• e.g., Which of the following two statements best represents your
views on taxation in our country? 1) I believe our tax dollars are not used wisely by
our government.2) I believe our tax dollars are used appropriately
by our government. Flexible and easy to administer in written or verbal format
Quantitative Instruments
Dichotomous Choice/Trade-Off Provides limited information - Does not tell you the
extent of opinion Yields dichotomous data Can be used to compare response percentages and
to examine how different groups responded Best used to measure facts. Not good for
measuring opinions, attitudes
Quantitative Instruments
Multiple Choice Gives respondent a variety of responses to choose from
e.g., Which organization do you trust the most? (Oxfam, Save The Children, Catholic Charities)
e.g., What do you consider the most critical development challenge facing your country? (Poverty, Education, Corruption, etc.)
e.g., Which of the following describes your position? (Government official, Private sector, NGO, etc.)
Quantitative Instruments
Multiple Choice Very common, flexible and easily used in written
form; verbal ease depends on the number of choices Provides a great deal of information, quickly and
easily Yields categorical or ordered data Can be used to compare response percentages and
to examine how different groups responded
Quantitative Instruments
Multiple Choice Under some circumstances, you may want the respondent to
be able to choose more than one answer• e.g., In your opinion, which are the most important Bank programs in
your country? (choose up to 3) Response options must be mutually exclusive
• e.g., How much of your country’s GDP do you think is spent on foreign assistance:
< 1% 1-4% 5-8% 9+%
Quantitative Instruments
Multiple Choice Response options need to be exhaustive, but not
overwhelming• Include an “Other” choice in which respondents can write
in a response that you did not include When applicable, order choices from smallest to
largest, most negative to positive• “Other” and “Don’t Know” options should be last
Quantitative Instruments
Open-Ended Allows respondent to provide any response that they want
• e.g., Why do you think economic reforms have not been implemented by the government?
• e.g., How would you improve the effectiveness of Bank programs? Common, however, must be transcribed and, if not in English,
translated Limited to skill of interviewer and knowledge of respondent
Quantitative Instruments
Open-Ended Provides a great deal of information Yields data unique to each respondent;
once it is coded, response percentages can be computed and compared
• Risk: coding is subjective, time consuming, and expensive
Best used for exploratory research • Use when many answers are possible and multiple-choice options are
too extensive or unknown
Quantitative Instruments
RankingAsks respondent to rank a variety of responses
according to some criterion• e.g., Please rank the following in order of priority from 1-
10, 1 being top priority: Education, Communicable Disease, Weak Institutions, The Economy, Corruption, etc.
Less common, best used in written form, difficult to administer verbally
Quantitative Instruments
Ranking Can be difficult for respondents to complete Provides limited information; assumes that
respondents feel differently about each item Data is difficult to analyze and interpret Best used to measure limited attitudes and opinions
Quantitative Instruments
Order of Questions If necessary, screening questions should be first to
eliminate ineligible respondents early Then, using the funnel approach, proceed from
broad, general questions to more specific questions Opening questions should be easy and
non-threatening to keep respondents’ interest If possible, make the opening questions interesting
to peak respondents’ interest
Quantitative Instruments
Order of Questions Consider the advantages/disadvantages of starting a
questionnaire with an open-ended question Avoid questions that get monotonous Consider the advantages/disadvantages of completing all
questions on an issue before moving to a new issue Order issues in a logical fashion to facilitate the flow of the
questionnaire Obtain historical information in chronological order, either
forward or backward
Quantitative Instruments
Order of Questions Use transition statements to facilitate shifts from
one issue to another or one series of questions to another
Eliminate order bias due to the sequencing of questions as much as possible• e.g., do not ask about economic growth as a major
development challenge before you ask respondents what they consider to be the major development challenges for their country
Quantitative Instruments
Order of Questions Do not ask sensitive or difficult questions until a rapport has
been established with the respondent Any demographic/classification questions should be asked at
the end of the questionnaire, unless required for screening or cultural tradition
• Demographic/classification questions should be ordered from the least to the most sensitive items
Avoid complex or frustrating skip patterns (if self-administered or face-to-face)
Quantitative Instruments
Length of Questionnaire Don’t burden respondents with a long questionnaire
• The shorter the questionnaire, the higher the quality of the data Only ask questions that meet your specific research
objectives• For each question, ask is this question necessary?
Pre-test your questionnaire, even if it is just among your colleagues, to determine how long it will take respondents to complete it• Language expansion factor• Cultural traditions of long introduction
Quantitative Instruments
Administering the QuestionnaireConfidentiality must be ensuredConsider not using a political polling firmKeep in mind how you will be administering the
questionnaire (e.g., mail-in, phone, in person, within groups, electronic) when deciding on the wording and length of the questionnaire
Quantitative Instruments
Administering the Questionnaire Mail-in Questionnaires:
• Best for longer questionnaires that may include sensitive issues for which respondents will want to maintain confidentiality
• Should include postage-paid return envelopes to encourage participation
Phone Surveys:• Best for moderate length questionnaires, around twenty minutes• Should be done at appropriate times with interviewers fluent in the
respondents’ language
Quantitative Instruments
Administering the Questionnaire In-person Surveys:
Best for longer questionnaires Should be done at appropriate times with local
interviewers who are fluent in the respondents’ language Skill of the interviewer will influence the quality of the data
Hands-on-Work:Designing a Quantitative Questionnaire
Designing Qualitative Instruments
Sharon Felzer
Qualitative Instruments
Designing a Guideline – Keep in Mind: This is a questionnaire for discussion Don’t forget the big picture
• Follow your objectives and hypotheses to determine the content of your questions
The quality of your research depends on the quality of your guideline and how well the guideline is administered
Ask others to review your guideline Utilize other sources to assist you in designing your guideline
• Previous studies, expert input, etc.
Qualitative Instruments
Designing a Guideline – Step by Step: IntroductionWording questionsOrder of questionsLength of guidelineAdministration of guideline
Qualitative Instruments
IntroductionThe point of the introduction period is to make the
respondent(s) feel more comfortable in the situation The researcher should introduce themselves and the
research Discuss anonymity and confidentiality Introduce respondent(s)
Qualitative Instruments
Wording QuestionsWriting questions for a guideline is much like writing
a questionnaire with all open-ended questions; therefore, many of the recommendations are the same, keeping in mind:
• You want a logical, systematic conversation• Replicate as much as possible by other interviewers
Qualitative Instruments
Wording QuestionsMost importantly:
• Your questions must encourage discussion:– Why? – How? – What do you think about…?– In what ways…?– What do you know about…?
» e.g., What is your perspective on privatizing the airlines in our country?
Qualitative Instruments
Wording QuestionsProvide probes for the researcher to use if the initial question does not elicit sufficient discussion. For instance:
– Question: What do you think are the major development challenges facing your country?
– Probe: Why are these more pressing than other challenges?
– Probe: Describe how the government is meeting these challenges (probe: resources, partnerships, etc.?)
Qualitative InstrumentsWording Questions
Occasionally, a close-ended question may be necessary, just be sure to include follow-ups
e.g., Do you believe families should have to pay for their children’s education? Why/why not?
Although you want to encourage discussion, make sure your questions are sufficiently focused that respondents do not go too far astray
You should set approximate time allotments for each section to be sure the researcher covers all the issues
For certain questions, you may want to take notes on a board for the respondents to see; indicate this in the guideline
Qualitative Instruments
Order of Questions As with quantitative questionnaires, use the funnel
approach, proceeding from broad, general questions to more specific questions
Opening questions should be easy and non-threatening
Qualitative Instruments
Length of Guideline Keep in mind, the objective of this research is to encourage
discussion. You should not expect to get through a lengthy guideline• The shorter the guideline, the more you will be able to encourage a
thorough discussion Only ask questions that meet your specific research
objectives. For each question, ask:• Is this question necessary?• Will this question further the relevant discussion?• Will it meet my objectives?
Qualitative Instruments
Administering the Guideline A skilled moderator/interviewer is essential to gathering
quality data Confidentiality must be ensured Consider not using political polling firms Keep in mind how you will be administering the
guideline (e.g., focus group, in-depth interview) when deciding on the wording and length of the guideline
Qualitative Instruments
Administering the GuidelineFocus Group:
Professional moderators Best for encouraging discussion of a range of respondents’
opinions Guidelines should be short to moderate in length.
To keep a lively discussion, focus groups should not go too long (about 90 minutes, depending on the culture)
Be careful not to include sensitive issues about which respondents will want to maintain confidentiality and may be reluctant to discuss
Qualitative Instruments
Administering the GuidelineFocus Group:
Moderators should be instructed on what to do if the discussion goes too far astray
Moderators must also maintain objectivity, so that respondents do not feel that they should respond in a certain manner
Moderators must be fluent in the respondents’ language
Qualitative Instruments
Administering the GuidelineIn-depth Interview:
Professional interviewer Best for a longer guideline, a guideline that does not
lend itself to group discussion, or when a group format would not be appropriate with the desired respondents (e.g., elite government officials)
Confidentiality must be ensured Should be done at appropriate times with interviewers
fluent in the respondent’s language
Qualitative Instruments
Administering the GuidelineIn-depth Interview:
Be careful asking about sensitive issues as responses may be vague or respondent may be uncomfortable responding, impacting the entire interview
As with focus groups, interviewers should be instructed on what to do if the discussion goes too far astray and how to maintain objectivity
Hands-on-WorkDesigning a Qualitative Guideline