Questionnaire Development

Health Survey Research Methods

Susan Sherman

December 6, 2010

Lecture Objectives

1. Learn about the sampling universe, sample, and sampling methodology– Random– Nonrandom

2. Describe strengths and weaknesses of different approaches.

3. To conceptualize, operationalize, and specify research questions;

Learning Objectives

4. To understand how questions, responses, instructions and the questionnaire can effect the meaning of a question.

5. To recognize and revise unbalanced, loaded, or double-barreled questions.

6. Hear about the block

Study Population

• The population (universe, target population) is the entire set of individuals to which findings of the survey are to be extrapolated– Members of the population are elements

• Often cannot sample elements directly (not available, too expensive), but are associated with other units, enumeration units or listing units.

SAMPLING

Why sample?

• Economy!• No need to determine all possible

responses• Can use probability and statistics to assist

making an informed judgment• Sampling frames are out of date as soon

as they are developed

The population and the sample

• The population (universe, target population) is the entire set of individuals to which findings of the survey are to be extrapolated

• Members of the population are elements• Often cannot sample elements directly

(not available, too expensive), but are associated with other units, enumeration units or listing units.

The sample

Probability and non-probability sampling• Probability sample: every element in the

population has a known, nonzero probability of being included in the sample

• Non-probability sampling does not have this feature, but is commonly used in market research and public opinion polls (time, expense, not feasible) – quota surveys

Sampling frame

• Reliance on known probability of being selected (vs. marketing research)

• Provides means of identifying and locating population elements.

– Often contains additional information that can be used for stratification and clustering

– Organization of frame exerts strong influence over sample design.

Sampling frame

• “Ideal frame” lists each population element once and contains no other listings (rare)

• Kish’s classification of possible frame problems:

1. Missing elements (not in the frame)2. Clusters3. Blanks and foreign elements4. Duplicate listings

Problems constructing sampling frames

• Household information may be dated• New households may have been added• Older houses may have been demolished• Block listings date quickly• Organizational lists may be fraught with

missing information• Clustering• Substitution

Forms of Probability Sampling

• Simple random sampling (rarely done)• Systematic sampling• Stratified sampling• Cluster Sampling• Serpentine fashion order

Unknown Source Population• Absence of a sampling frame; unknown

boundaries and size of target population

• Privacy concerns; illegal or stigmatized activitiesi.e., Hidden populations– Men who have sex with men (MSM)– Injection drug users (IDU)– Commercial sex workers– Migrant workers

• Relatively small groups

Sampling Methods

• Facility-Based • Snowball• Targeted• Time-Location (TLS) or Venue-Based

(VBS)• Respondent-Driven (RDS)

Facility-Based Sampling

• Sample of clients from facilities serving the target population

• Examples:– Jails/Prisons– STI clinics– Drug Treatment Centers

• Biased sample based on service seeking• Considered convenience sample

Snowball Sampling

• “Random” selection of “seeds”• Seeds refer others with outcome/exposure

of interest• Endpoint: sample size or sample

saturation• Considered convenience sample• Example: ALIVE Study looking at the

natural history of HIV/AIDS among IDU

Targeted Sampling

• Formative research to identify networks of outcome/exposure of interest

• Different networks treated as sampling strata

• Systematic sampling within strata• Practically treated as a convenience

sample• Heavily dependent on extensive formative

research

Venue-Based Sampling (VBS)

• Sampling of physical venues attended by target population

• Formative research identifies public/private venues and days/times of attendance

• Venue-Day-Times (VDT) enumerated for eligibility and viability

• Sampling frame consists of VDTs; random selection of VDTs to construct sampling event calendar

• Individuals systematically recruited at sampling events

Respondent-Driven Sampling (RDS)

• Type of chain referral sampling to reach hidden populations (Markov chain)

• Begin with a set of non randomly selected seeds• Seeds recruit peers, who recruit peers, etc.• Recruits are linked by coupons with unique

identifying numbers• Recruitment quota through coupons• Incentives provided for completed survey and for

each successful recruit

Heckathorn 1997; Heckathorn & Salganik, 2004; Broadhead et al. 1998

RDS Recruitment Network

Bias (systematic error)

• Sampling and non-sampling biases• Sampling biases come from the sampling

processes themselves or from the statistical estimation process

• Frame biases are the most problematic– Inappropriate selection procedure– Elements appear > 1 time– Non-random ordering

Non-sampling biases

• Account for largest source of total survey error, most often ignore, unappreciated

• Observational biases: caused by obtaining and recording observations incorrectly– Field errors (data collection, enumeration,

measurement) – Processing and data analysis errors

Overall Conclusions• Sampling methods have improved ability to arrive

at valid inferences• Each method has to be considered and applied

based on objectives and target population• Formative research is vital for implementation and

interpretation• These active surveillance and research efforts can

greatly supplement and enhance passive surveillance

• Infrastructures can be used for prevention efforts

RESEARCH QUESTIONS AND MEASUREMENT

To start: what are your goals?

Research Question: A statement that identifies the phenomenon to be studied.

• What are the units or entities being studied?

• What variables will be compared across those units?

• What relationships do you want to examine?

What relationships do you want to examine?

• Associative – correlate

• Causal – question a direction– temporality

• Mediating

Example

To determine if the relationship between exercise and obesity varies by race/ethnicity among 15-24 year olds in Baltimore, MD.

•What are the units?•What are the variables of interest?•What are the relationships between variables of interest?

Measurement Process

Measurement is the process of assigning numbers or labels to units of analysis in order to represent conceptual properties.

(Singleton, ‘93)

3 steps…1.Conceptual Definition2.Operational Definition3.Variable Definition

1. Conceptual Definition

Process of formulating and clarifying concepts of interest– Refines problem statements or hypotheses

which can be vague

Example: Obesity

2. Operational Definition

Questions asked to obtain information on concept or issue

Example: 1. Do you consider yourself overweight,

underweight, or just about right?2. a. About how tall are you without

shoes?b. About how much do you weigh without shoes?

Measurement Process

Measurement is the process of assigning numbers or labels to units of analysis in order to represent conceptual properties.

(Singleton, ‘93)

3. Variable Definition

Variable constructed from questions to be used in the analysis of the data.

Example: 1. Obesity:

1 = overweight

2 = underweight

3 = about right

2. Obesity: Construct index of obesity based on BMI, calculated as weight divided by height squared.

Note of Caution

• No indicator can perfectly represent a single concept.

• No two indicators measure a single concept exactly the same.

DEVELOPING SURVEY QUESTIONS

New vs. Existing QuestionsPros of Existing: enhances quality and applicability of

items, enables comparisons across studies

Consider:– Tested for validity and reliability– Evidence of methodological problems—missing

values– How questionnaire was delivered (self administered,

ACASI, etc)– Study population – Social changes (time period)

Four elements that effect meaning

1. Questions

2. Responses

3. Instructions

4. Questionnaires

1. Questionsa. Wording

• Unclear word choice

b. Phrasing • Unbalanced question• Loaded question• Double barreled question

c. Sentence• Wordiness (short vs. long)

d. Question• Irrelevant to population or research question

1a. Wording

• Avoid jargon

• Avoid acronyms

• Be precise

• Ensure there is adequate knowledge

1b. Phrasing

• Unbalanced Question

• Double-barreled Question

• Loaded Question

Unbalanced Question

Definition: Both sides of a question are not adequately represented.

Example: Do you agree that medical marijuana is bad?

Loaded Question

Definition: A question that encourages participants to respond to the question in a certain way

Example: There are many people who believe that medicinal marijuana should e available. Are you one of them?

Revised: Medicinal marijuana has positive medical properties. [strongly agree… strongly disagree]

Double-barreled Question

Definition: A question that has more than one question embedded within it. A red flag is the word “AND”

Example: Do you agree that medical marijuana should be legal and that you would vote for it on a ballet?

1c. Sentences

Avoid wordiness and confusing sentence structure.

Example: Do you believe that the parking situation on campus is problematic or difficult because of the lack of spaces and the walking distances or do you believe that the parking situation on campus is ok?

1d. Question

Be sure that questions directly relate to research questions.

Example (for research on parking): Do you like or dislike the bus system?

2. Responses

• Open ended– comprehensiveness

• Close ended (rating, ranking)– Should be completely exhaustive– Should be mutually exclusive

Open-ended vs. Closed-ended

Open-ended

• Enables participant to talk about what comes to mind first

• Respondents can provide a comprehensive and diverse array of answers

• Difficult to code and process

Closed-ended

• Dependent on structure of responses: some categories may be inadvertently omitted

• May be effected by # and type of response categories, presence of a neutral, “don’t know” category

• Easier to code and may be more reliable across respondents and interviewers

3. Instructions

Purpose • to ensure that the question or questionnaire is answered

in the way that it should be.

Usage• As part of question itself• To introduce or close questionnaire• To make meaningful transitions between topics• To guide respondent/interviewer on skip patterns

Designation • parentheses, all capital letters or some other type face

To introduce or close questionnaire

We are now done with the questionnaire. If you have any questions about this interview, please feel free to ask me now. [PAUSE FOR QUESTIONS]. If there are no questions, I'd like to thank you for participating in this interview.

To make meaningful transitions between topics

The next set of questions is about using drugs. Please remember that your answers are strictly confidential. Your name is not on this form. No one can trace these answers back to you. If you do not want to answer certain questions you don’t have to, but please answer all the questions that you can.

4. Questionnaire

The order and context in which items are placed has an impact on the meaning of certain questions and how respondents answer them.

Sequencing of Questions

1. Begin with behavioral questions about the present and then the past.

2. Start with questions that make them feel comfortable.

3. Place sensitive questions in the middle.

TYPES OF SURVEY QUESTIONS

Categories of Health Questions

1. Socio-demographics: e.g., age

2. Behavior: e.g., condom use

3. Knowledge: e.g., HIV transmission,

4. Attitudes: e.g., towards HIV positive individuals

1. Socio-demographics

• Use of standardized measures:– Reduce time and effort– Permit direct comparison– May have documented validity and reliability

• Adapt US Census, federally sponsored surveys

2. Behaviors

• Prone to over and under reporting due to recall errors

• Recall errors are a function of: – time period over which events are to be

remembered– Salience or significance of event to person

Minimizing Recall Bias: Memory Aid Procedures

• Aided recall: Clues provided in question

• Records: Ask respondents to consult personal records, e.g. checkbooks, doctor or hospital bills

• Diaries: Respondents use calendars to record relevant events; respondent consults diary when interviewed about these events

3 -4. Knowledge and Attitudes

• Knowledge: more objective b/c a “right” answer is presumed to exist

• Opinion: refers to views about particular objects such as a person or policy

• Attitude: Bundle of opinions that are more or less coherent and are about some complex object

Minimize Threat: Knowledge Questions

• Threat stems from fear of being perceived as ignorant

• To minimize threat, phrase questions as opinions (e.g., “do you think…”)

• To accurately capture knowledge: – Ask more than one question– Keep responses open ended

Rules to Write By Be mindful of 4 elements that effect the meaning of a

question: question, response, instructions, questionnaire

Write in everyday terms

Avoid unbalanced questions

Avoid loaded questions

Avoid double-barreled questions

Write short, simple questions

Ensure questions are relevant to research question

Minimize social desirability bias

Minimize recall bias