Chapter 4

pp. 87-98

Survey methodology Used to find out about behaviors and attitudes

Types of surveys Mail (or on-line!) survey

Telephone survey

Personal interview

Census: data collection on entire population Note: 2000 census cost $7 billion ($25/person); 2010 projected $10 billion

Survey construction Writing questions

Types of questions

Wording and content of questions

Types of answers

Arranging questions

Scoring survey

Create a survey of student alcohol use

Define “alcohol use”; what do you want to measure?

What questions would you ask?

How would you word those questions?

What scale would you provide for answers?

How would you arrange your questions?

How would you select your sample?

How do you increase the truthfulness of responses or willingness to respond?

Writing questions Use clear, simple language to min confusion, max

truthfulness, and increase willingness to respond

Types of questions Open-ended

Ss generate answer w/o limitations

Control length by amount of room for response

Qualitative answer

Closed-ended Ss choose from limited number of possible responses; Quantitative answ

Can use rating scale or “Likert scale”

Partially open-ended Choice w/ open-ended space (e.g. “other” choice)

Advantages/disadvantages of each? Open: difficult to analyze

Closed: limits responses participants can give (limits validity)

Writing questions Ways questions can go wrong or mislead the participant

Loaded question “includes nonneutral or emotionally laden terms”; conveys opinion

Leading question Wording implies a correct answer

Double-barreled question Asks about more than one thing; that need to be separated

Nature and # of alternatives in close-ended questions Choices imply average responses; # depends on what is meaningful

Other issues Inclusion of “N/A” or “No opinion” response

Avoid double-negatives; use “not agree” rather than “disagree”

Include multiple versions of same question so can check reliability (ask positive vs negative version)

Measurement errors Errors based on poor measuring instruments or human error

Response bias Tendency for Ss to give same answer to all items

Acquiescence = tendency to agree with question no matter what

Socially desirability bias Ss give inaccurate response b/c other seems inappropriate

Can get non-responses; or make up response b/c didn’t know about topic

Interviewer bias Researcher can influence Ss answers (e.g. by nodding or frowning)

Sampling bias Non-random sample of Ss; one group overrepresented

Memory bias Error due to inaccurate memory

Other: Response may not produce intensity of behavior, attitude, or feeling

Difference to use computer vs. pen & paper?

Perception of purpose of questionnaire

Construct validity Do survey questions adequately measure

construct?

E.g. survey on adolescent smoking

Kovar (2000): 4 million adolescents smoke

What is definition of “adolescent” and “smoking”?

“adolescent” = 12-17 yr-old

“smoking” = 1 puff w/in past 30days - 4 packs/day

Quote provides incomplete picture; need more detail in operational definitions

Arrangement of questions

Randomized order Not always ideal!

Present related questions in subsets Decision: Is it good or bad that the underlying issue is

obvious to Ss?

Sensitive questions at end of survey Increase likelihood to answer

Demographic questions at end of survey So don’t loose interest at start

Previous responses can affect later responses Remember responses and want to appear consistent

Scoring surveys

Decide meaning of high or low score

E.g. high = depression; low = no depression

If have positively and negatively worded questions

Use reverse scoring procedure

Provide min and max score

Reverse scoring procedure Geriatric Depression Scale (GDS)

Choose the best answer for how you have felt over the past week: YES / NO

1. Are you basically satisfied with your life?

2. Have you dropped many of your activities and interests?

3. Do you feel that your life is empty?

4. Do you often get bored?

5. Are you in good spirits most of the time?

6. Are you afraid that something bad is going to happen to you?

7. Do you feel happy most of the time?

8. Do you often feel helpless?

9. Do you prefer to stay at home, rather than going out and doing new things?

10. Do you feel you have more problems with memory than most?

11. Do you think it is wonderful to be alive now?

12. Do you feel pretty worthless the way you are now?

13. Do you feel full of energy?

14. Do you feel that your situation is hopeless?

15. Do you think that most people are better off than you are?

Considerations for ethics in survey research

Anonymity and Confidentiality

Only report data in aggregate (summary)

Ensure voluntary participation; Ss can decline participation or terminate responses at any time

Recognize that your presence can influence responses

Sampling Example: Voter opinions (Beins, 2009)

Approx 100 million potential voters in U.S.

If you want to find out opinion w/ reasonable accuracy, how do you sample?

Typical political polls sample 1,000 people

(Note: that’s .001% of population)

How accurate are they?

In 2000 presidential election difference between poll and result differed by 1.1%

Sampling Representative sample

When sample is similar to population we can be confident in generalization of results

Probability sampling

Nonprobability sampling

Probability sampling Each member of population has equal chance of being selected

for sample Ideal, but can be very difficult to employ

Random selection (or srs - “simple random sampling”) Use random number generator (pp.375) e.g. polls – randomly call households Still could be biased (who ends up in sample?), so the more people the

more likely it is representative of a population

Systematic sampling Create list of population sample every “X”th member

Stratified random sampling Ensure sample represent same proportions of groups as are in population Divide population in to subgroups; randomly select within each

Cluster sampling Identify groups that are representative of population; randomly select

from that group

Nonprobability sampling Individual members of population do not have equal likelihood of

being selected Less expensive and more practical Greatest problem is nonsampling error: Is sample representative? Are the differences between sample and population important to the research?

Convenience sampling Randomly select from participants that are available (but not necessarily

representative of population)

Quota sampling Convenience sampling of specific groups; ensure sample is like population on

certain characteristics

Purposive sampling Ss selected for some particular characteristic or expertise

Chain-referral sampling If difficult to have way to contact Ss (from “hidden populations”) – each Ss

selected identifies another participant Snowball sampling: volunteer identifies # of people that researcher contacts; then

each of those are asked to identify more people, etc. Key informant sampling: outsider who knows group provides names

Incentives Do you compensate Ss for participation?

Internal motivation

External motivation

Internal motivators

Adding to knowledge-base

Learn more about self/others

If external, how do you compensate?

Monetary amount (by hour; flat rate)

Gift

Gift to organization of choice

Student alcohol questionnaire (SAQ) http://www.indiana.edu/~engs/quest/saq.html

Engs (1977): Drinking patterns and drinking problems of college students.

Method:

13 of 62 schools agreed to participate in 1975-6 school-year

11 large universities (10,000+); 5 in big city; rep all areas of US

Each school randomly selected 100 undergrads

N = 1128; 41% male; 79% white; 35% freshman

Results:

79% drank at least 1x/year; 57% at least 1x/month; 70% drank beer

82% of men; 75% of women drank at least 1x/year

20% report no problems as a result of drinking; 50% 4+ problems (most common: hangovers, nausea, fighting, prob’s w/ grades)

Survey construction principles

Make sure questions match research objectives

Understand research participants

Use natural and familiar language

Write items that are clear, precise, and relatively short

Do not use “leading” or “loaded” questions

Use mutually exclusive and exhaustive response categories for close-ended questions

Consider different types of response categories available for close-ended questions

Use caution if you reverse the wording in some items

Develop a questionnaire that is easy to use

Always pilot test your questionnaire