Designing & Implementing Surveys

Anthony M. Paolo, Ph.D.Office of Medical Education &

Teaching and Learning Technologies

2

Overview

• What is a survey?

• Designing a survey

• KUMC survey tools

3

What is a Survey?Method of collecting information.

Good at gathering “how” and/or “what” people think/feel about issues.

Not good at getting at “why” people think the way they do.

4

Stages of Implementing Surveys

1. Defining objectives

2. Method of data collection

3. Selecting a sample

4. Writing questions

5. Designing the survey

6. Cleaning and verifying data

7. Analyzing data & reporting results

5

Define ObjectivesPrecisely defined objectives are the basis for the Population/sample, Questions, Analysis, and Reporting.

Good objectives have 5 elementsWho will do how much (how well) of what by when

“Determine the quality of education at KU”

“Student satisfaction with the course syllabus”

6

Stages of Implementing Surveys

1. Defining objectives

2. Method of data collection

3. Selecting a sample

4. Writing questions

5. Designing the survey

6. Cleaning and verifying data

7. Analyzing data & reporting results

7

Method of Data Collection1

• Hand out survey

• Self-administered mailed survey

• Self-administered web-based survey

• Face-to-face interview

• Telephone interview

• Method used determined by objectives, available time, resources, and sample.

8

Method of Data Collection2

Web-based / e-mail surveys • Advantages

– Very cost effective– Easy to develop, collect, analyze and respond to– Reduce bias since everyone gets the same item

presentation– Less intrusive then phone or face-to-face interviews– Get more and longer written comments then mailed

surveys– Becoming more popular

9

Method of Data Collection3

Web-based / e-mail surveys

• Disadvantages– Tend to have lower response rates then

mailed surveys– Cannot probe for deeper responses– Assume person sent the survey to is the one

that completed it. – People that have e-mail/web access may not

be demographically representative of general population

10

Stages of Implementing Surveys

1. Defining objectives

2. Method of data collection

3. Selecting a sample

4. Writing questions

5. Designing the survey

6. Cleaning and verifying data

7. Analyzing data & reporting results

11

Selecting a Sample1

• Define Population

• Survey entire population or just a sample?– Depends on resources, time, & size of

population

• The best samples are randomly selected and representative of the population on important characteristics.

12

Selecting a Sample2

Sample Types1. Probability

• Systematic random• Stratified random

2. Non-probability• Convenience• Snowballing

– Main disadvantage: Difficult to generalize results

13

Selecting a Sample3

How big of a sample?

• The bigger the sample, the more reliable the results.

• Depends on your objectives– Comparing groups?

• Can do a Power Analysis

• General Guideline– Have at least 30 in each subgroup, 50 to 100

is better.

14

Stages of Implementing Surveys

1. Defining objectives

2. Method of data collection

3. Selecting a sample

4. Writing questions

5. Designing the survey

6. Cleaning and verifying data

7. Analyzing data & reporting results

15

Writing Questions1

• Based on the objectives

• When in doubt, write fewer questions rather than more.

• Questions should be short & clear.

16

Writing Questions2

• Ask for an answer on only one dimension– Do you like the texture & flavor of the food?

• Accommodate all possible answers– What brand of computer to you own?

A. IBM PCB. Apple

• Have mutually exclusive options– Where did you grow up?

A. CountryB. FarmC. City

17

Writing Questions3

• Do not assume a certain state of affairs– Are you satisfied with your car insurance?

A. Yes B. No

• Avoid abbreviations and double negatives.

• Avoid emotionally loaded or vaguely defined terms.

– Crisis, failure, superb– Usually, frequently, rarely

Item Writing – Vague Terms

18

19

Writing Questions4

• Write your items to at most an 8th grade reading level.

• Do not imply a desired answer– Don’t you think congress spends too much

money?

• Do not ask respondents to order or rank more than 5 items. If must rank, better to have subjects rate each object separately and you rank them.

20

Writing Questions5

Response Scales

• How many options?– 5 to 7 is most adequate

• Include Middle/Neutral Option?

• Include Don’t Know/NA?

21

Writing Questions6

Response Scales

• Positive on left or right?

22

Stages of Implementing Surveys

1. Defining objectives

2. Method of data collection

3. Selecting a sample

4. Writing questions

5. Designing the survey

6. Cleaning and verifying data

7. Analyzing data & reporting results

23

Designing the Survey1

Make respondent’s task as easy and simple as possible.

Items on survey should be grouped in coherent sections.

Questions that use the same response format should appear together.

Include open-ended questions.

24

Designing the Survey2• Provide a cover letter (no more than 1 page)

– Describe purpose of survey and it’s importance to your organization and the respondent

– Mention how long it will take to complete the survey– Mention incentives, if applicable– Encourage prompt response and mention when survey closes– Describe any confidentiality/anonymity issues– Give information of contact person– Thank respondent for participating

• Provide simple and clear directions• Pre-notification letters increase response rate• Reminders to non-responders that include a copy of the

survey increase response rate.• Longer surveys tend to get lower response rates

Keep It Short!!!

25

Designing the Survey3

• Pilot Testing– Should be done with sample similar in

characteristics to the population you are surveying.

– Use small groups and be present when they complete the survey.

– “Talk Out Loud” method

26

Designing the Survey4

Reliability - A reliable survey will provide a consistent measure of important characteristics.

Validity – Are you measuring what you want to measure?

27

Stages of Implementing Surveys

1. Defining objectives

2. Method of data collection

3. Selecting a sample

4. Writing questions

5. Designing the survey

6. Cleaning and verifying data

7. Analyzing data & reporting results

28

Cleaning & Verifying Data• Check for:

– Missing data– Out of range data– Duplicates – Multiple response items– Inconsistent responses– Don’t Know, NA

29

Stages of Implementing Surveys

1. Defining objectives

2. Method of data collection

3. Selecting a sample

4. Writing questions

5. Designing the survey

6. Cleaning and verifying data

7. Analyzing data & reporting results

30

Analyzing & Reporting Results1

• Response rate The higher the better 80% or better - no worries < 50% - worry

• Representativeness – Compare respondents and non-respondents

on important characteristics.

31

Analyzing & Reporting Results2

How results are reported depends on the type of questions and the audience.

Statistical Analyses

Sampling error [(P(1-P)) / n] 1/2

Margin of Error – Sampling Error

n 95% CI

10 31%

30 18%

50 14%

100 10%

200 7%

300 6%

500 4%

1000 3%

1500 3%

2000 2%

32

33

Analyzing & Reporting Results3

• Statistical Significance p-value < .05Statistical significance means that the finding is rare, probably not due to chance or error.

It does not mean the finding is important or even meaningful

34

Analyzing & Reporting Results4

Suppose you find:• Males have a GPA of 3.50 (SD = .30)

• Females have a GPA of 3.54 (SD = .30)

• Is the difference of .04 meaningful?

• Is the difference statistically significant?

It is not meaningful (Effect size = .13), but is statistically significant (p < .05) with 500 per group

35

Software Available for Surveys• Angel• Vovici • SurveyMonkey

36

References• Fink A, The survey kit, Sage • Fink A, How to conduct surveys, Sage• Fowler FJ, Improving survey questions, Sage• Schuman H, & Presser, S. Questions & answers

in attitude surveys, Sage• Henerson ME, Morris LL, Fitz-Gibbon CT, How

to measure attitudes, Sage• Dillman DA, Mail & internet surveys, John Wiley

& Sons

37

Questions / Comments?