“And The Survey Says…” Designing Surveys to Effectively Measure

Outcomes in Educational Programs

Daniel C. West, MD

Timothy W. Kelly, MD University of California, San Francisco

APPD/COMSEP Workshop April 13, 2013

First, a few moments of levity—and for some of us…nostalgia… A video clip of a ‘Family Feud’ episode Even “simple” questions have challenges! …Please focus on the “gasoline” question and answers

“And the Survey Says…”

“And the Survey Says…” Pair off for the following exercise

Discuss with your partner what you thought of the question…

“Name a brand of gasoline” Consider the following: 1. Why do you suppose both contestants answered the

way they did?... “Regular” and “Ethyl” 2. Identify at least one thing that could be done to

improve the question

Introductions A little about you…

Today’s Goals What would you like to get out of this session?

Survey Skills How will you use what you learn?

Plan for Today 1. Review key principles of survey design 2. Learn a step-by-step method for writing survey

questions (i.e. items) 3. Review types of questions and response scales 4. Practice applying a standardized approach to survey

question quality control 5. In small groups, practice writing survey questions 6. Review a few basic principles of survey

administration (i.e. collection)

Designing Surveys Step 1: Understand the Big Picture

• What do you want to study? – Broad study question and hypothesis

• How are you going to use the survey? – Consequences of results determines

importance of validity measures

Writing Survey Questions Step 2: Developing Potential Items

• Determine Objective First – What specific thing are you trying to measure? – Items or questions follow next – Every question MUST map back to an objective – Single biggest mistake is to skip this step!

• What type of objective is it? – Factual: Objective things or events – Subjective state: Opinions, attitudes, feelings

Designing Survey Questions Step 2: Developing items using worksheet

Writing Survey Questions/Items Key Principles 1. Ask questions that respondent can answer

– Are you asking the right person in the right way? – Could they know? Consider time frame and recall

2. Ensure everyone answers the same question – Define key concepts and terms – Goal is to measure real differences—not differences in

understanding of question

3. Frame question to get accurate (i.e. valid) response

4. Use experts to review and revise

Writing Survey Questions/Items Framing survey questions

• Clarify purpose – Tell respondent why you are asking and how you intend to

use the information

• Social desirability – Normalization: Give respondent permission to give what may

be perceived as an undesirable response

• Order and context of questions – Group questions about similar things (i.e. construct) together

• Uniform response format – Group related questions with same response format together

Framing Survey Questions/Items Example of clarifying purpose

Framing Survey Questions/Items Example of normalization with framing

Response Scales Issues to consider

• Continuous vs. Ordinal vs. Categorical – Factual answers often categorical or ordinal – Subjective answers often on continuous scale

• Design response format so respondent can… – Answer the question (e.g. recall timeframe) – Place their response on your scale

• Save yourself some grief…Whenever possible use already validated scales!

Common Subjective Response Scales Generally considered continuous scales

Writing Survey Questions Step 3: Quality control step is ESSENTIAL!

Must do this for every single item…every single one…no exceptions!

Factual Questions Special considerations

Consider how you will use the information – How specific does it need to be? – Consider how you will analyze or display information (e.g.

will you need a continuous measure?) – Almost always best to use the most direct /continuous

measure and categorize later

Validate answers at time of response – Limit the mistakes the respondent can make – Example: provide categories or limit free text responses

Factual Questions Examples Objective: To learn the location of the training programs

of course participants Type of Question: Factual Question: Free text question

Factual Questions What result did I get?

Factual Questions Examples

Objective: To learn the location of the training programs of course participants

Type of Question: Factual Question: Free text question Problems:

– Results not very discriminating – Root of problem is that objective was not clear

Factual Questions Alternative version of where is your program located?

Factual Questions Another example

Objective: To know the size of training programs of course participants

Type of Question: Factual Question: Free text with validation (e.g. whole numbers)

Factual Questions Size of training program-what did I get?

Factual Questions Another example

Objective: To know the size of training programs of course participants

Type of Question: Factual Question: Free text with validation (e.g. whole numbers) Problem:

– Can be interpreted in several ways (e.g. do I mean your class or the total number of all classes?)

– Could I be asking the wrong people?

Factual Questions A potentially better way? Objective: To know the size of the training programs

of course participants Type of Question: Factual Question: Narrow respondents with skip logic,

clarify who to count and use categories

Factual Questions A potentially better way?

Factual Questions A potentially better way? [data]

Factual Questions Summary of key principles

• Carefully define your objective – Precision in language is critical – Pilot test! Look at what you get and revise!

• Determine how you will use the information in advance – How will you analyze or summarize the information – Usually better to use most direct/continuous measure and

then categorize later • Make sure you are asking the right people

– Can use skip logic step to increase the chances of that

Subjective Questions Key Principles Terms and concepts must be clear

– Subjective thoughts can vary depending on wording – Goal is shared understanding of question to minimize

error (i.e. want everyone to answer same question) – Ask one thing at a time

Response format is critically important – Match response to question/objective – Goal is different responses distributed across scale – Respondent places their response on your scale – Normalize the range of responses (i.e. any response

should be equally acceptable)

Common Response Scales Usually Subjective, but occasionally factual

Subjective Questions Example Objective: Understand how much experience course

participants have with survey design

Problems and strengths: Asks more than one thing and terms not adequately defined …but uses well established response scale

Subjective Questions Alternative questions Objective: Understand how much experience course

participants have with survey design Alternative question:

How FAMILIAR are you with the concepts and principles of optimal survey design and collection? Extremely, very, somewhat, slightly, not at all

In thinking about your skills in all aspects of survey design, collection, and analysis, where would you place yourself along the path from beginner to expert? Beginner, advanced beginner, competent, proficient, expert

Comparison of Question/Response Scale

Subjective Questions Alternative questions

Objective: Understand how much experience course participants have with survey design

Alternative question: Advantages: Breaks up question up and asks about specific aspects

of survey design and collection separately Problems: Response scale may be difficult for some respondents

Self-Rating of Survey Skills How would you summarize the response data?

Skill Mean Rating (Scale 1-7)

Standard Deviation

Survey Design 2.65 1.43 Question Writing 2.68 1.41 Response Scales 2.50 1.33 Collection 3.00 1.75 Analysis 2.82 1.54

Self-Rating of Survey Skills Does the mean and standard deviation describe these data?

Skill Mean (Scale 1-7) Median Mode

Survey Design 2.65 2.32 1 Question Writing 2.68 2.34 1 Response Scales 2.50 2.00 1 Collection 3.00 3.00 1 Analysis 2.82 2.41 2

Self-Rating of Survey Question Writing Skill Distribution of question writing skill rating data

Self-Rating of Survey Collection Skill Distribution of collection skill rating data

Subjective Questions Agree/Disagree Response Scales

• Potentially good if you want opinions – Don’t have to develop unique scales or behavioral anchors

• Statement or “stem” needs to be polarizing – Otherwise cannot interpret – Neutral statements encourage agreement bias

• Should avoid complex (“double negatives”) statements

• If possible, use a more direct and descriptive scale (…likely yields more useful information)

Subjective Question Response Scales Comparison of Descriptive vs. Agree/Disagree

VS

Subjective Question Response Scales Descriptive vs. Agree/Disagree

Direct Rating Scale Agree/Disagree Scale

Rate your skill in “survey question writing” with 1=Novice to 7=Expert

Agree or disagree with “I consider myself an expert in survey question writing”

Agree/Disagree Response Scales Potential problems with neutral statements

Subjective Question Response Scales Summary Points

• Almost always better to use direct measure instead of agree/disagree scale if you want to… – Use mean values or other continuous measures – Establish thresholds or explore association of response with

an outcome – Measure change in some characteristic over time

• Agree/Disagree very useful when… – Direct scale difficult to create – Truly want to know about opinion about a particular

statement or point of view

Response Scales Free text responses • Potential for rich information

– Especially exploratory information • Significant responder burden and can be

difficult to analyze and summarize • Use validation rules when possible • Useful approach is single word responses

• Single most important thing • What one word comes to mind? • List one but not more than three…

Free Text Response Responses

Free Text Response Analysis of responses [data]

Free Text Response Other examples

Validity and Reliability A unified definition

• Validity is not intrinsic to survey instrument • Scores derived from a survey have evidence

supporting the validity for a specific purpose or interpretation

• Scores should measure what they are intended to measure

• Scores should be dependably and reproducibly produced

Validity (make this consistent) Building evidence to support validity

• Are the items measuring what you intend? • Are scores from different measures of the same thing (i.e.

construct) consistent? • Are scores from measures of different constructs independent? • How do scores of the survey compare with a criterion or “gold”

standard? • Are scores from survey consistent with current (i.e. expected) or

future performance? • Can the survey be administered in way that produces scores that

measure what you intend, and not something else?

Survey Administration Things to consider • Respondent burden

– Length and difficulty of task (make it simple) – High quality questions will help

• Timing of survey – Ask when respondent most interested and most

likely to know the correct answer • Electronic vs. paper vs. interview • Incentives

Survey Administration What happened with our survey?

Type of contact Response Rate Initial email invitation and reminder from course administrator 63%

Second “gentle” reminder 71%

Third “stern” reminder 83%

Final response rate 88%

Data Analysis and Presentation General Principles

• Always keep questions linked to objectives – This will drive analysis – Disconnect leads to problems

• Continuous measures usually better – Can calculate means and compare easily – Can convert to categorical variables later

• Present with images, figures and tables – Don’t burry important data in text of a manuscript!

Survey Questions Step 3: More practice with quality control

Assessing Quality of Survey Questions Alternative measure of feelings about your clinical training

Assessing Quality of Survey Questions Alternative measure of feelings about your clinical training [data]

Quality of Survey Questions Direct measure of quality of medical training

Quality Survey Questions Direct measure of quality of medical training

Quality of Survey Questions A couple more… Question: Are mechanisms within the

institution available to you so that you may raise and resolve issues without fear of intimidation or retaliation? -at all times, some of the time, none of the time

Biggest problems: – Asking more than one thing – Mixed factual and subjective – Disconnected response scale

Quality of Survey Questions One more…

Question: Do rotations and other major assignments emphasize clinical education over any other concerns, such as fulfilling service obligations? -always or usually, sometimes, rarely or never

Fundamental problems: Disconnect from objective and terms poorly defined

Survey Design and Collection Common pitfalls

Common Pitfall Result or Effect

Purpose/objective not clear Survey questions difficult to develop Questions not linked to objective Results of survey don’t achieve objective

Result of survey cannot be interpreted Question terms and concept not well defined

Not all respondents answer the same question Difference in response due to difference in understanding of the question (error)

Question asks more than one thing Question does not measure what is intended Difference in response due to difference in understanding of the question (error)

Question mixes factual and subjective objectives

Question does not measure what is intended Response scale won’t match

Response scale does not match objective and question

Question does not measure what is intended Respondent frustration

Respondent cannot put their response onto the question response scale

Question does not measure what is intended Respondent frustration

Summary • Always keep every question linked to objective • Follow stepwise approach to writing each question • Always do quality control step…with every

question • Remember that response scale requires respondent

to put their response on your scale • Strive to make sure that everyone answers the

same question (a key to validity/reliability) • Pilot test! Practice, Practice, Practice