Building a Better Survey Katie Busby, PhD Director of Student Affairs Assessment & Planning...
21
Building a Better Survey Katie Busby, PhD Director of Student Affairs Assessment & Planning University of Alabama Keston Fulcher, PhD Director of Assessment, Evaluation and Accreditation Christopher Newport University International Assessment & Retention Conference 2007
-
Upload
elwin-lyons -
Category
Documents
-
view
213 -
download
0
Transcript of Building a Better Survey Katie Busby, PhD Director of Student Affairs Assessment & Planning...
Building a Better Survey
Katie Busby, PhDDirector of Student Affairs Assessment & PlanningUniversity of Alabama
Keston Fulcher, PhDDirector of Assessment, Evaluation and Accreditation Christopher Newport University
International Assessment & Retention Conference 2007
Survey
Please take a few minutes to complete this survey. It has some errors, note them when you read.
Overview of Session
• Introduce the assessment cycle as a foundation upon which to build assessment efforts
• Complete an exercise that will identify strong and weak components of a survey
• Discuss, in detail, the steps that are necessary to develop a good survey instrument
• Share practical techniques that can be applied to overcome common missteps in survey development
Definition of Assessment
• The systematic process of determining educational objectives, gathering, using, and analyzing information about student learning outcomes to make decisions about programs, individual student progress, or accountability
- T. Dary Erwin (1991)
Assessment ModelConceptualize and Establish Intended Learning Outcomes
Select Assessment
Methods
Collect Data
Report and Use Results
Analyze and Interpret
Select Assessment
Methods
Rules for Developing Survey
• Avoiding the don’ts and doing the do’s
Rules for Developing Surveys
• Be sure a survey is the way to go– Is there existing data you can use?– Is the survey the best method of collecting
data?– Is there an existing survey you can use?
• Design your assessment project first– Identify your research questions– Determine what you want to know– Clarify your goals
Rules for Developing Surveys
• Survey development is a team sport– Include other experts in the process– Involve relevant decision makers– Combine efforts with others who are
developing a survey instrument
Rules for Developing Surveys
• You never get a second chance to make a first impression– Keep survey as short as possible– Have a logical flow to questions– Ensure a good layout design– Requests for participation, title, and
instructions should be clear and concise
Rules for Developing Surveys
• It’s all about the respondent– Keep the respondent in mind during
the development phase– Make it easy for the respondent to
respond– Consider offering incentives – Pilot the survey
Specific Hint 1
• Keep sensitive items toward the end of survey (i.e., warm up the responder).
• Note: Do you need IRB approval?
Specific Hint 2
• If intended, make sure response options are mutually exclusive.
• Bad: – 0-15k– 15k-30k
• Better:– Under $15,000 per year– $15,001 - $30,000 per year
Specific Hint 3
• Group items with similar content and/or similar item types together.
• Notice that Item 2 and Item 4 are needlessly separated on Bogus Survey.
Specific Hint 4
• “Don’t Know” or “Not Applicable” options should be included at the end of or aside from other response options (READ: NOT IN THE MIDDLE)
Specific Hint 5
• Avoid Double-Barreled Items• Bad:
– (4) I enjoy working in the field of assessment and analyzing data.
• Better– (4) I enjoy working in the field of
assessment.– (5) I enjoy analyzing data.
Specific Hint 6
• Response options should accommodate all reasonable responses.
• On Bogus Survey, response options to Item 5 do not cover all responses.
Specific Hint 7
• Attempt to maximize variability with responses
• For Item 6, the vast majority of attendees would mark the middle option.
Specific Hint 8
• Proofread• As with any document, errors
connote a lack of professionalism.
Questions about survey development
• What challenges have you faced?• How can we improve our “survey”?
Contact us
Katie Busby, PhDDirector, Student Affairs Assessment and PlanningUniversity of AlabamaBox 870116Tuscaloosa, AL [email protected]
Keston H. Fulcher, PhDDirector of Assessment, Evaluation, and Accreditation Christopher Newport UniversityMcMurran Building Annex, Room 101-A1 University PlaceNewport News, Virginia 23606(757) 594-7392 [email protected]
Resources for Survey DesignAlreck, P. L., & Settle, R. B. (1995). The survey research
handbook, 3rd edition. New York: McGraw Hill. Bradburn, N. M., Sudman, S., & Wansink, B. (1996). Asking
questions: The definitive guide to questionnaire design – for market research, political polls, and social and health questionnaires. San Francisco: Jossey Bass..
Dillman, D. A. (2006). Mail and Internet surveys: The tailored design method 2007 update with new Internet, visual, and mixed-mode guide, 2nd edition. New York: John Wiley & Sons.
Rea, L. M., & Parker, R. A. (1997). Designing and conducting survey research: A comprehensive guide. San Francisco: Jossey-Bass.
Sudman, S., Bradburn, N. M., & Schwartz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey Bass.
