Building a Better Survey

Katie Busby, PhDDirector of Student Affairs Assessment & PlanningUniversity of Alabama

Keston Fulcher, PhDDirector of Assessment, Evaluation and Accreditation Christopher Newport University

International Assessment & Retention Conference 2007

Survey

Please take a few minutes to complete this survey. It has some errors, note them when you read.

Overview of Session

• Introduce the assessment cycle as a foundation upon which to build assessment efforts

• Complete an exercise that will identify strong and weak components of a survey

• Discuss, in detail, the steps that are necessary to develop a good survey instrument

• Share practical techniques that can be applied to overcome common missteps in survey development

Definition of Assessment

• The systematic process of determining educational objectives, gathering, using, and analyzing information about student learning outcomes to make decisions about programs, individual student progress, or accountability

- T. Dary Erwin (1991)

Assessment ModelConceptualize and Establish Intended Learning Outcomes

Select Assessment

Methods

Collect Data

Report and Use Results

Analyze and Interpret

Select Assessment

Methods

Rules for Developing Survey

• Avoiding the don’ts and doing the do’s

Rules for Developing Surveys

• Be sure a survey is the way to go– Is there existing data you can use?– Is the survey the best method of collecting

data?– Is there an existing survey you can use?

• Design your assessment project first– Identify your research questions– Determine what you want to know– Clarify your goals

Rules for Developing Surveys

• Survey development is a team sport– Include other experts in the process– Involve relevant decision makers– Combine efforts with others who are

developing a survey instrument

Rules for Developing Surveys

• You never get a second chance to make a first impression– Keep survey as short as possible– Have a logical flow to questions– Ensure a good layout design– Requests for participation, title, and

instructions should be clear and concise

Rules for Developing Surveys

• It’s all about the respondent– Keep the respondent in mind during

the development phase– Make it easy for the respondent to

respond– Consider offering incentives – Pilot the survey

Specific Hint 1

• Keep sensitive items toward the end of survey (i.e., warm up the responder).

• Note: Do you need IRB approval?

Specific Hint 2

• If intended, make sure response options are mutually exclusive.

• Bad: – 0-15k– 15k-30k

• Better:– Under $15,000 per year– $15,001 - $30,000 per year

Specific Hint 3

• Group items with similar content and/or similar item types together.

• Notice that Item 2 and Item 4 are needlessly separated on Bogus Survey.

Specific Hint 4

• “Don’t Know” or “Not Applicable” options should be included at the end of or aside from other response options (READ: NOT IN THE MIDDLE)

Specific Hint 5

• Avoid Double-Barreled Items• Bad:

– (4) I enjoy working in the field of assessment and analyzing data.

• Better– (4) I enjoy working in the field of

assessment.– (5) I enjoy analyzing data.

Specific Hint 6

• Response options should accommodate all reasonable responses.

• On Bogus Survey, response options to Item 5 do not cover all responses.

Specific Hint 7

• Attempt to maximize variability with responses

• For Item 6, the vast majority of attendees would mark the middle option.

Specific Hint 8

• Proofread• As with any document, errors

connote a lack of professionalism.

Questions about survey development

• What challenges have you faced?• How can we improve our “survey”?

Contact us

Katie Busby, PhDDirector, Student Affairs Assessment and PlanningUniversity of AlabamaBox 870116Tuscaloosa, AL 35487-0016205-348-7939kbusby@sa.ua.edu

Keston H. Fulcher, PhDDirector of Assessment, Evaluation, and Accreditation Christopher Newport UniversityMcMurran Building Annex, Room 101-A1 University PlaceNewport News, Virginia 23606(757) 594-7392  keston.fulcher@cnu.edu

Resources for Survey DesignAlreck, P. L., & Settle, R. B. (1995). The survey research

handbook, 3rd edition. New York: McGraw Hill. Bradburn, N. M., Sudman, S., & Wansink, B. (1996). Asking

questions: The definitive guide to questionnaire design – for market research, political polls, and social and health questionnaires. San Francisco: Jossey Bass..

Dillman, D. A. (2006). Mail and Internet surveys: The tailored design method 2007 update with new Internet, visual, and mixed-mode guide, 2nd edition. New York: John Wiley & Sons.

Rea, L. M., & Parker, R. A. (1997). Designing and conducting survey research: A comprehensive guide. San Francisco: Jossey-Bass.

Sudman, S., Bradburn, N. M., & Schwartz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey Bass.