SurveysCynthia J. Miller, Ph.D.

Assistant ProfessorDept. of Physiology & Biophysics

University of Louisville

Why should you use surveys?

To reach a large number

of participants

To obtain baseline data and monitor

changes

To gauge motivations, beliefs, and

attitudes

To evaluate the

effectiveness of teaching practices

Time-efficient,

convenient, and familiar

To gather demographic

s

To examine alignment

between faculty and students

Developing Surveys Search for established and/or

validated instrumentso Consult the literature in your fieldo Verify that the instrument is

beneficial for your useo Example: SALG (Student

Assessment of their Learning Gains) Survey

If an appropriate instrument does not exist:o AKA “back to square one”o Example: CREATE Surveyo Define your objectiveso Work backwards-determine the

data that you need to gather

Question Development Forced vs. open-response items

o Forced, Likert-scale most commonly usedo Forced responses can be easier to analyzeo Open responses can be useful due to multiple

definitions/perceptions of questions, but also increases difficulty of analysis

Try to ensure anonymity if possible, guarantee confidentiality

Use simple terminology and unambiguous language

Check for bias-ensure you are not asking leading questions

Question Development Ask questions from multiple perspectives and in

different parts of the survey to ensure validity Hoskins Paper

o “The scientific literature is difficult to understand”

o “I am confident in my ability to critically review scientific literature”

o Negative statements are reverse-scored when quantifying data

Question Development

Revisit original goals/hypotheses and ensure alignment

Pilot questions with your research team, peers, a sample of target respondents, and a statistician

Survey Distribution

Pen-and-Paper Benefits: Can ensure

completion by respondents

Limitations: Must compile data by hand, lower perception of anonymity (particularly with open-response questions)

Online Programs such as

SurveyMonkey, SurveyGizmo, Qualtrics

Most institutions have programs available for free or reduced prices

Benefits: Compile data, Produce graphs or charts, Statistical analysis, Larger potential sample sizes

Limitations: May lower response rates

Effective DistributionChallenges

Low Response Rateso “Survey Fatigue”o Poor timing

Respondents not carefully completing survey

Suggestions Consider providing incentives for

participation Emphasize importance of results Complete during class time or

scheduled meetings Endorsement by respected

authorities

Potential Limitations

Samples may not be representative of entire population

Self-reported data is often biasedo Past eventso Behaviors that may be

negatively perceived Respondents may feel the

need to “tell you what you want to hear”

Triangulation Use multiple methods of data collection to

analyze the same research question

Faculty Survey

s

Student Surveys

ExamScore

s

Student

Focus Groups

Faculty Surveys

Classroom Observation

s

CREATE Method Papers

Student Surveys (Pre/Post

)

Mentioned past studies on critical-thinking skills,

interviews, and validated surveys

Student

Surveys

Student Exams

(Pre/Post)

Student Article

Critiques

Segura-Totten and DalmanHoskins, Lopatto, and Stevens

Resources Survey Design

o www.statpac.com/surveys Describing & Measuring Undergraduate Stem Teaching Practices

o http://ccliconference.org/files/2013/11/Measuring-STEM-Teaching-Practices.pdf

Duke Initiative on Survey Methodologyo http://dism.ssri.duke.edu/question_design.php

General Introductiono Fink AG. How to Conduct Surveys: A Step-by-Step Guide. Sage: Thousand

Oaks, CA, 2008. Question Development and Distribution Methods

o DeVellis RF. Scale Development Theory and Application, 3rd Edition. Sage: Thousand Oaks, CA, 2012.

o Dillman DA. Mail and Internet Surveys: The Tailored Design Method, 2nd Edition. John Wiley & Sons, Inc.: Hoboken, NJ, 2007.

Triangulationo Creswell N, Clark V. Designing and Conducting Mixed Methods Research.

Sage: Thousand Oaks, CA, 2007.