Incorporating an Evaluation Plan into Program Design:Using Qualitative Data
Connie Baird Thomas, PhDLinda H. Southward, PhD Colleen McKee, MS
Social Science Research CenterMississippi State UniversityPositioned for Progress Conference April 30, 2014Jackson, MS
Workshop Goals
• Understand the difference between qualitative and quantitative data
• Learn how to capture qualitative data in a rigorous objective way
• Explore best practices related to qualitative data collection
PROJECT ABC – Literacy Intervention Program
Conceptualize—“If we do X, then Y will result”
ACTIVITIES: (X) Pilot program in 10 schools Group tutoring One-on-one tutoring Curriculum Take home materials Parent participation
Trained facilitators
OUTCOME (Y): Proficient reading by fourth grade
What is Qualitative Research?
Data collection method in which the researcher:• Seeks an in-depth, “rich” view of topic• Explores an idea or topic to describe or give it
meaning• Asks participants broad, general questions• Collects detailed views of participants or subject• Analyzes and codes data for description or
themes
Characteristics of Qualitative and Quantitative Research
Qualitative• Goal: describe, discover• Design: flexible, emergent• Sample: small, non-random• Data collection: interviews,
observation• Data interpretation:
subjective
Quantitative• Goal: predict, confirm• Design: structured• Sample: random, large
representative• Data collection: surveys,
questionnaires• Data interpretation:
objective
Qualitative Research …
• Does – Explain “how” and
“what”– Describe “what
happened”– Explore processes over
time– Explore meaning of
event to those involved
• Does not– Prove/disprove
hypothesis– Measure variables– Test theories– Show relationship
between variables– Make generalizable
statements
Methods of Qualitative Research
Methods• In-depth Interview
• Focus group
• Case Study
• Direct Observation
Data“Words don’t fly individually- they fly in flocks.” Geoffrey Nunberg- University of CA, Berkley School of Information
MethodologyIn-depth Interview
• Advantages• Open structure allows for individual to
express detailed opinions and experiences• Interviewer can easily follow-up and/or
clarify statements • Appropriate for sensitive or controversial
topics or special populations
• Limitations• Requires skilled interviewer• Responses may be biased• Not generalizable• Time/labor intensive
One-on-one conversation between interviewer and respondent regarding a specific topic
Sometimes audio/video recorded
Methodology
Focus Group
Advantages• Group setting often stimulates
additional thoughts and ideas• Gives more insight into why certain
opinions are held• Can better understand processes
(changes that have occurred over time)• Observe nonverbal responses
Limitations• Group/time management • Peer pressure among respondents• May stress resources
Focused small group (usually 6-10 participants) discussion guided by a leader or moderator
Useful for populations with shared experiences
Methodology
Case Study
Advantages• Provides rich, detailed information• Lends to understanding of processes,
interactions and context• Uses different methods of data collection
(interviews, observations, document analysis, etc.)
Limitations• Applicable only to site or event under study
(limited generalizability)• Requires extended time and effort
In-depth study of a single unit- can be a person, organization, program, etc.
Sometimes referred to as “telling a story”
When to Use Case Study*
• Is the program is unique?• Is it premature to establish impact
measures?• Is there unexplained variation in
program impacts?• Will understanding the program
implementation in detail help design future programs?
*E. Balbach. Using Case Studies to do Program Evaluation. (1999) California Department of Health.
MethodologyDirect Observation
Advantages• Helps to understand the social, physical,
political, and economic context in which participants live
• Can provide important information that was previously unknown
• Observe participants in their natural settings• Complements other forms of data collection
(triangulation)
Limitations• Time consuming/resource intensive• Difficult to document everything- may have to
rely on memory• Objectivity is a challenge (observer bias)• Observer effect
Observing the participants in their natural environment and setting- either as becoming a part of the setting or being strictly an observer
Issues to Consider
• Purpose of research (re: evaluation)• Subjective method (reliability and validity)• Researcher bias• Labor intensive• Cost
Words fly in flocks…
Example:
Assessing the Impact of the Mississippi Healthy Student Act: Year Four Report
Questions
Contact us at:[email protected]
duwww.kidscount.ssrc.msstate.
eduTel: 662-325-7127
Connie Baird Thomas, PhD.Tel: 601-407-2742Email: [email protected]
Linda H. Southward, PhDTel: 662-325-0851Email: [email protected]
Colleen McKee, MSTel: 662-325-0851Email: [email protected]
Thank You
Top Related