Smart Evaluation Considerations
-
Upload
eurekafacts -
Category
Economy & Finance
-
view
2.110 -
download
1
description
Transcript of Smart Evaluation Considerations
![Page 1: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/1.jpg)
Planning Campaigns with Evaluation in mind
By Jorge Restrepo M.G.A
Eureka Facts LLC
![Page 2: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/2.jpg)
Design• Clearly define the intended outcomes of a program and
ensure that clear measures can be obtained. The key challenges in program evaluation stem from unclear goals or promised outcomes that go beyond the scope of a grant proposal.
• Educational, language and cultural barriers are necessary considerations in good evaluation design.
• Know what data you need that the respondent will not answer, and develop alternative methods of collection or inference.
![Page 3: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/3.jpg)
Design
• The time crunch – this most often impacts design and instrument testing. Carefully consider timelines for data collection- especially when programs require pre-post measures. Consider other methods to make maximum use of the time required for program deployment.
• Pre-post evaluation are complex so consider their use before promising a specific method in a campaign proposal.
• Evaluate outcome not just activity– changes in awareness, knowledge, behavior, are also good indicators of the extent of a campaign’s outcome.
![Page 4: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/4.jpg)
Methods - data collection
Tips for in-person data collection• Protocols and interviewer training are key• Supervision and redundant verifications are
important • Dress-codes and interviewer profile• Compensation/ Investment• Controls• Timing• Response rate
![Page 5: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/5.jpg)
Methods - data collection
Mail surveys• Lists – sample frame• Who is your respondent?• Sponsor, package, timing, incentive• Respondent burden• Education/language• Coding visibly, invisible coding• Flexibility• Timing• Response rate - examine non-response closely
![Page 6: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/6.jpg)
data collectionOnline
• Who is your respondent? - Your sample frame is everything
• Spam filters are a bigger concern than non-response
• Web links create disproportionate sampling• Make web links and pop-ups as unobtrusive
as possible• Respondent fatigue• CANSPAM • COPA
![Page 7: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/7.jpg)
data collectionTelephone
• Who is your respondent? - Your sample frame is everything
• RDD sampling – often no longer representative• Combine methods • No land line households• Reach• Incentives• Cooperation• Do not call registry
![Page 8: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/8.jpg)
Challenges in Analysis• Most distributions – especially in public education
efforts are not normal so common centrality statistics (mean, median) are not very meaningful.
• People often articulate categories (i.e. well, not well) better than continuum scales.
![Page 9: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/9.jpg)
Analysis• Think outside the box – examine
findings spatially
Stars represent respondents to a campaign
![Page 10: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/10.jpg)
Analysis• Think outside the box –
examine findings spatially
Flat findings examined by other means revealed strong differences by county
![Page 11: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/11.jpg)
Simplify constructs• Build your assessment on existing frameworks
and models, but create high level aggregates as well
Mean Levels of Knowledge and Action
Source: Random survey
State
SCNCGAFL
Me
an
8
6
4
2
0
Know ledge
Action
This evaluation measures knowledge and action
![Page 12: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/12.jpg)
High and Low Impact Groups• A powerful tool is the use of classification trees
to identify pockets of a population with high or low impact.
![Page 13: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/13.jpg)
Compare - Always
Mirrored instruments are especially valuable in examining outcomes
Parents can move their child to a different school when their school has a failing grade.
0
20
40
60
80
100
Month
Random survey
Callback survey
Response cards
![Page 14: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/14.jpg)
Segment for Success
Outcomes are rarely evenly spread out throughout a population
![Page 15: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/15.jpg)
Common cross-tabbing analysis
• Ethnicity
• Hispanic Origin
• Gender
• Age
• Education
• Urban locale
• Generational cohort
• Socioeconomic status
• Life stage, career stage
• Primary language
• Geography
![Page 16: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/16.jpg)
Who we are
Eureka Facts brings you the
smart marketing Information
you need to
discover opportunities
maximize results
identify issues
make decisions
reduce market risks
![Page 17: Smart Evaluation Considerations](https://reader035.fdocuments.in/reader035/viewer/2022081602/549871d2b47959514d8b54eb/html5/thumbnails/17.jpg)
Smart Evaluation
www.EurekaFacts.com
EurekaFacts LLC451 Hungerford Drive
Suite 515 Rockville MD 20850
(301) 610-0590