CNCS Evaluation Highlights Carla Ganiel, Senior Program and Project Specialist AmeriCorps State and...
-
Upload
cora-burke -
Category
Documents
-
view
214 -
download
0
Transcript of CNCS Evaluation Highlights Carla Ganiel, Senior Program and Project Specialist AmeriCorps State and...
CNCS Evaluation HighlightsCarla Ganiel, Senior Program and Project Specialist
AmeriCorps State and National
Federal Policy ContextPresidential Administrations Federal Guidance
President Clinton (1993 – 2001)
Government Performance and Results Act of 1993(GPRA)
President Bush (2001 – 2009)
Program Assessment Rating Tool
President Obama(2009 – 2017)
• GPRA Modernization Act of 2010• Office of Management and Budget
Memoranda• M-10 -01 Increased Emphasis on Program
Evaluation• M-12-14 Use of Evidence and Evaluation in
the 2014 Budget• M-13-17 Next Steps in the Evidence and
Innovation Agenda• M-14-07 Fiscal Year 2016 Budget Guidance,
Evidence and Evaluation
Federal Evidence Initiatives
• Tiered Evidence Initiatives – Direct more resources to initiatives with strong evidence
– Study and scale the most promising program models
– CNCS Social Innovation Fund, Department of Education Investing in Innovation Fund (i3)
• Pay for Success– Federal funds invested only after programs demonstrate
results
• Evidence Clearinghouses– Repositories of evidence on existing program models
– CNCS Evidence Exchange, Department of Education What Works Clearinghouse
CNCS Evaluation Requirements
• Code of Federal Regulations (45 C.F.R. §§2522.500-.540 and .700-740)
• Finalized July 8, 2005
CNCS Long-Term Evaluation Agenda
Implement research and evaluation to:• Advance the agency’s mission• Accommodate agency-wide evidence priorities• Illuminate the agency’s most effective policies,
programs and practices
-Gather evidence supporting the intervention-Design/Adopt a strong program-Develop a Logic Model-Create Implementation Materials-Pilot implementation
-Document program process(es)-Ensure fidelity in implementation-Evaluate program’s quality and efficiency-Establish continuous process improvement protocols [Performance Measures - Outputs]
-Develop indicators for measuring outcomes-Conduct pre-/post- intervention evaluation to measure outcomes-Conduct process evaluation[Performance Measures - Outcomes]
-Examine linkage between program activities and outcomes -Perform multiple pre- and post-evaluations (time series design)-Conduct independent (unbiased) outcome evaluation(s) -Conduct meta-analysis of various studies
-Establish causal linkage between program activities and intended outcomes/impact (e.g. Conduct quasi-experimental evaluation using a comparison group , evaluation with random assignment (RCT), regression analysis, or other appropriate study design) -Conduct Multiple independent evaluations using strong study designs-Measure cost effectiveness compared to other interventions addressing same need
Identify a strong program design
Attain strong evidence of positive program
outcomes
Assess program’s outcomes
Ensure effective implementation
Obtain evidence of positive program
outcomes
Building Evidence of Effectiveness
Evidence Informed
Evidence Based
AmeriCorps Grant Making
• NOFO – Levels of Evidence– More points for stronger evidence
– Theory of Change – Effectiveness of intervention
• Evaluation requirements and plans are not scored
• CNCS provides feedback on evaluation reports and evaluation plans after funding decisions have been made
Common Challenges
• Staff Capacity– Understanding evaluation requirements and how to meet them
– Hiring/communicating with evaluators
– Fear
• Cost– Average amount budgeted was $10,000
– Median amount budgeted was $3,000
– Realistic evaluation costs are 15-25% of total program budget
• Timing– Year 1: Planning; Engage external evaluator if applicable
– Year 2: Conduct Evaluation
– Year 3: Analyze and Complete Report; Submit to CNCS
Challenges, continued
• Data collection systems are not adequate– If a program is struggling to collect performance
measurement data, it will struggle with evaluation
– Focus on setting up solid data collection systems in first three years of grant
• Program model is not clearly defined– Multiple, loosely defined interventions
– Lack of standardization in program delivery across sites
– Multi-site, multi-focus intermediaries
Successful Evaluation Approaches
• Focus the evaluation on one (or a few) key research questions
• Budget adequately
• Hire an external evaluator you are comfortable with and stay engaged in the process
• Plan your evaluation with the goal of using results to improve your program
Successful Approaches, continued
• Use existing data collection instruments, processes or administrative data to lower data collection costs
• Build a long-term research agenda designed to increase evidence base over time
CNCS Evaluation Resources
• Evaluation Core Curriculum– Ongoing webinar series
– Courses and other resources online: http://www.nationalservice.gov/resources/evaluation
• Technical Assistance
Other Resources
• Results for America: http://results4america.org/• The Arnold Foundation:
http://www.arnoldfoundation.org/initiatives/• Grantmakers for Effective Organizations: http://
www.geofunders.org/smarter-grantmaking/learn-for-improvement