Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and...
-
Upload
ellen-cobb -
Category
Documents
-
view
222 -
download
0
Transcript of Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and...
![Page 1: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/1.jpg)
Introduction to theSystems Evaluation Protocol
William TrochimProfessor, Policy Analysis and Management
Director of Evaluation for Extension and Outreach
Cornell University
eXtension Webinar on Research and Evaluation
October 21, 2009
![Page 2: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/2.jpg)
Overview
• Systems Thinking• The Systems Evaluation Protocol (SEP)• Integrating the SEP and Systems Thinking• The “Netway” Cyberinfrastructure• Research on Evaluation: The NSF Project
![Page 3: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/3.jpg)
Evolutionary & Ecological Systems Thinking
Every program has a lifecycle (ontogeny), a progression through phases or stages…
…and we can see the emergent evolutionary family “tree” (phylogeny) of programs and their theories
…over time there is an evolution of programs and program theories…
![Page 4: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/4.jpg)
4
The Systems Evaluation Protocol
![Page 5: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/5.jpg)
global (resources/policy)information for deciding how and whether to support this whole category of programs
Systems Evaluation – Stakeholder Analysis
The evidence you want depends on your place in the network of stakeholders
Program
local (delivery/implementation) information that makes running the program easier
![Page 6: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/6.jpg)
Systems Evaluation – Stakeholder Analysis
Government, especially county
Board Onondaga Staff
Program Partners: citizen groups,
agencies, (e.g. Head Start)
Researcher (e.g. Cornell,
SU, SUNY system)
Funder (e.g. federal foundation)
Program Participants (e.g. associations
by topic, geography, teens)
Citizens
Mike(Cornell Extension)
Team Leaders,
Executive Director
Local Government
Other Counties
County Legislators
Future Funders The Environment
The Press
OnondagaCounty
CooperativeExtension
![Page 7: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/7.jpg)
Systems Evaluation – Boundary Analysis
• Key Distinctions– part-whole – local-global
• Definitional Challenges – Program– Project– Activity
• Motivations– Narrow boundaries –program deliverers feel like they are
doing more but program funders feel like there’s “no there there”
– Broader boundaries – funders feel like they are funding something important but program deliverers feel the program description doesn’t capture the nuances
![Page 8: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/8.jpg)
Systems Analysis – Boundary Analysis
• The Goldilocks Criterion– Too narrow – need separate descriptions, logic models,
and evaluations for each program– Too broad – descriptions and models don’t describe the
program in sufficient detail; logic model is difficult to complete because there are too many activities and outcomes that only pertain to specific subprograms
– Just right – the program description and logic model capture both the common nature of the programs and the important details
Just Right!
![Page 9: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/9.jpg)
Systems Evaluation – Program Analysis
• Logic models– Components
• Assumptions
• Context
• Inputs/Resources
• Activities
• Outputs
• Outcomes (short, mid, long)
![Page 10: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/10.jpg)
Developing a Program Model: NYC 4-H After-School Clubs and Special Interest Projects
Context / Assumptions
Inputs Activities Outputs Short Medium Long
Outcomes
Youth Leadership Academy
Public Presentation
ProgramStudents make
public presentations to
a group
Youth interest in
science
Increased interest in
science education
Develop leadership
skills
Curricula
FTE Extension Educators
Partnerships (community, sponsors)
State & National 4-H
Initiatives
Volunteers
Agilent Science & Tech
Volunteer Orientation &
Trainings
Students complete
engineering kits
Increased rates of higher
education & career
goal attainment
Funder mandates time & budgetary constraints
Staff is proficient at train-the trainer skills
A Program Logic Model: NYC 4-H After-School Clubs and Special Interest Projects
![Page 11: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/11.jpg)
Systems Evaluation – Program Analysis
• Pathway models– Come from
• Causal Modeling (Path Analysis)
• Program Theory
• Theory of Change
– Show network of causal linkages– Primary pathways and nodes– “Throughlines”
![Page 12: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/12.jpg)
Short-Term OutcomesOutputsActivity
Systems Evaluation – Program Analysis
Legend: Medium-Term Outcomes Long-Term Outcomes
Increased rates of higher
education & career
goal attainment
Agilent Science & Tech
Public Presentation
Program
Volunteer Orientation &
Trainings
Youth Leadership Academy
Students complete
engineering kits
Students make public
presentations to a group
Youth interest in
science
Develop leadership
skills Increased interest in
science education
A Program Pathway Model: NYC 4-H After-School Clubs and Special Interest Projects
![Page 13: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/13.jpg)
Systems Evaluation – Lifecycle Analysis
• Program Lifecycle– Phases
• Initiation• Growth• Maturity• Translation/dissemination/regeneration
– Similar to the system of phased clinical trials in medicine• Evaluation Lifecycle
– Phases• Description and Rapid Feedback• Relationships and Change• Causation and Effectiveness• Generalizability and Meta-analysis
– Similar to many taxonomies of research methods• Determining the Lifecycles
– program phase– evaluation phase– synchronization of program and evaluation phases
![Page 14: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/14.jpg)
Initiation Development Maturity Translation &Dissemination
Farmer’s Market
EFNEP
Project Bridge
Systems Evaluation – Lifecycle Analysis
![Page 15: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/15.jpg)
Initiation Development Maturity Translation &Dissemination
Farmer’s Market
Program Phase
Evaluation Phase
Project Bridge
EFNEP
Systems Evaluation – Lifecycle Analysis
![Page 16: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/16.jpg)
• The Netway is an internet-based database tool for monitoring programs and their evaluations
• Features of the Netway
– Stores program information
– Enables logic models and pathway models
– Provides platform for developing evaluation plans
The Netway Cyberinfrastructure
![Page 17: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/17.jpg)
Netway = Networked Pathways
![Page 18: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/18.jpg)
The Netway: Tracking Programs
![Page 19: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/19.jpg)
The Netway: Logic Models
![Page 20: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/20.jpg)
The Netway: Pathway Models
![Page 21: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/21.jpg)
The Netway: Evaluation Plans
![Page 22: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/22.jpg)
Research on Evaluation
![Page 23: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/23.jpg)
Research on Evaluation
23
2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Cornell Cooperative Extension funding National Science Foundation Grant No. EREC-0535492, Systems Evaluation and Evaluation Systems plus Supplements
NSF Award #0814364 - A Phase II trial of the Systems Evaluation Protocol for Assessing and Improving STEM Education Evaluation
![Page 24: Introduction to the Systems Evaluation Protocol William Trochim Professor, Policy Analysis and Management Director of Evaluation for Extension and Outreach.](https://reader036.fdocuments.in/reader036/viewer/2022062309/56649e9d5503460f94b9d84b/html5/thumbnails/24.jpg)
Examples and Experiences with the
Systems Evaluation Protocol