What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the...
-
Upload
duongkhanh -
Category
Documents
-
view
221 -
download
4
Transcript of What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the...
![Page 1: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/1.jpg)
© FSG |
What Makes Evaluating Complexity Different? WEBINAR | MARCH 11, 2015
![Page 2: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/2.jpg)
© FSG |
Panelists
2
Hallie Preskill, FSG
Moderator
Srik Gopal, FSG
Katelyn Mack, FSG
Cris Kutzli, Grand Rapids
Community Foundation
Mona Jhawar, The California Endowment
![Page 3: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/3.jpg)
© FSG |
We hope you enjoy today’s presentation
Today’s slides will be available after the presentation on fsg.org
Enter your question for the panelists
Make the presentation full screen (Press ESC to return)
![Page 4: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/4.jpg)
© FSG |
Webinar agenda
1 About complexity and evaluation
2 Example: Challenge Scholars
3 Example: Building Healthy Communities
4
![Page 5: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/5.jpg)
© FSG |
Thinking about complexity
5
Consider an initiative or program you are supporting and ask: • Are there many components or parts? • Is the issue you’re working on related to other issues or problems? • Is the context continuously changing (e.g., culture, economy, politics,
demographics)? • Are relationships important to success? • Do things change just as you’ve thought you’ve gotten a handle on
them? • Is there a straight path from implementation to outcomes – are the
outcomes predictable? • Is it taking place in multiple locations? • Are many individuals and organizations involved? Across different
sectors? Are there different definitions of success? • Are all the outcomes known or knowable?
![Page 6: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/6.jpg)
© FSG |
What we mean by “complexity”
Complex situations are characterized by high uncertainty and high social conflict…the outcomes of interventions aimed at solving problems under conditions of complexity are unpredictable. So many factors and variables are interacting, many of then not only unknown but unknowable, that there can be no recipe for success.
Michael Q. Patton, 2011, p.90
6
![Page 7: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/7.jpg)
© FSG |
We act differently to address complex problems
7
Complex systems are:
• Nonlinear, always changing
• Unpredictable, never static
• Uncontrollable
• Highly connected and interdependent
• Emergent
Addressing complex problems requires:
• Pattern recognition
• Working with paradoxes
• Acting & learning at the same time
• Unique, evolving solutions
• Connections and relationships
![Page 8: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/8.jpg)
© FSG |
Evaluations of complex initiatives, or within complex environments should:
8
Be adaptive, flexible, and iterative 1
Describe the whole system, including components and connections 2
Strengthen feedback loops and improve access to information 3
Pay attention to context and changes as they occur 4
Look for effective principles of practice in action 5
Identify points of energy and influence, momentum, and power 6
Focus on the nature of relationships and interdependencies within the system 7
Explain the non-linear and multi-directional relationships between the initiative and its outcomes 8
Watch for patterns, both one-off and repeating 9
![Page 9: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/9.jpg)
© FSG |
Important to remember…
9
Complexity is a source of innovation and adaptability supported by diversity and distributed problem solving
We need to be more skilled at expecting, seeing, and responding to surprise…
Brenda Zimmerman, June 2014
![Page 10: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/10.jpg)
© FSG |
Example: Challenge Scholars
10
Srik Gopal, FSG
Cris Kutzli, Grand Rapids
Community Foundation
![Page 11: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/11.jpg)
© FSG |
About the Challenge Scholars initiative
11
Challenge Scholars… • Aims to increase the number of first generation, low-income students
that successfully complete a degree or high quality credential
• Is a partnership between the Grand Rapids Community Foundation and Grand Rapids Public Schools, with many other stakeholders engaged
• Offers a scholarship upon graduating, as well as early supports:
Physical/behavioral health; human services
Instructional/academic support
College and career pathways
Family engagement
![Page 12: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/12.jpg)
© FSG |
Guiding evaluation questions
1 How is a college and career culture emerging within the schools and broader West Side community?
2 How are parents and families being engaged as part of Challenge Scholars?
3 How are students becoming more college and career ready?
4 What does exemplary teaching and learning look like in a college and career culture?
5 How are the community, businesses, and higher education responding to Challenge Scholars?
12
![Page 13: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/13.jpg)
© FSG |
Be adaptive, iterative, flexible
13
Evaluation Design The evaluation plan was co-created with the working group
1
Evaluation Implementation
Data collection methods and sources were prioritized
based on real time client needs
![Page 14: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/14.jpg)
© FSG |
Describe the whole system, including components and connections
14
2
Systems Mapping Activity
Who are the various actors?
What is their connection to
each other and to the key activities?
![Page 15: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/15.jpg)
© FSG |
What we learned evaluating Challenge Scholars
15
• Co-create, iterate, rinse, repeat
• Find ways to have an “ear” to the ground
• Build trust between evaluators and the program team
![Page 16: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/16.jpg)
© FSG |
Example: Building Healthy Communities
16
Katelyn Mack, FSG
Mona Jhawar, The California
Endowment
![Page 17: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/17.jpg)
© FSG |
The California Endowment (TCE) is pursuing a multi-level policy and systems change strategy
Goals include: 1. Community and
environmental change 2. State and local policy and
systems change 3. Increased community
capacity
Healthy Communities
Healthy California
Campaigns
Building Healthy Communities
17
![Page 18: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/18.jpg)
© FSG |
Guiding evaluation questions
1 To what extent and in what ways are TCE structures and processes affecting [our] ability to effective provide oversight, management, and support? Enabling [us] to adapt and respond to [communities’ needs]? Affecting alignment between Healthy Communities and Healthy California?
2 To what extent and in what ways are Healthy Communities and Healthy California efforts aligned and working together to proactively address a specific issue, policy, or practice related to BHC goals?
3 What factors are supporting and hindering power building among residents and youth to advance BHC goals of policy and systems change?
4 To what extent and in what ways are the Hubs and other collaborative structures developing effective partnerships and increasing community capacity to influence policy and systems changes?
5 What changes are being realized at a local and state level as a result of the BHC work?
18
![Page 19: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/19.jpg)
© FSG |
Pay attention to context and changes as they occur
19
Sacramento BHC Youth at Gabriel Brower Rally, 2011
Santa Ana Youth and Residents at City Council Meeting
Sources: FSG. (2013) South Sacramento BHC Case Study. The California Endowment.; Santa Ana BHC.
4
![Page 20: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/20.jpg)
© FSG |
Look for effective principles of practice in action
20
How are collaboratives in BHC communities structured in ways that
contribute to policy and systems change?
We asked and examined…
Rather than…
Do local collaborative structures adhere to a certain set of criteria or requirements that TCE has laid out?
5
![Page 21: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/21.jpg)
© FSG |
What we learned evaluating Building Healthy Communities
21
• Include diverse voices and perspectives in the design and
implementation of the evaluation
• Take (more) time to analyze and test findings, looking for
alternate hypotheses
• Be proactive in discussing and using evaluation findings
![Page 22: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/22.jpg)
© FSG |
Audience Q&A
22
Hallie Preskill
Srik Gopal Moderator
Katelyn Mack
Cris Kutzli
Mona Jhawar
![Page 23: What Makes Evaluating Complexity Different? - fsg.org · PDF fileEnter your question for the panelists Make the presentation full screen (Press ESC to return)](https://reader033.fdocuments.in/reader033/viewer/2022052917/5a9f12fb7f8b9a7f178c533d/html5/thumbnails/23.jpg)
© FSG |