Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization...
Transcript of Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization...
![Page 1: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/1.jpg)
Combining Utilization Focused and Developmental Evaluation: The Case
of the Aflateen Social and Financial Education Program for Youth
Daniel Shephard & Simon Bailey
Aflatoun, Child Social & Financial Education
The Netherlands
![Page 2: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/2.jpg)
Background: Context
• Aflatoun
• Provides Social and Financial Education to Other Organizations (Social Franchise)
• History
• Started as a small education programme in Mumbai in early 90s
• Secretariat for Global Expansion started in Amsterdam in 2005
• Focus on primary education
• Has partners in over 100 countries and work with over 2 million children per year
• Aflateen
• New curriculum and first experience with age group
• Five year partnership with MasterCard Foundation to roll out to 50 countries
• Period of curriculum development, then piloting, then re development, then roll out
• Mid Term Evaluation
• MC requested a mid term evaluation, not part of original contract
• Provided opportunity for Secretariat to co design evaluation
• Secretariat Perspective
• Wanted approach that reflected the stage of programme we were in (re development)
• Interested in how the process of supporting organizations had worked
• Settled on utilization focused evaluation as a potential option
1
![Page 3: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/3.jpg)
Background: Process and Level of Assessment
• Selected Partnership for Evaluation • External
• Partnership of Utilization Focused External Evaluation team of two
• Internal
• Research Manager
• Partners & Consultants from 8 countries
• Secretariat level: • Assessment of the Aflateen curriculum development and pilot process to help the
Secretariat review and improve its rollout and support of the program
• 3 visits by Evaluator(s)
• (Co)writing
• 2 field visits by evaluators (focus groups with country/regional partners)*
• Surveys and Phone Interviews
• Implementation level (not covered in presentation) • Generating lessons for current and future partners.
• 2 field visits by evaluators (focus groups with country/regional partners)*
• 8 commissioned research projects (2 to 5K projects)
*This element provided data on both the secretariat functioning and program implementation
2
![Page 4: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/4.jpg)
Background: Methodology Combination
• Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and
facilitating use
• Developmental evaluation – Documenting a model in the making and learning from it
• UDFE instead of UFE
3
![Page 5: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/5.jpg)
Simplified Model of Secretariat Social Franchise
4
![Page 6: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/6.jpg)
Brainstorming & Clustering
5
![Page 7: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/7.jpg)
Intended Uses: Course Correction and Improvement
Secretariat (By Topic)
1. The curriculum development process
• Define our desired curriculum development process for the future
2. E-learning
• Create a plan to scale up and broaden e-learning.
3. Partners / dissemination
• Align Aflateen partner support practices with strategic goals
4. Training
• Differentiate training process for Aflateen vs. Aflatoun as needed
5. Evaluation
• A methodology for research calls that improves their program
6
![Page 8: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/8.jpg)
What was the Model Used?
7
![Page 9: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/9.jpg)
Intended Users
Primary Intended USERS
• Curriculum Manager* and Programme Head
• Specialized Staff, Department Head, and Research manager*
• Programme Head, 2 Programme Staff, and Research Manager*
• Training Manager, 2 Programme Staff and Research Manager*
• Department Head, and Research manager*
Other USERS
• Programme Staff
• Are what we doing effective? How to improve?
• Management
• How do we manage programme expansion? How do we change? How do we
secure more funding?
• MasterCard Foundation
• Is this approach applicable to our portfolio of grantees?
*This is the same staff member
8
![Page 10: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/10.jpg)
Actual Users
Primary Intended Users
• Not all primary Users available or full participants, traveling challenges
• Research Manager often mediator or primary participant
• Time pressure reduced input beyond research team
• Evaluators did think contribution was better than most cases!
Programme Staff
• Importance of in person discussions
• Team contributed (even beyond primary Users) and were affected
• Data collection and use process merged
• Often brought out and document tacit processes, perceptions, and practices
Management
• Process Ownership
• Reputation Management
• Follow Up on Recommendations
Mastercard
• Learning about methodology
• Interest in how organization used the approach and changed
9
![Page 11: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/11.jpg)
Finalization of Uses: Process
• External researcher provided recommendation domains based on findings
which were then adapted, replaced or expanded by the Users.
• The final recommendations for use were drafted by Users prior to a final visit by
the researcher
• Recommendations were finalized during the last visit with involvement of the
management and full staff.
• Management determined much of the wording of the final recommendations.
• Donor asked for formal timelines for implementation
10
![Page 12: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/12.jpg)
Finalization of Uses:
Curriculum Development
• Develop a Secretariat strategy to support country-level curriculum integration, including
national events and a guide for partners
• Modify and simplify the curriculum Contextualization Manual through the creation of simple
checklists, a list of most frequent adaptations, and FAQs, to increase use
• Document the process for the development of new curriculum based on experience from
Aflateen, Aflatoun and Aflatot as part of the Quality Assurance process
E-learning
• Disseminate Aflateen Digital learning to existing Aflateen partners
• Disseminate Aflateen Digital learning to youth organizations and other stakeholders who
are not currently partners
• Identify ‘Aflateen Digital Champions’ from amongst the youth who are actively using and
promoting the platform
• Optimize the platform on an ongoing basis via usability testing
Programme Support
• Strengthen national networks and partners that function as country level secretariats for
scale-up of Aflateen
• Strengthen direct linkages between partners regionally and globally by leveraging partner
skills and experience more strategically
• Ensure that Aflateen partners get more differentiated support and Aflateen specific partner
communication
11
![Page 13: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/13.jpg)
Cont’d
Training
• Update the Aflateen training to strengthen content related to facilitation skills for training
youth and then conduct new regional master trainings in 2014
• Expand and strengthen the pool of Aflateen specific trainers
• Revise monitoring and evaluation tools for both Secretariat and Partner led trainings to
facilitate learning
Evaluation (Changed question)
• To prepare and implement a research plan that privileges proof of concept, scaling and
performance while linking partners to external capacity development opportunities
• Complete the creation of the Implementation & Monitoring Guide for monitoring
programme numbers. This should also capture variability of programme implementation
• Produce user-friendly guides and integrate them into programmatic support for
conducting research of various types
12
![Page 14: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/14.jpg)
Actual Implementation of Uses
General Findings
• Some automatic or based on past understanding of issues
• Some instigated via the process itself
• Some came late in process after dealing with new information or having it presented to us
by the evaluator in challenging ways
Process
• Responsibility for driving the process shifted to Management level from research level
• Transition from recommendations with clear timelines to non-timelines in the final report
affected the actualization of recommendations
• To date, more likely to be completed if started in the evaluation or shortly thereafter than if
it was to be started at some time in the future
13
![Page 15: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/15.jpg)
Contributing Factors to Use
Supporting
• Willingness of funder to consider learning as end goal of evaluation
• Allow Secretariat to define its own key questions
• Ability of Lead Evaluator to communicate with both team and funder (in person)
• Dedicated time for reflection for staff to consider what they were doing as a group
• Facilitated time as well as internal discussions
• Length of time allowed for continued progress and incorporation of findings
• Ability to modify evaluation questions to use (ex. evaluation) while maintaining evaluation’s
focus on uses
• Ability to have internal evaluation staff as part of process helped mediate concerns on
both sides and keep process going
Challenging
• Balancing demands of outcome evaluation with aim of UFD evaluation
• Transition from evaluation to use after end of report
• Key responsibilities had moved within the organization
• Keeping focus on the programme vs expansion to the whole organization
Point of Discussion
• Developmental approach as a key requirement for Utilization Focused Evaluation to
achieve change?
14
![Page 16: Combining Utilization Focused and Developmental Evaluation: … · 2014. 3. 26. · • Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071219/6057099ec5442901756aacbc/html5/thumbnails/16.jpg)
Aflatoun, Child Social & Financial Education
Stichting Child Savings International | Po Box 15991 | Spaklerweg 14 | 1096 BA Amsterdam
The Netherlands
P: +31 20 760 1340 | F: +31 20 626 2118 | www.Aflatoun.org
Thanks…. Questions, Clarifications, Discussions
For more information on all Aflatoun evaluation work visit
www.aflatoun.org/evaluation