2. Evaluation design of the cofimvaba ict4 red initiative - Bridge 2014 version
-
Upload
benita-williams -
Category
Education
-
view
828 -
download
0
description
Transcript of 2. Evaluation design of the cofimvaba ict4 red initiative - Bridge 2014 version
Evaluation Design of the Cofimvaba e-Textbook Initiative
Benita Williams, Mario Marais, Mmamakanye Rampa
Project Aims
• The project aims to improve the teaching and learning environment
• Contribute towards – improved learner results and – the development of 21st century skills among
teachers and learners.
• It also aims to document a model for e-textbook rollout in schools, which could potentially be sustainably replicated in other educational districts.
12 Component Implementation Model
Merryl Ford, CSIR Meraka Institute
SCHOOL ICT ARCHITECTURE
DevicesWireless LAN
Storage and Power
CHANGE MANAGEMENTPeople (District, SMT)
TechnologyProcess
CONTENTStandardsConversion
Creation & Customisation
TEACHER DEVELOPMENTTraining
PreparationIn the classroom
OPERATIONS MANAGEMENT
Logistics Support & Maintenance
Distribution
RESEARCHMasters & PhD Technology R&D
ICT4E R&D
NETWORKWiFi Mesh
Backbone connectivityInternet
MONITORING & EVALUATION
LearnersTeachersSchool
COMMUNICATIONMarketing strategy
Social Media StrategyKnowledge management
COMMUNITY ENGAGEMENT
Learners & ParentsTeachers
Community
STAKEHOLDER MANAGEMENT
District/Circuit officialsLocal leadership
Provincial
PROJECT MANAGEMENTFinancial Management
ProcurementImplementation management
M&E Approach
• Developmental Evaluation• Principles of Utilization Focused Evaluation
• Why? Constantly evolving complex and innovative project
Framework Part 1
• Key Activities– Model Development
• Document Theory of Action & Theory of Change• Design Choice Review• Cost Study
– Monitoring & Learning• Implementation Monitoring
– Evaluation & Learning• Result Review (Reaction, Learning, Individual Behaviour,
Organizational change at level of learner, teacher, school, district, province)
Framework Part 2Component Evaluation QuestionsDocumentation of theory of action and theory of change
1. What is the initial design and significant modifications made to each of the 12 components (version 1 of model) required to address in e-textbook rollout?
Method: Participatory Workshop
Framework Part 2Component Evaluation Questions
Design choice review
Programme focus: 2. What were the significant design choices for each of the components, and what was the impact on relevance, sustainability and achievement of outcomes?Model focus: 3. Which design of the programme components, and programme overall, are likely to yield sustainable implementation by the DBE/ ECDOE ?
Methods: Documentation of Decisions in a decision template. Documentation of lessons in “Learning brief” template
Framework Part 2Component Evaluation Questions
Cost Study Programme focus: 4. What was the total cost of implementing the different designs of the programme ?Model focus: 5. What would the total cost be to implement such a programme on district wide level? (three scenarios, different designs, in comparison to existing LTSM delivery process)
Methods: GeSCI TCO toolProcess modellingScenario BuildingActivity based costing
Framework Part 2Component Evaluation Questions
Implementation monitoring
Programme focus:6. Was the programme components, and programme overall, feasible to implement?Model focus:7. What have we learnt about the implementation of this programme that should be transferred to others?8. Which factors support, inhibit or prevent implementation?
Methods: Textual analysis of Twitter feed, and Whatsap groupPost training trainer feedback formAttendance registersPost-training teacher feedback forms
Quarterly telephonic teacher surveyQuarterly telephonic school surveyDevice usage statisticsSchool functionality checklist
Framework Part 2Component Evaluation Questions
Results Review: Programme focus: 9. Did the programme deliver the anticipated educational outcomes (learning, behaviour change, organizational change) and necessary intermediate outcomes (reaction) at learner, teacher, school, district and province levels?10. What were the unanticipated results?Model focus: 11. What have we learnt about the results of this programme that should be transferred to others?12. How do the achieved outcomes in this project compare to the outcomes 13. Which factors support, inhibit or prevent the realisation of outcomes?
Framework Part 2• Post-training teacher feedback forms• Learner focus groups, Parent focus groups• Quarterly telephonic district interviews
• Teacher session assessment record (Badge tracker)• Teacher baseline questionnaire and follow up
• Learner 21st century assessment assignment with sample of learners• Assessment of learner marks (Matric results, ANAs) in comparison with other
matched schools • Ethnographic description of classroom observations • School Functionality checklist • Impact Story template• School leavers tracer study
Issues• Initiative is evolving, the M&E framework needs to be
adapted within budgetary and practical constraints • Operationalizing the evaluation required that other
component managers must help with the collection of data, interpretation of the data
• Evaluation information collected on some tools may have immediate uses for other components – how best to make it available
• Capturing learning in learning templates is useful for the team members who are more practice than research oriented.
• Challenges of collecting and packaging the information for purposes of research, evaluation, and sharing in media
Benefits
• The benefit of the evaluation approach include providing continuous feedback on various project components to the various evaluation users whilst also collecting information that could later be consolidated into a model.
• There is flexibility to incorporate methods with different ontological foundations e.g. RCT and thick ethnographic description in order to meet the needs to the diverse set of users.
• The framework is now also being adapted for the evaluation of other similar ICT programmes rolled out by the Meraka Institute