Literacy, Language and Learning Initiativepdf.usaid.gov/pdf_docs/PA00HV17.pdf · Performance...

Literacy, Language and Learning (L3) Initiative Cooperative Agreement: AID-696-A-11-00006

Literacy, Language and Learning Initiative

Performance Monitoring Plan (PMP) August 8, 2011 – August 7, 2016

Submitted by Education Development Center, Inc.

Cooperative Agreement: AID-696-A-11-00006

Literacy, Language and Learning (L3) Initiative


2

Contents Acronyms and Abbreviations ........................................................................................................................ 3

INTRODUCTION ............................................................................................................................................. 4

RESULTS FRAMEWORK.................................................................................................................................. 4

OVERVIEW OF PERFORMANCE MONITORING APPROACH ........................................................................... 6

PERFORMANCE MONITORING INDICATORS ................................................................................................. 6

PERFORMANCE EVALUATION ..................................................................................................................... 14

SAMPLING STRATEGIES ............................................................................................................................... 18

CRITICAL ASSUMPTIONS ............................................................................................................................. 18

Programmatic assumptions .................................................................................................................... 18

Assumptions with respect to targets ...................................................................................................... 19

PLANS FOR USING AND COMMUNICATING PERFORMANCE INFORMATION ............................................. 21

MONITORING AND EVALUATION STAFF ..................................................................................................... 22

DATA SYSTEMS ............................................................................................................................................ 22

DATA QUALITY ............................................................................................................................................ 24

Appendix A: Key activities (by sub intermediate result) ............................................................................. 26



3

Acronyms and Abbreviations ADS USAID Automated Directives System COP Chief of Party DQA Data quality assessment EDC Education Development Center, Inc. IR Intermediate result MINEDUC Ministry of Education M&E Monitoring and evaluation NGO Non-governmental organization PMP Performance Monitoring Plan SO Strategic Objective TA Technical assistance USAID United States Agency for International Development USG United States Government



4

INTRODUCTION Rwanda aims to participate in the global knowledge-based economy, but it faces a huge gap between the skills required for such participation and the low literacy, numeracy and English skills of its population. The Ministry of Education has developed an array of policies to close this gap, but lacks the structures and technical and human resources to implement them. The Literacy, Language and Learning Initiative (L3) is designed to fill this gap by assisting the Ministry to implement a comprehensive program of support that will produce significant gains in early literacy/numeracy and English as a second language. L3 will work with Rwanda’s Ministry of Education (MINEDUC) to improve students’ reading and mathematical skills in grades one to four, as well as their English language proficiency. Working in collaboration with the MINEDUC, USAID and technical partners, the L3 Initiative works with pre-service and in-service facilitators to introduce proven reading and mathematics teaching strategies, and with community volunteers to support struggling learners. The initiative also aims to improve the availability and use of innovative reading and math instructional materials and to reinforce teachers’ and students’ reading, math and English language skills.

RESULTS FRAMEWORK The strategic objective of the L3 Initiative is to strengthen teaching and learning so that all children leave primary school with solid literacy and numeracy skills. The Results Framework on the following page outlines the five intermediate results that will contribute to the achievement this objective and shows hypothesized cause-and-effect linkages between these results and the overall objective. It also lists the sub intermediate results required to achieve the stated objective. The framework will serve as the primary L3 management tool, enabling the management team to define and organize its annual work plan around the desired end results, to gauge progress toward the achievement of results and to make appropriate adjustments to relevant programs and activities. (See appendix A for a list of activities that will be initiated to support each of the three sub intermediate results.) The framework will also function as effective communication tool, enabling the L3 team to succinctly capture and communicate the key elements of its strategy for improving the quality of teaching and learning.


USAID Goal: Improved reading skills for 100 million children in primary grades by 2015

USAID Rwanda Strategic Objective: Strengthen teaching and learning so that children leave primary

school with solid literacy and numeracy skills

Intermediate Result 3:

Support for English

3.1 Student access to supplementary ESL materials expanded

Intermediate Result 2: Improved availability

and use of instructional materials and

technology

2.1 Student access to supplementary reading and math materials and supporting technologies expanded

1.1 National standards for literacy, numeracy and ESL established.

Intermediate Result 1: Improved quality of

teaching

5.1 Girls’ access to gender-friendly materials and instructional practices increased

4.1 Ministry technical capacity to lead literacy/numeracy reform strengthened

Intermediate Result 5: Improved Equity in

Education Intermediate Result 4:

Strengthened Ministry capacity to improve the

quality of education

1.5 Teacher morale and working conditions improved

1.4 Teacher Training Colleges transformed into Centers of Excellence for Literacy/Numeracy

2.2 Students in isolated and low performing regions have access to additional materials

1.3 Effective literacy/numeracy instructional practices implemented in P1 –P4

1.2 Literacy/numeracy school based mentoring program implemented nationwide

3.2 Interactive English audio programs used regularly by students and teachers

3.3 Teachers’ English Language Skills improved

4.2 Student assessment systems and processes strengthened

2.3 Culture of reading begins to take root

5.3 National and district initiatives are support improved girls’ literacy implemented

4.3 System for evaluating changes in teachers’ instructional practices developed and implemented

5.2 Student teachers and parents trained in inclusive practices


OVERVIEW OF PERFORMANCE MONITORING APPROACH We propose a monitoring and evaluation (M&E) system1 that will provide a continuous and systematic flow of data on the implementation of the L3 Initiative. It will track key outputs and outcomes and provide information as to the quality of the activities undertaken. The data will provide the L3 management team and key stakeholders with up-to-date information about the progress of L3-related activities and be used to adjust and improve L3 delivery.

The M&E system is built on the following underlying principles:

• Participatory approach. The L3 monitoring and evaluation (M&E) system is geared toward engaging staff, stakeholders and beneficiaries in all aspects of implementation.

• Data quality. The M&E system places a heavy emphasis on ensuring data quality through activities including the documentation of data collection procedures, training of data collection and management staff, triangulation of data on key indicators, and monitoring or auditing of collected data. (See section: Data quality.)

• Gender equality. Wherever appropriate data will be disaggregated by sex. This will enable the L3 management team to monitor how inputs are received differently by females and males and if necessary to make appropriate adjustments.

• Performance management. M&E data will be reviewed regularly during management meetings with a view to influencing decision-making and resource allocation. It will be routinely discussed with implementing partners and be regularly shared with wider stakeholders, including the Government of Rwanda and USAID. (See section: Plans for using and communicating performance information)

• Flexibility and localization. The M&E system will be flexible enough to adapt to L3 Initiative partners’ and organizational beneficiaries’’ unique styles of working. L3 M&E staff will provide extensive and ongoing M&E technical assistance to implementing partners and organizational beneficiaries in how to collect and report data, and ground its approach in local expertise.

PERFORMANCE MONITORING INDICATORS Table 1 on the following page outlines the indicators that will be used to monitor the performance of the L3 initiative. The table follows USAID policy guidelines (ADS 2032 and Performance Management Toolkit,3). Each indicator is accompanied by the unit of measure, the data collection method; the frequency and schedule of data reporting; the data sources, the

1 This PMP was developed in alignment with USAID ADS 203 guidance. 2 USAID ADS 203, http://www.usaid.gov/policy/ads/200/203.pdf 3 Performance Management Toolkit, April 2003, http://www.dec.org/pdf_docs/PNACT871.pdf.

http://www.dec.org/pdf_docs/PNACT871.pdf



7

partner responsible for collecting and reporting the data, the disaggregation and the annual targets.

The indicators measure the most important and relevant aspects of the L3 Initiative’s performance. Most are standard indicators from the Foreign Assistance Framework (F-List) and are indicated as such. Others are customized and reflect the specific foci of the L3 initiative.

The data generated will provide both USAID and the Rwandan Ministry of Education with the means to make critical decisions about how resources should be allocated across and within sectors and/or how implementation should be modified to improve effectiveness.


Table 1. Performance Monitoring Indicators and Data Collection Plan

Indicator type Indicator description

Target 2012 Target 2013


Target 2016

Disaggregation

Fre-quency

Data source/ responsible Notes

F-Indicator 3.2.1-14: Number of learners enrolled in primary schools and/or equivalent non-school based settings with USG support (Type: Output)

Number of P1 to P6 learners benefitting from USG support (direct or indirect)

Tl 2,436,935

2,519,580 2,606,125 2,696,803 2,791,860 Gender,

Direct beneficiari

es and Indirect

beneficiaries

Annual

MINEDUC enrolment Statistics (projections based on 2011 figures, with extrapolations)

Includes students receiving direct and indirect support

F 1,240,400 1,282,466 1,326,518 1,372,673 1,421,057

IND 2,434,212 2,477,191 2,148,281 1,214,375 694,527

Dir 2,723 42,389 457,844 1,482,428 2,097,333

F-Indicator 3.2.1-27 Proportion of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text. (Type: Outcome/Impact)

Proportion and number of (P3) students who meet the MINEDUC threshold for reading with fluency and comprehension after three years of L3 support

M. 40.5% 77634

NA NA

75% 148961

75% 150734

Gender,

Twice (2015/6)

Ministry Fluency Assessment in Rwanda Schools

Baseline: 2011 EGRA Note: This calculation is different than the overall calculation for Goal 1. For that calculation, the % of students with improved reading skills in P3 (see concept paper) is applied to the overall number of students in P1 to P6, over the life of the project, receiving direct or indirect support, without double counting. Calculation is based on 75% of all projected P3 students as all students will be enrolments for those particular years as all P3 students will be receiving either direct or indirect

F 46.7% 92800

75% 154422

75% 156260

TOTAL

44% 170434 75% 303383

75% 306994



9




Target 2016

Disaggregation

Fre-quency


support from L3. Assumes 50.9% of students are female (MINEDUC 2011 Statistics)

Custom-Indicator Proportion of students who, by the end of grade 5, are able to read and demonstrate understanding as defined by a country curriculum, standards, or national experts(Type: Outcome/Impact)

Proportion and number of grade 5 students who, by the end of primary school, are able to read with comprehension according to Rwanda’s curricular goals and standards.

F 69.5% (115635)

NA NA NA

75% 165596

Gender Once (2016,)

Ministry Fluency Assessment in Rwanda Schools Project impact evaluation grade 5

Calculation is based on 75% of all projected P5 students as all students will be enrolments for those particular years as all P3 students will be receiving either direct or indirect support from L3. Assumes 50.9% of students are female (MINEDUC 2011 Statistics)

M 61.9% (99348)

75% 159740

T 214984 75% (325336)

Result 1: Improved quality of teaching

Custom- Indicator Number of learners receiving reading interventions at the primary level (Type: Output)

1.1 Number of P1 to P3 students who listen regularly to Kinyarwanda or English reading interactive audio programs

2043. 31,791 343,383 1,111,821 1,573,000 Gender Annual Listenership records

To avoid double counting, targets refer to number of learners in schools benefiting from interactive audio instruction; Assumes 75% of students regularly use programs i.e. 75% of direct each year

F-Indicator 3.2.1-31: Number of teachers/educators/teaching assistants who successfully completed in-service training or received

1.2 Number of mentors/teachers/head teachers/ trained by L3 initiative

130 13,023 16,975 18,901 19,277 Gender, District Annual

Listenership records (for teachers); attendance records for mentors, head teachers, TTC

For 2012, figures reflect teachers, mentors, head teachers, etc. receiving face to face training



10




Target 2016

Disaggregation

Fre-quency


intensive coaching or mentoring with USG support

instructors from L3

F 3.2.1-32 Number of teachers/educators/teaching assistants who successfully completed pre-service training with USG support (Type: Output)

1.3 Number of student teachers trained by L3 VSO Literacy Specialists assigned to TTCs

30 1,203 4,010 4,411 5,213 Gender Annual

Enrollment records (student teachers)

For 2013, includes student teachers trained in Writer’s workshops, as well as those in new courses co-taught by VSO specialists working with L3 Norma to Email Mary to get numbers.

Custom Indicator (Type: Outcome/Impact)

1.4 Proportion of teachers in SBMP using new literacy/ numeracy instructional practices and materials

0 50% 55% 60% 70% Gender, Zone Annual

Criterion-referenced Observation Sheet

Tool to be developed

Custom Indicator (Type: Output)

Number of PTAs or similar ‘school’ governance structures supported

6 90 738 1,938 2,838 Zone Annual PTA/PTC log of initiatives


Number of PTAs/PTCs that undertake initiatives- to support increased teacher motivation with USG assistance

6 90 738 1,938 2,838 Zone Annual PTA/PTC log of initiatives


Result 2. Improved availability and use of instructional materials and technology F-Indicator 3.2.1-33 Number of textbooks and other teaching and learning materials (TLM) provided

2.1 Number of learning and teaching materials produced and distributed

8,500 431,071 339,710 387,194 10,700 Type Semi annual

Distribution records for print and audio materials, supporting

Disaggregated by type of materials and source; Cost share documentation



11




Target 2016

Disaggregation

Fre-quency


with USG assistance (Type: Output)

technologies, etc.

(in case of BBF donations)

2.2 Number of different book/story titles and problem sets authored locally

91 175 240 170 80 Type Semi annual

List of works produced to support new Kinyarwanda and English reading programs; list of problem sets; list of works produced via Writer’s Workshop

Indicator F3.2.1-36 Number of schools using Information and communication technology due to USG support (Output)

2.3 Number of schools receiving MP3 players, video players or other technologies

6 90 738 1938 2714 Region Annual Distribution records

2.4 Number of schools using MP3 players, video players or other technologies

6 90 738 1938 2714 Region Annual Distribution records

Result 3. Support for English

Custom Indicator (Type: Outcome/Impact)

3.1 Improved English language skills among primary teachers benefiting from L3 Initiative support (percentage increase over baseline)

Baseline NA +10% TBD TBD Gender ,

Zone TBD

English-language testing completed in collaboration with DFID

Results to be provided by DFID testing, with additional P1-P3 teachers added to sample if needed to provide robust sample. Results to be provided in 2015 and 2016 if decision is made to extend DFID testing to these years.

Result 4. Strengthened Ministry capacity to improve the quality of education F-Indicator 3.2.1-37: Number of impact evaluations

Number of impact evaluations conducted

0 0 1 0 1 NA

Ministry National Reading

2014: Small Impact Evaluation.



12




Target 2016

Disaggregation

Fre-quency


conducted (Type: Output)

assessment program

2016: Nationally representative sample.

4.2 Number of completed reports submitted to MINEDUC on the impact of L3 initiatives, during grade-specific action research cycles, as well as during nationwide roll out

0 0 0 1 1 Annual Report submitted to Ministry

Baseline: 2011 National assessment instruments (LARS, EGRA/EGMA), programs and policies

Result 5. Improved Equity in Education

Custom Indicator (Type: Outcome)

5.1 Average increase in parity of students completing P3 in L3 supported schools4

F`

83.9%

NA 5% increase

in parity from 2012

10% increase in parity from

2012

15% increase in parity

from 2012

Gender, Zone,

(urban/ rural)

Annual

Annual EMIS Data (parity index)

Baseline is 2011 EMIS data from Ministry

M 81.2%

Difference of 2.7%

Custom Indicator (based on UNESCO indexes) (Type: Outcome)

5.2 Average increase in parity of student’s performance in literacy at P3 level in L3 supported schools, as measured by % of students who meet Rwandan minimal standards for fluency (correct words per minute)

NA NA

10% increase in parity from

2012

15% increase in parity

from 2012

Gender, Zone

Twice (2015, 2016)

EGRA fluency scores, analyzed to indicate % of students who meet minimal standards for fluency (Note change)

Baseline is 2011 EGRA data (note change) NOTE: Need to code schools in 2011 EGRA baseline according to rural and urban, according to Ministry definition, and then redo baseline

F 46.7%

M 40.5%

Gender differenc

e 6.2% Gender

5.58% Gender 5.27%

U

4 This indicator assumes that the Ministry’s information management system is able to extract the required data.



13




Target 2016

Disaggregation

Fre-quency


R

assessment according to those definitions

Rural/urban

Difference

Rural/urban


PERFORMANCE EVALUATION The L3 Performance Monitoring Plan (PMP) will provide descriptive and normative data to measure progress with respect to expected results. It weaves together output and outcome/impact evaluations; the latter designed to provide rigorous data as to changes in students’ reading and numeracy skills - as well as teacher’s pedagogical content knowledge and instructional practices - that can be attributable to L3 interventions.

OUTPUT EVALUATION The M&E team will collect annual data to determine whether the L3 initiative has met targets with respect to the provision of services, materials and resources. The outcome indicators are identified in the performance monitoring table (Table 1) and summarized in the box 1 above.

OUTCOME EVALUATION5 The performance monitoring table also identifies the outcomes that will be evaluated.6 This includes measuring whether, over time, disparities in student completion rates are decreasing between rural and urban zones and whether teachers’ English language skills improve over time.

Good data on student learning outcomes will provide the basis for determining the value added of L3 interventions. For that reason, the design of the L3 Initiative includes impact evaluations to provide counterfactual evidence of: 1) the improvements in students’ reading and numeracy skills and 2) changes in teachers’ knowledge, attitudes and instructional practices as a result of L3 interventions.

Student Impact Evaluation (Counterfactual) The student impact evaluation will be carried out at two different points and for two different purposes:

5 Outcome is defined as the state of the target population or the social conditions that a project is expected to have changed. From Rossi, Lipsey and Freeman, Evaluation: A Systematic Approach, SAGE Publications, 2004, p. 204.

Box 1: KEY OUTPUT QUESTIONS 1. How many activities are completed? 2. How many participants were

engaged/trained? 3. How many materials were

disseminated and used?

Box 2: KEY IMPACT QUESTIONS 1. Does teachers’ pedagogical content

knowledge increase as a result of L3 initiatives?

2. Do teachers’ reading and math instructional practices change as a result of L3 initiatives?

3. Do students’ reading and math skills change as a result of L3 initiatives?

4. What differences are demonstrated between: a) girls compared with boys; b) students in urban and rural areas?

5. Are students’ meeting the Ministry’s targets in terms of reading comprehension?



15

• As part of the “field test” of grade-specific materials and trainings. This impact evaluation will provide the Ministry with empirical data on the value added of L3 materials and teacher trainings on students’ learning prior to roll out. As the materials for each grade level are being developed, the L3 initiative will work with the Ministry to identify “action research” sites for the field test (urban and rural primary schools in close proximity to Kigali), as well as matched “control sites”. Data on students’ reading and math skills will be collected in both sites at the beginning, middle and end of the year. The data collected will enable the L3 Initiative to provide the Ministry with empirical evidence of the value added of L3 literacy and numeracy resources and trainings. The proposed calendar for the collection of impact data for the grade specific field tests is as follows:

o 2012: P1, P2 o 2013: P3 o 2014: P4

• As part of an overall evaluation of the impact of L3 supports during the national roll-

out process. In 2015 and 2016, the L3 initiative will collect data on P3 students’ reading and math skills in a nationally representative sample of schools/students and compare the results with the 2011 baseline studies (EGRA/EGMA7 and/or LARS). If necessary, the L3 Initiative will support the collection of data in additional schools so as to examine the correlation between L3 inputs and students’ reading skills. The results will permit USAID and the Ministry to identify the proportion (and by extrapolation, the number) of students who have met fluency and comprehension thresholds or benchmarks for P3. The evaluation will also to examine the correlation between students’ learning results and the degree to which they have been benefited from L3 support.

In 2016, the L3 Initiative will also examine the results of the P5 LARS and/or EGRA to determine the proportion (and by extrapolation, the number) of students who, by the end of P5, are able to read with comprehension, according to their countries' curricular goals. If necessary, the L3 Initiative will support the collection

7 See Early Grade Reading/Math Assessment (EGRA/EGMA) Study carried out by RTI International and the 2011 Learning Assessment for Rwandan Schools (LARS) assessment carried out by the MINEDUC in 2011.The decision to use an Early Grade Reading Assessment or the LARS to measure progress in students’ learning will be made in collaboration with the MINEDUC.

Table 2. Impact testing with the national rollout Year of entry

into P1 Access to L3 program support

2011 2012 2013 2014 2015 2016 Baseline P3 Cohort 1 (2012) -- P1, P2 P3 P4 Cohort 2 (2013) P1 P2 P3 P4 Cohort 3 (2014) P1 P2 P3 Cohort 4 (2015) P1 P2 Cohort 5 (2016) P1



16

of data in additional schools so as to examine the correlation between L3 inputs and students’ reading skills.8

SPSS or STATA will be used to analyze impact data. Simple statistical analysis will be performed to assess differences between participant and comparison groups, as well as differences among groups (boys versus girls, students in urban versus rural areas, etc). To detect the independent contribution of each variable in the model the team will employ more advanced inferential statistical techniques including Ordinary Least Squares regression to control for variables that could explain differences (such as gender, zone) and to examine changes during the L3 Initiative.

The student impact evaluation will be conducted in close collaboration with Government of Rwanda partner, using MINEDUC verified and approved tools and instruments. A simple system for ensuring that student data are analyzed and reviewed in a timely manner, and disseminated to institutions at the central level as well as TTCs and District Education Offices, will be developed. (See Section: Plans for Communicating and Using Results.)

Teacher Impact Evaluation (Counterfactual) The teacher impact evaluation will measures changes in teachers’ knowledge base (cognitive changes), as well as changes in their reading and mathematics pedagogical practices (immediate behavior change).

In the case of cognitive change, a sample of teachers (in L3 supported schools as well as in control schools) will be surveyed using questionnaires, face-to-face interviews, SMS technology, or other methods as appropriate. Baseline data will be collected in 2012 (prior to the start of the national rollout) and annually after that in L3 treatment and control schools. The results will be disaggregated by gender, zone, and grade level.

To determine whether L3 initiatives result in changes in teachers’ instructional practices, L3 program staff will work with the MINEDUC to develop appropriate criterion-referenced instruments and an accompanying research protocol. The Standards-based Classroom Observation Protocol (SCOPE) will serve as model for the MINEDUC to consider (see text box). As is the case

8 The PMP assumes that the Ministry will continue its commitment to building and implementing a strong student learning assessment program to monitor progress in reading and mathematics at the P3 and the P5 level, using either EGRA/EGMA or LARS or some combination of the two, and that the L3 Initiative will be able to use this assessment to monitor progress.

Box 3: SCOPE The SCOPE was developed by EDC with USAID funding for Egyptian Education Reform Program and adapted subsequently for Mali, Benin and the Philippines. It evaluates the same practices captured by the LOS: teachers’ use of instructional time, ability to promote problem solving and critical thinking and to foster a collaborative, equitable learning environment. It also examines how learning materials and instructional technologies are used.



17

with the measure of cognitive change, baseline data will be collected in 2012 (prior to the start of the national rollout) and annually after that in L3 treatment and control schools. The results will be disaggregated by gender, zone, and grade level.

The teacher impact evaluations will be conducted in close collaboration with MINEDUC partners at all steps and the results made available to National Mentors, School-based Mentors, KIE and TTC instructors and District Education Offices. The co-development of tools and the implication of Ministry representatives in the data collection and analysis process will help ensure the sustainability and institutionalization of the tools and systems and contribute to IR 4 (Strengthen Ministry Capacity).

PROCESS EVALUATION The L3 Initiative will collect data annually on beneficiaries’ and partners’ perceptions of the quality of supports and services provided. The process evaluation will include two parts:

1. interviews with beneficiaries, implementing partners and MINEDUC representatives and to monitor perceptions of how the L3 initiative is being implemented (process). The interviews will follow the semi-structured protocol described above, to ensure consistency and comparability of the collected data.

2. a review of challenges identified in the implementation of L3 activities, modifications proposed and lessons learned (implementation process). Data will be collected via questionnaires and semi structured interviews9 with L3 managers, implementing partners and L3 beneficiaries.

The data collection process will allow L3 staff to triangulate data from various sources, including implementing partners and organizational beneficiaries. Participants’ feedback will be aggregated, to protect their confidentiality.

Findings from the annual process evaluation will be merged with findings from the output and outcome evaluations and incorporated into the L3 initiative annual report. They will also be discussed during senior management meetings and used to improve program quality and effectiveness.

9 The use of semi-structured interviews will assure consistency and comparability of the collected data.

Box 4: SAMPLE PROCESS QUESTIONS 1. How are proposed activities being

implemented? 2. What are beneficiaries’ or partners’ perceptions

of the quality or value-added of L3 materials, activities or services?

3. How is the project tailoring the intervention to the cultural context?



18

SAMPLING STRATEGIES The L3 Initiative will work with over 2500 primary schools. It is not possible, given the target population, to collect data from every teacher and student. When properly conducted, a probability sample provides reliable information with a very small margin of error for the whole population. The smaller sampling operation lends itself to the application of more rigorous controls, thus ensuring better accuracy. These rigorous controls can serve to reduce non sampling errors such as interviewer bias and mistakes, non response problems, questionnaire design flaws, and data processing and analysis errors.

For that reason, and in the interest of using available resources as effectively as possible, whenever the target population exceeds 200, the L3 Initiative will establish a properly designed probability sample as a means of inferring information about the entire population. In the case of student assessments, for example, the L3 Initiative will establish a sample of schools (control/treatment) in targeted regions, as well as a random sample of students in these schools. In the case of interactive audio/radio listenership, L3 will establish samples at the provincial level and use data from these schools to extrapolate overall listenership rates for each province.

To the extent possible, probability sampling (simple random or stratified samples) will be used, to allow the L3 Initiative to generalize from the sample to all members of the L3 population group.

CRITICAL ASSUMPTIONS Programmatic assumptions The achievement of the results outlined in the results framework, and of the targets specified in the performance monitoring table, is predicated on a number of critical assumptions. These include:

• The Ministry’s commitment to establishing in place a strong, sustainable school-based mentoring system where high performing Rwandan educators are trained to be effective mentors and released from other responsibilities to be fulltime mentors.

• The Ministry’s willingness to have the focus of the school-based mentoring program expanded to include effective early grade literacy and numeracy instructional practices.10

• The Ministry’s willingness to consider implementing a rewards- or incentive-based professional development system, whereby teachers who engage in professional development experiences and are able to demonstrate that they have implemented the

10 The Ministry’s current plan calls for the school-based mentoring program to focus exclusively on reinforcing teachers’ English language skills.



19

instructional practices presented in their own classrooms, are rewarded professionally or otherwise. Without such a program, it is difficult to imagine teachers making the effort to participate in local professional development sessions or to implement the new ideas presented in their classroom.

• The ability to form neighbourhood clusters with schools that are within a manageable walking distance from each other. The neighbourhood cluster constitutes that primary vehicle for re-energizing local professional development initiatives and for nurturing a culture of sharing and mutual support among the teaching population. If teachers are not able to attend the cluster meetings, and avail themselves of the technology (video-based self directed learning modules), the ability of the L3 program to significantly impact their instructional practices will be compromised.

• The ability to train National Mentors and School-based mentors to be true “facilitators” of teacher learning, as opposed to the more traditional work-shop leader/dispenser of knowledge. Although Ministry discourse differentiates “facilitation” from “transmission”, there has not necessarily been a corresponding shift in practices in the field. If the L3 Initiative is not successful in creating a clear understanding of the role of Master trainers and school-based mentors in helping teachers build their own understandings of effective reading and mathematics teaching and learning there is a danger that the current “bottom-up” and teacher centered professional development program may develop into a tradition cascade model.

Should any of these assumptions become a reality, the targets outlined in the PMP table will be seriously compromised

Assumptions with respect to targets

The teacher training targets in the PMP table are based on the census data provided by the MINEDUC in December, 2010 and number of teachers projected to have used at least 24 hours (equivalent of 3 days) of in-class distance-education training through interactive audio. The targets assume an average of 3.5 P1 and P2 teachers per school and 2.5 P3 and P4 teachers per school. They also assume that the radio programs will be made available to all schools in the country the first year that they are ready for broadcast/usage. Student targets are based on an average of 45 students per class.



20

Table 3. Proposed broadcast schedule

Table 4. Listenership targets, based on above table (75% listenership rate)

Yr

Schools

No. Teaches per grade level No. students If 75% use program P1 P2 P3 P4 Total P1 P2 P3 P4 Total Teachers Students

2012 6 22 16 38 1532 1118 2650 29 1987

2013 2663 9853 7190 17043 679864 496117 1175981 12782 881986

2014 2716 10050 7334 5976 17384 693461 506039 412328 1611829 13038 1208872

2015 2771 10251 7481 6095 5541 17732 707330 516160 420575 382341 2026406 13299 1519805

2016 2826 10456 7630 6217 5652 18086 721477 526483 428986 389988 2066934 13565 1550201

Note: Based on an average of 3.7 P1 teachers, 2.7 P2 teachers, 2.2 P3 teachers and 2 P4 teachers per school and an average of 69 students per teacher. Number of schools increases by an average of 5% per year.

Finally, the targets for instructional resources assume that the Ministry will support the production of the following resources:

Table 5. Numbers of teaching and learning materials distributed to P1/P4 students and teachers

IRI broadcast schedule

Grade Year 1 Year 2 Year 3 Year 4 Year 5 P1& P2 Radio Radio and MP3 MP3 MP3 MP3

P3 Radio MP3 MP3

P4 Radio MP3

Teaching and learning materials Year 1 Year 2 Year 3 Year 4 Year 5 Teachers' manuals 0 28 380 14 190 11 352

Decodable texts 0 383 130 255 420 255 420 Read aloud texts 0 9460 9460 9460

Books for portable library 2500 10 000 10 000 10 000 10 000 Books for book fair 0 0 50 000 100 000

Solar radios/MP3 players 5400 0 0 0 0 Video viewing systems (iPOD + Pico projector) 0 56 270 431 300

Cell phones 0 45 370 531 400 TOTAL 7900 431071 339710 387194 10700


PLANS FOR USING AND COMMUNICATING PERFORMANCE INFORMATION Program management The indicators will serve both formative and summative functions. The results will allow the L3 management team to track progress toward planned results and to make informed decisions about program implementation and resource allocation. They will also provide stakeholders with accurate and reliable summative information to evaluate progress and goal achievement, thus ensuring accountability for project implementation. Communication and dissemination of results Relevant results will be share with USAID and governmental partners on a quarterly basis, as part of the project’s quarterly reporting requirement. In the annual report (final quarterly report of the fiscal year), the L3 Initiative will include a summary of progress achieved for the year. Also, on an annual basis, the L3 Initiative will share progress reports with all stakeholders, both in the form of a progress report and in the form of annual results sharing meetings. Impact evaluation results (student learning assessments, teacher instructional practices, etc), will be shared as widely as possible, with a commitment to full and active disclosure. In addition, because improving students’ early literacy and early numeracy skills is a priority for many countries in sub Saharan Africa, the L3 Management team will encourage the Ministry to consider presenting baseline and impact data at regional and international conferences. The lessons learned in Rwanda can contribute to extending the overall body of research in best practices. Table 6: Overview of communication and dissemination of L3 results

Activity Timing Staff responsible

Regular meetings with implementing partners

Once every two weeks/quarterly L3 Management Team

Regular site visits Monthly L3 technical staff

Update on progress with respect to indicators

Quarterly (with quarterly report) L3 COP

PMP review meeting and planning session with implementing partners

Annually L3 COP/DCOP

Review meeting with stakeholders to disseminate L3 results

Annually L3 COP

Production/dissemination of annual progress report

Annually L3 COP

Dissemination of results and lessons learned at regional/international conferences

Periodically L3 Management Team



22

MONITORING AND EVALUATION STAFF

The proposed M&E staffing structure includes on international M&E expert (Dr. Diane Shilder) to provide short term technical assistance; one local, fulltime M&E specialist (M&E Manager) and one local, fulltime M&E assistant (M&E Coordinator). The team will be assisted by a local, fulltime data management systems specialist who will build and maintain the data base system required to input and analyze data.

The staff will be supported by the L3 Initiative’s management team (COP, DCOP, Technical Director) and Literacy, Numeracy and School-based Management Task Leaders. Together, these individuals will be responsible for developing and finalizing the M&E system, indicators, targets, database and reporting system. They are also responsible for assuring the quality and reliability of data and for training implementing partners to complete L3 Initiative reporting requirements. Staff will visit partners to provide guidance, to ensure quality control and in the process, collect data using monthly and quarterly tracking forms.

DATA SYSTEMS Levels of Performance Data. Data will be collected at three levels:

- L3 level, - Implementing partners’ level, - Institutional (ministries) and organizational beneficiaries’ level.

L3 staff, implementing partners and organizational beneficiaries will collect data at the participant level. L3 staff will monitor the all data collection activities and aggregate, analyze and verify all data submitted.

Data system. EDC has allocated resources the development of an internal L3 data system to house all monitoring and evaluation data and facilitate data analysis and reporting. This will ensure that information about L3 activities is captured on time and that it is used to make programmatic decisions. Technical assistance will be offered implementing partners and beneficiaries to ensure that any data received and entered into the system is of quality

The data system will contain all partner-level and participant-level data and will be organized using unique identifiers. It will be developed to ensure an easy data transfer from implementing partners and organizational beneficiaries and to the on-line Partner Reporting and Performance Management System (PRPMS). Training on data process and data utilization will be included in capacity building activities for all institutional and organizational beneficiaries.

The L3 Initiative Data Management Specialist, M&E Manager and M&E Coordinator will be responsible for ensuring the data system operates effectively and efficiently.

Transfer of data to Partner Reporting and Performance Management System (PRPMS). The M&E manager will oversee and coordinate the transfer data from the L3 initiative (both EDC data as well as L3 partner data) to the PRPMS system on a regular basis, according to a schedule established in collaboration with MEMs representatives in year 1.



23

Security of L3 data EDC maintains all its field data on secured computers whose physical access is limited. The following is a summary of the way EDC protects the M&E data. This policy is amended regularly to remain current with industry standards concerning network and data security.

• Only authorized staff have access to the M&E files and data. • The L3 Project uses a firewall to prevent unwanted Internet access to the field

computers. The field data are backed up as detailed in the Data Security section and in accordance with the P&P.

• The L3 Project retains current software to protect systems from viruses and spyware.

EDC also trains staff in data maintenance and in data safety. Data safety is also outlined in the EDC Rwanda Policies and Procedures (P&P) Manual. All data access is password protected and all transmission is encrypted. L3 personnel will follow a regular schedule for backing up data, for storing a copy off-site, and for sending critical data to the US:

• Every week a designated M&E staff member will be responsible for electronically backing up the database - as well as all relevant Monitoring and Evaluation documents - on password protected storage space on L3 hard drives or storage system. External back- ups will be stored off-site at a secure place designated by the Chief of Party. Hard copies of data will be stored in a secured place with limited access.

• On a quarterly basis M&E staff will scan the hard copies of critical M&E documents and email them to a designated EDC home office staff who will be responsible for: 1) cataloguing and archiving the scanned materials on a password protected EDC shared drive and 2) backing all scanned materials up on an external storage system.

As part of M&E capacity building activities, EDC will work with our local implementing partners to review or develop hard copy and electronic data safety procedures during an emergency. With the understanding that participant-level information can be potentially damaging to participants should it falls into wrong hands, our staff will place particular emphasis on developing data and network safety procedures and will train implementing partners’ M&E managers in the use of these procedures.

Safety of the Data/Network in the case of an emergency. Our team is dedicated to keeping our operations ongoing to the extent possible in the case of a security emergency or national crisis. EDC maintains an active security plan, a list of security policies, and an emergency action plan, which combines willing and able project staff to undertake immediate response to mitigate the impact on project assets and personnel. This system helps ensure that EDC is able to quickly resume operations and deliver L3 objectives. In the event of a major impact on our operations, EDC’s Crisis Management Team will mobilize all available resources and efforts to support field personnel and to ensure their safety and security. In the case of an emergency, senior L3 personnel will also be responsible for keeping the back-up data safe.



24

DATA QUALITY Table 7 below describes the mechanisms the L3 data system will put in place to ensure high quality11 of M&E data.

Table 7 Operational Definitions of Data Quality

Dimension Operational definition L3 Initiative’s Data System

Accuracy (Validity)

Accurate data are considered correct: the data measure what they are intended to measure. Accurate data minimize error (e.g., recording or interviewer bias, transcription error, sampling error) to a point of being negligible.

To assure data accuracy, monitoring indicators were developed with the input of L3 staff and EDC evaluation specialists. Experienced STTA12 will help the L3 M&E staff operationalize each indicator when the tools are developed.

Reliability The data generated by a program’s information system are based on protocols and procedures that do not change according to who is using them and when or how often they are used. The data are reliable because they are measured and collected consistently.

Forms and protocols for data capturing will be developed early in the implementation process to assure data consistency and reviewed by representatives from the Monitoring and Evaluation Management Services (MEMS) Project to ensure compatibility with on-line Partner Reporting and Performance Management System (PRPMS). The Technical Team and Senior M&E staff will train staff and partners in data collection procedures and reporting systems, including how to enter data into the PRPMS system. Periodic refresher sessions on data collection procedures and ongoing support with data collection will be organized.

Completeness

Completeness means that an information system from which the results are derived is appropriately inclusive: it represents the complete list of eligible persons or units and not a fraction of the list.

Procedures to ensure completeness of the data will be included in the data collection procedures developed for each data collection activity.

Precision Precision means that the data have sufficient detail. An information system lacks precision if it is not designed to record variables that may be required later for disaggregation.

Disaggregation categories for each indicator have been established during the preparation of this PMP and reflect the L3 Initiative’s goals and objectives.

Timeliness Data are timely when they are up-to-date (current) and when the information is available on time. Timeliness is affected

Routine program monitoring data will be collected as activities occur and according to the schedule. Specific procedures for timely

11 The table is based on the six dimensions of data quality as outlined in Data Quality Assurance Tool for Program-Level Indicators, USAID, 2007. 12 Dr. Diane Schiller, EDC Impact Evaluation Specialist, will lead all L3 student impact evaluations. She will be assisted in the design phase by Dr. Nancy Clark-Chiarelli, EDC Literacy Expert, and Dr. Paul Goldenberg, EDC Numeracy Expert.



25

by 1) the rate at which the program’s information system is updated; 2) the rate of change of actual program activities; and 3) when the information is actually used or required.

data collection will be developed. The M&E Manager will coordinate data collection and data entry into the L3 database and oversee data verification. He/she will also coordinate and oversee the transfer of data to the Partner Reporting and Performance Management System (PRPMS), according to the schedule established with MEMs representatives.

Integrity Integrity is when data are generated by a program’s information system are protected from deliberate bias or manipulation for political or personal reasons.

Data integrity will be assured through manual verification of entered data by staff other than the data entry clerks, with the M&E manager’s oversight.

Assessing L3 data systems In the second quarter of YR1, the M&E Manager will take the lead in organizing an appropriate team to review the L3 Initiative’s data systems and M&E capacity. The team will include senior management representatives, as well as EDC and L3 partner technical staff who have program performance monitoring roles. The Monitoring and Evaluation Readiness Questionnaire/Self Assessment Tool developed by Social & Scientific Systems Inc. (2009) will serve as the basis for the self assessment. The exercise will result in the identification of the strengths and weaknesses with respect to M&E and the development of a Action Item List (AIL) to monitor progress towards improving M&E interventions. The AIL will be shared with Monitoring and Evaluation Management Services Project Staff who will provide TA as required to address identified challenges. A report on progress with respect to addressing the AIL will be developed and filed b Q4 of Year 1.

The process described above will be repeated in Year 3 of the L3 Initiative to ensure ongoing assessment/evaluation of L3 data systems.


Appendix A: Key activities (by sub intermediate result)

INTERMEDIATE RESULT

Sub intermediate result KEY ACTIVITIES

IR1: Improved quality of teaching

1.1 National standards for literacy, numeracy and ESL established.

• Develop national standards for literacy and numeracy and grade specific benchmarks for key competencies

• Develop teacher standards for literacy and numeracy • Develop criteria of effective literacy/numeracy environments • Develop tools to measure progress with respect to the implementation of the standards and criteria

1.2 Literacy, numeracy school based mentoring program (SBMP) implemented nationwide

• Support the development of an institutional framework for the SBMP • Develop video-modules to train National Mentors and school-based mentors in effective mentoring

practices, and in effective literacy-numeracy instructional practices • Provide initial and ongoing training to National Mentors and School-based mentors • Provide NMs and SBMs with the resources (video viewing systems, cell phones, etc) and support

necessary to implement effective school-based trainings 1.3 Effective

literacy/numeracy instructional practices implemented in P1 –P4

• Train head teachers in effective pedagogical management (to improve literacy and numeracy results) • Ensure that teachers have access to ongoing training and support via school-based mentoring program • Ensure that teachers have access to and training in how to use L3 supplementary instructional

materials

1.4 Teacher Training Colleges transformed into Centers of Excellence for Literacy/Numeracy

• Facilitate the incorporation into the KIE Tutor training program of courses on reading and math methodology

• Facilitate the inclusion in the TTC curriculum of courses on reading and math methodology • Assign VSO literacy specialists to each TTC for a two-year period to co-teach the methodology courses • Organize sessions in the TTC Teacher Resource Center (TRC) on how to use L3 literacy/numeracy

support materials, how to build and use low-cost/no-cost materials, and how to use clastools to assess students’ reading and numeracy skills

• Institute literacy and numeracy “out reach” programs, lead by TTC student teachers, to provide support



27

to teachers and students in the zone surrounding the TTC

1.5 Teacher morale and working conditions improved

• Include in school yearly action plans activities to recognize teachers in the community • Organize annual festivals to recognize teachers, head teachers, SBM and NMs and communities that

have encouraged girls to stay in school and to improve their performance. • Train PTCs in how to identify and implement teacher-motivation activities related to literacy/numeracy

and English, including launching income-generating activities to subsidize teacher housing, professional development activities or income levels.

• Train recent graduates and community volunteers in how to make low-cost instructional materials, use L3 instructional materials to support student learning, and use simple L3 tools to assess students’ early reading and numeracy skills.

IR2: Improved Availability of Teaching and Learning Materials

2.1 Student access to supplementary reading and math materials and supporting technologies expanded

• Develop and distribute a comprehensive package of supplementary math and reading materials to every primary school in the country (decodable texts, read aloud texts, audio stories, home-school reading trackers, diagnostic tasks, teachers’ guides, interactive audio programs)

• Identify education videos to upload to school-based mentor video viewing system and train teachers in how to use video in the classroom

• Distribute MP3 players to all primary schools in the country • Organize regular “make and take” sessions for teachers to build lo-cost/no-cost instructional materials

for reading and math classes • Organize regular “Writers’ Workshops” to produce culturally and developmentally appropriate reading

texts for students • Organize regular “Math camps” to introduce teachers to open ended problems for the primary class

2.2 Students in isolated and low performing regions have access to additional materials

• Implement portable lending libraries in isolated and low-performing regions • Organize book fairs in isolated and low-performing regions • Implement volunteer program in isolated and low-performing regions

2.3 Culture of reading begins to take root

• Research study completed to identify positive practices with respect to literacy, as well as barriers at the school, classroom, home and community level

• Key messages identified for inclusion in national and regional campaigns



28

• Launch of local, regional and national campaigns to promote reading: student/school story book writing competitions, story reading on local radio stations, student teacher literacy/numeracy outreach programs in neighbouring schools, etc.

• Training of parents in positive literacy practices

IR 3: Support for English

3.1 Student access to supplementary ESL materials expanded

• Develop supplementary print and interactive audio materials to reinforce teachers’ and students’ English language skills and to transition children to English as a language of instruction

• Develop system for evaluating improvements in teachers’ English language skills • Revise, if necessary, primary ESL curriculum

3.2 Interactive English audio programs used regularly by students and teachers

• Develop carefully sequenced interactive audio programs to increase students’ and teachers’ English language skills and to prepare the transition to English as a language of instruction

3.3 Teachers’ English

Language Skills improved

• Ensure that English Interactive audio programs are distributed to all schools • Offer an English language School-based mentoring program to reinforce teachers’ English language

skills • Produce English language resource materials for SBMP • Upload additional English programs on MP3 players for teachers to use independently or with

colleagues to reinforce language skills

IR4: Strengthen Ministry Capacity

4.1 Ministry technical capacity to lead literacy/numeracy reform strengthened

• L3 technical staff and literacy/numeracy specialists embedded in REB • EDC literacy and numeracy experts twinned with KIE literacy and numeracy experts to provide ongoing

technical support • VSO literacy volunteers embedded in TTCs • VSO curriculum and instructional materials specialists embedded in KIE and NCDC • Short term VSO volunteers mobilized to respond to Ministry emerging technical needs

4.2 Student assessment systems and processes strengthened

• Provide short-term technical assistance provided to MINEDUC to reinforce student literacy/numeracy assessment programs, particularly in the area of mathematics

4.3 System for evaluating changes in teachers’ instructional practices developed and implemented

• System to collect data on teachers’ instructional practices developed in collaboration with Ministry an used to measure changes in teachers’ literacy and numeracy instructional practices



29

IR5: Improved Equity in Education

5.1 Girls’ access to gender-friendly materials and instructional practices increased

• Key social messages (gender and others) identified for inclusion in new L3 Instructional materials • Positive images of girls and other marginalized groups and key social messages embedded in new

instructional materials

5.2 Student teachers and

parents trained in inclusive practices

• Module on inclusive education in pre-service program/TTCs and in Parent Teacher Committee training program

5.3 National and district

initiatives are support improved girls’ literacy implemented

• Grants provided to districts to implement activities to address literacy-related gender disparities

Literacy, Language and Learning Initiativepdf.usaid.gov/pdf_docs/PA00HV17.pdf · Performance...

Documents

Transcript of Literacy, Language and Learning Initiativepdf.usaid.gov/pdf_docs/PA00HV17.pdf · Performance...