USAID SINDH READING PROGRAM EARLY GRADE READING … · USAID SINDH READING PROGRAM EARLY GRADE...

USAID SINDH READING PROGRAM

EARLY GRADE READING ASSESSMENT MIDLINE COHORT REPORT

Contract No. AID-391-C-14-00001

Submission Date: 5/22/2017

This document was produced for review by the United States Agency for International Development. It was

prepared by Chemonics International, Inc. and School-to-School International.

EARLY GRADE READING ASSESSMENT MIDLINE STUDY COHORT REPORT

The final version of this report was submitted May 22, 2017 by STS. This updated report, submitted December

22, 2017, includes the following revisions: clarification on the letter sound subtask results and an update to the

methodology regarding Cohort 3.

Contracted under Order No. AID-391-C-14-00001

DISCLAIMER The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

SRP MIDLINE COHORT REPORT

MAY 2017 i

ACKNOWLEDGEMENTS

The School-to-School International and Chemonics teams would like to thank USAID/Pakistan for their support. We would also like to recognize the stellar faculty and staff from the Sindh Education and Literacy Department, the Bureau of Curriculum, the Provincial Education Assessment Centre (PEACE) Sindh, and the Provincial Institute of Teacher Education (PITE) Sindh. Finally, this endeavor would not have been possible without the dedication of the master trainers, quality control officers, data collection partners, subject experts, and administrators who lent their continuous support throughout the assessment activity.

The following persons played a major role in the process:

Mr. Zulifqar Bachani (Chemonics International) Ms. Shahbana Shaikh (Chemonics International) Mr. Abdul Rauf (Chemonics International) Mr. Muhammad Usman (Chemonics International) Mr. Aftab Nizamani (Chemonics International)

Dr. Mark Lynd (School-to-School International) Dr. Hetal Thukral (School-to-School International) Louise Bahry (School-to-School International) Randy Tarnowski (School-to-School International) Aftab Khushk (School-to-School International) Nadia Qasim (School-to-School International) Beth Odenwald (School-to-School International) Dr. Jasmina Josic (School-to-School International) Ami Kanani (School-to-School International) Colleen Komeyli (School-to-School International) Melyssa Sibal (School-to-School International) Dr. Sydney Merz (School-to-School International) Laura Conrad (School-to-School International)


MAY 2017 ii

CONTENTS

Contents ........................................................................................................................................................................ ii

Acronyms ...................................................................................................................................................................... v

Executive Summary .................................................................................................................................................... 6

Chapter 1: Introduction ............................................................................................................................................ 10

Chapter 2: Methodology ........................................................................................................................................... 14

Chapter 3: Cohort 1 Results—Sindhi .................................................................................................................... 19



Chapter 6: Cohort 1 Results—Urdu ...................................................................................................................... 45



Chapter 9: Overall Discussion and Recommendations....................................................................................... 68

Annexes ....................................................................................................................................................................... 70


MAY 2017 iii

List of Tables and Figures

Table 1: SRP Inclusion Criteria and Length of Interventions, by Cohort ........................................................ 13

Table 2: Student Sample by Cohort and Gender .................................................................................................. 16

Table 3: School Sample by Cohort and Language ................................................................................................ 16

Table 4: EGRA Test Structure ................................................................................................................................ 17

Table 5: Standards of Reading Performance and Proportion Students Meeting Standards at Baseline (2014) and Midline (2016)—Sindhi Cohort 1 .............................................................................................. 21

Table 6: Contextual Variables—Student Questionnaire and EGRA Performance—Sindhi Cohort 1 ....... 31

Table 7: Contextual Variables—Teacher & Head Teacher Questionnaire and EGRA Performance—Sindhi Cohort 1 ................................................................................................................................................. 31

Table 8: Standards of Reading Performance and Proportion Students Meeting Standards in 2016—Sindhi Cohort 2 ............................................................................................................................................................. 38

Table 9: Standards of Reading Performance and Proportion Students Meeting Standards 2016—Sindhi Cohort 3 ............................................................................................................................................................. 42

Table 10: Standards of Reading Performance and Proportion Students Meeting Standards at Baseline (2014) and Midline (2016)—Urdu Cohort 1 ................................................................................................ 47

Table 11: Contextual Variables—Student Questionnaire and EGRA Performance—Urdu Cohort 1 ....... 56

Table 12: Contextual Variables—Teacher & Head Teacher Questionnaire and EGRA Performance—Urdu Cohort 1 ................................................................................................................................................... 56

Table 13: Standards of Reading Performance and Proportion Students Meeting Standards in 2014 and 2016—Urdu Cohort 2...................................................................................................................................... 63

Table 14: Standards of Reading Performance and Proportion Students Meeting Standards in 2016—Urdu Cohort 3 ............................................................................................................................................................. 66


MAY 2017 iv

Figure 1. Evaluation Design and Cohort Participation in SRP Interventions ................................................. 13

Figure 2. Average EGRA Scores at Baseline (2014) and Midline (2016) by Task—Sindhi Cohort 1 ......... 20

Figure 3. Average EGRA Gain Scores from Baseline (2014) to Midline (2016)—Sindhi Cohort 1 ............ 20

Figure 4. Proportion of Students Meeting Standards at Baseline (2014) and Midline (2016) by Gender—Sindhi Cohort 1 ................................................................................................................................................. 22

Figure 5. Proportion of Students Receiving Zero Scores by EGRA Task at Baseline (2014) and Midline (2016)—Sindhi Cohort 1 ................................................................................................................................. 23

Figure 6. Proportion of Students Receiving Zero Scores at Midline (2016) by Task and Gender—Sindhi Cohort 1 ............................................................................................................................................................. 23

Figure 7. EGRA Average Gain Scores from Baseline (2014) to Midline (2016) by Gender—Sindhi Cohort 1 ........................................................................................................................................................................... 24

Figure 8. EGRA Average Gain Scores from Baseline (2014) to Midline (2016) by Location—Sindhi Cohort 1 ............................................................................................................................................................. 25

Figure 9. Proportion of Students Receiving Zero Scores at Midline (2016) by EGRA Task and School Location—Sindhi Cohort 1 ............................................................................................................................ 25

Figure 10. Average EGRA Scores in 2016 by Task—Sindhi Cohort 2 ............................................................ 37

Figure 11. Proportion of Students Meeting Standards in 2016 by Gender—Sindhi Cohort 2 ..................... 39

Figure 12. Proportion of Students Receiving Zero Scores by EGRA Task in 2016—Sindhi Cohort 2 ..... 39

Figure 13. Proportion of Students Receiving Zero Scores by EGRA Task in 2016 by Gender—Sindhi Cohort 2 ............................................................................................................................................................. 40

Figure 14. Average EGRA Scores in 2016 by Task—Sindhi Cohort 3 ............................................................ 41

Figure 15. Proportion of Students Meeting Standards in 2016 by Gender—Sindhi Cohort 3 ..................... 43

Figure 16. Proportion of Students Receiving Zero Scores by EGRA Task in 2016—Sindhi Cohort 3 ..... 43

Figure 17. Proportion of Students Receiving Zero Scores by EGRA Task in 2016 by Gender—Sindhi Cohort 3 ............................................................................................................................................................. 44

Figure 18. Average EGRA Scores at Baseline (2014) and Midline (2016) by Task—Urdu Cohort 1 ......... 45

Figure 19. Average EGRA Gain Scores from Baseline (2014) to Midline (2016)—Urdu Cohort 1 ............ 46

Figure 20. Proportion of Students Meeting Standards at Baseline (2014) and Midline (2016) by Gender—Urdu Cohort 1 ................................................................................................................................................... 48

Figure 21. Proportion of Students Receiving Zero Scores by EGRA Task at Baseline (2014) and Midline (2016)—Urdu Cohort 1 ................................................................................................................................... 48

Figure 22. Proportion of Students Receiving Zero Scores by EGRA Task at Midline (2016) by Task and Gender—Urdu Cohort 1 ................................................................................................................................. 49

Figure 23. EGRA Average Gain Scores from Baseline (2014) to Midline (2016) by Gender—Urdu Cohort 1 ........................................................................................................................................................................... 49

Figure 24. EGRA Average Gain Scores from Baseline (2014) to Midline (2016) by Location—Urdu Cohort 1 ............................................................................................................................................................. 50

Figure 25. Proportion of Students Receiving Zero Scores at Midline (2016) by Task and School Location —Urdu Cohort 1 .............................................................................................................................................. 50

Figure 26. Average EGRA Scores in 2016 by Task—Urdu Cohort 2 .............................................................. 62

Figure 27. Proportion of Students Meeting Standards in 2016 by Gender—Urdu Cohort 2 ....................... 63

Figure 28. Proportion of Students Receiving Zero Scores by EGRA Task in 2016—Urdu Cohort 2 ....... 64

Figure 29. Proportion of Students Receiving Zero Scores by EGRA Task and Gender in 2016—Urdu Cohort 2 ............................................................................................................................................................. 64

Figure 30. Average EGRA Scores in 2016 by Task—Urdu Cohort 3 .............................................................. 65

Figure 31. Proportion of Students Meeting Standards 2016 by Gender—Urdu Cohort 3 ........................... 66

Figure 32. Proportion of Students Receiving Zero Scores by EGRA Task in 2016—Urdu Cohort 3 ....... 67

Figure 33. Proportion of Students Receiving Zero Scores by EGRA Task and Gender in 2016—Urdu Cohort 3 ............................................................................................................................................................. 67


MAY 2017 v

ACRONYMS

CLNPM Correct Letter Names Per Minute

CNWPM Correct Nonwords Per Minute

CWPM

EDO

Correct Words Per Minute

Education District OfficersEGRA Early Grade Reading Assessment

GoS Government of Sindh

MT Master Trainer

PEACE Provincial Education Assessment Centre

PITE Provincial Institute of Teacher Education

PRP Pakistan Reading Project

QCO Quality Control Officer

SBEP Sindh Basic Education Project

SMC School Management Committee

SRP Sindh Reading Program

USAID

VTT

United States Agency for International Development

Voice Tel Tech


MAY 2017 6

EXECUTIVE SUMMARY

Under the umbrella of the Sindh Basic Education Project (SBEP), the Sindh Reading Program (SRP) is a five-year collaborative initiative of the Government of Sindh (GoS) and USAID/Pakistan funded by

United States Agency for International Development (USAID). SRP is designed to improve student performance in early grade reading for 400,000 children across eight districts in the province of Sindh in Pakistan. To assess the impact of SRP interventions on student performance, a study was conducted.

The study uses a non-experimental design with student and teacher data collected at three time points: 2014

(baseline), October/November 2016 (midline), and 2018 (endline). The purpose of this report is to

summarize findings from the midline study. To that end, data on students’ reading skills were collected in

the third year of SRP to 1) measure progress on improved reading skills since a baseline assessment was

performed in 2014 and 2) to inform the implementation of SRP interventions for the remaining years of

the project. The midline evaluation data collection was conducted in October–November 2016.

Prior to midline (and after baseline), SRP leadership and USAID advised researchers that SRP had adopted a cohort approach in the implementation of its interventions. Furthermore, SRP imposed selection criteria for inclusion of schools in cohorts one and two. As a result of this shift in SRP’s implementation model and the differences in selection criteria for inclusion in each cohort, the results of the midline study must be examined by cohort. Specifically, comparisons across cohorts are not feasible due to the differences in cohorts. Although Cohort 2 and, what we treat as Cohort 3, may appear to be comparison groups to Cohort 1 (since Cohort 1 has been the only group to receive interventions as of midline), they cannot be used as such because of the differences in selection criteria for Cohort 2 from Cohort 1 schools and the fact that Cohort 3 in this report consists of all remaining schools in a district. Since Cohort 3 consists of all schools in a district that were not included in Cohort 1 or Cohort 2 at the time of midline, the inclusion criteria for schools in SRP’s Cohort 3 will be an important consideration at endline reporting. In this updated report, the difference between Cohort 3 – as reported at midline – and the actual cohort of schools that will receive SRP interventions in Year 4 and 5 – are described in the methodology.

The Early Grade Reading Assessment (EGRA) was used to assess students’ reading skills. Additionally,

because the evaluation also sought to capture contextual variables that may be associated with strong

reading outcomes, separate questionnaires for students taking the EGRA, their teachers, and their head

teachers were included. This midline report summarizes results – by cohort – for students, teachers, and

head teachers from schools in SRP Cohorts 1, 2 and 3.

Cohort 1

A comparison of baseline to midline Cohort 1 data estimates the change over time for students whose teachers participated in SRP interventions between December 2015 and May 2016 (approximately 6 months of intervention). Results are summarized below for students in seven districts tested in Sindhi and students in Karachi tested in Urdu.

Sindhi-Medium Schools

Overall, students in Cohort 1 Sindhi-medium schools made notable gains from baseline to midline in reading fluency, decoding (nonword reading), and expressive vocabulary. However, while students made a significant gain on the reading comprehension task, they continue to struggle with the task with only an average of 1.4 correct responses out of 5. Students also continue to struggle with the phoneme isolation task, though there was a decrease in the proportion of students with zero scores from baseline to midline (i.e., more students were able to answer correctly at least one item on the task at


MAY 2017 7

midline compared to baseline)1. Students’ scores on the listening comprehension task remain relatively stable from baseline to midline, on average.

There was a statistically significant difference in mean scores among Cohort 1 students on five of the six tasks common to both the baseline and midline assessments: phoneme isolation, nonword reading, expressive vocabulary, passage reading, and reading comprehension. In all cases, the midline scores represented a gain over baseline results. In addition, there were increases in the proportions of students who met or exceeded the SRP standard on all tasks, except on listening comprehension.

Relationships between reading outcomes and SRP interventions were also explored for Cohort 1 students in Sindhi-medium schools. Specifically, oral reading fluency rates (based on results from the passage reading task) were higher among students who reported the following: a stronger disposition towards reading, better attendance, higher levels of parental literacy, more reading resources and support in the home and at school, and greater access to and use of SRP-provided materials. Oral reading fluency rates were also higher in these schools when teachers reported having more positive perceptions of SRP intervention effectiveness.

The following points elaborate on each of these relationships between student outcomes and student-, teacher- and head teacher survey responses at midline2 for Sindhi-medium Cohort 1 schools:

1. Most students, teachers, and head teachers in Sindhi-medium Cohort 1 schools reportedreceiving SRP reading materials; however, availability, and use, varied.

Three-quarters of all students recall receiving any of the six SRP readers that were intended toreach SRP schools. Of the six readers, the most commonly recalled reader was Kenh ja tangun ahen;roughly half of surveyed students recalled it. Mujhe aata hai was the least recalled reader; only one-fifth of students recalled it.

All head teachers indicated that they received at least one read-aloud reader from SRP. Lomri aa ja

school was reported as received by three-fourths of surveyed head teachers, and was currently

present in two-thirds of schools. At least three-quarters of all head teacher reported distributing

various readers.

Most head teachers also reported that they received leveled-reader books. The leveled-reader least

often reported as received was Ahmed badak ain topi; only two-thirds of head teachers reported

receiving it.

The majority of teachers indicated that they used the read-aloud readers provided by SRP daily or

two- to three- times per week.

The reading skill that teachers reported focusing on the least was orientation to print (38.6%) and

reading fluency (29.0%). Conversely, identifying letter-sounds (72.8%) and listening

comprehension (64.9%) were the most common concepts focused on by teachers.

2. The majority of Grade 2 teachers in Sindhi-medium Cohort 1 schools reported participatingin SRP-led trainings. A higher proportion of head teachers reported that their schools participatedin the training than teachers reported (91.7% of head teachers versus 79.7% of teachers). 10.5% ofteachers surveyed were not sure if they participated, and 9.8% were certain that they did not participate.In addition, 71.9% of head teachers reported that they also participated in the training. Only 38.6% ofteachers reported using an SRP lesson plan the day of the interview.

3. Grade 2 teachers in Cohort 1 Sindhi-medium schools reported higher expectations of theirstudents reading abilities at midline than baseline; however, classroom reading practices

1 Note that the EGRA task assesses students’ ability to identify the last sound, whereas SRP intervention materials focus on identifying the first sound. See pp. 36 for further discussion.

2 Note that students, teachers, and head teachers were interviewed 5 months after the SRP intervention was completed in schools.


MAY 2017 8

appear to remain unchanged. Teachers reported there was no change to how often their students repeated stories read them, re-told stories, sounded out familiar words, learned new words, or read aloud to teachers or each other. At the same time, teachers had higher expectations of their students, particularly their comprehension abilities.

4. In-class reading practices increased, including access to reading materials in Sindhi-medium

Cohort 1 schools. Half of students surveyed at midline indicated that they always practiced reading

aloud to the teacher or other students, compared to only two-fifths at baseline.

5. Students in Sindhi-medium Cohort 1 schools received better support for reading at home,including increases in access to materials and reading practice. Three-quarters of students atmidline indicated that they have reading materials at home, compared to two-thirds at baseline.Similarly, the number of students responded that they practice silent reading at home increased slightly.Additionally, two-thirds of students indicated at midline that they have someone at home read aloudto them, compared to one-half of students at baseline. Students are also reading aloud to someone athome more frequently; three-quarters of students reported reading aloud at midline compared to one-half at baseline.

Urdu-Medium Schools

Students’ performance on the EGRA in Cohort 1 Urdu-medium schools remained unchanged from baseline to midline in Karachi. Across all six EGRA tasks common to both assessments, there was no statistically significant difference between student performance at baseline and midline. Similarly, there was no significant reduction in the proportion of students receiving zero scores.

While baseline to midline comparisons were not significantly different, some notable results at midline were observed. First, the expressive vocabulary and letter name recognition tasks had the lowest proportions of zero scores of the seven EGRA tasks at midline. Second, the proportion of students meeting or exceeding SRP standards increased across all tasks except for phoneme isolation, which continues to prove difficult.

Students responses on the survey showed variability in some contextual factors associated with reading ability. For example, students reported increased support for reading in the home and in school. However, their reports of access to a library contradict reports given by their teachers; this discrepancy warrants further investigation. Students demonstrated that they receive more reading support at home and are assigned reading homework more frequently. A greater number of students reported that there is a library at their school at midline, but this is inconsistent with responses from head teachers. Although, it is possible that this contradiction is the result of more students becoming aware of pre-existing libraries at their schools. Teachers at midline reported higher expectations for students regarding the grade level in which they are expected to gain certain reading skills.

Relationships between reading outcomes and SRP interventions were also explored for Cohort 1 students in Urdu-medium schools. Specifically, fluency rates were higher among students who reported a stronger disposition toward reading, better attendance at school, more reading resources in the home, and more reading support at school. Two themes from the teacher and head teacher questionnaires were also significantly related to students’ fluency scores; more reading activities were related to higher fluency scores, but increased TLA involvement was related to lower fluency scores. The TLA-involvement result should be interpreted cautiously given the small sample size.


MAY 2017 9

Cohort 2 and Cohort 3

Cohort 2

Cohort 2 includes schools with at least one Grade 1 and one Grade 2 classroom located in geographic clusters around a “guide school.” Guide schools must have a teacher who can serve as “guide teacher.”3 Cohort 2 schools were chosen from half of remaining schools in each district, excluding Cohort 1 schools.

On average, students in Cohort 2 Sindhi-medium school read 13.3 correct nonwords per minute (CNWPM) and 30.1 correct words per minute (CWPM). They averaged 1.1 correct answers out of five questions on the reading comprehension task and 1.5 correct answers out of three question on the listening comprehension task. Finally, they identified an average of 37.4 correct letter names per minute (CLNPM).

Students in Cohort 2 Urdu-medium schools read 12.6 CNWPM and 20.1 CWPM on average. In comprehension, they averaged 0.5 correct answers on the reading comprehension task and 1.3 correct answers on the listening comprehension task. Finally, they identified 44.9 CLNPM an average.

Cohort 3

Cohort 3-for the purposes of this midline report only - includes all schools remaining in the district (i.e., not in Cohort 1 or Cohort 2). At the time the sample was drawn in September 2016, it was unclear which Cohort 3 schools would actually receive SRP interventions in Years 4 and 5. See methodology for a description of how Cohort 3 as reported in this document differs from Cohort 3 as treated by SRP post-midline.

Students in Cohort 3 Sindhi-medium schools read an average of 12.4 CNWPM and 30.5 CWPM. In comprehension, students in Cohort 3 schools had an average of 1.1 correct answers on the reading comprehension task and 1.6 correct answers on the listening comprehension task. Finally, students identified an average of 35.4 CLNPM.

Students in Cohort 3 Urdu-medium schools read 15.3 CNWPM and 28.0 CWPM on average. In comprehension, they averaged 0.8 correct answers on the reading comprehension task and 1.5 correct answers on the listening comprehension task. Finally, they identified an average of 47.7 CLNPM.

3 Criteria for guide teachers included 1) they were chosen through Sindh government’s new competitive hiring process for teachers and 2) they were under 45 years of age.


MAY 2017 10

CHAPTER 1: INTRODUCTION

1.1 The Sindh Reading Program (SRP)

Under the umbrella of SBEP, SRP is a five-year collaborative initiative of the GoS and USAID/Pakistan funded by USAID. SRP is designed to improve student performance in early grade reading for 400,000 children across eight districts in the province of Sindh in Pakistan.4 SRP’s main objectives are to:

1. Improve teacher competencies for effective early grade reading instruction;

2. Improve early grade reading assessment practices;

3. Improve student access to supplementary

reading materials; and

4. Enhance participation of parents and

caregivers in support of early grade literacy

for out-of-school children.

SRP is being implemented across eight districts in the province of Sindh in Pakistan: Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur, and five towns in Karachi (which are treated as a single district for the purposes of this report). By 2018 (year 5 of the program), one of SRP’s key indicators is to improve reading for at least 200,887 Grade 2 students.5

Through the end of the project, the Education and Literacy Department of GoS, USAID, and SRP will collaborate to implement multiple strategies to improve teaching practices and learning performance in early grade literacy with the ultimate goal of improving reading skills among students in participating schools.

SRP’s strategies include:

1. Design and implement a baseline study of student performance in early grade reading;2. Analyze current performance levels to target project interventions;3. Define “grade level” competencies through the development of performance standards;4. Implement follow-up studies to monitor project progress towards established goals; and5. Institutionalize early grade reading assessment practices at the government and school levels.

1.2 Overview of the Midline Evaluation

The purpose of this report is to summarize findings from the midline study, as part of the fourth strategy

listed above. To that end, data on students’ reading skills were collected in 2016, the third year of SRP to

4 At the start of the project, there was an additional focus on mathematics. The focus on mathematics has since been removed from the evaluation and program design. 5 The target reflects revisions as of April 2015.


MAY 2017 11

1) measure progress on improved reading skills since baseline and 2) to inform the implementation of SRP

interventions for the remaining years of the project.

The study uses a non-experimental design with student and teacher data collected at three time points—baseline, midline and endline—during SRP interventions. Comparisons across these three time points provide a measure of the growth made by Grade 2 students in early grade reading in schools participating in SRP interventions. 6 In this report, comparisons across time are made by SRP cohort. Following the baseline assessment, SRP adopted a cohort approach to its interventions, which was then accounted for by STS in the midline evaluation design. As a result of this shift in SRP’s implementation model and the differences in selection criteria for inclusion in each of three cohorts, the results of the midline study must be examined by cohort. Furthermore, comparisons across cohorts are not feasible due to the differences in cohorts. Additional details regarding the non-experimental design and how cohorts differ are discussed in the next section.

The tool used to assess students’ reading skills is called the Early Grade Reading Assessment (EGRA). In

addition to student assessment data, the evaluation also sought to capture contextual variables that may be

associated with strong reading outcomes, and thus included a questionnaire for students taking the EGRA,

a questionnaire for their teachers, and a questionnaire for their head teachers (found in Annex 4, Tables 1-

18). Student questions were related to language spoken at home, availability of reading materials at home,

study habits, access to communication technology, socioeconomic variables, teaching practices, and

feelings on reading. Teacher and head teacher questions related to qualifications, attendance at trainings,

time scheduling, level of support they receive, head teacher knowledge of student performance, classroom

observation procedures, School Management Committee (SMC) authority, facility improvements, and, for

those schools that have already participated in SRP trainings (SRP Cohort 1 schools), specific questions

around the training teachers and head teachers received from SRP.

The midline data collection was conducted in October-November 2016. Prior to the midline, a baseline evaluation was conducted in 2014; it assessed a sample of Grade 3 students throughout Sindh province so that independent baselines could be established in each of the eight SRP districts. The results from the baseline and midline are considered a measure of learning for students at the end of Grade 2, even though the evaluations were conducted with students at the beginning of Grade 3.7

As part of the SRP study, the following activities were carried out at baseline or at both baseline and midline. Each activity is described in greater detail below and in Chapter 2.

1. Evaluation design. At baseline and midline, a cross-sectional design was followed in accordancewith USAID guidelines.

2. Sampling. At baseline and midline, a stratified cluster random sampling method was used in orderto be able to generalize the results to the district level.

3. Instrument development. At baseline, the EGRA tool was developed for the Sindh provinceand piloted. The same instrument was used in 2016 with the addition of one task. Student, teacher,and head teacher questionnaires were also developed.

6 Adjustments made to SRP’s focus overall, following the collection of baseline data, dictated that only reading be assessed at midline and endline. 7 The decision to conduct the baseline with Grade 3 students was made because of the requirement to conduct the

baseline before the startup of project activities in Year 1. This timing presents a complication – the effects of “summer learning loss” – i.e., that students typically forget some of what they have learned in a given school year over the summer. The extent of this loss cannot be estimated and the results presented here may in fact be underestimating the actual performance of students prior to the break. In other words, assessments conducted after the summer break will reflect a short-term decline in student achievement. The midline evaluation was conducted at the same point in the school year (as will the endline) in order to ensure comparability of results across time points.


MAY 2017 12

4. Training. At baseline and midline, workshops were conducted to train master trainers (MTs),supervisors, enumerators, and quality control officers (QCOs). Enumerators and supervisors wereobserved to ensure clear understanding and skills adequate to implement the EGRA tool.

5. Implementation. At baseline and midline, the EGRA and questionnaires were administered in alltargeted schools in the sample.8 At the time of this writing, it is unclear which Cohort 3 schoolswill receive SRP interventions as SRP interventions are not intended to reach all schools in eachof the eight districts

6. Analysis. At baseline and midline, two types of analyses were conducted: analysis of test qualityand analysis of EGRA and questionnaire results to report findings (fluency rates for timed tasksand average correct response for untimed tasks; proportion of students with zero scores; andproportion of students meeting pre-specified standards).

7. Reporting. At baseline and midline, district-level reports were produced to summarize averagereading abilities among all students—those in SRP schools and those in non-SRP schools—in eachdistrict. Additionally, midline results by cohort are summarized in this report.

1.3 Theory of Change

SRP's approach involves four components of activities that promote a culture and set of supportive behaviors related to reading and learning in order to improve literacy outcomes. The following describes each of these components:

Component 1: Teachers utilize improved tools and techniques including TLMs to improve student participation, attendance, and reading outcomes.

Component 2: Continuous assessment of student reading abilities to support the improvement of teaching techniques so that teachers can address individual student needs; at the district-level, monitoring of district-to-district performance to support efficient allocation of resources.

Component 3. Providing access to reading material and reading spaces through support to public and hub school libraries and creation of classroom reading corners.

Component 4. Engage out of school children through non-formal education program to provide primary level literacy and mainstreaming opportunities.

This study primarily focuses on examining SRP’s activities under Component 1, with some results relevant to Components 2, 3 and 4.

SRP Interventions by Cohort

Prior to midline (and after baseline), SRP advised researchers that SRP had adopted a cohort approach in the implementation of its interventions (Figure 1). Furthermore, SRP imposed selection criteria for inclusion of schools in one of three cohorts (Table 1). As a result of the shift in SRP’s implementation model and the differences in selection criteria for inclusion in each of three cohorts, the results of the midline study must be examined by cohort. Specifically, comparisons across cohorts are not feasible due to the differences in cohorts. Although Cohort 2 and 3 may appear to be a comparison group, they cannot be used as such because of the differences in selection criteria for Cohort 2 and 3 schools and Cohort 1 schools. As Figure 1 shows, comparison of baseline to midline data for students in Cohort 1 estimates the change over time for students whose teachers participated in SRP interventions between December 2015 and May

8 At baseline, the tools were administered on paper. Following baseline data collection, data entry was conducted. At midline, the tools were administered on tablets; therefore, no data entry was needed.


MAY 2017 13

2016 (teachers participated in approximately 6 months of intervention).9 2014 to 2016 comparisons for students in Cohorts 2 and 3 represent the change over time for students prior to their teachers beginning participation in SRP interventions.

FIGURE 1. EVALUATION DESIGN AND COHORT PARTICIPATION IN SRP

INTERVENTIONS

Note: Results from Grade 3 students at the beginning of the year are used to estimate reading abilities of students at the end of Grade 2. At baseline, Grade 5 students were also tested; and students in both Grade 3 and 5 were tested in Math. Teachers who taught Grade 2 in the previous year were surveyed. Inclusion criteria for Cohorts are described in detail above.

TABLE 1: SRP INCLUSION CRITERIA AND LENGTH OF INTERVENTIONS, BY

COHORT

Inclusion criteria applied SRP Interventions

Cohort 1a (Pilot) 512 schools

Selected at random, stratified by district. January 2014–May 2016 (17 months)

Cohort 1 (Pilot plus 920 schools) 1,432 schools

At least one Grade 1 and one Grade 2 classroom December 2015–May 2016 (5 months)

Cohort 2 2,173 schools

1. At least one Grade 1 and one Grade 2 classroom2. Located in geographic clusters around a “guide school”3. “Guide schools” must have a teacher who can serve as “guide

teacher.”10

4. Schools were chosen from half (geographically) of the schoolsin each district not previously included in Cohort 1.

December 2016–May 2017 (5 months)

Cohort 311 SRP selection criteria TBD. For the purposes of this report, Cohort 3 results apply to all schools in a district that are not Cohort 1 or Cohort 2. At the time of sampling, it was assumed that

TBD

9 Pilot schools received additional interventions as described in the previous section. 10 Criteria for guide teachers included 1) they were chosen through Sindh government’s new competitive hiring process for teachers and 2) they were under 45 years of age. 11 At the time of midline sampling, the intended selection criteria for SRP interventions in Year 4 and 5 (Cohort 3) was expected to be the same as that for Cohort 2 (chosen from the remaining half of each district, excluding schools from Cohort 1). Based on this assumption at midline, Cohort 3 as reported in this document is the population of remaining schools from which SRP’s Cohort 3 will be selected. See methodology for additional details.

Grade 3

Grade 5

BASELINE

2014 (Y1)

MIDLINE

2016 (Y3)

ENDLINE

2018 (Y5)

Grade 3 Grade 3

2015 (Y2)

Cohort 1a (Pilot) &

1b

2017 (Y4)

Cohort 2

Cohort 3


MAY 2017 14

these remaining schools would constitute the population of schools from which a Cohort 3 sample will be drawn.

This report includes results – by cohort – for students, teachers and head teachers from schools in Cohorts 1, 2, and 3 across two time points: 2014 and 2016. Chapter 2 details the methodology for the study. EGRA and survey results for Cohort 1-Sindhi are presented in Chapter 3, Cohort 2-Sindhi in Chapter 4, and Cohort 3-Sindhi in Chapter 5. Chapters 6–8 present results for the cohorts in Karachi tested in Urdu language. Additional tables are provided in the annexes at the end of this report.

CHAPTER 2: METHODOLOGY

This chapter describes the SRP sample, data collection tools and process, and data analysis approach.12 Additional details can be found in Annex 1.

2.1 Evaluation Design

As part of the request for proposal (RFP), USAID required a cross-sectional design, assessing students at the same grade levels (including equal representation by gender) over the course of SRP. The design features Grade 3 students at three time points (baseline, midline, and endline) and was revised prior to the midline to reflect the cohort design for the rollout of SRP interventions (adopted by SRP after the baseline was completed).

The intent of the midline evaluation is to determine gains in reading across two time points (October 2014 and October 2016) among students in SRP schools; at endline, comparisons will be made across the three time points assuming the same intervention design and theory of change for SRP. In the baseline and midline evaluations, results from students in Grades 3 are used to estimate student performance at the end of Grade 2. All students were assessed using the same EGRA instrument (one Sindhi-medium, one Urdu-medium) that measured student’s ability at the Grade 2 level.

Research Questions

As noted above, the evaluation sought to answer the following research questions:

1. How do students reading performance (as measured by EGRA) change from baseline to midline,

in Sindhi-medium and Urdu-medium Cohort 1 schools?

2. How do midline scores among SRP Cohort 1 students (1-2 year of SRP interventions) compare

by subgroup within each language group?

3. How do general reading practices among students, teachers and head teachers change baseline to

midline? How do they compare by subgroup (urban/rural)?

4. How do EGRA scores vary by general reading practices among students, teachers and head

teachers, within each language group, in Cohort 1? In Cohort 2? In Cohort 3?

Note that the research questions do not compare cohorts. The differences in SRP interventions and differences in the criteria for selection across cohorts limits the comparability of cohorts as well as the population of schools that results can be generalized to. In other words, results of this report can be generalized to all Grade 2 students in schools in each specific cohort and language group (i.e., results for Cohort 1 students in Sindhi can be generalized to all students in Cohort 1 in Sindhi).

12 For more details on the tool development process and implementation of testing, see Chemonics (2014) “Process Report for EGRA/EGMA Baseline Study 2014.”


MAY 2017 15

The evaluation utilizes a non-experimental pre-test/post-test design, in which students participating in the intervention are assessed prior to the start of the interventions (pre-test) and following the conclusion of the interventions (post-test); additionally, a mid-point measure is also collected. This design was appropriate for this study since an adequate comparison group (i.e. students in schools not participating in SRP interventions, but similar otherwise) was not feasible. At the time of the baseline, SRP interventions were intended to reach all schools in all participating districts, and as such, the baseline sampling methodology drew from all schools in the district. SRP interventions, at the time of midline, are intended to reach approximately 60% of schools in the eight districts and using a cohort design. With the introduction of a cohort design, as well as differing selection criteria for each cohort, a comparison group is unfeasible for this study. Without a comparison group, it is not possible to isolate the impact of the interventions (in other words, to be able to attribute all the changes from pre-test to post-test among participating students to the SRP interventions).

2.2 SRP Sampling Approach

The study uses a stratified cluster random sampling method to meet two SRP requirements: (1) that results be generalizable to the population of students in the eight districts; and (3) that results at the provincial level be generalizable to the population of students participating in each of SRP’s three intervention cohorts. The sampling frame is based on the population of schools in the target districts in SEMIS as of June 30, 2016; the frame is available upon request. In all, the population included 12,254 schools, including 1,299 Cohort 1 schools, 2,063 Cohort 2 schools and 8,947 remaining schools (treated as Cohort 3 in this midline report). See annex 1.

In seven of the SRP intervention districts (Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot and Sukkur), 70 schools from the population list of each district were selected—25 from the total pool of Sindhi-speaking schools that participated in Cohort 1 interventions, 25 from the total pool of Sindhi-speaking schools that are participating in Cohort 2 interventions, and 20 from the remaining pool of Sindhi-speaking schools in each district (these schools may or may not participate in SRP’s interventions in the future). Similarly, 100 schools were selected in total from the Urdu-speaking schools in the five towns of Karachi City being served by SRP (for the purposes of this report, the five towns of Karachi City were treated as a district)—40 schools that had participated in Cohort 1 interventions, 40 schools that are participating in Cohort 2 interventions, and 20 schools from the remaining pool (again, these schools may or may not participate in SRP’s interventions in the future).

During data collection, an additional 24 schools were assessed due to issues with replacement schools and cohort stratification; enumerators were uncertain whether a certain school met all the criteria and, in an effort to be thorough, tested students in additional schools. During data cleaning, these additional schools were not excluded since schools met the cohort and stratification criteria.

Update on sampling frame: In December 2017, SRP’s selection criteria for Cohort 3 was finalized. This selection criteria is important to consider at the time of endline, as it will affect the comparability of results from midline to endline for these schools. As described above, the Cohort 3 schools in this report consists of all schools remaining in a district, after removing those selected for Cohorts 1 and 2. These remaining schools were assumed to be the population of schools that SRP’s interventions would reach in Years 4 and 5 of the project (i.e, SRP Cohort 3). Upon finalization of the selection criteria for this cohort, this update provides a comparison of the population of Cohort 3 schools assumed at midline (and to which the midline results are generalizable) and the population of Cohort 3 schools that will actually receive SRP Cohort 3 interventions (and to which the endline results are generalizable).

Cohort 3 population assumed in this midline report is all remaining schools in the district, after removing schools in Cohort 1 or 2. The Cohort 3 population of schools as finalized by SRP in December 2017 includes 2,570 schools. However, in contrast to the assumption made at the time of sampling (i.e., no overlap with Cohorts 1 or 2 schools), 887 of the selected Cohort 3 schools had also been in Cohort 1 while 112 of Cohort 3 schools had also been in Cohort 2. This represents 38.8% of Cohort 3 schools as also


MAY 2017 16

having been schools originally included in the population of schools for Cohort 1 or 2.13 For reference, only 55 schools overlapped across cohorts in the population list used at midline (see Annex 1) and were treated as occurring in the first cohort they were assigned to. Therefore, at midline, each cohort was treated as mutually exclusive and afforded generalizability of results to all schools in the population of that cohort. In other words, midline results for Cohort 1 are generalizable to the population of 1,432 schools receiving Cohort 1 interventions; and for Cohort 2, to the 2,176 schools slated to receive interventions; for Cohort 3, results are generalizable for all remaining non-Cohort 1 and 2 schools in the districts.

At endline, however, generalizability of results will need to be based on the actual sample drawn at endline, accounting for the significant overlap in cohorts. If sampling is conducted to assume mutually exclusive groups, then how these 999 schools that received multiple years of interventions, are categorized, will be an important decision. Further analysis will need to be done to compare the population of Cohort 3 schools at endline to the population of Cohort 3 schools at midline to determine the generalizability of results across time points.

2.3 Study Sample

At midline, the EGRA tool was administered to 7,075 students across eight districts. The target sample consisted of 70 schools from each Sindhi district and 100 schools from Karachi.14 Within each school, 12 students from Grade 3 were assessed.

After data cleaning was completed, the final sample used for analysis consisted of 6,851 students. Results for surveys from teachers and head teachers were also used from this final sample of 610 schools (which included valid results from the additional 24 schools tested). The final analytical sample, by cohort and gender, is shown in Table 2, and by school and language is shown in Table 3.

TABLE 2: STUDENT SAMPLE BY COHORT AND GENDER

Cohort 1 Cohort 2 Cohort 3

n n % n n % n n %

Sindhi Boys 1343 62.7% 1343 64.6% 1000 66.9%

Girls 799 37.3% 735 35.4% 495 33.1%

Total Sindhi 2142 100.0% 2078 100.0% 1495 100.0%

Urdu Boys 202 45.0% 252 55.0% 109 47.6%

Girls 247 55.0% 206 45.0% 120 52.4%

Total Urdu 449 100.0% 458 100.0% 229 100.0%

TABLE 3: SCHOOL SAMPLE BY COHORT AND LANGUAGE

Cohort 1 Cohort 2 Cohort 3 Total

n n n

13 2 schools were found to be in all three Cohort lists based on the final list for Cohort 3 received in December 2017.

14 The original target sample was 590 schools, but data was collected in a total of 614 schools. An additional 24 schools were visited during data collection to maintain the sample’s cohort stratification in the face of school replacements.


MAY 2017 17

Sindhi 188 182 142 512

Urdu 41 40 20 101

Total 229 222 162 613

2.4 Data Collection Tools

The EGRA, or Early Grade Reading Assessment, was developed in consultation with cognitive scientists, early grade reading experts, research methodologists, and assessment experts with funding assistance from USAID, the World Bank, and other international donors. This tool measures student performance on the basic foundational skills required for fluency in reading, and it assesses the skills needed for reading acquisition. Although many students are not yet fluent readers in the early grades, EGRA allows researchers to capture what students—even the “nonreaders”—can do and where they are in the developmental path to becoming fluent readers. At midline, the same EGRA tool that was administered at baseline was used. An additional reading passage was administered at midline to confirm that test leakage had not occurred. The assessment was administered in Sindhi in the seven districts of Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur, and in Urdu in five towns of Karachi (treated as one district).

At baseline, six reading skills were assessed using the EGRA: phonemic awareness, phonics, vocabulary, fluency, reading comprehension, and listening comprehension. At midline, a letter name identification task was added to the end of the EGRA. Some tasks were timed to determine the speed with which students could decode words (phonics), read connected text (fluency), and identify letters (letter name recognition). A summary of these tasks and the skills they were designed to assess are presented in Table 4.

TABLE 4: EGRA TEST STRUCTURE

Core Reading Skills SRP EGRA Tasks (Task Number and Task Name)

Task Length

1. Phonemic Awareness 1. Phoneme isolation Untimed

2. Phonics 2. Nonword reading Timed (1 minute)

3. Vocabulary 3. Expressive vocabulary Untimed

4. Fluency (Grade 2-level text) 4a. Passage reading Timed (1 minute)

5. Reading comprehension(Grade 2-level text)

4b. Reading comprehension Untimed

6. Listening Comprehension 5. Listening comprehension Untimed

7. Alphabet Knowledge 6. Letter name recognition* Timed (1 minute)

*Only included on midline assessment. See the “Updating Tools in 2016” section below.

At each school visit, three tools were administered in addition to the EGRA: student survey, teacher survey

and head teacher survey. Students were administered the EGRA followed by a student survey. Student

questions related to language spoken at home, availability of reading materials at home, reading support

received at home, reading practice and behaviors at school, homework, school attendance, school

resources, and the SRP interventions. Questions on the availability of resources were used as a proxy

measure of socio-economic status.

Teacher and head teacher questions related to qualifications, school information, scheduled time for

reading instruction, methods to evaluate student learning in reading, language, head teacher knowledge of

student performance, teacher observation procedures, SMC authority, facility improvements, expectations

of students’ reading abilities, and participation in and perception of SRP interventions.


MAY 2017 18

2.5 Data Collection

Preparation for the midline data collection commenced in June 2016. A timeline was developed to ensure the timely procurement of services and materials, the training of assessment personnel, and the execution of the data collection. To adequately prepare for the data collection, subcontractors were selected to oversee data collection in each of the eight districts; the EGRA and survey instruments were digitized into an electronic data capture software application; MT and enumerator training was conducted; and, final enumerators were selected. During operational data collection, inter-rater reliability (IRR) was assessed and the level of agreement between observations was computed using the Kappa coefficient. Each of these tasks (and IRR results) are described in Annexes 1 and 2.

2.6 Data Analysis

In November 2016, EGRA statisticians developed a research plan for the student, teacher, and head teacher data. This plan included computing test quality statistics and analysis of EGRA and questionnaire data.

To answer the study’s research questions, the following summary scores were computed and analyses conducted: task scores for student groups (overall and by gender); proportion of students with zero scores; and, proportion of students who met pre-established thresholds or SRP standards (see section 3.2). Student, teacher, and head teacher questionnaire data on teacher and classroom reading practices were also summarized, and their relationship to student assessment results were analyzed. Throughout, changes between baseline and midline were examined to estimate changes in average reading abilities among Grade 2 students in each cohort. All analyses in this report use weighted data for both the baseline and midline results.

Descriptive and inferential statistics were computed by task and gender to summarize results, measure significance of differences in EGRA scores between groups and across time, and determine the strength of the relationship between midline EGRA scores and contextual variables from student, teacher, and head teacher questionnaires. Data were cleaned and analyzed using SPSS statistical software. Final graphs presented in this report were generated using Word and Excel.

2.7 Quality of Assessment Tools

Overall, the quality of the EGRA in measuring reading skills among students tested in 2016 was found to be very strong. See Annex 2 for details and the 2014 reports for additional statistics.


MAY 2017 19

CHAPTER 3: COHORT 1 RESULTS—SINDHI

This section summarizes Grade 2 students’ reading abilities in Cohort 1 across all SRP districts except Karachi where the EGRA is administered in Urdu; those results are presented in Chapter 6. The midline EGRA assessment in 2016 consisted of seven tasks; the baseline EGRA in 2014 consisted of six tasks (the letter name recognition task was added to the midline assessment).

Results in this chapter are presented by task and disaggregated further by gender and school location (urbanicity). First, overall results across all tasks are presented using mean scores (or fluency rates) followed by gain scores (midline mean minus baseline mean). Second, the proportion of students meeting or exceeding SRP performance standards is presented for both baseline and midline. Third, the proportion of students who were unable to answer a single question on the task (or zero scores) at midline compared to baseline, are reported by task. Fourth, results are discussed by task and gender and urbanicity. The section concludes with recommendations for supporting students in Cohort 1 to improve their reading.

3.1 Overall EGRA Results- Cohort 1 Sindhi

At baseline, the EGRA assessments were administered to 5,445 students in 489 schools across the seven Sindhi districts. Almost half of the baseline sample (53.2%) was boys. For the midline, the sample size for Cohort 1 was smaller with a total of 2,141 students in 188 schools across the seven Sindhi districts. At midline, there was an increase in the proportion of boys in the sample, up to approximately two-third of the sample (62.7%). Additionally, at midline, teacher questionnaires were administered to 188 Grade 3 teachers and to 188 head teachers in Cohort 1.

The EGRA includes two types of tasks—those that are timed and those that are untimed. Timed tasks measure a student’s fluency. Fluency rates were calculated for three EGRA tasks: nonword reading, passage reading, and letter name reading.15 passage reading (or oral reading) fluency is a reader’s ability to read accurately with speed and proper pacing; it is a common measure for determining reading skill. Fluency rates are presented as the number items on the task read correctly in one minute. Untimed tasks are reported as the number of items a student could complete correctly. Untimed tasks include phoneme isolation, reading comprehension and listening comprehension.

In all cases, these differences were gains from baseline results. Figure 2 below shows the average results from baseline and midline by EGRA task (tasks are shown in the order they were administered). An asterisk next to the task name indicates that the difference in mean scores between baseline and midline was statistically significant.

There was a statistically significant difference in mean scores among students in Cohort 1 on five of the six tasks common to baseline and midline—phoneme isolation, nonword reading, expressive vocabulary, passage reading and reading comprehension. In all cases, these differences were gains from baseline results.

15 The letter naming task was only administered at midline.


MAY 2017 20

FIGURE 2. AVERAGE EGRA SCORES AT BASELINE (2014) AND MIDLINE (2016) BY

TASK—SINDHI COHORT 1

Figure 3 below shows the average gain scores from baseline to midline.16

As noted above, the average score on six tasks increased significantly; only listening comprehension remained stable from baseline to midline. Specifically, on the phoneme isolation task, students answered 0.3 more items at midline than at baseline. On the nonword reading task, students at midline had an average of 3.4 more CNWPM than at baseline, having an average 11.4 CNWPM at baseline and 14.8 CNWPM at midline. Scores on expressive vocabulary increased at midline over baseline—with students answering 0.4 more items correctly at midline. Also, scores on oral reading fluency increase by 9.0 CWPM from baseline to midline. Finally, for reading comprehension, students responded correctly to an average of 1.0 items at baseline and an average of 1.4 correct answers at midline for a growth of an average of 0.4 more correct responses at midline.

FIGURE 3. AVERAGE EGRA GAIN SCORES FROM BASELINE (2014) TO MIDLINE

(2016)—SINDHI COHORT 1

16 Gains were calculated by subtracting the average weighted task score at baseline from the average weighted task score at midline.

When found to be statistically significant, positive gains indicate an increase in the average task score at midline, and negative gains indicate a decrease in the average task score at midline. When the difference is not statistically significant, however, the scores are said to be comparable, or stable over time. Throughout this report, an asterisk is used to indicate that a significant difference between the mean task score at baseline and midline was observed.

0.6

11.46.2

26.6

1.0 1.60.9

14.86.6

35.6

1.4 1.6

39.4

PhonemeIsolation (correct

out of 10)*

NonwordReading (CorrectNonwords Per

Minute)*

ExpressiveVocabulary

(correct out of10)*

Passage Reading(Correct WordsPer Minute) *

ReadingComprehension(correct out of

5)*

ListeningComprehension(correct out of 3)

Letter Name(Correct Letter

Names PerMinute)

Baseline Midline

0.3

3.4

0.4

9.0

0.4 0.0

Phoneme Isolation* Nonword Reading*Expressive

Vocabulary* Passage Reading *Reading

Comprehension*Listening

Comprehension

*Indicates the average weighted scores from baseline to midline were significantly different at p<0.05. Baseline scores are weighted although scores reported in the baseline report were not weighted. Notes: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Baseline n = 5,554; Midline n = 2,142.

*Indicates the average weighted scores from baseline to midline were significantly different at p<0.05. Notes: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Baseline n = 5,445; Midline n = 2,141.


MAY 2017 21

3.2 Results by SRP Standards

Standards of reading performance were established by the Sindh Reading Program (SRP) and the Pakistan Reading Program (PRP) throughout 2015, in close consultation with Sindh Education and Literacy Department (ELD) officials. The standards are in the process of being validated through a series of district- and provincial-level workshops including Sindh ELD officials, teachers and others.

While these standards were developed as part of the SRP activity, the standards of reading performance established through the project are expected to be adopted across all schools in Sindh province. As such, results for all students in Sindhi—presented in this report—are also examined against these standards. Table 5 presents the Standards of Reading Performance, the related EGRA task, the EGRA standard (or benchmark), and the proportion of students in Cohort 1 who met or exceeded the standard at baseline and midline.

TABLE 5: STANDARDS OF READING PERFORMANCE AND PROPORTION

STUDENTS MEETING STANDARDS AT BASELINE (2014) AND MIDLINE (2016)—

SINDHI COHORT 1

Performance Standard EGRA Task

EGRA Standard

Proportion of Students Meeting EGRA

Standard, 2014


Standard, 2016

Correctly remove, 70% to 80% of the time, an initial sound in a word to produce a new word.

Correctly blend, 70% to 80% of the time, individual phoneme sounds together to form a word.

Phoneme isolation 7 Correct

Items (of 10) 2.8% 3.9%

Correctly identify, 70% to 80% of the time: a word, a sentence, a letter.

Letter name recognition

70% Correct Letters

n/a 11.4%

Correctly read a list of grade-appropriate invented words at a rate of 25 to 35 correct nonwords per minute.

Nonword reading* 25 Correct Nonwords per Minute

20.8% 26.9%

Fluency: Read Grade 2 level text at a rate of 50 to 80 correct words per minute.

Accuracy: Correctly answer, orally, 4 out of 5 literal comprehension questions on a Grade 2 level text or story read independently or with support from teacher.

Make a reasonable connection between a character or event in a story or information in a text and his/her own life.

Passage reading* 50 Correct Words per

Minute 24.0% 32.0%

Reading comprehension

4 Correct Items (of 5)

11.5% 14.6%

Correctly answer, orally, 4 out of 5 simple literal comprehension questions on a grade-2 level text or story read aloud or told by the teacher.

Listening comprehension


59.7% 59.0%

*Indicates the average percentage of students meeting the standard at baseline and at midline were significantly differentat p<0.05. Note: Baseline n = 5,554; Midline n = 2,142.

The proportion of students meeting or exceeding the standards increase between baseline and midline for nonword reading and passage reading. For the other three common tasks, we can say that the results


MAY 2017 22

remained stable between the baseline and midline. At both baseline and midline, the largest proportion of students met the standard of at least 6 items correct for vocabulary (Baseline= 67.4%, Midline=73.3%). The proportion of students meeting or exceeding the standard for nonword reading and passage reading also increased significantly from baseline to midline for Cohort 1. In passage reading, for example, 24.0% of students met the standard at baseline compared to 32.0% at midline.

Figure 4 shows the proportion of students by gender at baseline and midline meeting or exceeding the standards. For girls, a significant increase was observed for the phoneme isolation, nonword reading, passage reading and reading comprehension tasks; boys saw a significant increase on the passage reading task. As a result, the observed gap between boys and girls who met or exceeded the standard narrowed from baseline to midline.

FIGURE 4. PROPORTION OF STUDENTS MEETING STANDARDS AT BASELINE

(2014) AND MIDLINE (2016) BY GENDER—SINDHI COHORT 1

3.3 Proportion of Students Receiving Zero-Scores

The results presented thus far examine the percentage of items answered correctly or fluency rates—i.e., what students were able to do, on average. The proportion of zero scores, however, provides information on the number of students who were unable to correctly answer even a single item on a task. zero scores provide critical warning bells to a system, highlighting students who will continue to be left behind if not provided significant remedial help.

Figure 5 shows the proportion of students who were identified as zero score students by EGRA task at baseline and midline. The proportion of students who were zero scores decreased in all tasks with significant decreases in phoneme isolation, nonword reading, and passage reading. expressive vocabulary and listening comprehension present the lowest proportion of students with zero scores, not allowing for much decrease in zero scores. This is particularly the case with expressive vocabulary with only 6.1% of zero score students at baseline and 4.5% at midline.

4.0%

25.3%28.2%

13.6%

59.5%

3.7%

11.4%

27.8%32.4%

13.6%

59.4%

1.4%

15.4%

19.0%

9.0%

59.6%

4.1%

11.4%

25.4%

31.3%

16.2%

58.3%

Phoneme Isolation* Letter NameRecognition

Nonword Reading * Passage Reading * ReadingComprehension *

ListeningComprehension

Baseline-Boys Midline-Boys Baseline-Girls Midline-Girls

*Indicates a significant difference between baseline and midline for at least one gender at p<0.05. Note: Baseline-Male n = 2,898; Midline-Male n = 1,343; Baseline-Female n = 2,547; Midline-Female n = 799.


MAY 2017 23

FIGURE 5. PROPORTION OF STUDENTS RECEIVING ZERO SCORES BY EGRA

TASK AT BASELINE (2014) AND MIDLINE (2016)—SINDHI COHORT 1

The largest proportions of zero scores were observed in the phonemic isolation task with 74.3% of students unable to answer a single question correctly at midline, representing a significant decrease from 85.4% at baseline. The second highest proportion of zero scores were observed in the reading comprehension task, with 46.3% of students receiving a zero score at midline, representing a significant decrease from 61.8% at baseline. The third highest proportion of zero scores was observed in the nonword reading task with 33.3% of students unable to answer a question correctly compared to 48.4% at baseline.

Passage Reading had the largest decrease in proportion of zero scores from baseline (41.8%) to midline (20.0%), representing a decrease of over 20%. Finally, letter name recognition results showed 9.3% of students receiving a zero score at midline (the task was not administered at baseline).

Figure 6 presents the proportion of zero score students by gender at midline only. The proportion of zero scores was lower for boys than girls on all tasks on the midline EGRA. However, this difference in the proportion of zero scores between boys and girls was not statistically significant for all seven tasks. The task with the largest difference between boys and girls was in reading comprehension, with 50.1% of girls receiving a zero score compared to 43.9% of boys.

FIGURE 6. PROPORTION OF STUDENTS RECEIVING ZERO SCORES AT MIDLINE

(2016) BY TASK AND GENDER—SINDHI COHORT 1

85.4%

48.4%

6.1%

41.8%

61.8%

14.6%

74.3%

33.3%

4.5%20.0%

46.3%

14.3% 9.3%

PhonemeIsolation*

NonwordReading *


PassageReading*

ReadingComprehension*


Letter Name

Baseline Midline

*Indicates the proportion of zero scores between boys and girls were significantly different at p<0.05.Note: Midline-Male n = 1,343; Midline-Female n = 799.

*Indicates the proportion of zero scores at baseline and midline were significantly different at p<0.05.Note: Baseline n = 5,445; Midline n = 2,142.

74.2%

31.3%

4.4%17.9%

43.9%

14.1% 9.1%

74.6%

36.4%

4.6%

23.4%

50.1%

14.6%9.6%

Phoneme Isolation* Nonword Reading* ExpressiveVocabulary*

Passage Reading* ReadingComprehension*


Letter NameRecognition

Boys at Midline Girls at Midline


MAY 2017 24

3.4 EGRA Results by Gender and Urbanicity

EGRA results were further disaggregated by gender to determine how girls’ and boys’ performance on each task compared; with sufficient sample sizes for disaggregation by location, results for students in urban and rural settings are also provided.

Figure 7 shows that, on the phoneme isolation task, mean scores differed from baseline to midline for both boys and girls. Girls saw a statistically significant increase in the average number of correct items (Average Gain = 0.5) from baseline to midline while boys saw an increase in average number of correct items of only 0.1 from baseline to midline. A similar trend was observed on nonword reading with an average gain of 2.3 CNWPM for boys and 4.4 CNWPM for girls. For passage reading, girls also showed a larger average gain of 12.1 CWPM compare to boys with 6.1 CWPM.

FIGURE 7. EGRA AVERAGE GAIN SCORES FROM BASELINE (2014) TO MIDLINE

(2016) BY GENDER—SINDHI COHORT 1

For expressive vocabulary and reading comprehension, the increase was similar and statistically significant for both boys and girls. On the expressive vocabulary task, the boys showed an average gain of 0.5 correct items between baseline and midline while the girls’ average gain was of 0.4 correct items. For the reading comprehension task, the average gain for boys was 0.3 correct items, and for girls was 0.5 correct items.

Finally, for the listening comprehension task, neither gender saw a significant gain in scores from baseline to midline. Girls saw stable results from baseline to midline, whereas boys saw a small change in average score from baseline to midline. Boys saw an increase in average correct items of 0.1 from baseline (Mean = 1.5) to midline (Mean = 1.6).

Midline results for the letter name recognition task show that boys and girls have similar average scores on this task (not shown on Figure 7). On average, boys had a fluency rate of 39.5 CLNPM and girls had a fluency rate of 39.2 CLNPM.

Figure 8 shows the average gain score from baseline to midline, by location and task.

0.12.3

0.5

6.1

0.3 0.10.5

4.4

0.4

12.1

0.5 0.0


Vocabulary * Passage Reading *Reading


Comprehension

Boys Girls

*Indicates a significant difference between baseline and midline for at least one gender at p<0.05. Note: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Note: Baseline-Male n = 2,898; Midline-Male n = 2,547; Baseline-Female n = 1,343; Midline-Female n = 799.


MAY 2017 25

Average gains across tasks were not consistent across rural and urban schools. For students living in rural location, differences between baseline and midline were significant for five of the tasks while there were no significant differences between baseline and midline gains for students living in urban areas.


(2016) BY LOCATION—SINDHI COHORT 1

Proportions of students receiving zero scores were also compared between students by location. Figure 9 presents the proportion of students with zero scores at midline by task and location. While the proportions of students with zero scores were slightly higher for students at rural schools at midline, they showed a statistically significant reduction in zero scores from baseline for four tasks. Students attending urban schools did not show any significant decreases from baseline.

FIGURE 9. PROPORTION OF STUDENTS RECEIVING ZERO SCORES AT MIDLINE

(2016) BY EGRA TASK AND SCHOOL LOCATION—SINDHI COHORT 1

Compared to baseline, there was a 5.7% (Baseline= 88.3%, Midline=82.6%) decrease in students with zero scores at rural schools for the phoneme isolation task. The decrease in the proportion of students attending rural schools with zero scores was similar for nonword reading (Baseline= 54.0%, Midline=39.2%) and reading comprehension (Baseline= 67.8%, Midline=52.9%) with a significant decrease of 14.8%. Finally, passage reading was associated with the biggest decrease in the proportion of zero scores for students at rural schools, decreasing from 47.2% at baseline to 27.0% at midline.

0.1

3.10.6

8.7

0.3 0.0-0.1 -0.4 0.3-1.8

0.0 0.1


Vocabulary * Passage Reading *Reading


Comprehension

Rural Urban

82.6%

39.2%

4.8%

27.0%

52.9%

16.5%10.1%

80.3%

35.7%

4.5%

26.8%

51.3%

12.2% 9.2%

PhonemeIsolation*

NonwordReading*


Passage Reading* ReadingComprehension*



Rural at Midline Urban at Midline

*Indicates a significant difference between baseline and midline for rural school locations p<0.05.Note: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Note: Baseline-Rural n = 5,071; Midline-Rural n = 4,544; Baseline-Urban n = 384; Midline-Urban n = 1,171.

*Indicates a significant difference between baseline and midline for rural school locations p<0.05.

Note: Baseline-Rural n = 5,071; Midline-Rural n = 4,544; Baseline-Urban n = 384; Midline-Urban n = 1,171.


MAY 2017 26

3.5 Description of SRP Participation & Contextual Variables

(Cohort 1, Sindhi)

The SRP midline study included questionnaires in which a series of questions were posed to the students taking the EGRA test and to their teachers and head teachers. The questions were designed to capture contextual variables such as reading behaviors in the home and at school, and student, teacher, and head teacher characteristics to examine the relationships between these contextual variables and students’ reading performance. Responses to these items are summarized below by theme. Many of these themes were also asked about at the baseline assessment, and where this comparison was feasible, changes in the responses from baseline to midline are discussed. Questions regarding the SRP intervention were only asked at midline and are therefore, reported for only that time point.

Student Background

Socio-Economic Status This theme asks questions that are indicators of the student’s socioeconomic status. At baseline, 76.3% of students indicated they watched television at home, 32.5% listened to the radio, and 13.5% said they have a computer at home. Results were similar at midline with 78.0% of students watching television at home, 40.6% listening to the radio, and 19.7% having a computer at home.

Disposition to Reading This theme asks students three questions about their confidence and affinity for reading. At midline, nearly all students indicated that they enjoyed their reading period in the last school year (Grade 2) and that they enjoy reading in general. When asked how students felt when learning to read, the majority (82.7%) said they felt confident, 12.7% said they felt anxious, and only 15 total students indicated that they do not like learning to read.

Attendance This theme asks questions surrounding student attendance over the last week. On average, students at baseline missed less school with an average of 2.3 days missed, compared to 2.7 days at midline. A similar proportion of students at midline (43.0%) also indicated that they missed any school in the last week compared to baseline (42.6%).

Parental Literacy This theme asks students about their parents’ literacy. At midline, only 39.8% of Cohort 1 students reported that their mother can read and write. A much larger proportion, 68.0%, indicated that their father can read and write.

Student Language Consistency

This theme includes questions regarding the language most often spoken at home and at school. At both baseline and midline, the most commonly spoken language at home was Sindhi. However, the proportion of students assessed at midline indicating Sindhi as the language their family speaks at home is slightly larger (77.2%) than at baseline (79.3%). At both time points, Balochi was the second largest proportion with 9.7% at midline and 10.9% at baseline. Also at both time points, Sindhi was most often identified as the language spoken at school (Midline=95.5%; Baseline= 97.7%).

Student— Reading at Home

Reading Resources at Home This theme includes questions regarding the student reading habits at home on their own. At midline, 23.1% of students indicated that they do not have any reading materials at home, which is a large decrease from baseline, where 38.5% indicated that they did not have any reading material at home. There was also a slight increase in the number of students responding that they practice silent reading at home, with 88.7% at midline and 81.0% at baseline.


MAY 2017 27

Reading Support at Home The Reading Support at Home theme includes items intended to get at the level of involvement in reading at home of the child’s family. At midline, 67.6% of students indicated that they have someone at home read aloud to them, compared to 49.8% at baseline. There was also an increase in the number of students responding that they practice reading aloud to someone at home, with 72.8% at midline and 50.6% at baseline.

Student— Reading at School

Reading Practice in School This theme includes items regarding the activities the students engage in during their reading lessons. At midline, 50.6% of students indicate that they always practiced reading aloud to the teacher or other students, and 46.5% said they sometimes did. At baseline, only 40.6% of students indicated that they always practiced reading aloud to the teacher or other students, and 49.1% said they sometimes did. Homework This theme includes items regarding the assignment of reading homework by the teacher. There was a large increase in the proportion of students at midline (78.2%) indicating that their teacher always assigned reading homework, compared with baseline (57.3%). School Resources This theme covers questions about the school and school library. At baseline, only 9.2% of students indicated that their school had a library, and 76.6% of the students with access to a library said they take books home from school. At midline, this increased to 32.3% of students indicating they have access to a library at school, and 65.0% indicating they take books home to read. It should be noted that students were asked specifically about the presence of a library within the school. Students may be reporting on the presence of reading corners or simply the presence of books in the classroom, as formal libraries are not present in primary schools in Sindh province.

Student Use of SRP Materials

This theme includes seven questions asked of the students regarding their recollection of SRP-provided readers in the classroom. Although students were surveyed 5 months after the completion of the intervention, three-quarters of all students in Cohort 1 reported that they recalled receiving any one of the six SRP readers asked about. Of the six readers, the most commonly recalled reader was Kenh ja tangun ahen with 45.9% of students recalling it. Mujhe aata hai was the least recalled reader with only 18.1% of students recalling this reader.

School Background

Demographics The Demographics theme includes items mostly about the gender makeup of the schools. At baseline, 56.2% of the Head Teachers who indicated their gender in the survey were male, where 77.8% indicated male at midline. A similar pattern followed for the gender of teachers surveyed, with 43.9% indicating male at baseline, but 80.5% indicating male at midline. Language of Instruction Language includes items about the language of instruction used in the schools surveyed. At baseline, 34.4% of Head Teachers indicated Sindhi as the language of instruction, and 20.4% indicated Urdu. At midline, 96.2% indicated Sindhi as the language of instruction, and only 1.1% indicated Urdu. Background This theme includes items regarding the academic background and tenure of the teachers and head teachers surveyed. At both baseline and midline, most head teachers interviewed were acting head teacher and not deputy head teacher. At baseline, head teachers had been in their position an average of 12.4 years, and at


MAY 2017 28

midline an average of 12.2 years. Head teachers surveyed at midline tended to have a little bit more education with 38.3% having Master’s degrees compared to 34.1% at baseline. Finally, more schools at midline tended to have a timetable (72.4%) compared to baseline (66.8%). School Information This theme includes items regarding school buildings and library access. At both baseline (4.1%) and midline (8.1%), very few head teachers indicated the presence of a library in the school. Of those schools with a library at baseline, half had a scheduled library time, and 61.9% indicated that students were able to take books home from the library. At midline, 60% of schools with a library had scheduled library time and 80% indicated that students could take books home. As with the student responses to these items, teachers and head teachers were asked specifically about libraries in the schools. At both time points, more than half of the head teachers indicated that there is clean drinking water at the school (Midline=60.0%; Baseline= 69.7%). Over two thirds of schools had toilet facilities, 72.4% at midline and 71.8% at baseline, and there was an increase in separate girls’ toilet facilities from baseline (39.7%) to midline (55.9%). School Management Committee This theme includes questions about the presence of a SMC and their role in the schools. The majority of schools at both baseline (96.5%) and midline (89.2%) have an SMC, and most meet once every two to three months (Midline=49.1%; Baseline= 44.8%). At both time points, the general purpose noted of the SMC was surrounding budget review, SMC fund spending, and the management of school infrastructure and equipment. However, from baseline to midline, a larger proportion of head teachers indicated SMC involvement in student-related issues, including: discussing student problems and solutions (Midline=75.8%; Baseline= 59.1%), discussing school curriculum (Midline=45.5%; Baseline= 34.3%), and managing the procurement and distribution of textbooks (Midline=50.0%; Baseline= 37.5%).

Teaching and Evaluating Reading in School

Timetable and Reading This theme includes items regarding the available timetable at the school and scheduled reading classes. When asked at baseline, 81.7% of head teachers indicated that the timeline for Grade 2 included periods for reading, and this increased to 94.0% at midline. However, at baseline, 62.2% of head teachers indicated that there were 5+ reading periods per week, but only 28.6% of head teachers at midline said the same. Expectations of Reading Skills This theme includes items in which teachers indicate the grade level in which they expect students to have gained certain reading skills. In general, teachers had much higher expectations of students at midline than baseline. The largest differences were for students to understand stories they read (Midline=79.7%; Baseline= 47.8%) and to understand stories they hear (Midline=77.3%; Baseline= 50.8%). Student Activities During Lesson This theme includes questions to the teacher about what the student activities include during the reading lesson. From baseline to midline, there was no change in the frequency of students repeating stories that teacher read them, students re-telling stories, students sounding out familiar words, students learning of new words, or students reading aloud to teachers or other students. However, there was an increase in the proportion of students copying words from the blackboard or textbook five days per week (Midline=67.4%; Baseline=46.5%), re-telling a story that they heard (Midline=39.5%; Baseline=30.2%), and answering comprehension questions based on text read aloud to them five days per week (Midline=51.2%; Baseline=43.3%), and answering comprehension questions based on text they read themselves five days per week (Midline=43.6%; Baseline=36.8%). Methods to Evaluate Student Learning of Reading This theme includes items asked of both the teacher and head teacher about evaluating student reading skills, and use of evaluation results. The use of written evaluation, oral evaluation, checking of exercise books, and the checking of homework were consistently noted as used in over 90.0% of the schools


MAY 2017 29

surveyed at both baseline and midline. The use of a lesson plan did decrease from baseline to midline from 69.8% to 55.1%. Teacher Observation This theme includes items regarding who is responsible for observing Grade 2 teachers in the classroom during reading period, and how often observation occurs in a school year. At both baseline (89.2%) and midline (94.6%), head teachers were most often those responsible for observing Grade 2 teachers in the classroom. Most head teachers at midline (51.9%) stated that teachers were observed four times throughout the school year compared to 59.9% at baseline.

SRP Intervention with Teachers and Head Teachers

SRP Participation This theme includes two items asked of head teachers and the same two items asked of teachers, regarding their recollection of participation in the SRP training. In Cohort 1, 91.7% of head teachers were certain that their schools participated in the training, and 79.7% of teachers surveyed were certain that they themselves participated in the training (10.5% were not sure, and 9.8% were certain that they did not participate). In addition, 71.9% of head teachers reported that they also participated in the training. Although teachers were surveyed 5 months after the intervention completed, 38.6% of teachers used an SRP lesson plan the day of the interview in their reading class. SRP Activities This theme includes 14 items in which teachers reported on their use of the read alouds provided by SRP as part of the Let’s Read Faster program, as well as their focus on reading concepts throughout the year. The majority of Cohort 1 teachers indicated that they used the read alouds provided by SRP daily, or two to three time per week (86.8%), and less than one percent said they never used the materials. The least commonly reported focuses of teachers last year were on the concepts Orientation to Print (38.6%) and Reading Fluency (29.0%). Conversely, Identifying Letter Sounds (72.8%) and listening comprehension (64.9%) were the most commonly reported concepts on which teachers focused. SRP Materials This theme includes items asked of both the teacher and head teacher regarding receipt, distribution, and use of the SRP materials. None of the head teachers surveyed indicated that they did not receive a single read aloud from SRP. However, Lomri aa ja school was received by only 72.4% of head teachers. In addition, this same reader was only currently present in 68.6% of schools. And, only 70.5% and 86.7% of head teacher reported distributing the various readers. In terms of the leveled readers, one head teacher reported that they did not receive any of the books, and the least often reader reported to have been received was Ahmed badak ain topi (65.7%). For teachers, 79.0% were using an SRP lesson plan the day of the interview, and 84.2% reported that they used the leveled readers daily or two to three times per week in the last year. One teacher reported that they did not receive any of the read alouds, and like head teachers, Lomri aa ja school was identified as the read aloud least often used with only 59.7% of teachers reporting that they used it last year. The read aloud most often identified as used in the classroom was Watayo faqeer ai khairat ji mani with 93.0% of teachers reporting that they used it last year. The leveled reader Kenh ja tangun ahen was reported as most often used (90.4%), and Mujhe aata hai was used least (65.8%). In terms of supplementary materials, 12.3% of teachers reported that they did not receive any supplementary materials. The most commonly reported supplementary materials received were letter cards (58.8%), vocabulary cards (50.0%), and reading cards (49.1%), and these were also the materials most reported as used in the classroom in the last year (60.5%, 44.7%, and 44.7%, respectively). Most of these materials were also reported as being present with the teacher on the day of the survey as well. Many of these supplementary materials were created with guidance from the TLAs. Read aloud books (90.4%),


MAY 2017 30

lesson plans (76.3%), and leveled readers (73.7%) were most often reported as present the day of the survey. However, 25.2% of the teachers also reported that they do not still use the materials with their students. Teaching and Learning Assistant (TLA) Involvement This theme includes items asked of both teachers and head teachers regarding the appointment of a TLA to their school, the number of visits the TLA made in the last year, and the type of feedback and support offered to the teachers. The majority (82.0%) of head teachers reported that they met their TLA, and none reported that they did not receive any visits by the TLA in the last year. TLAs were expects to visit schools twice per month over a minimum period of five months (for a total of 10 visits). The average number of months head teachers reported receiving a TLA visit was 2.4, and ranged from 0 to 7 months. Similarly, 88.0% of teachers reported a visit by the TLA to their classroom, with 2 reporting no visits at all. The average number of months teachers reported having visits by the TLA was 2.6, and ranged from 0 to 7 months. The majority of teachers (88.0%) also reported that they received guidance from their TLA in a pre-observation session, and the most commonly reported guidance from the TLA was regarding their teaching methodology (68.9%) and the content of the lessons (62.1%). SRP Effectiveness This theme includes 4 items regarding the teachers’ perception of the effectiveness of the components of the SRP program and materials. Over half of the teachers reported that they noticed a noticeable improvement (58.7%), and 36.1% noted a slight improvement in their students’ ability to read and comprehend text while using SRP materials and TLA supports. Decoding (30.8%) and reading fluency (21.1%) were reported by teachers as showing the greatest improvements in student performance. Interestingly, teachers were split regarding the most and least effective area of support by the TLA. While 50.4% (the largest proportion) of teachers reported that TLA support with effective focused instruction on at least one critical area of reading was the most effective, and 26.5% (the largest proportion) reported that this support was the least effective.

3.6 Relationship between Contextual Variables, SRP Participation

and EGRA Performance

As noted, the SRP 2016 study included questionnaires in which a series of questions were posed to the students taking the EGRA and to their teachers and head teachers. The questions were designed to capture contextual variables with the intention of examining the relationships between these contextual variables and students’ reading performance. To this end, responses to the questionnaire are summarized by theme and correlated with the EGRA oral reading fluency scores from the 2016 administration. Analyses of these variables revealed significant relationships for student, teacher, and head teacher themes. These relationships are useful to examine contextual factors that may positively, or negatively, influence student performance. Table 77 summarize results from student, teacher, and head teacher themes (each is discussed in detail following the tables). It is important to note that sample sizes can strongly effect the statistical significance of correlations; therefore, it is recommended to consider a correlation of 0.1 as small, 0.3 as moderate, and 0.5 as strong. Generally, all correlations were small. Positive correlations should be interpreted such that the scores on the two variables (in this case, midline average passage reading fluency scores, and a composite score) increase together. Conversely, negative correlations should be interpreted such that as one of the variables’ scores increases, the other variables’ scores decrease. Note that correlations that were not statistically significant are reported as 0.00 in the tables below as they should be interpreted as ‘no relationship’.


MAY 2017 31

TABLE 6: CONTEXTUAL VARIABLES—STUDENT QUESTIONNAIRE AND

EGRA PERFORMANCE—SINDHI COHORT 1

Student Survey Themes Correlation

Student Background

Socio-Economic Status 0.00

Reading Disposition 0.13*

Attendance -0.12*

Mother Literate 0.00

Father Literate 0.08*

Language Consistency 0.00

Reading Support in the Home 0.09*

Reading Resources in the Home 0.12*

Reading Support at School 0.08*

Reading Practice at School 0.08*

SRP Materials 0.11* *Composite score for theme is significantly correlated with the oral reading fluency 2016 score (p < .05). Non-significant correlations are shown above as 0.00.

TABLE 7: CONTEXTUAL VARIABLES—TEACHER & HEAD TEACHER

QUESTIONNAIRE AND EGRA PERFORMANCE—SINDHI COHORT 1

Teacher and Head Teacher Themes Correlation

Theme 1: School Background Staff Background 0.00

School Information 0.00

School Management Committee 0.00

Timetable 0.00

Teacher Age 0.00

Theme 2: Teaching and Evaluating Reading in School Expectations of Reading Skills 0.00

Reading Activities During Class 0.00

Teacher Practice -0.19*

Teaching Reading Strategies 0.00

Student Activities During Reading Lesson 0.00

Methods to Evaluate Student Learning of Reading 0.00

Homework 0.00

Teacher Observation 0.00

Head Teacher Involvement 0.00

Theme 3: SRP Intervention ‘Dosage’ 0.00 Participation in SRP Training 0.00

Use of SRP Materials in the Classroom 0.00

Availability of SRP Materials 0.00

TLA Involvement 0.00

SRP Effectiveness 0.15* *Composite score for theme is significantly correlated with oral reading fluency 2016 score (p < .05)

The sections below discuss each of these themes in detail for Cohort 1- Sindhi.


MAY 2017 32

Student Themes and EGRA Performance—Sindhi Cohort 1

The same students who participated in the EGRA in Cohort 1 were also surveyed via the student questionnaire; therefore, the relationship between their response on the survey and EGRA assessment were explored. Generally, when students had a higher disposition to reading language, resources and support for reading at home and at school, higher attendance, and access to more of the SRP materials, they also had higher passage reading (or oral reading) fluency scores.

Student Background Students who had a positive disposition to reading, had been absent fewer days from school, and whose fathers were literate, also had higher oral reading fluency scores.

Student reading disposition was computed using responses to three items: did you enjoy reading during the reading period last year; did you enjoy reading in general; and, did you feeling confident when learning to read. The composite score ranged from 0 (a student reported ‘no’ to all three questions) to 3 (a student reported ‘yes’ to all three questions)17. Students who had higher scores on the reading disposition composite (i.e., more positive disposition to reading) also had higher oral reading fluency scores. Student attendance was examined with one question—did you miss any school days last week; and if yes, how many days? The number of days students reported being absent ranged from 0 to 7. Students who reported higher numbers of days absent last week (closer to 7) had significantly lower reading fluency scores than students who reported fewer numbers of days absent last week (closer to 0).

In addition, students were asked if their father could read and write, and this response was correlated directly with fluency scores. Students who responded that their father was literate had significantly higher fluency scores than those students who said their father could not read and write.

Language Consistency Language consistency was examined by computing a composite score of three items—whether the student spoke Sindhi at home, read in Sindhi at school, and spoke in Sindhi at school. The composite score ranged from 0 (student reported ‘no’ to all three items) to 3 (student reported ‘yes’ to all three items).

The language consistency composite score was not significantly related to student fluency scores; students who used Sindhi at home and in the school had similar oral reading fluency scores to those who spoke other languages. It should be noted, however, that the majority of Cohort 1 students spoke Sindhi in the home and at school, and this lack of variability can make it difficult to see a relationship.

Reading at Home The availability of reading resources and the level of reading support received in the home were examined using two composite scores. The first—the availability of reading resources in the home—consisted of two items: do you have any reading materials at home; and, do you have books or reading materials for yourself at home. The second composite score—the level of reading support—consisted of four items: do you get help with reading at home; does someone read stories aloud to you at home; do you practice reading stories aloud to someone at home; and, do you practice silent reading at home. Composite score range was from 0 (student responded ‘no’ to all four items) to 4 (student responded ‘yes’ to all four items).

Both composite scores—reading support in the home and reading resources in the home—were positively and significantly correlated with student oral fluency scores. Specifically, students who had higher rates of literacy support in the home (closer to 4) and those who reported having reading resources generally in the home and specifically for themselves (scores closer to 2) had significantly higher oral fluency rates than

17 Note that for the third item, students were asked ‘how do you feel when you are learning to read’ and responses were dichotomized into students who reported feeling confident and those who reported feeling anxious/do not like to read.


MAY 2017 33

those with a lower score on the composites (closer to 0). These two composite variables were also significantly and positively correlated with one another. That is, students who reported having more reading resources in the home also reported increased reading support.

Reading at School Reading practices and the level of support received at school were examined in two composite scores. A composite score for reading practices at school was computed using three items: during your reading period last year, did you ever practice reading aloud to your teacher or other students; did you ever practice silent reading; and, did you listen to other students read aloud. This composite score ranged from 0 (student responded ‘no’ to all three items) to 3 (student responded ‘yes’ to all three items).

The school reading support composite score was computed based on the following six items: during your reading period last year, did your teacher talk to you about why reading is important; did your teacher help you when you made a mistake in reading; did your teacher ask you questions related to something you read; did your teacher help you read better; did your teacher help you write better; and, did your teacher assign reading for you to do at home. The composite score ranged from 0 (student reported ‘no’ on all six items) to 6 (student reported ‘yes’ on all six items).

Both composite scores—reading practices in the school and reading support in the school—were positively and significantly correlated with student oral fluency scores. Specifically, students who had higher rates of reading practices in school (closer to 3) and those who reported having higher levels of reading support in school (scores closer to 6) had significantly higher oral fluency rates than those with a lower score on the composites (closer to 0). These two composite variables were also significantly and positively correlated with one another. That is, students reporting increased reading support at school also reported increased reading practice.

SRP Materials SRP Materials was examined using six survey items. The items asked students if they recall working with the six distributed leveled readers, all identified by name. The composite scores ranged from 0 (students recalled none of the leveled readers) to 6 (students recalled all the leveled readers). This composite score was positively and significantly related to student fluency scores. Specifically, students who reported that they recalled more of the readers (closer to 6) had significantly higher fluency scores than those recalling fewer of the readers (closer to 0).

Teacher and Head Teacher Themes and EGRA Performance—Sindhi Cohort 1

In Cohort 1, 185 teachers and 185 head teachers were surveyed on questions pertaining to qualifications, attendance at training sessions, and school-related factors such as scheduling, teacher support, head teacher’s knowledge of student performance, teacher observation procedures, SMC authority, and facility improvements. Of the 185 teachers surveyed, 41 were female and 144 were male. Of the 185 head teachers surveyed, 41 were female and 144 were male head teachers. Below is a discussion of themes which were found to be significantly related to 2016 oral reading fluency scores.

School Background School background was examined using 5 composite scores. The first—staff background— consisted of 5 items surrounding the educational background of the teacher and head teacher, as well as the number of years of experience. Higher scores on this composite (closer to 5) indicate more education and experience on the part of the teachers and head teachers. The second—school information—also included 5 items regarding the availability of toilets and clean drinking water at the school, and whether the school has electricity. Higher scores on this composite (closer to 5) indicate higher levels of school infrastructure in place. Third—the SMC—includes 12 items regarding the presence of an SMC in the school, their frequency of meetings, and the types of content covered in the SMC meetings. Higher composite scores on the SMC composite (closer to 12) indicate that the SMC presented in the school meets more frequently and are more involved in a larger number of management concerns.


MAY 2017 34

The fourth composite—the timetable—includes 9 items that ask the teacher and head teacher about the use of a timetable and scheduled reading lessons. Higher scores on the timetable composite indicate the presence of a formal timetable that includes reading lessons, and that the frequency of reading lessons is higher. Finally, teacher age was represented by a single item wherein teachers were asked their year of birth and this was transformed to age and this relationship with reading fluency was investigated. None of these composite scores proved to be significantly related to student fluency scores. In fact, all correlations were close to zero indicating that there is no relationship between passage reading fluency and school and teacher background characteristics.

Teaching and Evaluating Reading in School Teaching and evaluating reading in school was examined using nine composites. The first—expectations of reading skills—includes 8 items asked of teachers regarding their expectations of basic reading skills. Higher values on this composite score (closer to 8) indicate that teachers had higher expectations of their Grade 2 students’ reading abilities. Second—reading activities during class—includes 2 items: ‘How often do your students read a story or text out loud in class?’ and ‘How often do you listen to individual students read out loud?’. Higher values on this composite (closer to 2) indicate that a teacher does both of these activities on a regular basis. Third—teacher practice—includes 9 items around support for struggling students and encouraging students to embrace reading. Higher values on this composite (closer to 9) indicates that teachers take varied actions when assisting struggling readers, and regularly encourage their students to read. The fourth composite—teaching reading strategies—includes 2 items: ‘Last year, how often did you conduct an activity with your students that included playing with sounds?’ and ‘Last year, how often did you conduct an activity with your students that included playing with letters?’. Higher values on this composite (closer to 2) indicate that a teacher does both of these activities on a regular basis. Fifth—student activities during reading lesson—includes 10 items asking teachers to identify the type of activities (retelling stories, reading aloud, etc.) they use during a lesson. Higher values on this composite (closer to 10) indicate that teachers use a greater number of strategies more regularly. The sixth composite—methods to evaluate student learning—includes 18 items asked of teachers and head teachers around the types of assessment they use to evaluate learning (written evaluations, oral evaluations, etc.) and how frequently they use them. Higher values on this composite (closer to 18) indicate that teachers and head teachers use varied approached to assessment regularly. The seventh composite—homework—includes 2 items: ‘Students were assigned reading to do on their own during school time.’ and ‘Students were assigned reading to do on their own at home.’. Higher values on this composite (closer to 2) indicate that teachers assigned reading to students to do on their own regularly. Eighth—teacher observation—was assessed with a single item: In an academic year, how often is a grade 2 teacher observed in her classroom?’. The responses on this item ranged from 0 to 4. Therefore, higher values on this composite (closer to 4) indicate that head teachers observed their grade 2 teachers’ reading lessons more frequently in the last year. Finally, ninth—head teacher involvement—includes 3 items asked of teachers and head teachers around the number of observations made in a year and the level of feedback provided to the teachers after observation. Higher values on this composite (closer to 3) indicate that teachers are observed more often and provided feedback as a result. Teacher practice was the only composite with a significant correlation to reading fluency. This relationship was negative, which indicates that teachers in schools with lower fluency scores reported using more varied strategies with their struggling readers.

SRP Intervention The relationship between SRP intervention and reading fluency was examined using 5 composites. First—Participation in SRP Training—includes 4 items around the reported participation in the face-to-face training by both the teachers and head teachers. The higher the value of this composite (closer to 4), the greater the participation in the SRP intervention. Second—Use of SRP Materials in the Classroom—includes 13 items regarding the teachers’ reported use of the SRP-provided read alouds and leveled readers


MAY 2017 35

in their classrooms. Higher values on this composite (closer to 13) indicate that teachers used more of the materials in the classroom. Third—Availability of SRP materials—includes 90 items asked of teachers and head teachers about their receipt of SRP materials last year. Higher values on this composite (closer to 90) are indicative of teachers and head teachers receiving, using, and retaining, the SRP-provided read alouds, leveled readers, lesson plans, and supplementary materials.

The fourth composite—TLA involvement—includes 37 items asked of both the teachers and head teachers regarding the number of visits last school year by the TLA, and the feedback provided to the teachers during pre-observation conferences. Higher values on this composite (closer to 37) indicate more frequent visits by a TLA, and more varied feedback provided to the teachers and head teachers. Fifth—SRP effectiveness—includes one item ‘Last year, did you see an improvement in your students’ ability to read and comprehend text while you were using the lesson plans and receiving TLA support visits?’. Higher values on this composite indicate that teachers perceived a greater improvement in their students’ reading ability during the SRP intervention.

SRP effectiveness was found to be significantly, and positively, correlated with student fluency scores. That is, the more effective the teachers perceived the intervention to be, the greater the associated fluency scores. It should be noted that scores cannot be attributed directly to teachers, as we do not know if the teachers completing the survey were also the grade two teachers of the students assessed by EGRA.

While none of the other four SRP composites were significantly related to fluency scores, there were significant relationships with others.18 For example, SRP participation was significantly related to SRP materials. This relationship is not surprising, as participation is difficult if materials are not available. TLA involvement is also positively related to SRP activities. This indicates that the more involved TLAs were within a school, the more likely teachers were to report using the SRP materials in their classrooms.

There were also significant relationships between SRP intervention composites and the other composites. SRP participation was positively related to Head Teacher Involvement, Teaching Reading Strategies, and Timetable use. That is, the greater the participation in the SRP intervention reported by teachers and head teachers, the more involved head teachers in the grade 2 reading classes, the more varied the strategies used in teaching reading, and the more likely there is a timetable used in the school that designates a reading period.

The composite SRP activities was also significantly related to Teacher Age, Homework, Teacher Practice, Teaching Reading Strategies, and SMC involvement. That is, the older the teacher, the more reading homework given to the students, and the more varied the strategies and activities used in the reading lessons, the more likely they were to use the SRP materials in the classroom. Increased TLA involvement was positively related to teacher practice, but was negatively related to head teacher involvement. That is, the more involved the TLA in terms of the number of visits and amount of feedback provided to the teachers, the more variable the activities used in their reading lessons. And, the more involved the TLA, the less involved the head teachers were in the reading teachers’ lessons. Finally, the greater the perceived effectiveness of SRP, the higher the value of the teaching reading strategies composite. Specifically, the more effective teachers perceived the intervention, the more varied their reported strategies for teaching reading in their classrooms.

3.7 Summary and Recommendations – Cohort 1 Sindhi

Cohort 1 results provide useful information about the average reading abilities as well as how much change has been observed between baseline and midline for the cohort across all seven Sindh districts. A summary of findings is presented below followed by recommendations.

18 The restricted number of contextual factors that have been found to be significant is explain in part by the size of the sample

that reduce the power of hypothesis testing. Power is defined as the capacity of a test to detect significant results and is directly related with the size of the sample were bigger sample allow to detect smaller relationship and smaller sample only large relationship. As a reminder, the size of the sample for the teacher and head teacher in cohort 1 was 188 while we have 2142 students.


MAY 2017 36

Summary

Midline results showed that the task with the largest average gain from baseline is passage reading, with half the proportion of lower scores compared to baseline. Students in the cohort have also significantly increased their average scores in nonword reading at midline compared to baseline, and showed a large decrease in the proportion of students receiving zero scores. However, students continue to struggle still with the phoneme isolation task, though there was a decrease in the proportion of zero scores from baseline to midline. Finally, the expressive vocabulary and Reading comprehension tasks saw a significant increase from baseline to midline.

When gain scores were compared across gender, results showed that girls had larger average gains than boys, especially for the passage reading and the nonword reading tasks. In addition, students who attended schools in rural areas also showed larger average gains from baseline to midline than those attending schools in rural areas. As with the gender differences, the gains are more pronounced for the passage reading and the nonword reading tasks.

Some changes in the contextual factors have also been observed from baseline to midline. First, students reported receiving more support in their reading at home, as well as at school where they report more reading practice and homework. Head teachers also reported more access to books. At the school level, head teachers report that their schools tend to use timetables more often.

For students, several themes were found to have a significant relationship with reading fluency at midline. Among those factors, it was found that fluency rates were higher when students reported: a stronger disposition toward reading, better attendance, parental literacy, reading resources and support in the home and at school, and had greater recollection of using SRP provided materials. For teachers and head teachers, there were also several themes found to be significantly related to student fluency scores. Of note, teachers’ increased perceptions of SRP intervention effectiveness on increasing student reading and comprehension was related to increased fluency scores.

Recommendations

Specific trends observed in the data from baseline to midline for Grade 2 students in Cohort 1 are discussed along with recommended next steps:

The phoneme isolation task appears to continue to be problematic for students in Cohort 1 atmidline. No improvement was observed for this task while tasks representing more “advanced”skills in reading such as nonword and passage reading have demonstrated improvement. Onepossible reason for this result may be that the phoneme isolation task on the EGRA assessesstudents’ ability to identify the last sound of a word, while the SRP materials (including lesson plans)focus instruction on identifying the first sound of a word instead. Due to inconsistent use ofdiacritics in the classroom, it is possible that students are not able to transfer their knowledge offirst sounds to identifying last sounds. These differences should be further investigated prior toendline.

A significant increase was observed on the passage reading task, but very little change in readingcomprehension was observed. It appears that students are reading more fluently, but theircomprehension of the text is poor. A deeper analysis of the comprehension questions or a moredeveloped study on reading comprehension could be conducted in order to shed more light onthe lack of gain on this important skill.

Girls in Cohort 1 seem to have “closed the gap” present at baseline with the boys. At baseline,boys were performing better than the girls, and at midline, the comparison of gains from baselineshows that girls are showing much greater improvement over time.


MAY 2017 37


The SRP midline study included two different cohorts of students who have not been exposed—as of November 2016—to SRP interventions. Students in Cohort 2 and some of Cohort 3 schools are expected to participate in a new round of program implementation in the next two years of the project. The selection criteria for these cohorts varied from Cohort 1, and are discussed in Chapter 2 of this report.

This chapter presents results for students in Cohort 2 in 2016—representing learning levels prior to the start of SRP interventions (essentially, the results from this midline evaluation represent the baseline for Cohort 2 students). At endline, data will be collected once again in Cohort 2 schools to compare to results from 2016.

Results for students in 2016 in Cohort 2 are based on EGRA assessments administered to 2,078 students in 182 schools across the seven Sindhi districts. Almost two-third of the students in this sample were boys and most of the students came from rural locations.

4.1 Overall EGRA Results

Students from Cohort 2 present an average of 0.6 correct answers out of 10 for the phoneme isolation task and 6.4 correct answers out of 10 for the expressive vocabulary task. On average, they can read 13.3 CNWPM and 30.1 CWPM. In comprehension, they have an average of 1.1 correct answers out of 5 for Reading comprehension and 1.5 out of 3 for listening comprehension. Finally, they have identified correctly an average of 37.4 CLNPM.

FIGURE 10. AVERAGE EGRA SCORES IN 2016 BY TASK—SINDHI COHORT 2

Comparisons with midline scores of Cohort 1 have been conducted. Significant differences between the results of the two cohorts have been observed for phoneme isolation, passage reading and Reading comprehension. Students in Cohort 1 have been able to identify an average of 0.9 correct items compared to Cohort 2 with an average of 0.6. For passage reading, students from Cohort 1 have significantly higher average fluency with approximately 5 more CWPM. Finally, in Reading comprehension, students in Cohort 1 have been able to produce an average of 1.4 correct answers compared to 1.1 for Cohort 2.


Table 8 presents the proportion of students in Cohort 2 who met or exceed the SRP standard in 2016. expressive vocabulary is the task that shows the highest proportion of students who met the standard with 70.1% of students. The second largest proportion is observed for listening comprehension with 55.2%. For nonword reading (23.6%) and passage reading (25.2%) about one out of four students met the

0.6

13.36.4

30.1

1.1 1.5

37.4


out of 10)


Minute)


(correct out of10)

Passage Reading(Correct Words

Per Minute)

ReadingComprehension(correct out of 5)



Names PerMinute)

Note: 2016 - n = 2078.


MAY 2017 38

standards. The tasks that present the lowest proportions are Letter Name Recognition and Reading comprehension with 8.6% and phoneme isolation with only 2.5% of students meeting the standard.


STUDENTS MEETING STANDARDS IN 2016—SINDHI COHORT 2

Performance Standard EGRA Task

EGRA Standard


Standard, 2016




Items (of 10) 2.5%



70% Correct Letters

8.6%


Nonword reading 25 Correct Nonwords per Minute

23.6%

Fluency: Read Grade 2 level text at a rate of 50 to 80 correct words per minute.

Accuracy: Correctly answer, orally, 4 out of 5 literal comprehension questions on a Grade 2 level text or story read independently or with support from teacher.


Passage reading 50 Correct Words per

Minute 25.2%



8.6%




55.2%

Keeping in mind the differences in selection criteria for the three cohorts, a comparison of the proportion of students in each cohort meeting the standards was conducted. Compared to Cohort 1 students, a significantly smaller proportion of Cohort 2 students met the standards for passage reading and Reading comprehension. For passage reading, 32.0% of students met the standard from Cohort 1 compared to 25.2% in Cohort 2. This difference cannot be attributed to SRP participation, but may be due to SRP participation and differences in the criteria for the two cohorts. Similarly, in Reading comprehension, 14.6% of students in Cohort 1 met the standard compared to 8.6% in Cohort 2. For all other tasks, the differences between the two cohorts were not statistically significant.

By gender, comparable proportions of boys and girls met or exceeded the standard across all EGRA tasks (Figure 11).


MAY 2017 39

FIGURE 11. PROPORTION OF STUDENTS MEETING STANDARDS IN 2016 BY

GENDER— SINDHI COHORT 2


The task with the highest proportion of zero scores was the phoneme isolation task with 82.1% of students with zero scores; similarly, on Reading comprehension more than one student out of two were zero scores. Tasks that were associated with the lowest proportion of zero scores included expressive vocabulary (6.3%) and Letter name identification (10.3%). nonword reading had 37.5% of students who were zero score, while passage reading had one quarter of students who were zero score (26.4%) and listening comprehension had one-fifth of students who were zero scores (19.1%) (Figure 12).


TASK IN 2016- SINDHI COHORT 2

Differences between Cohort 1 and Cohort 2 in the proportion of students who were zero scores was significant for phoneme isolation, passage reading, reading comprehension and listening comprehension. Again, these differences may be due to SRP participation and/or differences in criteria for selection in each cohort. For these four tasks, the proportions of students with zero scores were lower in Cohort 1.

2.6% 5.9%

23.5% 26.0%

9.5%

53.9%

2.2%

13.5%

23.9%29.6%

7.1%

57.7%

Phoneme Isolation Letter NameRecognition

Nonword Reading Passage Reading ReadingComprehension


Boys Girls

82.1%

37.5%

6.3%

26.4%

55.1%

19.1%10.3%

PhonemeIsolation

Nonword Reading ExpressiveVocabulary

Passage Reading ReadingComprehension


Letter Name

Note: 2016 - n = 2078.

Note: Boys n = 1,065; Girls n =579.


MAY 2017 40


TASK IN 2016 BY GENDER—SINDHI COHORT 2

The proportions of zero score for boys was comparable to the proportion of girls receiving zero scores on all tasks.

82.0%

36.1%

6.4%

24.9%

53.7%

20.2%11.6%

82.3%

40.1%

6.2%

29.1%

57.5%

16.8% 8.1%

PhonemeIsolation

NonwordReading





Boys Girls

Note: Boys n = 1343; Girls n = 735.


MAY 2017 41


Most schools in Cohort 3 are not expected to participate in SRP interventions. As with Cohort 2, Cohort 3 cannot serve as a comparison group to isolate effects of SRP interventions on students. See methodology for the population of Cohort 3 schools to which results are generalizable. Note that Cohort 3 consists of all schools that remained in the population after removing Cohorts 1 and 2. Following midline, actual selection criteria for inclusion in SRP interventions were determined and a cohort of schools were identified to receive interventions in Years 4 and 5. This updated population list will be the population from which endline results will be captured. This chapter presents results for students in Cohort 3 and may be useful at endline to compare gains for any schools that are in fact included in SRP interventions prior to the end of the project- the actual comparability of these data at endline will be based on a review of the populations of schools assumed for Cohort 3 at midline and the actual list of Cohort 3 schools as of endline. The results presented here are based on a sample of 1,495 students in 142 schools across the seven Sindhi districts. Almost two-third of the students in this sample were boys and most of the students came from rural locations.


Students from Cohort 3 had an average of 0.5 correct answers out of 10 for the phoneme isolation task and 6.6 correct answers out of 10 for the expressive vocabulary task. On average, they were able to read 12.4 CNWPM and 30.5 CWPM. In comprehension, Cohort 3 students had an average of 1.1 correct answers out of 5 for Reading comprehension and 1.6 out of 3 for listening comprehension. Finally, students were able to identify an average of 35.4 CLNPM (Figure 14).

FIGURE 14. AVERAGE EGRA SCORES IN 2016 BY TASK—SINDHI COHORT 3

Keeping in mind the difference in selection criteria for Cohort 1 and 3, differences in EGRA performance

between the two Cohorts has been conducted. Specifically, significant differences between the results of

the two cohorts have been observed for phoneme isolation, nonword reading, passage reading, Reading

comprehension and Letter Name Recognition. For phoneme isolation, students in Cohort 1 have been able

to identify an average of 0.9 correct items compared to Cohort 3 with an average of 0.2. On the nonword

reading task, students in Cohort 1 outperform students in Cohort 3 with an average of 14.8 correct items

compared to 12.4 for Cohort 3. For passage reading, students from Cohort 1 have a significant higher

average fluency with about 5 more CWPM than students in Cohort 3. In reading comprehension, students

in Cohort 1 have been able to produce an average of 1.4 correct answers compared to 1.1 for Cohort 3.

Finally, for letter name recognition, students in Cohort 1 present an average of 39.4 correct items compared

to 35.4 for Cohort 3. Again, these comparisons are provided in the context of the selection criteria for

Cohorts 1 and 3 and should not be used to interpret effects of SRP participation.

0.5

12.46.6

30.5

1.1 1.6

35.4


out of 10)


Minute)


(correct out of10)


Per Minute)




Names PerMinute)

Note: 2016 - n = 1495.


MAY 2017 42


Table 9 presents the proportion of students in Cohort 3 who meet or exceed the SRP standard at midline. The largest proportion of students who meet or exceed a standard was observed for listening comprehension with 58.5%. For nonword reading (21.3%) and passage reading (26.4%) about one out of four students met the standards. The tasks that present the lowest proportions are Letter Name Recognition with 7.8%, Reading comprehension with 10.3% and phoneme isolation with only 2.2% of students meeting the standard.


STUDENTS MEETING STANDARDS 2016—SINDHI COHORT 3

Performance Standard

EGRA Task EGRA

Standard


Standard, 2016




Items (of 10) 2.2%



70% Correct Letters

7.8%



21.3%

Fluency: Read Grade 2 level text at a rate of 50 to 80 correct words per minute. Accuracy: Correctly answer, orally, 4 out of 5 literal comprehension questions on a Grade 2 level text or story read independently or with support from teacher.



Minute 26.4%



10.3%




58.5%

Compared to Cohort 1 students, this cohort had a significantly lower proportion of students who met the SRP standard in phoneme isolation, Letter Name Recognition, nonword reading, passage reading and Reading comprehension. The largest differences were for nonword reading and passage reading with 5.6% less students in Cohort 3 that met the standard. There were also 4% less students in Cohort 3 who met the standard for Reading comprehension. For Letter Name Recognition, 11.4% of students in Cohort 1 met the standard compared to 7.8% for this cohort. Finally, 3.9% of students met the standard in phoneme isolation in Cohort 1 and 2.2 in Cohort 3.


MAY 2017 43


GENDER—SINDHI COHORT 3

By gender, comparable proportions of boys and girls met or exceeded the standard for all EGRA tasks.


The highest proportion of zero scores was observed on the phoneme isolation task with 83.8% of students with zero scores. Similarly, on Reading comprehension, more than one student out of two was a zero score. Tasks that were associated with the lowest proportion of zero scores were expressive vocabulary (4.3%) and Letter Name (10.0%). Nonword reading presents 40.3% of zero score, passage reading 28.4% and listening comprehension 15.5%.


TASK IN 2016 – SINDHI COHORT 3

Differences between Cohort 1 and Cohort 3 were significant for phoneme isolation, nonword reading, passage reading and Reading comprehension. For those four tasks, the proportions of students with zero scores were lower in Cohort 1.

1.8%6.9%

22.5%26.2%

10.4%

58.1%

2.7%9.9%

18.7%

26.9%

9.9%

59.2%




Boys Girls

83.8%

40.3%

4.3%

28.4%

53.2%

15.5%10.0%

PhonemeIsolation




Letter Name

Note: 2016 - n = 1495.

Note: Boys n = 1000; Girls n =495.


MAY 2017 44


TASK IN 2016 BY GENDER – SINDHI COHORT 3

85.2%

38.0%

3.6%

26.6%

52.2%

14.8% 9.7%

80.4%

45.6%

6.1%

32.3%

55.6%

17.1%10.7%

PhonemeIsolation





Boys Girls

Note: Boys n = 1000; Girls n =495.


MAY 2017 45

CHAPTER 6: COHORT 1 RESULTS—URDU

6.1 Overall EGRA Results- Cohort 1 Urdu

This section summarizes Grade 2 students’ reading abilities in Cohort 1 in Karachi district only. In this district the EGRA was administered in Urdu. Like the Sindhi version, the midline EGRA assessment in 2016 consisted of seven tasks; the baseline EGRA in 2014 consisted of six tasks. At baseline, the EGRA assessments were administered to 749 students in 70 schools across Karachi district. Almost half of the baseline sample (53.4%) was boys (see Table 3). For the midline, sample size for Cohort 1 is smaller with a total of 449 students in 42 schools across the seven Sindhi districts. The proportion of girls in the midline sample is higher than what have been observed at baseline. Additionally, teacher questionnaires were administered to 42 Grade 3 teachers and to 42 head teachers in Cohort 1. There was a statistically significant difference in mean scores among students in Cohort 1 on two of the six tasks common to baseline and midline—expressive vocabulary and listening comprehension.

FIGURE 18. AVERAGE EGRA SCORES AT BASELINE (2014) AND MIDLINE (2016)

BY TASK —URDU COHORT 1

The average score on two tasks increased significantly while scores on other tasks doesn’t show significant differences between midline and baseline. Specifically, student scores on expressive vocabulary increased at midline over baseline—students answered 2.1 more items correctly and in listening comprehension students got an average of 1.2 correct items at baseline while the average increased to 1.4 correct items at midline.

0.7

17.6

5.2

24.2

0.7 1.20.6

15.87.4

25.7

0.9 1.4

46.5


out of 10)

Nonword Reading(Correct

Nonwords PerMinute)


(correct out of10)*


Per Minute)



(correct out of 3)*


Names PerMinute)

Baseline Midline

*Indicates the average weighted scores from baseline to midline were significantly different at p<0.05. Baseline scores are weighted although scores reported in the baseline report were not weighted. Notes: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Baseline n = 749; Midline n = 449.


MAY 2017 46

FIGURE 19. AVERAGE EGRA GAIN SCORES FROM BASELINE (2014) TO MIDLINE

(2016)—URDU COHORT 1


Table 10 presents the Standard of reading performance, the related EGRA task, the EGRA standard (or benchmark), and the proportion of students in Cohort 1-Urdu who met or exceeded the standard at baseline and midline. In both 2014 and 2016, the largest proportion of students met the standard of at least 6 items correct for Vocabulary. In 2014, 48.9% of students met or exceeded the standard for Vocabulary as measured by the expressive vocabulary task, and this increased to 83.4% in 2016. listening comprehension present an important increase in the proportion of students who met the standard with 29.8% at baseline compare to 45.6% at midline. For all other tasks, there is no significant difference between the two times. Figure 19 shows the proportion of students by gender in 2014 and 2016 meeting or exceeding the standards. The increase between baseline and midline were significant for both gender for expressive vocabulary with an increase of 23% for boys and 47% for girls between baseline and midline. listening comprehension was also significant for bot gender. At baseline 30% of boys and 30% of girls met the standard while at midline 53% of boys and 39% of girls met the standard. Compared to boys, girls saw a significant increase in reading comprehension. At baseline 4% of boys and 6% of girls met the standard while at midline those proportion were of 10% for boys and 11% for girls. For this task, even though there was an increase for both genders, results were only significant for girls.

-0.2 -1.8 2.1

1.5

0.2 0.3

PhonemeIsolation

NonwordReading

ExpressiveVocabulary* Passage Reading

ReadingComprehension

ListeningComprehension*

*Indicates the average weighted scores from baseline to midline were significantly different at p<0.05. Baseline scores are weighted although scores reported in the baseline report were not weighted. Notes: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Baseline n = 749; Midline n = 449.


MAY 2017 47


STUDENTS MEETING STANDARDS AT BASELINE (2014) AND MIDLINE (2016)—

URDU COHORT 1


EGRA Task EGRA

Standard


Standard, 2014


Standard, 2016




Items (of 10) 3.5% 2.5%



70% Correct Letters

n/a 15.2%



23.6% 28.9%




Minute 16.5% 23.8%



5.3% 10.7%


Listening comprehension*


29.8% 45.6%

*Indicates the average percentage of students meeting the standard at baseline and at midline were significantly different at p<0.05. Note: Baseline n = 749; Midline n = 449.


MAY 2017 48

FIGURE 20. PROPORTION OF STUDENTS MEETING STANDARDS AT BASELINE

(2014) AND MIDLINE (2016) BY GENDER—URDU COHORT 1


Figure 20 shows the proportion of students who were identified as zero score students by EGRA task at baseline and midline. Overall, there was no task that showed significant increase or decrease between 2014 and 2016.


TASK AT BASELINE (2014) AND MIDLINE (2016)— URDU COHORT 1

Figure 21 disaggregates the proportion of zero score students by gender at midline only. The proportions of zero scores were lower for girls than boys on all tasks on the midline EGRA except for listening comprehension and Letter Name Recognition. Like the overall results, differences between baseline and midline were not significant for all common tasks. Proportion of zero scores seems to remain stable between baseline and midline for both boys and girls.

4.2%

17.8%

10.5%

4.1%

29.5%

2.7%

11.4%

25.9%

20.0%

10.2%

52.7%

2.6%

30.3%

23.3%

6.4%

30.3%

2.2%

18.7%

31.8%27.3%

11.1%

38.8%

Phoneme Isolation* Letter NameRecognition

Nonword Reading * Passage Reading * ReadingComprehension *


Baseline-Boys Midline-Boys Baseline-Girls Midline-Girls

83.8%

26.6%

5.8%

49.6%

67.7%

17.6%

86.1%

32.3%

3.8%

43.7%

67.0%

18.8%

2.7%

Phoneme Isolation Nonword Reading ExpressiveVocabulary



Letter Name

Baseline Midline

*Indicates the proportion of zero scores at baseline and midline were significantly different at p<0.05. Note: Baseline n = 749; Midline n = 449.

*Indicates a significant difference between baseline and midline for at least one gender at p<0.05. Note: Baseline-Male n = 400; Midline-Male n = 202; Baseline-Female n = 349; Midline-Female n = 247.


MAY 2017 49


TASK AT MIDLINE (2016) BY TASK AND GENDER—URDU COHORT 1

6.4 Results by Gender and Urbanicity

EGRA results were further disaggregated by gender to determine how girls’ and boys’ performance on each task compared. With a sufficient sample size, results were also disaggregated by urban/rural. Figure 22 shows the average gain score, by gender and task.

Increases in scores were not similar for boys and girls. For boys, the increases were significant for expressive vocabulary and listening comprehension while for girls the decrease was significant for nonword reading.


(2016) BY GENDER—URDU COHORT 1

EGRA results were further disaggregated by location (urban/rural) to determine how urban and rural student’s performance on each task compared. Figure 23 shows the average gain score, by location and task. Increases in scores are not similar for students in rural or urban location. For students living in rural location, there is a significant difference between average at baseline and midline for the phoneme isolation task. For students living in urban area, the differences were significant for nonword reading, expressive vocabulary and listening comprehension.

89.0%

40.8%

5.2%

50.0%

70.8%

14.8%1.6%

83.2%

24.1%

2.5%

37.6%

63.2%

22.7%

3.7%





Boys at Midline Girls at Midline

-0.4 -0.4

1.6

2.6

0.3 0.40.0

-3.7

2.7

-0.6

0.0 0.1

Phoneme Isolation Nonword ReadingExpressive

Vocabulary * Passage ReadingReading

ComprehensionListening

Comprehension*

Boys Girls

*Indicates the proportion of zero scores at between boys and girls were significantly different at p<0.05. Note: Midline-Male n = 202; Midline-Female n = 247.

* Indicates a significant difference between baseline and midline for at least one gender at p<0.05. Note: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Note: Baseline-Male n = 400; Midline-Male n = 202; Baseline-Female n = 349; Midline-Female n = 247.


MAY 2017 50


(2016) BY LOCATION—URDU COHORT 1

Proportions of students with zero scores were also compared between rural and urban students. Figure 24 presents the proportion of students with zero score at midline for both task and location. At midline proportions are similar for Reading comprehension and Letter Name Recognition. Proportion is higher for rural students in phoneme isolation while they are higher for urban students for all other tasks.

FIGURE 25. PROPORTION OF STUDENTS RECEIVING ZERO SCORES AT

MIDLINE (2016) BY TASK AND SCHOOL LOCATION —URDU COHORT 1

Compared to baseline, there are 16.4% more students with zero scores in rural areas on the phoneme isolation task. For urban students, the only significant difference between baseline and midline was for nonword reading— 46.5% at baseline down to 35.1% at midline.

6.5 Description of SRP Participation and Contextual Variables

(Cohort 1, Urdu)

The SRP midline study included questionnaires in which a series of questions were posed to the students taking the EGRA test and to their teachers and head teachers. The questions were designed to capture contextual variables, reading behaviors in the home and at school, and student, teacher, and Head Teacher characteristics to examine the relationships between these contextual variables and students’ reading performance. Some of the questions were also asked at baseline. To this end, responses to the items asked both at baseline and midline are summarized below by theme.

-0.7

3.50.5

13.2

0.2 0.40.1

-5.0

2.0

-1.8

0.0 0.3

PhonemeIsolation* Nonword Reading

ExpressiveVocabulary * Passage Reading

ReadingComprehension

ListeningComprehension*

Rural Urban

91.5%

24.4%

1.8%

36.0%

69.9%

15.8%0.3%

84.2%

35.1%

7.4%

45.8%

70.0%

20.3%

1.7%

PhonemeIsolation*

NonwordReading*





Rural at Midline Urban at Midline

*Indicates a significant difference between baseline and midline for at least one location at p<0.05. Note: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Note: Baseline-Rural n = 150; Midline-Rural n = 167; Baseline-Urban n = 604; Midline-Urban n = 969.

*Indicates a significant difference between baseline and midline for at least one location at p<0.05. Note: Baseline-Rural n = 150; Midline-Rural n = 167; Baseline-Urban n = 604; Midline-Urban n = 969.


MAY 2017 51

Student Background

Socio-Economic Status This theme asks questions that are indicators of the student’s socioeconomic status. At baseline, 73.4% of students indicated they watched television at home, 23.3% listened to the radio, and 22.2% said they have a computer at home. Results were similar at midline with 74.8% of students watching television at home, 37.0% listening to the radio, and 26.2% having a computer at home.

Disposition to Reading This theme asks students three questions about their confidence and affinity for, reading. At midline, nearly all students indicated that they enjoyed their reading period in the last school year (Grade 2) and that they enjoy reading in general. When asked how students felt when learning to read, the majority (89.1%) said they felt confident, 7.6% said they felt anxious, and only 6 total students indicated that they do not like learning to read.

Attendance This theme asks questions surrounding student attendance over the last week. On average, students at baseline missed less school with an average of 2.1 days missed, compared to 2.4 days at midline. Fewer students at midline (51.5%), however, indicated that they missed any school in the last week compared to baseline (52.1%). Parental Literacy This theme asks students about their parents’ literacy. At midline, 64.4% of Cohort 1 students reported that their mother can read and write. A much larger proportion, 76.8%, indicated that their father can read and write.

Student Language Consistency

This theme includes questions regarding the language most often spoken at home and at school. At both baseline and midline, the most commonly spoken language at home was Urdu. However, the proportion of students assessed at midline indicating Urdu as the language their family speaks at home is slightly larger (42.3%) than at baseline (33.3%). At both time points, Pashto was the second largest proportion with 19.2% at midline and 18.3% at baseline. Also at both time points, Urdu was most often identified as the language spoken at school (Midline=95.8%; Baseline= 97.6%).

Student— Reading at Home

Reading Resources at Home This theme includes questions regarding the student reading habits at home on their own. At midline, 24.3% of students indicated that they do not have any reading materials at home, which is a decrease from baseline, where 30.4% indicated that they did not have any reading material at home. There was also a slight increase in the number of students responding that they practice silent reading at home, with 79.5% at midline and 77.4% at baseline. Reading Support at Home The Reading Support at Home theme includes items intended to get at the level of involvement in reading at home of the child’s family. At midline, 71.7% of students indicated that they have someone at home read aloud to them, compared to 53.4% at baseline, a moderate increase. There was also an increase in the number of students responding that they practice reading aloud to someone at home, with 74.3% at midline and 51.7% at baseline.


MAY 2017 52

Student— Reading at School

Reading Practice in School This theme includes items regarding the activities the students engage in during their reading lessons. At midline, 37.9% of students indicate that they always practiced reading aloud to the teacher or other students, and 51.5% said they sometimes did. At baseline, only 32.5% of students indicated that they always practiced reading aloud to the teacher or other students, and 51.9% said they sometimes did.

Baseline 83.6%.and midline 86.2% results showed almost identical proportions of students indicating that they practiced silent reading during their lesson at

Homework This theme includes items regarding the assignment of reading homework by the teacher. There was a large increase in the proportion of students at midline (62.5%) indicating that their teacher always assigned reading homework, compared with baseline (40.2%).

School Resources This theme covers questions about the school and school library. At baseline, only 10.2% of students indicated that their school had a library, and 75.6% of the students with access to a library said they take books home from school. At midline, this increased to 36.2% of students indicating they have access to a library at school, and 63.0% indicating they take books home to read.

Student Use of SRP Materials

This theme includes 7 questions asked of the students regarding their recollection of SRP-provided readers in the classroom. Nearly one half (43.4%) of all students in Cohort 1 reported that they did not recall receiving any of the 6 SRP readers asked about. Of the six readers, the most commonly recalled reader was Baba with 40.3% of students recalling it. Mujhe aata hai was the least recalled reader with only 18.0% of students recalling this reader.

School Background

Demographics The Demographics theme includes items mostly about the gender makeup of the schools. At baseline, 40.0% of the Head Teachers indicated their gender in the survey were male, where 51.2% indicated male at midline. An increase in proportion is also observed for the gender of teachers surveyed, with 43.8% indicating male at baseline, but 80.5% indicating male at midline.

Language of Instruction Language includes items about the language of instruction used in the schools surveyed. At baseline, 97.1% of Head Teachers indicated Urdu as the language of instruction, and 4.3% indicated Sindhi. At midline, every Head Teachers indicated Urdu as the language of instruction.

Background This theme includes items regarding the academic background and tenure of the teachers and head teachers surveyed. At both baseline and midline, most Head Teachers interviewed were acting Head Teacher and not Deputy Head Teacher. At baseline, Head Teachers had been in their position an average of 11.9 years, and at midline an average of 10.2 years. Head Teachers surveyed at midline tended to have a little less education with 17.1% having Master’s degrees compared to 21.4% at baseline. Finally, a slightly smaller proportion of schools at midline tended to have a timetable (82.9%) compared to baseline (85.7%).

School Information This theme includes items regarding school buildings and library access. At both baseline (8.6%) and midline (12.2%), very few Head Teachers indicated the presence of a library in the school. Of those schools with a library, at baseline half had a scheduled library time, and 33.3% indicated that students were able to take books home from the library. At midline, all head teachers stated that they have scheduled library time


MAY 2017 53

and 60% that students could take books home. As previously noted, teachers and head teachers were asked specifically about the presence of libraries in the schools. As primary schools do not tend to have libraries, it is possible that some teachers and head teachers interpreted this as reference to the reading corners or sets of reading materials present in the classroom. At both time points, more than two third of the Head Teachers indicated that there is clean drinking water at the school (Midline=70.7%; Baseline= 71.4%). Over two thirds of schools had toilet facilities, 75.6% at midline and 78.6% at baseline, and there was a decrease in separate girls’ toilet facilities from baseline (50.0%) to midline (18.8%).

School Management Committee This theme includes questions about the presence of a SMC and their role in the schools. The majority of schools at both baseline (98.6%) and midline (97.6%) have an SMC, and most meet once a month (Midline=50.8%; Baseline= 35.0%). At both time points, the general purpose noted of the SMC was surrounding budget review, SMC fund spending, and the management of school infrastructure and equipment. However, from baseline to midline, a larger proportion of Head Teachers indicated SMC involvement in student-related issues, including: discussing student problems and solutions (Midline=71.0%; Baseline= 39.1%), discussing school curriculum (Midline=45.5%; Baseline= 24.6%), and managing the procurement and distribution of textbooks (Midline=50.0%; Baseline= 18.8%).

Teaching and Evaluating Reading in School

Timetable and Reading This theme includes items regarding the available timetable at the school and scheduled reading classes. When asked at baseline, 78.3% of Head Teachers indicated that the timeline for Grade 2 included periods for reading, and this increased to 85.3% at midline. At baseline, 37.0% of Head Teachers indicated that there were 5+ reading periods per week, a similar proportion is observed at midline with 37.9%.

Expectations of Reading Skills This theme includes items in which teachers indicate the grade level in which they expect students to have gained certain reading skills. In general, teacher had much higher expectations of students at midline than baseline. The largest differences were for students to understand stories they read (Midline=76.9%; Baseline= 50.6%) and to understand stories they hear (Midline=66.7%; Baseline= 52.9%).

Student Activities During Lesson This theme includes questions to the teacher about what the student activities include during the reading lesson. From baseline to midline, there was a decrease in every activities done 5 days a week included in the questionnaire: repeating sentences that teacher tells (Midline=30.8%; Baseline=50.2%), copying words from the blackboard or textbook (Midline=46.2%; Baseline=66.3%), re-telling stories (Midline=15.4%; Baseline=23.3%), re-telling a story that they read (Midline=10.3%; Baseline=26.7%), sounding out unfamiliar words (Midline=20.5%; Baseline=37.2%), learning of new words (Midline=20.5%; Baseline=50.0%), students reading aloud to teachers or other students (Midline=48.7%; Baseline=64.0%), and answering comprehension questions based on text read aloud to them Midline=25.64; Baseline=46.5%), answering comprehension questions based on text they read themselves (Midline=18.0%; Baseline=39.5%).

Methods to Evaluate Student Learning of Reading This theme includes items asked of both the teacher and Head Teacher about evaluating student reading skills, and use of evaluation results. The use of written evaluation, oral evaluation, checking of exercise books, and the checking of homework were consistently noted as used in over 90.0% of the schools surveyed at both baseline and midline. The use of a lesson plan did decrease from baseline to midline from 85.7% to 55.1%.


MAY 2017 54

Teacher Observation This theme includes items regarding who is responsible for observing grade 2 teachers in the classroom, and how often observation occurs in a school year. At both baseline (91.9%) and midline (90.2%), Head Teachers were most often those responsible for observing Grade 2 teachers in the classroom. Most Head Teachers at midline (70.7%) stated that teachers were observed four times throughout the school year compared to 68.1% at baseline.

SRP Intervention with Teachers and Head Teachers

SRP Participation This theme includes 2 items asked of head teachers and the same 2 items asked of teachers, regarding their recollection of participation in the SRP training. In Cohort 1, 95.4% of head teachers were certain that their schools participated in the training, and 80.0% of teachers surveyed were certain that they themselves participated in the training (4.6% were not sure, and 15.4% were certain that they did not participate). In addition, 60.3% of head teachers reported that they also participated in the training. However, only 24.1% of teachers reported using an SRP lesson plan the day of the interview.

SRP Activities This theme includes 14 items in which teachers reported on their use of the read alouds provided by SRP as part of the Let’s Read Faster program, as well as their focus on reading concepts throughout the year. Most Cohort 1 teachers indicated that they used the read alouds provided by SRP daily, or two to three time per week (88.9%), and one teacher said they never used the materials. The least commonly reported focus of teachers last year was on the concept Orientation to Print (57.4%). Conversely, Identifying Letter Sounds (85.2%), Word Recognition (81.5%) and Reading Fluency (74.1%) were the most commonly reported concepts that teachers focused on.

SRP Materials This theme includes items asked of both the teacher and head teacher regarding receipt, distribution, and use of the SRP materials. Four of the head teachers surveyed indicated that they did not receive a single read aloud from SRP. Lomri aa ja school was received by only 65.0% of head teachers, the least commonly receive of all read alouds. In addition, this same reader was only currently present in 60.0% of schools. In terms of the leveled readers, the least often reader reported to have been received was Ahmed badak ain topi (68.3%). This was also the reader least often in the school currently.

For teachers, 57.1% were using an SRP lesson plan the day of the interview, and 81.5% reported that they used the leveled readers daily or two to three times per week in the last year. One teacher reported that they did not receive any of the read alouds, and like head teachers, Lomri aa ja school was identified as the read aloud least often used with only 44.4% of teachers reporting that they used it last year. The read aloud most often identified as used in the classroom was Watayo faqeer ai khairat ji mani with 88.9% of teachers reporting that they used it last year. The leveled reader Baba was reported as most often used (90.7%), and Ahmed badak ain topi was used least (55.6%).

In terms of supplementary materials, 12.3% of teachers reported that they did not receive any supplementary materials. The most commonly reported supplementary materials received were teaching and learning aids that the teachers created themselves (64.8%). Letter cards (44.4%) and reading cards (40.7%) were the next most common materials received. The materials reported as used most often in the classroom in the last year were Letter Cards (63.0%), Vocabulary Cards (59.3%), and Word Cards (57.4%). Most of these materials were also reported as being present with the teacher on the day of the survey as well. Read aloud books (83.3%), Lesson Plans (79.6%), and leveled readers (74.1%) were most often reported as present the day of the survey. However, 18.9% of the teachers also reported that they do not still use the materials with their students.


MAY 2017 55

Teaching and Learning Assistant (TLA) Involvement This theme includes items asked of both teachers and head teachers regarding the appointment of a TLA to their school, the number of visits the TLA made in the last year, and the type of feedback and support offered to the teachers.

The majority (95.2%) of head teachers reported that they met their TLA, and none reported that they did not receive any visits by the TLA in the last year. TLAs were expects to visit schools twice per month over a minimum period of five months (for a total of 10 visits). The average number of months head teachers reported receiving a TLA visit was 2.4, and ranged from 0 to 7 months. Similarly, 96.9% of teachers reported a visit by the TLA to their classroom, with 2 reporting no visits at all. The average number of months teachers reported having visits by the TLA was 2.6, and ranged from 0 to 7 months. The majority of teachers (79.4%) also reported that they received guidance from their TLA in a pre-observation session, and the most commonly reported guidance from the TLA was regarding their teaching methodology (88.0%), the content of the lessons (78.0%), and the use of teaching and training materials (78.0%).

SRP Effectiveness This theme includes 4 items regarding the teachers’ perception of the effectiveness of the components of the SRP program and materials. Over half of the teachers reported that they noticed a noticeable improvement (60.0%), and 30.8% noted a slight improvement in their students’ ability to read and comprehend text while using SRP materials and TLA supports. Sight Recognition (21.5%) and reading fluency (20.0%) were reported by teachers as showing the greatest improvements in student performance. The largest proportion of teachers (34.9%) reported that TLA support with effectively focusing instruction on at least one critical area of reading was the most effective, and 23.8% (the largest proportions) reported that the least effective support from the TLA was on asking critical questions about reading materials to promote higher levels of thinking and learning, and on creating supplementary reading materials for the classroom.

6.6 Relationship between Contextual Variables, SRP Participation

and EGRA Performance

As noted, the SRP 2016 study included questionnaires in which a series of questions were posed to the students taking the EGRA and to their teachers and head teachers. The questions were designed to capture contextual variables with the intention of examining the relationships between these contextual variables and students’ reading performance. To this end, responses to the questionnaire are summarized by theme and correlated with the EGRA oral reading fluency scores from the 2016 administration.

Analyses of these variables revealed significant relationships for student, teacher, and head teacher themes. These relationships are useful to examine contextual factors that may positively, or negatively, influence students reading performance. Table 11 and Table 12 summarize results from student, teacher, and head teacher themes. It is important to note that sample sizes can strongly effect the statistical significance of correlations; therefore, it is recommended to consider a correlation of 0.1 as small, 0.3 as moderate, and 0.5 as strong. Generally, correlations were small. Positive correlations should be interpreted such that the scores on the two variables (in this case, midline average passage reading fluency scores, and a composite score) increase together. Conversely, negative correlations should be interpreted such that as one of the variables’ scores increases, the other variables’ scores decrease. Note than correlations that were not statistically significant were reported as 0.00 below, as they should be interpreted as ‘no relationship’.


MAY 2017 56

TABLE 11: CONTEXTUAL VARIABLES—STUDENT QUESTIONNAIRE AND

EGRA PERFORMANCE—URDU COHORT 1

Student Survey Themes Correlation

Student Background

Socio-Economic Status 0.00

Reading Disposition 0.11*

Attendance -0.11*

Mother Literate 0.00

Father Literate 0.00

Language Consistency 0.00

Reading Support in the Home 0.00

Reading Resources in the Home 0.14*

Reading Support at School 0.14*

Reading Practice at School 0.00

SRP Materials 0.00 *Composite score for theme is significant correlated with the oral reading fluency 2016 score (p < .05)

TABLE 12: CONTEXTUAL VARIABLES—TEACHER & HEAD TEACHER

QUESTIONNAIRE AND EGRA PERFORMANCE—URDU COHORT 1

Teacher and Head Teacher Themes Correlation

Theme 1: School Background Staff Background 0.00

School Information 0.00

School Management Committee 0.00

Timetable 0.00

Teacher Age 0.00

Theme 2: Teaching and Evaluating Reading in School Expectations of Reading Skills 0.00

Reading Activities During Class 0.32*

Teacher Practice 0.00

Teaching Reading Strategies 0.00

Student Activities During Reading Lesson 0.00

Methods to Evaluate Student Learning of Reading 0.00

Homework 0.00

Teacher Observation 0.00

Head Teacher Involvement 0.00

Theme 3: SRP Intervention ‘Dosage’ Participation in SRP Training 0.00

Use of SRP Materials in the Classroom 0.00

Availability of SRP Materials 0.00

TLA Involvement -0.42*

SRP Effectiveness 0.00 *Composite score for theme is significant correlated with oral reading fluency 2016 score (p < .05)

The sections below discuss each of these themes in detail for Cohort 1- Urdu.


MAY 2017 57

Student Themes and EGRA Performance—Urdu Cohort 1

The same students who participated in the EGRA in Cohort 1 were also surveyed via the student questionnaire. Generally, when students had a higher disposition to reading language, resources and support for reading at home and at school, higher attendance, and access to more of the SRP materials, they also had higher passage reading (or oral reading) fluency scores.

Student Background Students in Cohort 1 who had a positive disposition to reading and were absent fewer days the previous week also had higher oral reading fluency scores.

Student reading disposition was computed using responses to three items: did you enjoy reading during the reading period last year; did you enjoy reading in general; and, did you feeling confident when learning to read. The composite score ranged from 0 (a student reported ‘no’ to all three questions) to 3 (a student reported ‘yes’ to all three questions)19. Students who had higher scores on the reading disposition composite (i.e., more positive disposition to reading) also had higher oral reading fluency scores.

Student attendance was examined with one question—did you miss any school days last week; and if yes, how many days? The number of days students reported being absent ranged from 0 to 7. Students who reported higher numbers of days absent last week (closer to 7) had lower reading fluency scores than students who reported fewer numbers of days absent last week (closer to 0), and this relationship was significant.

In addition, students were asked if their mother and/or father could read and write, and these responses was correlated directly with fluency scores. Parental literacy was not significantly related to fluency scores for Cohort 1.

Language Consistency Language consistency was examined by computing a composite score of three items—whether the student spoke Sindhi at home, read in Sindhi at school, and spoke in Sindhi at school. The composite score ranged from 0 (student reported ‘no’ to all three items) to 3 (student reported ‘yes’ to all three items). The language consistency composite score was not significantly related to student fluency scores; students who used Urdu at home and in the school had similar oral reading fluency scores to those who spoke other languages. It should be noted, however, that the majority of Cohort 1 students spoke Urdu both in the home and at school, and this lack of variability can make it difficult to see a relationship.

Reading at Home The availability of reading resources and the level of reading support received in the home were examined using two composite scores. The first—the availability of reading resources in the home—consisted of two items: do you have any reading materials at home; and, do you have books or reading materials for yourself at home. The second composite score—the level of reading support—consisted of four items: do you get help with reading at home; does someone read stories aloud to you at home; do you practice reading stories aloud to someone at home; and, do you practice silent reading at home. Composite score range was from 0 (student responded ‘no’ to all four items) to 4 (student responded ‘yes’ to all four items).

Only one composite score—reading resources in the home—was positively and significantly correlated with student oral fluency scores. Specifically, students who reported having reading resources generally in the home and specifically for themselves (scores closer to 2) had significantly higher oral fluency rates than those with a lower score on the composites (closer to 0). However, these two composite variables were

19 Note that for the third item, students were asked ‘how do you feel when you are learning to read’ and responses were dichotomized into students who reported feeling confident and those who reported feeling anxious/do not like to read.


MAY 2017 58

also significantly and positively correlated with one another. That is, when students reported having greater reading resources in the home, they also reported more support.

Reading at School Reading practices and the level of support received at school were examined in two composite scores. A composite score for reading practices at school was computed using three items: during your reading period last year, did you ever practice reading aloud to your teacher or other students; did you ever practice silent reading; and, did you listen to other students read aloud. This composite score ranged from 0 (student responded ‘no’ to all three items) to 3 (student responded ‘yes’ to all three items).

The school reading support composite score was computed based on the following six items: during your reading period last year, did your teacher talk to you about why reading is important; did your teacher help you when you made a mistake in reading; did your teacher ask you questions related to something you read; did your teacher help you read better; did your teacher help you write better; and, did your teacher assign reading for you to do at home. The composite score ranged from 0 (student reported ‘no’ on all six items) to 6 (student reported ‘yes’ on all six items).

Only one of these composite scores—reading support in the school—was positively and significantly correlated with student oral fluency scores. Specifically, students who reported having higher levels of reading support in school (scores closer to 6) had significantly higher oral fluency rates than those with a lower score on the composites (closer to 0). However, these two composite variables were significantly and positively correlated with one another. That is, when students reported more reading support in the school they also reported increased reading practice.

SRP Materials SRP Materials was examined using 6 survey items. The items asked students if they recall working with the six distributed leveled readers, all identified by name. The composite scores ranged from 0 (students recalled none of the leveled readers) to 6 (students recalled all the leveled readers). This composite score was not significantly related to student fluency scores. That is, students who reported that they recalled more of the readers (closer to 6) did not have increased fluency scores than those recalling fewer of the readers (closer to 0).

Teacher and Head Teacher Themes and EGRA Performance—Urdu Cohort 1

In Cohort 1, 41 teachers and 41 head teachers were surveyed on questions pertaining to qualifications, attendance at training sessions, and school-related factors such as scheduling, teacher support, head teacher’s knowledge of student performance, teacher observation procedures, SMC authority, and facility improvements. Of the 41 teachers surveyed, 32 were female and 9 were male. Of the 41 head teachers surveyed, 21 were female and 20 were male head teachers. Below is a discussion of themes which were found to be significantly related to 2016 oral reading fluency scores.

School Background School background was examined using 5 composite scores. The first—background—consisted of 5 items surrounding the educational background of the teacher and head teacher, as well as the number of years of experience. Higher scores on this composite (closer to 5) indicate more education and experience on the part of the teachers and head teachers. The second—school information—also included 5 items regarding the availability of toilets and clean drinking water at the school, and whether the school has electricity. Higher scores on this composite (closer to 5) indicate higher levels of school infrastructure in place. Third—the SMC—includes 12 items regarding the presence of an SMC in the school, their frequency of meetings, and the types of content covered in the SMC meetings. Higher composite scores on the SMC composite (closer to 12) indicate that the SMC presented in the school meets more frequently and are more involved in a larger number of management concerns. Fourth—the timetable—includes 9 items that ask the teacher and head teacher about the use of a timetable and scheduled reading lessons. Higher scores on the timetable composite indicate the presence of a formal timetable that includes reading lessons, and that


MAY 2017 59

the frequency of reading lessons is higher. Finally, teacher age was represented by a single item wherein teachers were asked their year of birth and this was transformed to age and this relationship with reading fluency was investigated. None of these composite scores proved to be significantly related to student fluency scores. In fact, all correlations were close to zero indicating that there is no relationship between passage reading fluency.

Teaching and Evaluating Reading in School Teaching and evaluating reading in school was examined using 8 composites. The first—expectations of reading skills—includes 8 items asked of teachers regarding their expectations of basic reading skills. Higher values on this composite score (closer to 8) indicate that teachers had higher expectations of their grade 2 students’ reading abilities. Second—reading activities during class—includes 2 items: ‘How often do your students read a story or text out loud in class?’ and ‘How often do you listen to individual students read out loud?’. Higher values on this composite (closer to 2) indicate that a teacher does both activities on a regular basis. Third—teacher practice—includes 9 items around support for struggling students and encouraging students to embrace reading. Higher values on this composite (closer to 9) indicates that teachers take varied actions when assisting struggling readers, and regularly encourage their students to read. The fourth composite—teaching reading strategies—includes 2 items: ‘Last year, how often did you conduct an activity with your students that included playing with sounds?’ and ‘Last year, how often did you conduct an activity with your students that included playing with letters?’. Higher values on this composite (closer to 2) indicate that a teacher does both activities on a regular basis. Fifth—student activities during reading lesson—includes 10 items asking teachers to identify the type of activities (retelling stories, reading aloud, etc.) they use during a lesson. Higher values on this composite (closer to 10) indicate that teachers use a greater number of strategies more regularly. The sixth composite—methods to evaluate student learning—includes 18 items asked of teachers and head teachers around the types of assessment they use to evaluate learning (written evaluations, oral evaluations, etc.) and how frequently they use them. Higher values on this composite (closer to 18) indicate that teachers and head teachers use varied approached to assessment regularly. The seventh composite—homework—includes 2 items: ‘Students were assigned reading to do on their own during school time.’ and ‘Students were assigned reading to do on their own at home.’. Higher values on this composite (closer to 2) indicate that teachers assigned reading to students to do on their own regularly. Eighth—teacher observation—was assessed with a single item: In an academic year, how often is a grade 2 teacher observed in her classroom?’. The responses on this item ranged from 0 to 4. Therefore, higher values on this composite (closer to 4) indicate that head teachers observed their grade 2 teachers’ reading lessons more frequently in the last year. Finally, ninth—head teacher involvement—includes 3 items asked of teachers and head teachers around the number of observations made in a year and the level of feedback provided to the teachers after observation. Higher values on this composite (closer to 3) indicate that teachers are observed more often and provided feedback as a result. The composite Reading Activities During Class was significantly related to fluency scores. Specifically, the greater the number of student activities completed during a reading lesson as reported by the teachers, the higher the associated fluency scores.

SRP Intervention The relationship between SRP intervention and reading fluency was examined using 5 composites. First—SRP participation—includes 4 items around the reported participation in the face-to-face training by both the teachers and head teachers. The higher the value of this composite (closer to 4), the greater the participation in the SRP intervention. Second—SRP activities—includes 13 items regarding the teachers’ reported use of the SRP-provided read alouds and leveled readers in their classrooms. Higher values on this composite (closer to 13) indicate that teachers used more of the materials in the classroom. Third—


MAY 2017 60

SRP materials—includes 90 items asked of teachers and head teachers about their receipt of SRP materials last year. Higher values on this composite (closer to 90) are indicative of teachers and head teachers receiving, using, and retaining, the SRP-provided read alouds, leveled readers, lesson plans, and supplementary materials. The fourth composite—TLA involvement—includes 37 items asked of both the teachers and head teachers regarding the number of visits last school year by the TLA, and the feedback provided to the teachers during pre-observation conferences. Higher values on this composite (closer to 37) indicate more frequent visits by a TLA, and more varied feedback provided to the teachers and head teachers. Fifth—SRP effectiveness—includes one item ‘Last year, did you see an improvement in your students’ ability to read and comprehend text while you were using the lesson plans and receiving TLA support visits?’. Higher values on this composite indicate that teachers perceived a greater improvement in their students’ reading ability during the SRP intervention.

TLA involvement was found to be significantly, and negatively, correlated with student fluency scores. That is, those teachers in schools with lower fluency scores reported higher levels of TLA involvement. It should be noted that scores cannot be attributed directly to teachers, as we do not know if the teachers completing the survey were also the grade two teachers of the students assessed by EGRA. While none of the other SRP composites were significantly related to fluency scores, there were significant relationships with others. The composite SRP effectiveness was significantly related to SRP materials. This indicates that the more SRP materials the teachers and head teachers reported having received, used, and retained, the more effective they perceived the intervention in student learning. There were also significant relationships between SRP intervention composites and the other composites. SRP materials was positively related to methods to evaluate student learning. That is, the greater the number of SRP materials reported to have been received, used, and retained, the greater and more varied the methods teachers reported using for evaluation of student reading in the classroom. Finally, increased values in the SRP activities composite are significantly related to higher composite values on teacher practice. That is, the higher the reported use of the SRP-provided read alouds and leveled readers in their classrooms, the more teachers report taking varied actions when assisting struggling readers, and regularly encourage their students to read.

6.7 Summary and Recommendations – Cohort 1 Urdu

These SRP Urdu-medium Cohort 1 results provide useful information about the average reading abilities in Urdu, as well as how much change has been observed between baseline and midline across the Karachi district. A summary of findings is presented below followed by recommendations.

Summary

Compared to the baseline, midline results show that there are no significant differences across all the tasks. That is, across all 6 tasks, students are performing about the same at midline as they were at baseline. The same trend was observed for the proportion of students receiving zero scores, as there was no significant reduction in their proportions. The expressive vocabulary and Letter Name Recognition tasks had the lowest proportions of zero scores of the seven tasks. The gender difference analysis shows that boys and girls show similar average gains from baseline to midline. For the expressive vocabulary task, however, girls showed a greater improvement than boys. Similar trends were found for the analysis across students attending schools in rural areas vs. urban areas. Some changes in the contextual factors have been observed between baseline and midline. First, students are receiving more reading support at home. Also, a larger proportion of students reported at midline that they are receiving reading homework. A greater number of students at midline report that there is a library at their school, but this is not consistent with responses from head teachers. It is possible that students are


MAY 2017 61

becoming more aware of libraries in their schools. Finally, teachers at midline have higher expectations of students regarding the grade level in which they have gained certain reading skills. For students, several themes were found to have a significant relationship with reading fluency at midline. Among those factors, it was found that fluency rates were higher when students reported: a stronger disposition toward reading, better attendance, reading resources in the home, and reading support at school. For teachers and head teachers, there were also two themes found to be significantly related to student fluency scores. Interestingly, increased TLA involvement was related to lower fluency scores. This result should be interpreted cautiously, however, given the small sample size. In addition, the greater the number of reading activities teachers report completing in the classroom, the higher the associated fluency scores.

Recommendations

Specific trends observed in the data from baseline to midline for students in Cohort 1 are discussed along

with recommended next steps:

Students showed a slight improvement on pre-reading skills as shown in gains on the task expressive vocabulary, and the higher fluency rates for the Letter Name Recognition task. It is possible that teachers have been focusing more on pre-literacy skills than developing fluency and comprehension. However, this is contrary to the reported activities taking place in reading lessons, wherein teachers report student reading and the learning the meaning of new words much more frequently than pre-reading tasks such as sounding out new words.

Performance overall for Cohort 1 was not as strong as for the Sindhi group. There may be many reasons for this difference, but a thorough analysis of the fidelity of implementation across the two language groups could shed some light on these differences. On the surface, the Urdu Cohort overall reported lower levels of receipt and use of SRP provided materials. That is, students in the Urdu cohort recalled fewer of the leveled readers, and teachers reported receiving and using fewer of the read aloud, leveled readers, and supplementary materials. While there are many factors at play, a lack of materials could be contributing to the lack of gains seen in this cohort.


MAY 2017 62


This chapter presents results for students in Cohort 2 in Karachi in 2016—representing learning levels prior to the start of SRP interventions (essentially, the results from this midline evaluation represent the baseline for Cohort 2 students). At endline, data will be collected once again in Cohort 2 schools to compare to results from 2016. In 2016, the Urdu EGRA assessments were administered to 449 students in 40 schools across the Karachi districts. Almost half of the students in this sample were boys and most of the students came from urban locations.


Students from Cohort 2 had an average of 0.4 correct answers out of 10 for the phoneme isolation task and 6.8 correct answers out of 10 for the expressive vocabulary task. On average, they are able to read 12.6 CNWPM and 20.1 CWPM. In comprehension, they have an average of 0.5 correct answers out of 5 for Reading comprehension and 1.3 out of 3 for listening comprehension. Finally, they have identified correctly an average of 44.9 CLNPM.

FIGURE 26. AVERAGE EGRA SCORES IN 2016 BY TASK —URDU COHORT 2

Comparisons with midline scores of Cohort 1 were conducted. For all tasks of the EGRA, no statistically significant differences were found between results of Cohort 1 and Cohort 2 Urdu.


Table 13 presents the proportion of students in Cohort 2 who met or exceed the SRP standard at midline for Urdu. expressive vocabulary had the highest proportion of students who met the standard with 74.0% of students. The second largest proportion was observed for listening comprehension with 43.0%. The tasks with the lowest proportion of students who meet or exceed the standard were phoneme isolation with only 2.9% of student meeting the standard and reading comprehension with 3.2% of students meeting the standard. Letter name recognition had 10.9% of students meeting the standard while that proportion was 19.0% for nonword reading and 16.3% for passage reading. Compared to Cohort 1 students, this cohort present a significant lower proportion of students who met the standard only for Reading comprehension. For this task, 10.7% of students met the standard in Cohort 1 while this proportion is only 3.2% for Cohort 2. For all other task, the differences between the two cohorts were not statistically significant.

0.4

12.66.8

20.1

0.5 1.3

44.9


out of 10)


Minute)


(correct out of10)


Per Minute)




Names PerMinute)

Note: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Note: 2016- n = 458.


MAY 2017 63


STUDENTS MEETING STANDARDS IN 2014 AND 2016— URDU COHORT 2


EGRA Task EGRA

Standard


Standard, 2016




Items (of 10) 2.9%



70% Correct Letters

10.9%



19.0%




Minute 16.3%



3.2%




43.0%


GENDER— URDU COHORT 2

2.00%8.20%

13.50%10.50%

3.40%

45.40%

4.17%

15.10%

27.90% 25.80%

2.80%

39.10%




Boys Girls

Note: Boys-n = 252; Girls-n =206.


MAY 2017 64

For most of the EGRA tasks, a higher proportion of girls met the standard than boys. The exception was Reading comprehension where the proportions were comparable and listening comprehension where a higher proportion of boys met the standard than girls. The largest difference was for passage reading where 11% of boys met the standard comparing to 26% for girls.


Phoneme Isolation is the task that presents the highest proportion of zero scores with 91.0% of students with zero scores. Reading comprehension also has a rather large proportion with more than three students out of four who got zero scores on this task. Tasks that are associated with lowest proportion of zero scores are expressive vocabulary (4.9%) and Letter Name (0.8%). nonword reading presents 31.7% of zero score, passage reading 46.8% and listening comprehension 25.6%.


TASK IN 2016—URDU COHORT 2

Differences between Cohort 1 and Cohort 2 were significant only for listening comprehension. For this task, 25.6% of students in Cohort 2 had zero score compared to 18.8% of students in Cohort 1. For all other task, the differences between the two cohorts were not statistically significant.


TASK AND GENDER IN 2016—URDU COHORT 2

The proportions of students with zero scores were higher for boys for phoneme isolation, nonword reading, passage reading and Reading comprehension. For listening comprehension, the girls had the highest proportion of zero scores. The biggest difference observed between genders was for passage reading where 55.3% of boys have zero scores comparing with girls with 32.6%.

91.0%

31.7%

4.9%

46.8%

76.0%

25.6%

0.8%

PhonemeIsolation




Letter Name

92.6%

39.0%

4.4%

55.3%

84.2%

24.3%

0.9%

88.4%

19.4%5.7%

32.6%

62.5%

27.8%

0.6%





Boys Girls

Note: 2016-n =458.

Note: Boys-n = 252; Girls-n =206.


MAY 2017 65


Most schools in Cohort 3 are not expected to participate in SRP interventions. As with Cohort 2, Cohort 3 cannot serve as a comparison group to isolate effects of SRP interventions on students. See methodology for the population of Cohort 3 schools to which results are generalizable. Note that Cohort 3 consists of all schools that remained in the population after removing Cohorts 1 and 2. Following midline, actual selection criteria for inclusion in SRP interventions were determined and a cohort of schools were identified to receive interventions in Years 4 and 5. This updated population list will be the population from which endline results will be captured. This chapter presents results for students in Cohort 3 and may be useful at endline to compare gains for any schools that are in fact included in SRP interventions prior to the end of the project- the actual comparability of these data at endline will be based on a review of the populations of schools assumed for Cohort 3 at midline and the actual list of Cohort 3 schools as of endline. The results presented here for students in Cohort 3 are based on Urdu assessments administered to 229 students in 20 schools across Karachi district. Less than half of the students in this sample were boys and most of the students came from urban location.


Students from Cohort 3 had an average of 0.7 correct answers out of 10 for the phoneme isolation task and 6.8 correct answers out of 10 for the expressive vocabulary task. On average, they are able to read 15.3 CNWPM and 28.0 CWPM. In comprehension, they have an average of 0.8 correct answers out of 5 for Reading comprehension and 1.5 out of 3 for listening comprehension. Finally, they have identified correctly an average of 47.7 CLNPM.

FIGURE 30. AVERAGE EGRA SCORES IN 2016 BY TASK — URDU COHORT 3

Comparisons with midline scores of Cohort 1 were conducted. For all tasks of the EGRA, no statistically

significant differences were found between results of Cohort 1 and Cohort 3.


Table 14 presents the proportion of students in Cohort 3 who met or exceed the SRP standard in 2016. The largest proportion of students meeting or exceeding the SRP standard for the task was observed for listening comprehension with 49.8%. For nonword reading (23.6%) and passage reading (25.2%) less than one out of four students met the standards. The tasks with the lowest proportions were Letter Name Recognition with 15.6%, Reading comprehension with 8.9% and phoneme isolation with only 5.0% of student meeting the standard.

0.7

15.3

6.8

28.0

0.8 1.5

47.7


out of 10)


Minute)


(correct out of10)


Per Minute)




Names PerMinute)

Note: Mean score estimates are noted above each bar in the graph. The Confidence Interval (CI) is indicated by the lines at the top of each bar. CI indicates a range of values that’s likely to encompass the true value. Note: 2016- n = 229.


MAY 2017 66

Comparisons with midline scores of Cohort 1 were conducted. For all tasks of the EGRA, no statistically significant differences were found between results of Cohort 1 and Cohort 3.


STUDENTS MEETING STANDARDS IN 2016—URDU COHORT 3


EGRA Task EGRA

Standard


Standard, 2016




Items (of 10) 5.0%



70% Correct Letters

15.6%



23.3%




Minute 22.3%



8.9%




49.8%

FIGURE 31. PROPORTION OF STUDENTS MEETING STANDARDS 2016 BY

GENDER— URDU COHORT 3


MAY 2017 67

For all EGRA tasks, girls present a higher proportion of students who met the standard than boys. The largest difference is for Letter Name Recognition where 5% of boys met the standard comparing to 23% for girls.


Phoneme Isolation was the task with the highest proportion of zero scores (84.7%). Reading comprehension was also associated with rather large proportion with 69.0% of students with zero scores. Tasks that were associated with lowest proportion of zero scores included expressive vocabulary (6.7%) and Letter Name (1.3%). nonword reading presents 32.6% of zero scores, passage reading 42.3% and listening comprehension 17.6%.


TASK IN 2016—URDU COHORT 3

Comparisons with 2016 scores of Cohort 1 were conducted. For all tasks of the EGRA, no statistically significant differences were found between results of Cohort 1 and Cohort 3.


TASK AND GENDER IN 2016—URDU COHORT 3

87.6%

47.6%

62.9%

77.5%

24.5%

2.7%

82.8%

22.6%28.7%

63.4%

13.1%0.4%

Phoneme Isolation Nonword Reading Passage Reading ReadingComprehension



Boys Girls

84.7%

32.6%

6.7%

42.3%

69.0%

17.6%

1.3%

Phoneme IsolationNonword Reading ExpressiveVocabulary



Letter Name

Note: Boys- n = 109; Girls – n =120.

Note: 2016- n = 229.


MAY 2017 68

The proportions of students with zero scores are higher for boys for all tasks. The biggest difference observed between genders was for passage reading where 62.9% of boys have zero scores comparing with girls with 28.7%.

CHAPTER 9: OVERALL DISCUSSION AND

RECOMMENDATIONS

This section presents a discussion of the results of the EGRA and SMES in regard to overall outcomes and programmatic implications.

Overall Outcomes

1. There were improvements in reading skills, as well as a reduction in zero scores, seen in many areas of reading, from baseline to midline.

2. Across languages, phoneme isolation and reading comprehension continues to be an area of relative weakness for students, even at midline. Alternatively, listening comprehension is an area of relative strength, even prior to the intervention.

a. Sindhi-Medium Schools - The most consistent gains overall for Cohort 1, as well as by district,

were in the area of Passage Reading. There were also significant increases in phoneme isolation, nonword reading, passage reading, and reading comprehension. There was a significant reduction in zero scores for phoneme isolation, nonword reading, passage reading. Girls and students in rural areas made the most gains in all areas of reading, with the exception of listening comprehension.

b. Urdu-Medium Schools – The most consistent gains were in the area of expressive vocabulary. Girls made the most gains in the areas of vocabulary and listening comprehension. Students in rural areas made the most gains in phoneme isolation and listening comprehension, and students in urban areas made the most gains in vocabulary.

Programmatic Implications

1. Implementation of the SRP intervention was not uniformly high across the board. A high percentage of teachers attended the SRP training (80% or more across all schools). In addition, even 5 months after the completion of the SRP intervention in schools, a small proportion of teachers were using the SRP lesson plans on the day of the interview (38% Sindhi-medium schools; 24% Urdu-medium schools). A higher percentage of teachers were using the SRP materials than using the lesson plans

87.6%

47.6%

12.2%

62.9%

77.5%

24.5%

2.7%

82.8%

22.6%

3.0%

28.7%

63.4%

13.1%0.4%

Phoneme IsolationNonword Reading ExpressiveVocabulary




Boys Girls

Note: Boys- n = 109; Girls – n =120.


MAY 2017 69

(Sindhi-medium schools - 75% using materials; Urdu-medium schools - 82% using materials). While the persistence of materials use is promising, it is not consistent across schools or districts. Recommendation: Determine barriers to higher levels of implementation and address those concerns, such that level of implementation is uniformly high.

2. Positive feelings about the intervention resulted in positive outcomes for students. The more effective teachers felt the intervention was, the higher the fluency scores were for students. This result was significant for Sindhi and a positive correlation for Urdu. The perception of effectiveness for SRP appears to lead to the use of the SRP materials, which in turn correlates to positive student outcomes. Recommendation: Explore what elements correspond to teachers’ positive feelings about the intervention, and then replicate these elements.

3. Student outcomes were related to intervention implementation. For students in both Sindhi- and Urdu-medium schools, students who recalled that teachers were using the SRP readers (a proxy for intervention implementation), had higher fluency scores. Recommendation: Consider sharing with teachers the positive outcomes associated with using SRP so they understand that their use of the intervention can lead to positive outcomes for their students.

4. Teachers in low-performing schools reported higher rates of TLA involvement. A large percentage of teachers/head teachers met with a TLA (80% Sindhi-medium schools; 90% Urdu-medium schools) and received guidance on teaching methods and content of lessons. Those teachers at schools with the lowest fluency scores also reported the highest rates of TLA involvement throughout the intervention. This relationship is not well understood. It may be that the TLAs noted the need in lower performing schools and focused attention to meet increased needs. Recommendation: Consider exploring this result further to determine the way in which TLAs focus their resources and involvement.

5. When there is a focus on skills, there is an improvement in fluency. In some instances, as compared with students at baseline, students at midline were significantly higher in selected reading skills, when that skill was the focus of instruction. For example, for Sindhi-medium teachers/head teachers, there was a reported increase from baseline to midline in the amount of time spent answering comprehension questions, and this corresponded with an increase in reading comprehension scores for students overall, and more specifically for girls, and students attending schools in rural areas.

Alternatively, when a skill was not taught or the focus instruction, we see declines from baseline to midline. Specifically, teachers/head teachers reported a decrease in the amount of time for answering comprehension questions in the classroom, and the midline results showed a decline from baseline in reading comprehension for students at Urdu-medium schools overall, as well as for girls and students in rural areas.

Recommendation: Share this result with teachers so they recognize that their efforts result in positive outcomes for students.

6. Practice makes a difference. When students had access to reading materials and had time allocated to practice reading, these elements resulted in increases from baseline to midline in reading fluency. For example, reading support at home (Sindhi-medium schools), reading resources in the home (Sindhi- & Urdu-medium schools), reading support in school (Sindhi- & Urdu-medium schools), and reading practice at school (Sindhi) resulted in higher reading fluency scores from baseline to midline. These are practices that are increasing across the province. Recommendation: Share this result with teachers so they recognize that their efforts result in positive outcomes for students.


MAY 2017 70

ANNEXES

Annex 1. Additional Details on Study Methodology

TABLE 1.1: POPULATION OF SCHOOLS BY COHORT

Districts

Total No of Primary & Elementary

Schools*

Schools Covered in

Cohort 1

Schools Covered in Cohort 2

All Remaining

Schools (Cohort 3)

Dadu 1,856 188 299 1,369

Jacobabad 1,370 136 192 1,049

Kamber Shahdadkot 1,373 159 211 1,006

Karachi 1,069 177 201 699

Kashmore 1,182 154 171 876

Khairpur 3,113 215 461 2,446

Larkana 1,144 141 178 830

Sukkur 1,147 129 350 672

Grand Total 12,254 1,299 2,063 8,947

Note: 55 schools occurred in more than one group (C1, C2 or C3). They are shown here in both cohorts but were treated as occurring in the first Cohort in which they were placed.

Sampling Methodology

The following is a description of the procedure used for the selection of schools and students. Data from the 2016 Sindh Education Management Information System (SEMIS) were used as a basis for sampling. Schools that were Urdu or English medium in Sindhi districts were excluded.

1. All schools within the district were sorted by their location: urban or rural.

2. Schools were deselected if they did not meet the minimum requirement of at least 12 students for Grade 3.

3. A mix of boys’ schools and girls’ schools were sampled in the district, proportionate to urban or rural ratios. When there were not enough boys’ or girls’ schools in the district that met the sampling criteria, mixed-gendered schools were sampled.

4. Within each district, the sample was furthered stratified to support another study specific to the SRP intervention. The sample of schools was categorized into three cohorts: Cohort 1 (received SRP interventions between 2014-2016), Cohort 2 (started receiving SRP interventions beginning late 2016), and Cohort 3 (may or may not receive SRP interventions). Twenty-five of the 70 sampled schools were Cohort 1 schools (35%), 25 were from Cohort 2 schools (35%) and 20 were from Cohort 3 schools (30%). This stratification did not impact the generalizability of the results to each district.

5. Additional schools were selected using the same procedure (stratified by location, gender, and cohort) in case of inaccessibility or inaccurate SEMIS data. Note: most of these replacement schools were mixed-gender schools because an insufficient number of single-gender schools were available.

6. Within each school, 12 students in Grade 3 were randomly selected to represent learning levels of students at the end of Grade 2. Female students were sampled first in girls’ schools, and male students were sampled first in boys’ schools. If there were fewer than 12 students of the school’s


MAY 2017 71

gender designation present, then students from the other gender were chosen to reach a total of 12 students per school. Teachers who were most likely to be the Grade 2 classroom teacher for the students tested in Grade 3, were identified and administered the questionnaire.

7. Teams of QCOs visited the districts to verify the preliminary sample of schools, and replacements were made wherever the actual number of students was less than the target number. In some instances, previously disqualified schools were retained in the final sample if the number of students at each grade level was close to 12 and if the replacement schools did not have a larger number of students.

Subcontractor Selection

In August 2016, a request for proposals solicited services from firms to manage data collection. Thirteen firms submitted proposals. A committee composed of four senior technical staff evaluated each proposal using predefined criteria. Two firms, VTT (Voice Tel Tech) and I-SAPS, were selected to manage the recruitment of enumerators, provide administrative support, and supervise data collection in each district.

Data Collection Tools

The EGRA is an individually administered, oral assessment that requires approximately 20 minutes per student. The test is administered to one student at a time by a trained enumerator (not a teacher) in a location outside of the classroom. The enumerator begins by explaining the assessment to the student and asking if the student agrees to participate. Consent is always optional, and no student is required to take the assessment. The enumerator creates a relaxed environment for the student and assures the student that the assessment is not used for a grade. The enumerator then begins by asking the student questions aloud and having the student respond aloud. For certain subtasks, the enumerator places a paper stimulus containing letters or words in front of the student and asks the student specific questions about the stimuli. The EGRA administered at midline consisted of six subtasks which are described in more detail below. At midline, the same EGRA tool that was administered at baseline was used. Administering the same assessment again is the best way to ensure that the baseline and midline results are directly comparable; there is no risk of one assessment being harder than the other. The assessment tool was developed prior to data collection in 2014, and copies of all tests were kept secure. An additional reading passage was administered at midline to confirm that test leakage had not occurred. The assessment was administered in Sindhi in the seven districts of Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur, and in Urdu in the five towns of Karachi (treated as one district). Detailed descriptions of each task are provided below. Prior to the first data collection in 2014, an in-depth process was completed to develop the EGRA tool. The development was a multi-step process, including:

1. The choice of tasks to be included in the tools (the competencies to be assessed); 2. The development of the content for each of these tasks (the items); 3. The piloting of the tools in local schools; 4. The analysis of data from the pilot; 5. The determination of changes required to improve the tools; and, 6. The finalization of the tools.

Description of EGRA Tasks

Phonemic Awareness (phoneme isolation task): On this untimed task, students were asked to identify a phoneme—the smallest unit of sound in a word—at the end of 10 words. Phonics (nonword reading task): On this timed task, students were presented with 50 nonwords (words created for this exercise) and given one minute to read each using decoding skills.


MAY 2017 72

Vocabulary (expressive vocabulary task): On this untimed task, students were presented with 10 pictures and asked to identify what they saw. Fluency (passage reading task): On this timed task, students were given one minute to read a passage consisting of 60 words—Grade 3 students were tested using a passage at a Grade 2 reading level text. Reading Comprehension (reading comprehension task): On this untimed task, students were asked comprehension questions based on the same 60-word passage they read for the fluency task. Listening Comprehension (listening comprehension task): On this untimed task, students were read a story and then asked three questions to assess their understanding of the story’s meaning. Alphabet Knowledge (letter name identification task): On this timed task, students were presented with 100 letters and asked to read the letters aloud in one minute.

EGRA Tool Adaptation

The tool used to assess students at midline was the same tool used to assess students at baseline. To ensure that there was no concern with test leakage, a sub-group of students were administered a second reading passage. Student performance across the two reading passages did not indicate test leakage. The EGRA tool used at baseline was developed during a six-day workshop with 19 participants in April 2014. During that time, participants drafted the content for assessment tasks; developed questionnaires for students, teachers, and head teachers in Sindhi and Urdu; and finalized the instruments based on pilot data. See the baseline report for additional details on the development of the EGRA tool.

Updating Tools in 2016

To ensure comparability to the baseline data collection, the same EGRA tool was administered again in 2016 with one additional task: letter name recognition. This task was added to the EGRA tool in accordance with current EGRA guidance. To ensure that this new task did not disrupt comparability to the baseline results, it was added to the end of the assessment.

Digitization of Tools

The baseline tools were administered on paper while the midline tools were administered on tablets. Tools were transferred to electronic format because of the increased accuracy and efficiency afforded by electronic data capture. Collecting the data on tablets automates rules and skip patterns of the tools, reducing the possibility for human error during test or survey administration. It also eliminates the need for data entry and reduces data cleaning. At midline, the EGRA tool and survey instruments were administered on tablets using the electronic data capture software applications Tangerine for the EGRA and student survey and SurveyCTO for the teacher and head teacher surveys. Once the content of the tools was digitized, the software application was downloaded onto tablets. Extensive testing of the applications on the tablets was conducted prior to and during MT and QCO training.

Training of Master Trainers, Quality Control Officers, and Enumerators

Master Trainers and Quality Control Officers A six-day training was organized from September 26–October 1, 2016 in Karachi for the training of master trainers (MTs) and quality control officers (QCOs). The role of the MTs and QCOs is to monitor data collection and ensure that data collection methods are consistent across all enumerators. Ideal recruits had experience with EGRA or, at the very least, had experience in educational assessments, evaluation, or training. Nine out of the 17 MTs and QCOs had previously served in their same role during the 2014 data collection. MTs and QCO pairs were created so that MTs and QCOs who had served in this role in 2014 were paired with the candidates who had not previously worked on this assignment. Therefore, the majority


MAY 2017 73

of MTs and QCOs pairs were familiar with EGRA and had been previously trained on EGRA administration and data collection best practices. Additionally, to be selected for data collection, final candidates were required to demonstrate three attributes: adequate understanding of the tools and ability to administer the assessments with minimal support; high inter-rater reliability scores; and availability to monitor data collection activities. In final, sixteen individuals served as MTs or QCOs in teams of two. They supported field work for the teams of enumerators appointed to each district. Enumerators In each district, enumerators were recruited and selected using a similar approach as described above for MTs and QCOs. Enumerator candidates were proposed by VTT and I-SAPS; all CVs were reviewed and approved prior to engaging the individuals as enumerators. Enumerators were required to have a university degree and experience administering surveys; experience conducting surveys in schools or related to education was preferred. Enumerator performance was monitored regularly throughout training by the MTs. In addition to frequent feedback, MTs also led three formal sessions during training to measure consistency of scoring across all enumerators. The average consistency of enumerators across all three measures was 94%. Average consistency ranged from 91% to 98% for all subtasks.

Operational Data Collection

Data collection began on October 18, 2016 in all eight districts. Data collection was completed by November 5, 2016 with approximately 99% of the target number of Grade 3 students assessed. To ensure quality of data collection, supervisors as well as two QCOs monitored data collection and provided feedback to the enumerators during data collection in each district. At the end of each day, enumerators, MTs, and QCOs discussed progress and problems encountered that day. To record the level of enumerator consistency in scoring, measures of inter-rater reliability (IRR) were also captured during data collection. All data were uploaded from tablets daily, and school verification forms were collected. Overall, the Kappa coefficient (used to estimate the extent of agreement between two observers coding the same students’ responses) during operational data collection were above 0.8 on all but one task; expressive vocabulary (Kappa estimate 0.74); see table 2.3 in Annex 2. Despite several challenges—including low student attendance, security risks, and limited accessibility to sampled schools—each team visited one school per day and were able to reach most schools as planned. Where necessary, replacement schools were used.

Weighting

The procedure used to compute the weights for SRP follows the best practices for analyzing complex survey data. In this case, weights are applied at the student-level for analysis. The weight wij for a student (j) in school (i) is determined by two base weights, a school base weight and a within-school base weight. A school weight-trimming factor was also necessary in order to reduce any unexpected large values of a school base weight, as well as a student weight trimming factor to reduce the weight of student cases with very large values. Therefore, constructing the weights was a three-part process.

1) During sampling, schools were selected with equal probability within each stratum (district, school type, language of instruction). The school base weight is defined by the following equation:

𝑊𝑖 =𝑁

𝑛

where n is the number of sampled schools in a stratum and N is the total number of school in the stratum.


MAY 2017 74

2) Within each school, students were also selected with equal probability. Thus, the within-schoolbase weight is defined by the following equation:

𝑊𝑗 =𝑆𝑖𝑠𝑖

where Si is the total number of students in a specific school and si the number of students sampled from this school. Therefore, every student sampled within the same school has the same within-school weight.

3) Final weights for each student were compute by multiplying the school base weight and within-school base weight:

𝑊𝑖𝑗 = 𝑊𝑖 ∗ 𝑊𝑗

As noted, trimming factors were applied to the school base weight and within-school base weight. The calculation of both of those trimming factors followed the same rule: when a weight is more than three times larger than the median value of the weights from the same stratum, it was trimmed to be equal to three times the median weight for the stratum.

Annex 2. Data Quality Indices

Coefficient alpha

The reliability analysis below consists of analyzing the reliability of the full assessment using coefficient alpha with percent correct scores on each subtask. This index varies between 0 and 1, where a value closer to 1 indicates that the performance of the students can be easily generalized. This analysis treats each task as a single item with a score ranging from 0.0 to 1.0 for each student (their percent correct on the task). This method of measuring internal consistency may show lower estimates of alpha due to the small number of items being using in the calculation (equal to the number of subtasks administered and the low variability in scores in subtasks), as coefficient alpha is heavily dependent on the number of items used and the variability in responses to the items.

Tables 2 and 3 are the results of this analysis. The overall EGRA internal consistency in 2014 is 0.76 and in 2016 is 0.79. These coefficients of reliability shown below were calculated as described above; those reported in the 2014 report were calculated using a different approach that viewed subtask items as individual items—this approach benefited from an increased number of items and yielded comparable coefficients.

TABLE 2.1: RELIABILITY ESTIMATES - 2014

Reading subtasks Number of

students

Task-Total EGRA

correlation

Alpha if task is deleted

Phoneme Isolation (percentage correct)

6,209 0.29 0.77

Expressive Vocabulary (percentage correct)

6,209 0.30 0.78

Nonword Reading (percentage correct)

6,209 0.72 0.68

Passage Reading (percentage correct)

6,209 0.74 0.66


MAY 2017 75

Reading Comprehension (percentage correct)

6,209 0.74 0.67

Listening Comprehension (percentage correct)

6,209 0.34 0.78

Alpha 0.76

TABLE 2.2: RELIABILITY ESTIMATES - 2016

Reading subtasks Number of

students

Task-Total EGRA

correlation

Alpha if task is deleted

Phoneme Isolation (percentage correct)

6,851 0.31 0.79

Expressive Vocabulary (percentage correct)

6,851 0.31 0.79

Letter Name Recognition (percentage correct)

6,851 0.44 0.77

Nonword Reading (percentage correct)

6,851 0.74 0.71

Passage Reading (percentage correct)

6,851 0.75 0.70

Reading Comprehension (percentage correct)

6,851 0.75 0.71

Listening Comprehension (percentage correct)

6,851 0.34 0.80

Alpha 0.79

Kappa Coefficient

To record the level of enumerator agreement throughout data collection, enumerators undertook daily measures of inter-rater reliability (IRR) according to the following protocol. Enumerators worked in pairs to assess the first student of the day. During this first assessment, one enumerator acted as the “main enumerator”, administering the EGRA and scoring the student responses in his or her tablet. The second enumerator simultaneously listened and also scored the student responses in his or her tablet. Once the assessment was completed and the student had returned to class, the two enumerators compared and discussed their scoring of the student’s responses. Any points of disagreement or difference in marking were brought to the attention of supervisors and discussed during team meetings. Enumerator pairs took turns playing the role of main enumerator from each day to the next.

The percent agreement between enumerators and Kappa coefficient from the 2016 data collection are presented by subtask in Error! Reference source not found.4. Kappa is a measure of how many items the enumerators within a pair scored differently on an assessment out of the total number of items within a subtask. Kappa values greater than .75 are considered excellent.

Overall, among the IRR administrations conducted in during data collection, agreement was high on all of the seven subtasks.


MAY 2017 76

TABLE 2.3: INTER-RATER RELIABILITY ESTIMATES - 2016

Reading Subtasks Percent

Agreement Kappa

Phoneme Isolation 98.3% 0.85

Expressive Vocabulary 88.6% 0.74

Letter Name Recognition 93.9% 0.82

Nonword Reading 94.7% 0.86

Passage Reading 94.5% 0.89

Reading Comprehension 98.6% 0.95

Listening Comprehension 95.8% 0.91

*Note. N = 577.

Intra-Class Correlation

TABLE 2.4: INTRA-CLASS CORRELATIONS – 2014 & 2016 – BY COHORT

Sindhi Urdu

Cohort 1 Baseline 0.26 0.14

Midline 0.28 0.18

Cohort 2 Midline 0.19 0.16

Cohort 3 Midline 0.20 0.06

Annex 3. EGRA Results by Cohort – Additional Data

TABLE 3.1. AVERAGE EGRA SCORES AT BASELINE (2014) AND MIDLINE (2016)

BY TASK – SINDHI COHORT 1

Reading Subtasks

Baseline Midline Gain

N Mean SE of Mean N Mean

SE of Mean Mean

SE of Mean

Phoneme Isolation (correct answers) 5455 0.57 0.09 2142 0.87 0.08 0.30 0.12

Nonword Reading (words correct per minute) 5455 11.43 0.89 2142 14.83 0.60 3.39 1.08

Expressive Vocabulary (% correct of words) 5455 6.21 0.06 2142 6.64 0.09 0.44 0.11

Passage Reading (words correct per minute) 5455 26.61 2.16 2142 35.61 1.42 9.01 2.59

Reading Comprehension (correct answers) 5455 0.99 0.07 2142 1.41 0.07 0.41 0.10

Listening Comprehension (correct answers) 5455 1.58 0.02 2142 1.61 0.03 0.03 0.04


MAY 2017 77

Letter Name (letters correct per minute) 2142 39.38 0.91


MAY 2017 78

TABLE 3.2. PROPORTION OF STUDENTS MEETING STANDARDS AT BASELINE

(2014) AND MIDLINE (2016) BY GENDER – SINDHI COHORT 1

Reading Subtasks

Boys Girls

Baseline Midline Baseline Midline

N % Meeting Standards N

% Meeting Standards N


% Meeting Standards

Phoneme Isolation (correct answers) 2898 4.04% 1343 3.70% 2547 1.39% 1343 4.11%

Nonword Reading (words correct per minute) 2898 68.10% 1343 73.40% 2547 66.30% 1343 73.10%

Letter Name (letters correct per minute) 1343 11.40% 1343 11.40%

Expressive Vocabulary (% correct of words) 2898 25.30% 1343 27.80% 2547 15.40% 1343 25.40%

Passage Reading (words correct per minute) 2898 28.20% 1343 32.40% 2547 19.00% 1343 31.30%

Reading Comprehension (correct answers) 2898 13.60% 1343 13.60% 2547 9.00% 1343 16.20%

Listening Comprehension (correct answers) 2898 59.50% 1343 59.40% 2547 59.60% 1343 58.30%

TABLE 3.3. PERCENT ZERO-SCORE STUDENTS BY TASK AT BASELINE (2014)

AND MIDLINE (2016) – SINDHI COHORT 1

Reading Subtasks Baseline Midline

N % Zero Scores N % Zero Scores

Phoneme Isolation (correct answers) 5455 85.40% 2142 74.30%

Nonword Reading (words correct per minute) 5455 48.40% 2142 33.30%

Expressive Vocabulary (% correct of words) 5455 6.14% 2142 4.48%

Passage Reading (words correct per minute) 5455 41.80% 2142 20.00%

Reading Comprehension (correct answers) 5455 61.80% 2142 46.30%

Listening Comprehension (correct answers) 5455 14.60% 2142 14.30%


TABLE 3.4. PERCENT ZERO-SCORE STUDENTS AT MIDLINE (2016) BY TASK

AND GENDER – SINDHI COHORT 1

Reading Subtasks Boys Girls

N % Zero Scores N % Zero Scores









MAY 2017 79


BY TASK AND GENDER – SINDHI COHORT 1

Reading Subtasks

Boys Girls

Baseline Midline Gain Baseline Midline Gain


SE of Mean Mean

SE of Mean N Mean

SE of Mean N Mean

SE of Mean Mean

SE of Mean

Phoneme Isolation (correct answers) 2898 0.74 0.14 1343 0.86 0.09 0.11 0.08 2547 0.39 0.07 1343 0.89 0.11 0.51 0.06

Nonword Reading (words correct per minute) 2898 13.10 1.23 1343 15.37 0.73 2.27 0.69 2547 9.52 0.91 1343 13.96 0.83 4.44 0.61

Expressive Vocabulary (% correct of words) 2898 6.21 0.07 1343 6.69 0.12 0.48 0.07 2547 6.20 0.11 1343 6.58 0.13 0.38 0.09

Passage Reading (words correct per minute) 2898 30.08 3.16 1343 36.20 1.73 6.11 1.73 2547 22.62 1.84 1343 34.68 2.04 12.06 1.37

Reading Comprehension (correct answers) 2898 1.10 0.08 1343 1.43 0.08 0.33 0.06 2547 0.87 0.09 1343 1.36 0.11 0.49 0.07

Listening Comprehension (correct answers) 2898 1.57 0.03 1343 1.62 0.04 0.06 0.02 2547 1.61 0.04 1343 1.60 0.05 -0.01 0.03

Letter Name (letters correct per minute) 2898 1343 39.52 1.08 2547 1343 39.16 1.33


BY TASK AND LOCATION – SINDHI COHORT 1

Reading Subtasks

Rural Urban



SE of Mean Mean

SE of Mean N Mean

SE of Mean N Mean

SE of Mean Mean

SE of Mean

Phoneme Isolation (correct answers) 5071 0.44 0.00 1534 0.58 0.05 0.14 0.02 384 0.83 0.20 608 0.72 0.09 -0.11 0.10

Nonword Reading (words correct per minute) 5071 9.62 0.41 1534 12.70 0.51 3.08 0.33 384 14.85 1.55 608 14.47 1.18 -0.38 0.97

Expressive Vocabulary (% correct of words) 5071 5.98 0.06 1534 6.58 0.09 0.60 0.05 384 6.33 0.11 608 6.61 0.23 0.28 0.12

Passage Reading (words correct per minute) 5071 22.17 0.89 1534 30.84 1.05 8.67 0.69 384 34.95 3.74 608 33.13 3.02 -1.83 2.39

Reading Comprehension (correct answers) 5071 0.81 0.04 1534 1.15 0.05 0.34 0.03 384 1.34 0.10 608 1.30 0.14 -0.04 0.08

Listening Comprehension (correct answers) 5071 1.57 0.02 1534 1.58 0.03 0.01 0.02 384 1.62 0.06 608 1.70 0.05 0.08 0.04

Letter Name (letters correct per minute) 1534 36.00 0.86 608 39.51 2.23


MAY 2017 80


AND LOCATION – SINDHI COHORT 1

Reading Subtasks

Rural Urban


N % Zero Scores N

% Zero Scores N

% Zero Scores N

% Zero Scores








TABLE 3.8. AVERAGE SCORES AT MIDLINE (2016) BY TASK AND GENDER –

SINDHI COHORT 2

Reading Subtasks Boys Girls Total


SE of Mean N Mean

Phoneme Isolation (correct out of 10) 1343 .60 .07 735 .57 .11 0.59 0.06

Nonword Reading (Correct Nonwords Per Minute) 1343 13.43 .67 735 13.07 1.77 13.30 0.77

Expressive Vocabulary (correct out of 10) 1343 6.48 .15 735 6.32 .20 6.43 0.13

Passage Reading (Correct Words Per Minute) 1343 30.77 1.40 735 28.78 2.09 30.07 1.20

Reading Comprehension (correct out of 5) 1343 1.10 .07 735 .97 .13 1.05 0.06

Listening Comprehension (correct out of 3) 1343 1.51 .05 735 1.59 .09 1.54 0.05

Letter Name (Correct Letter Names Per Minute) 1343 35.34 .95 735 41.25 2.27 37.43 1.15

TABLE 3.9. PROPORTION OF STUDENTS MEETING STANDARDS AT MIDLINE

(2016) BY GENDER– SINDHI COHORT 2

Reading Subtasks

Boys Girls


% Meeting Standards









MAY 2017 81

TABLE 3.10. PERCENT ZERO-SCORE STUDENTS BY TASK AND GENDER AT

MIDLINE (2016) – SINDHI COHORT 2

Reading Subtasks

Boys Girls Total

N % Zero Scores N

% Zero Scores N

% Zero Scores

Phoneme Isolation (correct answers) 1343 82.00% 735 82.30% 2078 82.10%

Nonword Reading (words correct per minute) 1343 36.10% 735 40.10% 2078 37.50%

Expressive Vocabulary (% correct of words) 1343 6.40% 735 6.19% 2078 6.30%

Passage Reading (words correct per minute) 1343 24.90% 735 29.13% 2078 26.40%

Reading Comprehension (correct answers) 1343 53.70% 735 57.54% 2078 55.10%

Listening Comprehension (correct answers) 1343 20.20% 735 16.79% 2078 19.10%

Letter Name (letters correct per minute) 1343 11.60% 735 8.08% 2078 10.30%


SINDHI COHORT 3


N Mean SE of Mean N Mean SE of Mean N Mean SE of Mean

Phoneme Isolation (correct answers) 1000 0.503 0.0683 495 0.636 0.109 1495 0.869 0.0756

Nonword Reading (words correct per minute) 1000 12.84 0.717 495 11.25 1.173 1495 14.83 0.603

Expressive Vocabulary (% correct of words) 1000 6.649 0.135 495 6.565 0.165 1495 6.643 0.09

Passage Reading (words correct per minute) 1000 30.81 1.484 495 29.83 2.688 1495 35.61 1.424

Reading Comprehension (correct answers) 1000 1.161 0.089 495 1.118 0.129 1495 1.405 0.0697

Listening Comprehension (correct answers) 1000 1.611 0.0441 495 1.582 0.0675 1495 1.614 0.0319

Letter Name (letters correct per minute) 1000 34.91 1.109 495 36.55 2.101 1495 39.38 0.912


(2016) BY GENDER – SINDHI COHORT 3

Boys Girls

Reading Subtasks N % Meeting Standards N

% Meeting Standards









MAY 2017 82

TABLE 3.13. PERCENT ZERO-SCORE STUDENTS BY TASK AT MIDLINE (2016) –

SINDHI COHORT 3


N % Zero Scores N

% Zero Scores N

% Zero Scores

Phoneme Isolation (correct answers) 1000 85.20% 495 80.40% 1495 74.30%

Nonword Reading (words correct per minute) 1000 38.00% 495 45.60% 1495 33.30%

Expressive Vocabulary (% correct of words) 1000 3.59% 495 6.08% 1495 4.48%

Passage Reading (words correct per minute) 1000 26.60% 495 32.30% 1495 20.00%

Reading Comprehension (correct answers) 1000 52.20% 495 55.60% 1495 46.30%

Listening Comprehension (correct answers) 1000 14.80% 495 17.10% 1495 14.30%

Letter Name (letters correct per minute) 1000 9.70% 495 10.70% 1495 9.27%


BY TASK – URDU COHORT 1

Reading Subtasks

Baseline Midline Gain


SE of Mean Mean

SE of Mean

Phoneme Isolation (correct out of 10) 754 0.74 0.12 449 0.57 0.12 -0.18 0.17

Nonword Reading (Correct Nonwords Per Minute) 754 17.63 0.90 449 15.83 1.49 -1.80 1.74

Expressive Vocabulary (correct out of 10) * 754 5.25 0.15 449 7.35 0.21 2.11 0.26

Passage Reading (Correct Words Per Minute) 754 24.16 1.89 449 25.69 3.38 1.53 3.87

Reading Comprehension (correct out of 5) 754 0.69 0.07 449 0.87 0.21 0.18 0.22

Listening Comprehension (correct out of 3) * 754 1.16 0.04 449 1.42 0.07 0.26 0.08

Letter Name (Correct Letter Names Per Minute) 449 46.53 2.09

TABLE 3.15. PROPORTION OF STUDENTS MEETING STANDARDS AT BASELINE

(2014) AND MIDLINE (2016) BY GENDER – URDU COHORT 1

Reading Subtasks

Boys Girls





% Meeting Standards









MAY 2017 83

TABLE 3.16. PERCENT ZERO-SCORE STUDENTS BY TASK AT BASELINE (2014)

AND MIDLINE (2016) – URDU COHORT 1

Reading Subtasks

Baseline Midline

N % Zero Scores N

% Zero Scores

Phoneme Isolation (correct out of 10) 754 83.80% 449 86.09%

Nonword Reading (Correct Nonwords Per Minute) 754 26.60% 449 32.27%

Expressive Vocabulary (correct out of 10) * 754 5.77% 449 3.82%

Passage Reading (Correct Words Per Minute) 754 49.60% 449 43.70%

Reading Comprehension (correct out of 5) 754 67.70% 449 66.95%

Listening Comprehension (correct out of 3) * 754 17.60% 449 18.80%

Letter Name (Correct Letter Names Per Minute) 0.00% 449 2.69%


AND GENDER – URDU COHORT 1

Reading Subtasks

Boys Girls


N % Zero Scores N

% Zero Scores N

% Zero

Scores N % Zero Scores

Phoneme Isolation (correct out of 10) 400 81.30% 202 89.00% 349 86.30% 247 83.20%

Nonword Reading (Correct Nonwords Per Minute) 400 34.50% 202 40.80% 349 17.30% 247 24.10%

Expressive Vocabulary (correct out of 10) * 400 3.54% 202 5.20% 349 7.87% 247 2.50%

Passage Reading (Correct Words Per Minute) 400 56.62% 202 50.00% 349 41.30% 247 37.60%

Reading Comprehension (correct out of 5) 400 73.69% 202 70.80% 349 60.90% 247 63.20%

Listening Comprehension (correct out of 3) * 400 15.98% 202 14.80% 349 19.50% 247 22.70%

Letter Name (Correct Letter Names Per Minute) 400 202 1.61% 349 247 3.73%


BY TASK AND GENDER – URDU COHORT 1

Reading Subtasks

Boys Girls



SE of Mean Mean

SE of Mean N Mean

SE of Mean N Mean

SE of Mean Mean

SE of Mean

Phoneme Isolation (correct out of 10) 400 0.85 0.18 202 0.49 0.19 -0.36 0.13 349 0.63 0.14 247 0.64 0.14 0.02 0.10

Nonword Reading (Correct Nonwords Per Minute) 400 14.53 1.13 202 14.13 2.29 -0.40 1.21 349 21.21 1.00 247 17.47 1.38 -3.75 0.84

Expressive Vocabulary (correct out of 10) * 400 5.74 0.16 202 7.30 0.33 1.56 0.17 349 4.72 0.16 247 7.41 0.27 2.69 0.15

Passage Reading (Correct Words Per Minute) 400 18.19 2.47 202 20.81 5.40 2.62 2.78 349 31.02 2.34 247 30.39 3.25 -0.64 1.98

Reading Comprehension (correct out of 5) 400 0.52 0.09 202 0.81 0.31 0.28 0.14 349 0.88 0.11 247 0.93 0.15 0.05 0.09

Listening Comprehension (correct out of 3) * 400 1.19 0.08 202 1.57 0.12 0.38 0.07 349 1.14 0.04 247 1.28 0.08 0.14 0.04

Letter Name (Correct Letter Names Per Minute) 400 202 43.29 3.06 349 247 49.66 1.84


MAY 2017 84


BY TASK AND LOCATION – URDU COHORT 1

Reading Subtasks

Rural Urban



SE of Mean Mean

SE of Mean N Mean

SE of Mean N Mean

SE of Mean Mean

SE of Mean

Phoneme Isolation (correct out of 10) 150 1.22 0.30 77 0.48 0.09 -0.74 0.14 604 0.56 0.09 372 0.70 0.10 0.14 0.07

Nonword Reading (Correct Nonwords Per Minute) 150 14.95 1.64 77 18.48 3.18 3.53 1.71 604 18.66 0.99 372 13.65 1.08 -5.02 0.73

Expressive Vocabulary (correct out of 10) * 150 5.88 0.18 77 6.35 0.26 0.47 0.15 604 5.01 0.16 372 7.00 0.25 1.99 0.15

Passage Reading (Correct Words Per Minute) 150 19.02 3.98 77 32.22 6.85 13.20 3.83 604 26.13 2.01 372 24.36 2.32 -1.76 1.53

Reading Comprehension (correct out of 5) 150 0.61 0.14 77 0.84 0.17 0.22 0.11 604 0.72 0.08 372 0.74 0.08 0.02 0.06

Listening Comprehension (correct out of 3) * 150 1.35 0.11 77 1.70 0.18 0.35 0.10 604 1.09 0.04 372 1.36 0.07 0.27 0.04

Letter Name (Correct Letter Names Per Minute) 77 52.32 2.74 372 45.31 1.79


AND LOCATION – URDU COHORT 1

Reading Subtasks

Boys Girls


N % Zero Scores N

% Zero Scores N

% Zero Scores N

% Zero Scores

Phoneme Isolation (correct out of 10) 150 75.10% 77 91.50% 604 87.10% 372 84.15%

Nonword Reading (Correct Nonwords Per Minute) 150 31.00% 77 24.43% 604 24.90% 372 35.13%

Expressive Vocabulary (correct out of 10) * 150 2.50% 77 1.81% 604 7.02% 372 7.44%

Passage Reading (Correct Words Per Minute) 150 57.20% 77 36.02% 604 46.70% 372 45.79%

Reading Comprehension (correct out of 5) 150 67.30% 77 69.93% 604 67.90% 372 69.96%

Listening Comprehension (correct out of 3) * 150 8.30% 77 15.75% 604 21.20% 372 20.34%

Letter Name (Correct Letter Names Per Minute) 77 0.34% 372 1.74%


URDU COHORT 2

Reading Subtasks

Boys Girls Total


SE of Mean N Mean

SE of Mean

Phoneme Isolation (correct out of 10) 252 0.34 0.13 206 0.53 0.14 458 0.41 0.10

Nonword Reading (Correct Nonwords Per Minute) 252 10.13 1.54 206 16.77 1.26 458 12.63 1.53

Expressive Vocabulary (correct out of 10) * 252 6.73 0.41 206 6.82 0.46 458 6.76 0.33

Passage Reading (Correct Words Per Minute) 252 14.64 3.50 206 29.20 4.45 458 20.12 3.73

Reading Comprehension (correct out of 5) 252 0.38 0.14 206 0.73 0.10 458 0.51 0.12

Listening Comprehension (correct out of 3) * 252 1.37 0.08 206 1.24 0.17 458 1.32 0.09

Letter Name (Correct Letter Names Per Minute) 252 42.66 1.94 206 48.73 2.12 458 44.95 1.60


MAY 2017 85


(2016) BY GENDER – URDU COHORT 2

Reading Subtasks

Boys Girls


% Meeting Standards








TABLE 3.23. PERCENT ZERO-SCORE STUDENTS BY TASK AND GENDER AT

MIDLINE (2016) – URDU COHORT 2

Reading Subtasks

Boys Girls Total

N % Zero Scores N

% Zero Scores N

% Zero Scores

Phoneme Isolation (correct out of 10) 252 92.60% 206 88.37% 458 91.04%

Nonword Reading (Correct Nonwords Per Minute) 252 39.00% 206 19.41% 458 31.65%

Expressive Vocabulary (correct out of 10) * 252 4.44% 206 5.66% 458 4.90%

Passage Reading (Correct Words Per Minute) 252 55.30% 206 32.63% 458 46.79%

Reading Comprehension (correct out of 5) 252 84.20% 206 62.46% 458 76.02%

Listening Comprehension (correct out of 3) * 252 24.30% 206 27.79% 458 25.59%

Letter Name (Correct Letter Names Per Minute) 252 0.93% 206 0.56% 458 0.79%


URDU COHORT 3

Reading Subtasks

Boys Girls Total


SE of Mean N Mean

SE of Mean

Phoneme Isolation (correct out of 10) 109 .49 .12 120 0.86 0.14 229 0.71 0.11

Nonword Reading (Correct Nonwords Per Minute) 109 9.80 1.94 120 18.87 2.28 229 15.26 2.13

Expressive Vocabulary (correct out of 10) * 109 6.37 .61 120 7.00 0.37 229 6.75 0.30

Passage Reading (Correct Words Per Minute) 109 16.82 3.89 120 35.49 5.28 229 28.05 4.10

Reading Comprehension (correct out of 5) 109 .55 .13 120 0.97 0.11 229 0.80 0.09

Listening Comprehension (correct out of 3) * 109 1.40 .16 120 1.53 0.13 229 1.48 0.14

Letter Name (Correct Letter Names Per Minute) 109 36.51 2.18 120 55.19 2.47 229 47.74 2.76


MAY 2017 86


(2016) BY GENDER– URDU COHORT 3

Reading Subtasks

Boys Girls


% Meeting Standards








TABLE 3.26. PERCENT ZERO-SCORE STUDENTS BY TASK AT MIDLINE (2016) –

URDU COHORT 3

Reading Subtasks

Boys Girls Total

N % Zero Scores N

% Zero Scores N

% Zero

Scores

Phoneme Isolation (correct out of 10) 109 87.60% 120 82.80% 299 84.70%

Nonword Reading (Correct Nonwords Per Minute) 109 47.60% 120 22.60% 299 32.60%

Expressive Vocabulary (correct out of 10) * 109 12.20% 120 3.02% 299 6.68%

Passage Reading (Correct Words Per Minute) 109 62.90% 120 28.70% 299 42.30%

Reading Comprehension (correct out of 5) 109 77.50% 120 63.40% 299 69.00%

Listening Comprehension (correct out of 3) * 109 24.50% 120 13.10% 299 17.60%

Letter Name (Correct Letter Names Per Minute) 109 2.68% 120 0.35% 299 1.28%


MAY 2017 87

Annex 4. SRP Survey Results by Cohort – Additional Data

TABLE 4.1. STUDENT RECALL OF SRP MATERIALS – SINDHI COHORT 1

Questions N %

During your reading period last year, do you remember reading 1-Baba? No 1239 58.61%

Yes 875 41.39%

During your reading period last year, do you remember reading 2-Kis ne kitbag parhi?

No 1560 73.79%

Yes 554 26.21%

During your reading period last year, do you remember reading 3-Kenh ja tangun ahen??

No 1143 54.07%

Yes 971 45.93%

During your reading period last year, do you remember reading 4-Ye kis ki ke kaan hai?

No 1295 61.26%

Yes 819 38.74%

During your reading period last year, do you remember reading 5-Mujhe aata hai?

No 1732 81.93%

Yes 382 18.07%

During your reading period last year, do you remember reading 6-Ahmed badak ain topi?

No 1295 61.26%

Yes 819 38.74%

During your reading period last year, 7-I did not receive any of these books No 1567 74.12%

Yes 547 25.88%

TABLE 4.2. STUDENT RECALL OF SRP MATERIALS – URDU COHORT 1

Questions N %

During your reading period last year, do you remember reading 1-Baba? No 268 59.69%

Yes 181 40.31%

During your reading period last year, do you remember reading 2-Kis ne kitab parhi? No 342 76.17%

Yes 107 23.83%

During your reading period last year, do you remember reading 3-Kenh ja tangun ahen?? No 305 67.93%

Yes 144 32.07%

During your reading period last year, do you remember reading 4-Ye kis ki ke kaan hai? No 327 72.83%

Yes 122 27.17%

During your reading period last year, do you remember reading 5-Mujhe aata hai? No 368 81.96%

Yes 81 18.04%

During your reading period last year, do you remember reading 6-Ahmed badak ain topi? No 327 72.83%

Yes 122 27.17%

During your reading period last year, 7-I did not receive any of these books No 254 56.57%

Yes 195 43.43%


MAY 2017 88

TABLE 4.3. SRP PARTICIPATION AS REPORTED BY HEAD TEACHERS – SINDHI

COHORT 1

Questions N %

Do you believe your school participated in the project I just described?

No, I am certain that last year I did not participate in any of the activities described

3 2.26%

Yes, I am certain that last year I did participate in some or all of the types of activities described

122 91.73%

Maybe, I might have participated in some or all of the types of activities described

6 4.51%

I Don’t know 2 1.50%

Did you participate in the reading training that the grade two teachers attended as described?

No 36 28.13%

Yes 92 71.88%

Don’t know/no response 0 0.00%

TABLE 4.4. SRP PARTICIPATION AS REPORTED BY HEAD TEACHERS – URDU

COHORT 1

Questions N %

Do you believe your school participated in the project I just described?

No, I am certain that last year I did not participate in any of the activities described

2 3.08%


62 95.38%


1 1.54%

I Don’t know 0 0.00%

Did you participate in the reading training that the grade two teachers attended as described?

No 25 39.68%

Yes 38 60.32%



MAY 2017 89

TABLE 4.5. SRP PARTICIPATION AS REPORTED BY TEACHERS – SINDHI

COHORT 1

Questions N %

Do you believe you participated in the project I just described?

No, I am certain that, last year, I did not participate in any of the activities described

13 9.77%


106 79.70%


8 6.02%

I don’t know 6 4.51%

Did you teach the SRP reading lesson plan today?

No 70 61.40%

Yes 44 38.60%

TABLE 4.6. SRP PARTICIPATION AS REPORTED BY TEACHERS – URDU COHORT I

Questions N %

Do you believe you participated in the project I just described?

No, I am certain that, last year, I did not participate in any of the activities described

10 15.38%


52 80.00%


2 3.08%

I don’t know 1 1.54%

Did you teach the SRP reading lesson plan today?

No 41 75.93%

Yes 13 24.07%


MAY 2017 90

TABLE 4.7. SRP ACTIVITIES REPORTED BY TEACHERS – SINDHI COHORT 1

Questions N %

Last year, how often in a week did you use read alouds as part of the Lets Read Faster activity?

Never 1 0.88%

Daily 54 47.37%

Two to three times per week

45 39.47%

Once per week

8 7.02%

Once per month

4 3.51%

Less often than monthly

2 1.75%

Other 0 0.00%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Listening comprehension

No 31 27.19%

Yes 83 72.81%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Identifying letter sounds

No 47 41.23%

Yes 67 58.77%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Orientation to print

No 70 61.40%

Yes 44 38.60%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Letter recognition

No 42 36.84%

Yes 72 63.16%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Word recognition

No 64 56.14%

Yes 50 43.86%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Other

No 113 99.12%

Yes 1 0.88%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Decoding

No 48 42.11%

Yes 66 57.89%


No 40 35.09%

Yes 74 64.91%


No 59 51.75%

Yes 55 48.25%


No 44 38.60%

Yes 70 61.40%


No 58 50.88%

Yes 56 49.12%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Reading comprehension

No 66 57.89%

Yes 48 42.11%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Reading fluency

No 81 71.05%

Yes 33 28.95%


MAY 2017 91

TABLE 4.8. SRP ACTIVITIES REPORTED BY TEACHERS – URDU COHORT 1

Questions N %

Last year, how often in a week did you use read alouds as part of the Let’s Read Faster activity?

Never 1 1.85%

Daily 20 37.04%


28 51.85%

Once per week 5 9.26%

Once per month

0 0.00%


0 0.00%

Other 0 0.00%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Listening comprehension

No 15 27.78%

Yes 39 72.22%


No 8 14.81%

Yes 46 85.19%


No 23 42.59%

Yes 31 57.41%


No 9 16.67%

Yes 45 83.33%


No 10 18.52%

Yes 44 81.48%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Other

No 54 100.00%

Yes 0 0.00%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Decoding

No 20 37.04%

Yes 34 62.96%


No 13 24.07%

Yes 41 75.93%


No 24 44.44%

Yes 30 55.56%


No 12 22.22%

Yes 42 77.78%


No 13 24.07%

Yes 41 75.93%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Reading comprehension

No 15 27.78%

Yes 39 72.22%

Last year, when you conducted the LRF activity, which of the following concepts did you focus on? Reading fluency

No 14 25.93%

Yes 40 74.07%


MAY 2017 92

TABLE 4.9. SRP MATERIALS REPORTED BY HEAD TEACHERS – SINDHI COHORT I

Questions N %

Which of the following read alouds did you distribute to your grade two reading teachers last year? Watayo faqeer ai khairat ji mani

No 10 9.52%

Yes 95 90.48%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Keer Tapiyo

No 12 11.43%

Yes 93 88.57%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Karo bhoot

No 12 11.43%

Yes 93 88.57%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Kaan aa ja mitt maeth

No 9 8.57%

Yes 96 91.43%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Ahsan Jo Badlo

No 25 23.81%

Yes 80 76.19%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Lomri aa ja school

No 29 27.62%

Yes 76 72.38%

Which of the following read alouds did you distribute to your grade two reading teachers last year? I did not receive any of these books

No 105 100.00%

Yes 0 0.00%

Which of the following read alouds do you currently have in your school? Watayo faqeer ai khairat ji mani

No 13 12.38%

Yes 92 87.62%

Which of the following read alouds do you currently have in your school? Keer Tapiyo

No 18 17.14%

Yes 87 82.86%

Which of the following read alouds do you currently have in your school? Karo bhoot

No 16 15.24%

Yes 89 84.76%

Which of the following read alouds do you currently have in your school? Kaan aa ja mitt maeth

No 16 15.24%

Yes 89 84.76%

Which of the following read alouds do you currently have in your school? Ahsan Jo Badlo

No 26 24.76%

Yes 79 75.24%

Which of the following read alouds do you currently have in your school? Lomri aa ja school

No 33 31.43%

Yes 72 68.57%

Which of the following read alouds do you currently have in your school? I did not receive any of these books

No 103 98.10%

Yes 2 1.90%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Baba

No 15 14.29%

Yes 90 85.71%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Kis ne kitab parhi

No 15 14.29%

Yes 90 85.71%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Kenh ja tangun ahen?

No 14 13.33%

Yes 91 86.67%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Ye kis ki ke kaan hai

No 18 17.14%

Yes 87 82.86%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Mujhe aata hai

No 24 22.86%

Yes 81 77.14%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Ahmed badak ain topi

No 31 29.52%

Yes 74 70.48%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? I did not receive any of these books

No 105 100.00%

Yes 0 0.00%

No 14 13.33%


MAY 2017 93

Questions N %

Which of the following leveled readers do you currently have in your school? Baba

Yes 91 86.67%

Which of the following leveled readers do you currently have in your school? Kis ne kitab parhi

No 16 15.24%

Yes 89 84.76%

Which of the following leveled readers do you currently have in your school? Kenh ja tangun ahen?

No 17 16.19%

Yes 88 83.81%

Which of the following leveled readers do you currently have in your school? Ye kis ki ke kaan hai

No 19 18.10%

Yes 86 81.90%

Which of the following leveled readers do you currently have in your school? Mujhe aata hai

No 27 25.71%

Yes 78 74.29%

Which of the following leveled readers do you currently have in your school? Ahmed badak ain topi

No 36 34.29%

Yes 69 65.71%

Which of the following leveled readers do you currently have in your school? I did not receive any of these books

No 104 99.05%

Yes 1 0.95%

TABLE 4.10. SRP MATERIALS REPORTED BY HEAD TEACHERS – URDU

COHORT1

Questions N %

Which of the following read alouds did you distribute to your grade two reading teachers last year? Watayo faqeer ai khairat ji mani

No 12 20.00%

Yes 48 80.00%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Keer Tapiyo

No 20 33.33%

Yes 40 66.67%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Karo bhoot

No 14 23.33%

Yes 46 76.67%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Kaan aa ja mitt maeth

No 13 21.67%

Yes 47 78.33%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Ahsan Jo Badlo

No 13 21.67%

Yes 47 78.33%

Which of the following read alouds did you distribute to your grade two reading teachers last year? Lomri aa ja school

No 21 35.00%

Yes 39 65.00%

Which of the following read alouds did you distribute to your grade two reading teachers last year? I did not receive any of these books

No 56 93.33%

Yes 4 6.67%

Which of the following read alouds do you currently have in your school? Watayo faqeer ai khairat ji mani

No 12 20.00%

Yes 48 80.00%

Which of the following read alouds do you currently have in your school? Keer Tapiyo

No 23 38.33%

Yes 37 61.67%

Which of the following read alouds do you currently have in your school? Karo bhoot

No 19 31.67%

Yes

41 68.33%

Which of the following read alouds do you currently have in your school? Kaan aa ja mitt maeth

No 16 26.67%

Yes 44 73.33%

Which of the following read alouds do you currently have in your school? Ahsan Jo Badlo

No 18 30.00%

Yes 42 70.00%


MAY 2017 94

Questions N %

Which of the following read alouds do you currently have in your school? Lomri aa ja school

No 24 40.00%

Yes 36 60.00%

Which of the following read alouds do you currently have in your school? I did not receive any of these books

No 55 91.67%

Yes 5 8.33%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Baba

No 7 11.67%

Yes 53 88.33%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Kis ne kitab parhi

No 12 20.00%

Yes 48 80.00%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Kenh ja tangun ahen?

No 9 15.00%

Yes 51 85.00%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Ye kis ki ke kaan hai

No 10 16.67%

Yes 50 83.33%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Mujhe aata hai

No 17 28.33%

Yes 43 71.67%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? Ahmed badak ain topi

No 19 31.67%

Yes 41 68.33%

Which of the following leveled readers did you distribute to your grade two reading teachers last year? I did not receive any of these books

No 60 100.00%

Yes 0 0.00%

Which of the following leveled readers do you currently have in your school? Baba

No 11 18.33%

Yes 49 81.67%

Which of the following leveled readers do you currently have in your school? Kis ne kitab parhi

No 16 26.67%

Yes 44 73.33%

Which of the following leveled readers do you currently have in your school? Kenh ja tangun ahen?

No 9 15.00%

Yes 51 85.00%

Which of the following leveled readers do you currently have in your school? Ye kis ki ke kaan hai

No 12 20.00%

Yes 48 80.00%

Which of the following leveled readers do you currently have in your school? Mujhe aata hai

No 18 30.00%

Yes 42 70.00%

Which of the following leveled readers do you currently have in your school? Ahmed badak ain topi

No 19 31.67%

Yes 41 68.33%

Which of the following leveled readers do you currently have in your school? I did not receive any of these books

No 59 98.33%

Yes 1 1.67%


MAY 2017 95

TABLE 4.11. SRP MATERIALS REPORTED BY TEACHERS – SINDHI COHORT 1

Questions N %

Is the lesson an SRP lesson plan? No 12 21.05%

Yes 45 78.95%

Last year, how often in a week did you use leveled readers as part of the Let’s Read Together activity?

Never 1 0.88%

Daily 47 41.23%


49 42.98%

Once per week 10 8.77%

Once per month

5 4.39%


2 1.75%

Other 0 0.00%

Which of the following read alouds did you use last year? Watayo faqeer ai khairat ji mani

No 8 7.02%

Yes 106 92.98%

Which of the following read alouds did you use last year? Keer Tapiyo

No 16 14.04%

Yes 98 85.96%

Which of the following read alouds did you use last year? Karo bhoot

No 12 10.53%

Yes 102 89.47%

Which of the following read alouds did you use last year? Kaan aa ja mitt maeth

No 14 12.28%

Yes 100 87.72%

Which of the following read alouds did you use last year? Ahsan Jo Badlo

No 21 18.42%

Yes 93 81.58%

Which of the following read alouds did you use last year? Lomri aa ja school

No 46 40.35%

Yes 68 59.65%

Which of the following read alouds did you use last year? I did not receive any of these books

No 113 99.12%

Yes 1 0.88%

Which of the following leveled readers did you use with your students last year? Baba

No 13 11.40%

Yes 101 88.60%

Which of the following leveled readers did you use with your students last year? Kis ne kitab parhi

No 12 10.53%

Yes 102 89.47%

Which of the following leveled readers did you use with your students last year? Kenh ja tangun ahen?

No 11 9.65%

Yes 103 90.35%

Which of the following leveled readers did you use with your students last year? Ye kis ki ke kaan hai

No 19 16.67%

Yes 95 83.33%

Which of the following leveled readers did you use with your students last year? Mujhe aata hai

No 39 34.21%

Yes 75 65.79%

Which of the following leveled readers did you use with your students last year? Ahmed badak ain topi

No 36 31.58%

Yes 78 68.42%

Which of the following leveled readers did you use with your students last year? I did not receive any of these books

No 114 100.00%

Yes

0 0.00%

Which of the following supplementary reading materials did you receive last year? Letter cards

No 47 41.23%

Yes 67 58.77%

No 61 53.51%


MAY 2017 96

Questions N %

Which of the following supplementary reading materials did you receive last year? Posters

Yes 53 46.49%

Which of the following supplementary reading materials did you receive last year? Reading cards

No 58 50.88%

Yes 56 49.12%

Which of the following supplementary reading materials did you receive last year? Vocabulary cards

No 57 50.00%

Yes 57 50.00%

Which of the following supplementary reading materials did you receive last year? Word cards

No 60 52.63%

Yes 54 47.37%

Which of the following supplementary reading materials did you receive last year? Picture cards

No 60 52.63%

Yes 54 47.37%

Which of the following supplementary reading materials did you receive last year? Blackboard

No 92 80.70%

Yes 22 19.30%

Which of the following supplementary reading materials did you receive last year? Syllable cards

No 92 80.70%

Yes 22 19.30%

Which of the following supplementary reading materials did you receive last year? Sentence cards

No 86 75.44%

Yes 28 24.56%

Which of the following supplementary reading materials did you receive last year? Teaching and learning aids that I have created myself

No 75 65.79%

Yes 39 34.21%

Which of the following supplementary reading materials did you receive last year? None

No 100 87.72%

Yes 14 12.28%

Which supplementary reading materials did you use the most often with your students? Letter cards

No 45 39.47%

Yes 69 60.53%

Which supplementary reading materials did you use the most often with your students? Posters

No 75 65.79%

Yes 39 34.21%

Which supplementary reading materials did you use the most often with your students? Reading cards

No 63 55.26%

Yes 51 44.74%

Which supplementary reading materials did you use the most often with your students? Vocabulary cards

No 63 55.26%

Yes 51 44.74%

Which supplementary reading materials did you use the most often with your students? Word cards

No 73 64.04%

Yes 41 35.96%

Which supplementary reading materials did you use the most often with your students? Picture cards

No 65 57.02%

Yes 49 42.98%

Which supplementary reading materials did you use the most often with your students? Blackboard

No 93 81.58%

Yes 21 18.42%

Which supplementary reading materials did you use the most often with your students? Syllable cards

No 92 80.70%

Yes 22 19.30%

Which supplementary reading materials did you use the most often with your students? Sentence cards

No 82 71.93%

Yes

32 28.07%

Which supplementary reading materials did you use the most often with your students? Teaching and learning aids that I have created myself

No 83 72.81%

Yes 31 27.19%

Which supplementary reading materials did you use the most often with your students? None

No 102 89.47%

Yes 12 10.53%


MAY 2017 97

Questions N %

Which of the following SRP materials do you still HAVE today? Lesson Plans

No 27 23.68%

Yes 87 76.32%

Which of the following SRP materials do you still HAVE today? Read aloud books

No 11 9.65%

Yes 103 90.35%

Which of the following SRP materials do you still HAVE today? Leveled readers

No 30 26.32%

Yes 84 73.68%

Which of the following SRP materials do you still HAVE today? Supplementary materials (i.e. letter cards, etc.)

No 65 57.02%

Yes 49 42.98%

Which of the following SRP materials do you still HAVE today? I didn’t receive any of these materials

No 111 97.37%

Yes 3 2.63%

Which of the following SRP materials do you still HAVE today? None of the materials

No 111 97.37%

Yes 3 2.63%

Which SRP materials do you still use with your students in the current year? Lesson plans

No 49 44.14%

Yes 62 55.86%

Which SRP materials do you still use with your students in the current year? Read aloud books

No 37 33.33%

Yes 74 66.67%

Which SRP materials do you still use with your students in the current year? Leveled readers

No 56 50.45%

Yes 55 49.55%

Which SRP materials do you still use with your students in the current year? Supplementary materials (i.e. letter cards, etc.)

No 78 70.27%

Yes 33 29.73%

Which SRP materials do you still use with your students in the current year? None of the materials

No 83 74.77%

Yes 28 25.23%


MAY 2017 98

TABLE 4.12. SRP MATERIALS REPORTED BY TEACHERS – URDU COHORT 1

Questions N %

Is the lesson an SRP lesson plan? No 9 42.86%

Yes 12 57.14%

Last year, how often in a week did you use leveled readers as part of the Let’s Read Together activity?

Never 0 0.00%

Daily 14 25.93%


30 55.56%

Once per week

9 16.67%

Once per month

1 1.85%


0 0.00%

Other 0 0.00%

Which of the following read alouds did you use last year? Watayo faqeer ai khairat ji mani

No 6 11.11%

Yes 48 88.89%

Which of the following read alouds did you use last year? Keer Tapiyo

No 14 25.93%

Yes 40 74.07%

Which of the following read alouds did you use last year? Karo bhoot

No 17 31.48%

Yes 37 68.52%

Which of the following read alouds did you use last year? Kaan aa ja mitt maeth

No 10 18.52%

Yes 44 81.48%

Which of the following read alouds did you use last year? Ahsan Jo Badlo

No 9 16.67%

Yes 45 83.33%

Which of the following read alouds did you use last year? Lomri aa ja school

No 30 55.56%

Yes 24 44.44%

Which of the following read alouds did you use last year? I did not receive any of these books

No 53 98.15%

Yes 1 1.85%

Which of the following leveled readers did you use with your students last year? Baba

No 5 9.26%

Yes 49 90.74%

Which of the following leveled readers did you use with your students last year? Kis ne kitab parhi

No 10 18.52%

Yes 44 81.48%

Which of the following leveled readers did you use with your students last year? Kenh ja tangun ahen?

No 6 11.11%

Yes 48 88.89%

Which of the following leveled readers did you use with your students last year? Ye kis ki ke kaan hai

No 13 24.07%

Yes 41 75.93%

Which of the following leveled readers did you use with your students last year? Mujhe aata hai

No 21 38.89%

Yes 33 61.11%

Which of the following leveled readers did you use with your students last year? Ahmed badak ain topi

No 24 44.44%

Yes

30 55.56%

Which of the following leveled readers did you use with your students last year? I did not receive any of these books

No 53 98.15%

Yes 1 1.85%

No 30 55.56%


MAY 2017 99

Questions N %

Which of the following supplementary reading materials did you receive last year? Letter cards

Yes 24 44.44%

Which of the following supplementary reading materials did you receive last year? Posters

No 36 66.67%

Yes 18 33.33%

Which of the following supplementary reading materials did you receive last year? Reading cards

No 32 59.26%

Yes 22 40.74%

Which of the following supplementary reading materials did you receive last year? Vocabulary cards

No 33 61.11%

Yes 21 38.89%

Which of the following supplementary reading materials did you receive last year? Word cards

No 33 61.11%

Yes 21 38.89%

Which of the following supplementary reading materials did you receive last year? Picture cards

No 34 62.96%

Yes 20 37.04%

Which of the following supplementary reading materials did you receive last year? Blackboard

No 41 75.93%

Yes 13 24.07%

Which of the following supplementary reading materials did you receive last year? Syllable cards

No 40 74.07%

Yes 14 25.93%

Which of the following supplementary reading materials did you receive last year? Sentence cards

No 37 68.52%

Yes 17 31.48%

Which of the following supplementary reading materials did you receive last year? Teaching and learning aids that I have created myself

No 19 35.19%

Yes 35 64.81%

Which of the following supplementary reading materials did you receive last year? None

No 50 92.59%

Yes 4 7.41%

Which supplementary reading materials did you use the most often with your students? Letter cards

No 20 37.04%

Yes 34 62.96%

Which supplementary reading materials did you use the most often with your students? Posters

No 28 51.85%

Yes 26 48.15%

Which supplementary reading materials did you use the most often with your students? Reading cards

No 25 46.30%

Yes 29 53.70%

Which supplementary reading materials did you use the most often with your students? Vocabulary cards

No 22 40.74%

Yes 32 59.26%

Which supplementary reading materials did you use the most often with your students? Word cards

No 23 42.59%

Yes 31 57.41%

Which supplementary reading materials did you use the most often with your students? Picture cards

No 25 46.30%

Yes 29 53.70%

Which supplementary reading materials did you use the most often with your students? Blackboard

No 35 64.81%

Yes 19 35.19%

Which supplementary reading materials did you use the most often with your students? Syllable cards

No 29 53.70%

Yes

25 46.30%

Which supplementary reading materials did you use the most often with your students? Sentence cards

No 26 48.15%

Yes 28 51.85%

Which supplementary reading materials did you use the most often with your students? Teaching and learning aids that I have created myself

No 27 50.00%

Yes 27 50.00%


MAY 2017 100

Questions N %

Which supplementary reading materials did you use the most often with your students? None

No 51 94.44%

Yes 3 5.56%

Which of the following SRP materials do you still HAVE today? Lesson Plans

No 11 20.37%

Yes 43 79.63%

Which of the following SRP materials do you still HAVE today? Read aloud books

No 9 16.67%

Yes 45 83.33%

Which of the following SRP materials do you still HAVE today? Leveled readers

No 14 25.93%

Yes 40 74.07%

Which of the following SRP materials do you still HAVE today? Supplementary materials (i.e. letter cards, etc.)

No 29 53.70%

Yes 25 46.30%

Which of the following SRP materials do you still HAVE today? I didn’t receive any of these materials

No 53 98.15%

Yes 1 1.85%

Which of the following SRP materials do you still HAVE today? None of the materials

No 49 90.74%

Yes 5 9.26%

Which SRP materials do you still use with your students in the current year? Lesson plans

No 19 35.85%

Yes 34 64.15%

Which SRP materials do you still use with your students in the current year? Read aloud books

No 16 30.19%

Yes 37 69.81%

Which SRP materials do you still use with your students in the current year? Leveled readers

No 16 30.19%

Yes 37 69.81%

Which SRP materials do you still use with your students in the current year? Supplementary materials (i.e. letter cards, etc.)

No 29 54.72%

Yes 24 45.28%

Which SRP materials do you still use with your students in the current year? None of the materials

No 43 81.13%

Yes 10 18.87%


MAY 2017 101

TABLE 4.13. TLA INVOLVEMENT IN SCHOOL AS REPORTED BY HEAD

TEACHERS – SINDHI COHORT 1

Questions N %

Last year, which of the following months did you receive a visit by a TLA? Sep-15

No 76 72.38%

Yes 29 27.62%

Last year, which of the following months did you receive a visit by a TLA? Oct-15

No 68 64.76%

Yes 37 35.24%

Last year, which of the following months did you receive a visit by a TLA? Nov-15

No 71 67.62%

Yes 34 32.38%

Last year, which of the following months did you receive a visit by a TLA? Dec-15

No 56 53.33%

Yes 49 46.67%

Last year, which of the following months did you receive a visit by a TLA? Jan-16

No 54 51.43%

Yes 51 48.57%

Last year, which of the following months did you receive a visit by a TLA? Feb-16

No 53 50.48%

Yes 52 49.52%

Last year, which of the following months did you receive a visit by a TLA? Apr-16

No 98 93.33%

Yes 7 6.67%

Last year, which of the following months did you receive a visit by a TLA? I did receive a visit by a TLA, but it was during a different month than listed above

No 100 95.24%

Yes 5 4.76%

Last year, which of the following months did you receive a visit by a TLA? I did not receive any visits by a TLA between September 2015 and April 2016

No 103 98.10%

Yes 2 1.90%

Last year, did someone visit your school who may have introduced themselves as a Teaching and Learning Assistant (TLA)?

No 16 12.50%

Yes 105 82.03%



MAY 2017 102

TABLE 4.14. TLA INVOLVEMENT IN SCHOOL AS REPORTED BY HEAD

TEACHERS – URDU COHORT 1

Questions N %

Last year, which of the following months did you receive a visit by a TLA? Sep-15

No 27 45.00%

Yes 33 55.00%

Last year, which of the following months did you receive a visit by a TLA? Oct-15

No 22 36.67%

Yes 38 63.33%

Last year, which of the following months did you receive a visit by a TLA? Nov-15

No 23 38.33%

Yes 37 61.67%

Last year, which of the following months did you receive a visit by a TLA? Dec-15

No 20 33.33%

Yes 40 66.67%

Last year, which of the following months did you receive a visit by a TLA? Jan-16

No 20 33.33%

Yes 40 66.67%

Last year, which of the following months did you receive a visit by a TLA? Feb-16

No 20 33.33%

Yes 40 66.67%

Last year, which of the following months did you receive a visit by a TLA? Apr-16

No 33 55.00%

Yes 27 45.00%

Last year, which of the following months did you receive a visit by a TLA? I did receive a visit by a TLA, but it was during a different month than listed above

No 59 98.33%

Yes 1 1.67%

Last year, which of the following months did you receive a visit by a TLA? I did not receive any visits by a TLA between September 2015 and April 2016

No 60 100.00%

Yes 0 0.00%

Last year, did someone visit your school who may have introduced themselves as a Teaching and Learning Assistant (TLA)?

No 2 3.17%

Yes 60 95.24%

Don’t know/no response

1 1.59%


MAY 2017 103

TABLE 4.15. TLA INVOLVEMENT IN SCHOOL AS REPORTED BY TEACHERS –

SINDHI COHORT 1

Questions N %

Did you participate in a two-day, face-to-face orientation for Grade 1 and 2 teachers on reading in September 2015?

No 30 22.56%

Yes 103 77.44%


0 0.00%

Since you began to receive TLA support last year, do you believe you used the teaching techniques and strategies in the lesson plans more frequently?

No 1 0.85%

Yes, all the time

59 50.43%

Yes, sometimes

50 42.74%

Yes, but not very often

6 5.13%

I am not sure

1 0.85%

Last year, did you receive any visits from a TLA (Teaching Learning Advisor)? No 9 6.77%

Yes 117 87.97%


7 5.26%

Last year, during which of the following months did the TLA observe you during a reading lesson? Sep-15

No 67 57.26%

Yes 50 42.74%

Last year, during which of the following months did the TLA observe you during a reading lesson? Oct-15

No 69 58.97%

Yes 48 41.03%

Last year, during which of the following months did the TLA observe you during a reading lesson? Nov-15

No 73 62.39%

Yes 44 37.61%

Last year, during which of the following months did the TLA observe you during a reading lesson? Dec-15

No 66 56.41%

Yes 51 43.59%

Last year, during which of the following months did the TLA observe you during a reading lesson? Jan-16

No 76 64.96%

Yes 41 35.04%

Last year, during which of the following months did the TLA observe you during a reading lesson? Feb-16

No 78 66.67%

Yes 39 33.33%

Last year, during which of the following months did the TLA observe you during a reading lesson? Apr-16

No 81 69.23%

Yes 36 30.77%

Last year, during which of the following months did the TLA observe you during a reading lesson? I received a visit by a TLA, but during a different month than listed above

No 113 96.58%

Yes 4 3.42%

Last year, during which of the following months did the TLA observe you during a reading lesson? I did not receive any visits by a TLA

No 115 98.29%

Yes 2 1.71%

When you were observed last year, did you receive any guidance prior to your TLA observing your reading lesson? This may have been during a pre-observation conference.

No 14 11.97%

Yes

103 88.03%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Content of Lessons

No 39 37.86%

Yes 64 62.14%

No 32 31.07%


MAY 2017 104

Questions N %

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Teaching methodology (group work, participatory methods, etc.)

Yes 71 68.93%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Use of teaching and learning materials

No 42 40.78%

Yes 61 59.22%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Discipline of Students

No 60 58.25%

Yes 43 41.75%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Student assessment

No 63 61.17%

Yes 40 38.83%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Maintaining class records

No 67 65.05%

Yes 36 34.95%

TABLE 4.16. TLA INVOLVEMENT IN SCHOOL AS REPORTED BY TEACHERS –

URDU COHORT 1

Questions N %

Did you participate in a two-day, face-to-face orientation for Grade 1 and 2 teachers on reading in September 2015?

No 22 33.85%

Yes 43 66.15%

don’t know/no response

0 0.00%

Since you began to receive TLA support last year, do you believe you used the teaching techniques and strategies in the lesson plans more frequently?

No 5 7.94%

Yes, all the time

32 50.79%

Yes, sometimes

23 36.51%

Yes, but not very often

1 1.59%

I am not sure 2 3.17%

Last year, did you receive any visits from a TLA (Teaching Learning Advisor)? No 1 1.54%

Yes 63 96.92%

don’t know/no response

1 1.54%

Last year, during which of the following months did the TLA observe you during a reading lesson? Sep-15

No 24 38.10%

Yes 39 61.90%

Last year, during which of the following months did the TLA observe you during a reading lesson? Oct-15

No 20 31.75%

Yes 43 68.25%

Last year, during which of the following months did the TLA observe you during a reading lesson? Nov-15

No 23 36.51%

Yes 40 63.49%

Last year, during which of the following months did the TLA observe you during a reading lesson? Dec-15

No 22 34.92%

Yes 41 65.08%

Last year, during which of the following months did the TLA observe you during a reading lesson? Jan-16

No 28 44.44%

Yes 35 55.56%

Last year, during which of the following months did the TLA observe you during a reading lesson? Feb-16

No 24 38.10%

Yes 39 61.90%


MAY 2017 105

Questions N %

Last year, during which of the following months did the TLA observe you during a reading lesson? Apr-16

No 30 47.62%

Yes 33 52.38%

Last year, during which of the following months did the TLA observe you during a reading lesson? I received a visit by a TLA, but during a different month than listed above

No 63 100.00%

Yes 0 0.00%

Last year, during which of the following months did the TLA observe you during a reading lesson? I did not receive any visits by a TLA

No 63 100.00%

Yes 0 0.00%

When you were observed last year, did you receive any guidance prior to your TLA observing your reading lesson? This may have been during a pre-observation conference.

No 13 20.63%

Yes 50 79.37%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Content of Lessons

No 11 22.00%

Yes 39 78.00%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Teaching methodology (group work, participatory methods, etc.)

No 6 12.00%

Yes 44 88.00%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Use of teaching and learning materials

No 11 22.00%

Yes 39 78.00%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Discipline of Students

No 16 32.00%

Yes 34 68.00%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Student assessment

No 12 24.00%

Yes 38 76.00%

If Observed and received guidance prior to the observations, in what areas do you receive feedback? Maintaining class records

No 21 42.00%

Yes 29 58.00%


MAY 2017 106

TABLE 4.17. SRP EFFECTIVENESS AS REPORTED BY TEACHERS – SINDHI

COHORT 1

Questions N %

Last year, did you see an improvement in your students’ ability to read and comprehend text while you were using the lesson plans and receiving TLA support visits?

No, I haven’t noticed any improvements 7 5.26%

Yes, I notice a noticeable improvement 78 58.65%

Yes, I notice a slight improvement 48 36.09%

Last year, in which of the elements of reading do you believe students in your class made the greatest improvement?

Alphabetic principle 26 19.55%

Sight recognition 20 15.04%

Decoding 41 30.83%

Reading fluency 28 21.05%

Listening comprehension 9 6.77%

Vocabulary 3 2.26%

Reading comprehension 5 3.76%

Print concepts 1 0.75%

Overall, in which of the following areas did you receive the most effective support from the TLA last year?

Effectively focusing instruction on at least one critical area of reading (such as alphabetic principle, sight recognition

59 50.43%

Actively engaging students and at high levels during reading instruction

26 22.22%

Asking critical questions about reading materials to promote higher levels of thinking and learning

3 2.56%

Effectively using scripted lessons (including following activity sequence and direction as prescribed)

6 5.13%

Effectively using reading materials as prescribed by the lesson plan

21 17.95%

Effectively managing students for activities prescribed in the lesson plans

1 0.85%

Creating supplementary reading materials for my classroom (low-cost, no cost)

1 0.85%

Overall, in which of the following areas did you receive the least effective support from the TLA last year?


31 26.50%


20 17.09%


21 17.95%


9 7.69%


5 4.27%


MAY 2017 107

Questions N %


4 3.42%


27 23.08%

TABLE 4.18. SRP EFFECTIVENESS AS REPORTED BY TEACHERS – URDU

COHORT 1

Questions N %

Last year, did you see an improvement in your students’ ability to read and comprehend text while you were using the lesson plans and receiving TLA support visits?

No, I haven’t noticed any improvements 6 9.23%

Yes, I notice a noticeable improvement 39 60.00%

Yes, I notice a slight improvement 20 30.77%

Last year, in which of the elements of reading do you believe students in your class made the greatest improvement?

Alphabetic principle 9 13.85%

Sight recognition 14 21.54%

Decoding 10 15.38%

Reading fluency 13 20.00%

Listening comprehension 7 10.77%

Vocabulary 3 4.62%

Reading comprehension 8 12.31%

Print concepts 1 1.54%

Overall, in which of the following areas did you receive the most effective support from the TLA last year?


22 34.92%


9 14.29%


4 6.35%


8 12.70%


9 14.29%


7 11.11%


4 6.35%

Overall, in which of the following areas did you receive the least effective support from the TLA last year?


9 14.29%


8 12.70%


15 23.81%


MAY 2017 108

Questions N %


8 12.70%


3 4.76%


5 7.94%


15 23.81%


MAY 2017 109

Annex 5. Additional EGRA Results Analysis Requested

Following Provincial Dissemination Workshop

Multi-Grade Classrooms

In order to respond to interest in differences in student performance in classrooms with multiple grades vs. single grades, teacher survey data and student EGRA scores were examined. It is important to note that teachers were not asked if they had multi-grade classrooms specifically; but rather, teachers were asked from a list of grades, which they taught last year and which they are teaching this year. For the purposes of this analysis, a multi-grade classroom was then defined as one where a teacher indicated that they taught 2 or more consecutive grades, (e.g., Grades 1, 2 and 3; Grades 2, 3, and 4; Grades 2 and 3) and multi-grade status was determined for each teacher for both last year and this year.20 This approach provides a proxy for multi-grade classrooms that may have included Grade 2 students, and therefore allows us to examine whether there is a difference in EGRA scores between multi-grade and single grade classrooms. It should be noted that the teachers surveyed were not necessarily the teachers of the students completing the EGRA (although efforts were made to identify the teacher most likely to have been the Grade 2 teacher in the school, last year). Additionally, these results are not generalizable to all Cohort 1, 2, or 3 SRP schools since multi-grade status was not included as a sampling variable. Therefore, the results below should only be interpreted for the classrooms in the sample. Table 5.1 shows the proportions of teachers, by cohort, reporting that they taught in a multi-grade classroom last year.

TABLE 5.1. PROPORTION OF TEACHERS IN MULTI-GRADE CLASSROOMS LAST

YEAR BY COHORT

Language of

EGRA Administration

Single-Grade Classroom

Multi-Grade Classroom

Total

n n% n n% n n%

Cohort 1 Sindhi 162 87.6% 23 12.4% 185 100.0%

Urdu 37 90.2% 4 9.8% 41 100.0%

Cohort 2 Sindhi 148 83.6% 29 16.4% 177 100.0%

Urdu 36 94.7% 2 5.3% 38 100.0%

Cohort 3 Sindhi 93 68.4% 43 31.6% 136 100.0%

Urdu 18 94.7% 1 5.3% 19 100.0%

The smallest proportion of multi-grade classroom teachers was observed in Cohort 1, followed by Cohort 2, with the largest proportion in Cohort 3. Tables 5.2 through 5.7 present the mean EGRA subtask scores for Cohorts 1, 2 and 3, respectively, by multi-grade status and language. Cohort 1 showed no significant differences on any EGRA subtasks between multi-grade and single-grade classrooms for the Sindhi-medium schools. However, for the Urdu-medium schools in cohort 1, students in single-grade classrooms significantly outperformed students in multi-grade classrooms on the Letter Name subtask. For cohort 2 Sindhi-medium schools, students in single-grade classrooms significantly outperformed students in multi-grade classrooms on the Letter Name, Nonword Reading, and Passage Reading subtask. In the Urdu-medium schools, however, cohort 2 students in multi-grade classrooms had significantly higher average scores on the Phoneme Isolation task than students in single-grade classrooms. Finally, in cohort 3 schools, students in Sindhi-medium schools with single-grade classrooms outperformed

20 The proportion of teachers in the sample that reported not having taught Grade 2 last year was 0.9% (11 teachers) in Cohort

1, 8.2% (18 teachers in Cohort 2; and 4.4% (7 teachers) in Cohort 3.


MAY 2017 110

students in multi-grade classrooms on the Expressive Vocabulary task. There were no significant difference for the Urdu-medium schools in cohort 3.

TABLE 5.2. MEAN EGRA SUBTASK SCORES FOR COHORT 1 – SINDHI-MEDIUM

SCHOOLS

Single-Grade

Classroom Multi-Grade Classroom

n Mean SD n Mean SD

Phoneme Isolation (Correct out of 10)

162 0.91 0.95 22 0.68 0.56

Expressive Vocabulary (Correct out of 10)

162 6.64 1.39 22 6.84 1.54

Letter Name (Correct Letters per Minute)

162 38.81 12.36 22 39.73 10.48

Nonword Reading (Correct Nonwords per Minute)

162 14.94 7.70 22 12.74 8.01

Passage Reading (Correct Words per Minute)

162 35.09 18.53 22 31.91 19.44

Reading Comprehension (Correct out of 5)

162 1.39 0.90 22 1.26 0.95

Listening Comprehension (Correct out of 3)

162 1.63 0.42 22 1.56 0.44


SCHOOLS

Single-Grade


n Mean SD n Mean SD


148 0.55 0.66 29 0.51 0.67


148 6.37 1.66 29 5.91 1.61


148 36.17* 11.78 29 29.59 10.82


148 12.56* 6.66 29 9.52 4.85


148 29.12* 15.87 29 23.84 11.03


148 1.03* 0.73 29 0.79 0.50


148 1.53 0.48 29 1.44 0.49

*Indicates that the mean is significantly higher for the group at p<0.05.


MAY 2017 111


SCHOOLS

Single-Grade


n Mean SD n Mean SD


93 0.59 0.69 43 0.61 1.15


93 6.59* 1.61 43 5.84 1.73


93 36.46 12.70 43 32.94 12.35


93 12.59 7.49 43 10.19 6.16


93 31.59 17.20 43 25.84 16.70


93 1.20* 0.78 43 0.87 0.62


93 1.60 0.42 43 1.44 0.52


TABLE 5.5. MEAN EGRA SUBTASK SCORES FOR COHORT 1 –URDU-MEDIUM

SCHOOLS

Single-Grade


n Mean SD n Mean SD


37 0.60 0.79 4 0.95 0.77


37 7.25 1.08 4 7.71 1.10


37 45.43* 9.86 4 34.07 15.59


37 15.63 7.43 4 14.33 6.71


37 24.75 17.17 4 28.33 17.19


37 0.81 0.75 4 0.75 0.18


37 1.42 0.37 4 1.20 0.22



MAY 2017 112

TABLE 5.6. MEAN EGRA SUBTASK SCORES FOR COHORT 2 – URDU-MEDIUM

SCHOOLS

Single-Grade


n Mean SD n Mean SD


35 0.38 0.41 2 1.05* 0.19


35 6.97 1.27 2 7.45 1.16


35 45.90 8.67 2 42.32 9.45


35 15.05 6.23 2 11.00 3.60


35 24.39 15.19 2 18.50 7.90


35 0.67 0.53 2 0.50 0.32


35 1.30 0.38 2 1.36 0.26


TABLE 5.7. MEAN EGRA SUBTASK SCORES FOR COHORT 3 – URDU-MEDIUM

SCHOOLS

Single-Grade


n Mean SD n Mean SD


18 0.63 0.62 1 1.25 0.00


18 7.06 1.17 1 7.83 0.00


18 47.12 11.17 1 40.92 0.00


18 14.50 6.17 1 7.42 0.00


18 27.21 12.80 1 10.57 0.00


18 0.80 0.45 1 0.25 0.00


18 1.46 0.42 1 1.33 0.00


MAY 2017 113

Language of Instruction

In order to respond to interest in Language of Instruction across cohorts, the following results were produced. Table 5.8 shows the proportions of teachers and head teachers, by cohort and the language in which the EGRA was administered in the school, and their report of the school language of instruction. Results show that in all cases where the EGRA was administered in Urdu, teacher and head teachers also reported Urdu as the language of instruction. However, for those schools in which the EGRA was administered in Sindhi, there was a small proportion of teachers and head teachers across all cohorts that reported that Urdu, or that Urdu and Sindhi, were the language(s) of instruction. Due to the distribution of responses, no statistical comparisons can be made by language of instruction on EGRA scores.

TABLE 5.8. LANGUAGE OF INSTRUCTION AS REPORTED BY TEACHER AND

HEAD TEACHER BY COHORT

EGRA Completed

in Sindhi EGRA Completed

in Urdu

n n% n n%

Cohort 1

Language of Instruction - Head Teacher

Urdu 2 1.1% 41 100.0%

Sindhi 178 96.2% 0 0.0%

Urdu and Sindhi 5 2.7% 0 0.0%

Language of Instruction - Teacher

Urdu 3 1.6% 41 100.0%

Sindhi 181 97.8% 0 0.0%


Cohort 2


Urdu 0 0.0% 39 100.0%

Sindhi 173 97.2% 0 0.0%



Urdu 0 0.0% 38 100.0%

Sindhi 174 98.3% 0 0.0%


Cohort 3


Urdu 1 0.7% 19 100.0%

Sindhi 138 98.6% 0 0.0%



Urdu 0 0.0% 19 100.0%

Sindhi 134 98.5% 0 0.0%


Table 5.9 shows the proportion of students’ responses to three questions surrounding language and is reported by cohort, and the EGRA language of administration. The three questions were as follows:

1) In what language does your family read at home? 2) What language does your family speak most often at home? 3) What language do you speak most often at school?

The majority of students across all cohorts showed language consistency between home and school, and the language in which the EGRA was administered. For Cohort 1 results and a discussion on correlations between Language Consistency and student passage reading performance, see Chapter 3.6 (Sindhi-medium) and Chapter 6.6 (Urdu-medium).


MAY 2017 114

TABLE 5.9. LANGUAGE OF INSTRUCTION AS REPORTED BY STUDENT BY

COHORT

EGRA Completed


in Urdu

n n% n n%

Cohort 1

In what language does your family read at home?

Parents cannot/do not read

132 6.7% 31 6.9%

Urdu 14 0.7% 331 73.7%

Sindhi 1713 86.7% 10 2.2%

Pashto 1 0.1% 27 6.0%

Punjabi 2 0.1% 5 1.1%

Balochi 18 0.9% 7 1.6%

Siraeki 28 1.4% 0 0.0%

English 3 0.2% 8 1.8%

Other 4 0.2% 22 4.9%

Don't know/refuse/no answer

60 3.0% 8 1.8%

NO RESPONSE 0 0.0% 0 0.0%

What language does your family speak most often at home?

Urdu 8 0.4% 190 42.3%

Sindhi 1531 77.5% 25 5.6%

Pashto 3 0.2% 86 19.2%

Punjabi 6 0.3% 21 4.7%

Balochi 188 9.5% 48 10.7%

Siraeki 166 8.4% 9 2.0%

English 1 0.1% 1 0.2%

Other 67 3.4% 68 15.1%

Don't know 5 0.3% 1 0.2%


What language do you speak most often at school?

Urdu 15 0.8% 430 95.8%

Sindhi 1886 95.5% 3 0.7%

Pashto 1 0.1% 6 1.3%

Punjabi 1 0.1% 3 0.7%

Balochi 18 0.9% 2 0.4%

Siraeki 32 1.6% 1 0.2%

English 4 0.2% 0 0.0%

Other 4 0.2% 3 0.7%

Don't know 14 0.7% 1 0.2%


MAY 2017 115

EGRA Completed


in Urdu

n n% n n%


Cohort 2



148 7.5% 59 13.2%

Urdu 11 0.6% 269 60.3%

Sindhi 1696 85.7% 21 4.7%

Pashto 1 0.1% 12 2.7%

Punjabi 2 0.1% 5 1.1%

Balochi 16 0.8% 42 9.4%

Siraeki 19 1.0% 0 0.0%

English 9 0.5% 11 2.5%

Other 8 0.4% 9 2.0%


70 3.5% 18 4.0%



Urdu 3 0.2% 79 17.7%

Sindhi 1521 76.8% 50 11.2%

Pashto 0 0.0% 50 11.2%

Punjabi 8 0.4% 31 7.0%

Balochi 195 9.8% 169 37.9%

Siraeki 170 8.6% 6 1.3%

English 2 0.1% 1 0.2%

Other 72 3.6% 53 11.9%

Don't know 9 0.5% 7 1.6%



Urdu 0 0.0% 400 89.7%

Sindhi 1919 96.9% 2 0.4%

Pashto 4 0.2% 4 0.9%

Punjabi 0 0.0% 3 0.7%

Balochi 11 0.6% 27 6.1%

Siraeki 25 1.3% 1 0.2%

English 3 0.2% 1 0.2%

Other 3 0.2% 1 0.2%

Don't know 15 0.8% 7 1.6%

NO RESPONSE

0 0.0% 0 0.0%


MAY 2017 116

EGRA Completed


in Urdu

n n% n n%

Cohort 3



156 11.0% 19 8.3%

Urdu 11 0.8% 166 72.5%

Sindhi 1169 82.2% 2 0.9%

Pashto 0 0.0% 21 9.2%

Punjabi 0 0.0% 3 1.3%

Balochi 12 0.8% 0 0.0%

Siraeki 9 0.6% 0 0.0%

English 2 0.1% 3 1.3%

Other 5 0.4% 2 0.9%


58 4.1% 13 5.7%



Urdu 1 0.1% 135 59.0%

Sindhi 1029 72.4% 12 5.2%

Pashto 0 0.0% 51 22.3%

Punjabi 5 0.4% 13 5.7%

Balochi 191 13.4% 3 1.3%

Siraeki 123 8.6% 4 1.7%

English 1 0.1% 0 0.0%

Other 72 5.1% 7 3.1%

Don't know 0 0.0% 4 1.7%



Urdu 3 0.2% 204 89.1%

Sindhi 1379 97.0% 4 1.7%

Pashto 1 0.1% 12 5.2%

Punjabi 0 0.0% 0 0.0%

Balochi 9 0.6% 2 0.9%

Siraeki 20 1.4% 1 0.4%

English 0 0.0% 1 0.4%

Other 3 0.2% 1 0.4%

Don't know 7 0.5% 4 1.7%



MAY 2017 117

Teacher Qualifications

In order to respond to interest in Teacher Qualifications and the relationship with student outcomes, the following results were produced. Table 5.10 shows the proportions of teachers and head teachers, by cohort and language of EGRA administration, and their report of their highest academic qualifications. Results show that the majority of teachers and head teachers have either a Bachelor’s or a Master’s degree. For this reason, statistical tests were done comparing student outcomes for students in schools where the teacher had either a Bachelor’s degree or a Master’s degree, and the same for head teachers. Tables 5.11 through 5.16 show these results by cohort. Only one difference was statistically significant (see Table 5.11): for Cohort 1, Sindhi-medium schools with a head teacher with a Master’s degree had significantly higher average scores on Listening Comprehension than those with only a Bachelor’s degree. It must be noted that no results are generalizable, as the sampling for this study did not allow for adequate representation of teacher and head teacher qualifications.

TABLE 5.10. TEACHER AND HEAD TEACHER QUALIFICATIONS BY COHORT

EGRA Completed


in Urdu n n% n n%

Cohort 1

What is your highest academic qualification? – Head Teacher

Ph.D 0 0.0% 0 0.0%

M.Phil 1 0.5% 0 0.0%

M.A/M.Sc 70 37.8% 7 17.1%

B.A/B.Sc 87 47.0% 18 43.9%

F.A/F. Sc 10 5.4% 11 26.8%

Matric 5 2.7% 5 12.2%

Other 12 6.5% 0 0.0%

Subtotal 185 100.0% 41 100.0%

What is your highest academic qualification? – Classroom Teacher

Ph.D 0 0.0% 0 0.0%

M.Phil 3 1.6% 0 0.0%

M.A/M.Sc 58 31.4% 4 9.8%

B.A / B.Sc 95 51.4% 11 26.8%


MAY 2017 118

EGRA Completed


in Urdu

n n% n n%

F.A/F.Sc 15 8.1% 16 39.0%

Matric 3 1.6% 9 22.0%

Other 11 5.9% 1 2.4%

Subtotal 185 100.0% 41 100.0%

Cohort 2


Ph.D 0 0.0% 0 0.0%

M.Phil 0 0.0% 0 0.0%

M.A/M.Sc 71 39.9% 10 25.6%

B.A/B.Sc 83 46.6% 20 51.3%

F.A/F. Sc 9 5.1% 7 17.9%

Matric 5 2.8% 2 5.1%

Other 10 5.6% 0 0.0%

Subtotal 178 100.0% 39 100.0%


Ph.D 0 0.0% 0 0.0%

M.Phil 1 0.6% 0 0.0%

M.A/M.Sc 63 35.6% 3 7.9%

B.A / B.Sc 88 49.7% 12 31.6%

F.A/F.Sc 16 9.0% 13 34.2%

Matric 2 1.1% 10 26.3%

Other 7 4.0% 0 0.0%


MAY 2017 119

EGRA Completed


in Urdu

n n% n n%

Subtotal 177 100.0% 38 100.0%

Cohort 3


Ph.D 0 0.0% 0 0.0%

M.Phil 0 0.0% 0 0.0%

M.A/M.Sc 33 23.6% 4 21.1%

B.A/B.Sc 76 54.3% 7 36.8%

F.A/F. Sc 20 14.3% 8 42.1%

Matric 7 5.0% 0 0.0%

Other 4 2.9% 0 0.0%

Subtotal 140 100.0% 19 100.0%


Ph.D 0 0.0% 0 0.0%

M.Phil 1 0.7% 1 5.3%

M.A/M.Sc 39 28.7% 1 5.3%

B.A / B.Sc 61 44.9% 5 26.3%

F.A/F.Sc 24 17.6% 6 31.6%

Matric 7 5.1% 6 31.6%

Other 4 2.9% 0 0.0%

Subtotal 136 100.0% 19 100.0%


MAY 2017 120

TABLE 5.11. EGRA SUBTASK RESULTS BY TEACHER AND HEAD TEACHER

QUALIFICATIONS – COHORT 1 – SINDHI-MEDIUM SCHOOLS

Teacher Qualifications Head Teacher Qualifications

Bachelor's Degree Master's Degree Bachelor's Degree Master's Degree

n Mean SD n Mean SD n Mean SD n Mean SD


94 0.91 0.92 58 0.86 0.98 87 0.88 1.03 69 0.85 0.77


94 6.56 1.42 58 6.80 1.38 87 6.53 1.33 69 6.72 1.47


94 37.56 11.83 58 40.72 12.97 87 37.55 12.45 69 39.13 11.58


94 14.33 7.34 58 15.18 8.14 87 14.26 7.75 69 14.62 7.39


94 33.87 17.88 58 36.52 20.33 87 33.26 19.19 69 36.29 17.97


94 1.32 0.87 58 1.40 0.95 87 1.27 0.90 69 1.47 0.87


94 0.91 0.92 58 0.86 0.98 87 0.88* 1.03 69 0.85 0.77








88 0.54 0.71 63 0.58 0.62 83 0.49 0.63 71 0.60 0.66


88 6.41 1.62 63 6.16 1.70 83 6.30 1.80 71 6.31 1.56


88 35.20 12.45 63 34.32 11.80 83 33.71 12.85 71 36.66 10.98


88 11.83 6.53 63 11.74 6.76 83 12.47 6.76 71 11.68 6.11


88 27.96 15.67 63 27.19 14.92 83 30.00 16.30 71 26.84 13.22


88 1.04 0.71 63 0.90 0.68 83 1.08 0.75 71 0.89 0.61


88 1.57 0.50 63 1.46 0.49 83 1.54 0.51 71 1.53 0.48


MAY 2017 121







61 0.47 0.56 39 0.54 0.69 76 0.59 0.68 33 0.44 0.52


61 6.37 1.67 39 5.96 1.96 76 6.22 1.61 33 6.64 2.00


61 33.26 11.81 39 35.74 15.01 76 34.95 12.61 33 35.97 12.86


61 10.94 6.35 39 13.11 8.90 76 12.26 7.39 33 12.07 7.08


61 27.18 16.42 39 32.91 19.19 76 29.74 17.81 33 29.84 17.23


61 1.00 0.73 39 1.20 0.85 76 1.09 0.77 33 1.13 0.80


61 1.55 0.44 39 1.45 0.52 76 1.53 0.46 33 1.61 0.52


QUALIFICATIONS – COHORT 1





11 0.66 0.79 4 0.28 0.19 18 0.67 0.76 7 0.71 1.27


11 7.34 1.11 4 7.63 0.69 18 7.41 0.96 7 7.64 0.87


11 48.48 10.63 4 41.50 9.78 18 46.69 9.88 7 46.48 7.16


11 16.23 7.02 4 13.13 6.48 18 16.62 7.21 7 14.60 5.03


11 25.91 16.64 4 20.77 13.09 18 27.05 16.23 7 22.84 11.99


11 0.85 0.83 4 0.66 0.58 18 0.87 0.82 7 0.80 0.57


11 1.56 0.28 4 1.37 0.43 18 1.44 0.36 7 1.52 0.16


MAY 2017 122







12 0.48 0.45 2 1.00 0.47 19 0.49 0.46 10 0.46 0.48


12 6.61 1.46 2 6.67 0.59 19 7.04 1.38 10 6.83 1.24


12 44.95 5.28 2 42.65 2.81 19 44.13 8.16 10 44.05 6.31


12 16.22 5.86 2 19.17 3.18 19 14.69 5.61 10 12.86 4.91


12 25.78 14.25 2 31.66 5.32 19 24.53 12.83 10 17.71 9.01


12 0.79 0.52 2 1.00 0.12 19 0.63 0.40 10 0.46 0.36


12 1.23 0.35 2 1.17 0.47 19 1.27 0.40 10 1.27 0.30







5 0.87 0.26 1 0.67 0.00 7 0.43 0.39 4 1.03 0.64


5 7.00 0.67 1 5.42 0.00 7 6.39 1.11 4 7.20 0.63


5 40.75 9.82 1 44.50 0.00 7 40.47 8.90 4 53.77 9.41


5 10.69 7.64 1 11.92 0.00 7 11.46 3.51 4 14.28 5.19


5 18.05 14.75 1 20.63 0.00 7 20.29 7.81 4 25.59 11.80


5 0.40 0.34 1 0.83 0.00 7 0.59 0.47 4 0.77 0.39


5 1.37 0.48 1 1.25 0.00 7 1.28 0.52 4 1.50 0.16

USAID SINDH READING PROGRAM EARLY GRADE READING … · USAID SINDH READING PROGRAM EARLY GRADE...

Documents

Transcript of USAID SINDH READING PROGRAM EARLY GRADE READING … · USAID SINDH READING PROGRAM EARLY GRADE...