EdData II Task Order 27: Egypt Grade 3 Early Grade Reading ... · EdData II: Egypt Grade 3 Early...

EdData II

Task Order 27: Egypt Grade 3 Early Grade Reading Assessment (EGRA) Group Assessment Report

EdData II Technical and Managerial Assistance Contract Number: BPA#EHC-E-00-04-00004-00 Task Order Number: AID-263-BC-14-00002 (RTI Task 27) June 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International.

Task Order 27: Egypt Grade 3 Early Grade Reading Assessment (EGRA) Group Assessment Report June 2014 EdData II Task Order No. 27 Prepared for Education and Training USAID/EGYPT Unit 54902 APO AE 09839-4902 Prepared by RTI International 3040 Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 RTI International is a trade name of Research Triangle Institute.

The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

EdData II: Egypt Grade 3 Early Grade Reading – Pilot Group Assessment iii

Table of Contents Section Page List of Figures ........................................................................................................................... iv List of Tables ............................................................................................................................ iv Abbreviations ............................................................................................................................. v Acknowledgments..................................................................................................................... vi Executive Summary ................................................................................................................. vii

Sample Description .............................................................................................................. vii Assessment Approach .......................................................................................................... vii Results .................................................................................................................................. vii

Internal Consistency....................................................................................................... viii Concurrent Validity ....................................................................................................... viii Summary Conclusions ................................................................................................... viii

Implications and Recommendations .................................................................................. viii Methodology .............................................................................................................................. 1

The Schools and Student Sample ........................................................................................... 1 The Group Assessment Instrument ........................................................................................ 2 Training, Data Collection and Data Entry ............................................................................. 3

Results 3 Internal Consistency............................................................................................................... 6 Concurrent Validity ............................................................................................................... 7 Individual Item Assessment ................................................................................................... 9 Summary Conclusions ......................................................................................................... 13

Limitations, Recommendations and Further Research ............................................................ 13 Key Challenges to the Group Assessment ........................................................................... 13 Benefits of the Group Assessment ....................................................................................... 15 Recommendations ................................................................................................................ 16

Annex 1. 2014 EGRA Assessor Tool and Student Answer Sheet; Group Assessor Tool and Student Answer Sheet ............................................................................................... 19

iv EdData II: Egypt Grade 3 Early Grade Reading – Pilot Group Assessment

List of Figures Figure 1. Individual versus Group Assessment: Distribution of Non-Word Reading .......... 6 Figure 2. Individual Assessment Subtask Box and Whisker Plots...................................... 10 Figure 3. Group Assessment Subtask Box and Whisker Plots ............................................ 11 Figure 4. ORF and Reading Comprehension – Individual Assessment .............................. 12 Figure 5. ORF and Reading Comprehension – Group Assessment .................................... 13 Figure 6. Results of a Revised Group Assessment Design for 12 Students ........................ 17 List of Tables Table 1. Assessment Tasks: EGRA and Group Assessment Tools ................................... 2 Table 2. Descriptives for Group Assessment ....................................................................... 4 Table 3. Descriptives for Individual Assessment ................................................................. 5 Table 4. Internal Consistency for Individual Assessment .................................................... 6 Table 5. Internal Consistency for Group Assessment .......................................................... 7 Table 6. Pairwise Correlations of Subtasks for Individual Assessment .............................. 8 Table 7. Pairwise Correlations of Subtasks for Group Assessment ..................................... 8 Table 8. Concurrent Validity for Group and Individual Assessments ................................. 9

EdData II: Egypt Grade 3 Early Grade Reading – Pilot Group Assessment v

Abbreviations COP Chief of Party DCOP Deputy Chief of Party EGRA Early Grade Reading Assessment EGRP Early Grade Reading Program, Ministry of Education GILO Girls’ Improved Learning Outcomes Project, USAID project idara district-level ministry administration, GOE MOE Ministry of Education muderiya governorate-level ministry administration, GOE RTI Research Triangle Institute, a trademark of RTI International USAID Unted States Agency for International Development

vi EdData II: Egypt Grade 3 Early Grade Reading – Pilot Group Assessment

Acknowledgments The authors wish to acknowledge with appreciation and respect the contributions and support of education staff from USAID/Egypt and Egypt’s Ministry of Education (MOE) in the implementation of this pilot Group Assessment component of the 2nd national Early Grade Reading Assessment (EGRA) for Grade 3. At USAID/Egypt, we thank Lisa Franchette, Education Program Director, and Hala ElSerafy, Senior Education Specialist, for their strategic design and leadership of this second national assessment to include this Group Assessment component, and Mitch Kirby of USAID/Washington for his direction under the Data for Education Programming in Asia and the Middle East (DEP-AME) contract. Ms. Hanaa Qassem Hassanein, Head of MOE’s Early Learning Unit, was again instrumental to the successful implementation and mobilizing of ministry support at all levels for this 2nd national assessment of Grade 3 reading skills, including implementation of this Group Assessment. The Group Assessment instrument was developed by RTI International with technical assistance from Medina Korda and Drs. Jonathon Stern, Amy D. Mulcahy-Dunn, and Margaret Dubeck. Dr. Jonathon Stern was the chief author for this report. Drs. Samir Shafik and Ahmed Ismail of RTI International, and formerly COP and DCOP respectively of the GILO Project, liaised directly with the MOE in planning the Group Assessment, prepared and tested the Arabic version of the instrument, mobilized the MOE professionals with significant experience and training in early reading assessment, and then trained that cadre to conduct the Group Assessment. Dr. Robert J. LaTowsky, Field Director for the 2nd national EGRA for Grade 3, directed the field implementation and assisted in writing this report. At RTI International, Medina Korda served as the overall task leader, supported by Roxanne Zerbonia, and David Harbin who provided financial and contractual management. We sincerely thank them all for their contributions to the success of this pilot Group Assessment component of the 2nd national EGRA assessment for Grade 3 in Egypt.

EdData II: Egypt Grade 3 Early Grade Reading – Pilot Group Assessment vii

Executive Summary

Sample Description A total of 200 students were sampled from six schools for this pilot. The participating schools, all in Greater Cairo, were non-randomly selected by the MOE for the availability of empty classrooms or auditoriums in which to conduct the assessment. Schools and students were purposefully selected to ensure that the sample included only intermediate to strong readers.

In each of 6 schools, 30-35 students were selected to participate in the pilot. Of the 200 sampled students, 195 student observations without missing values were used for these analyses. Twelve additional students were tested in a final seventh school in order to assess the impact of slight revisions to the assessment.

Assessment Approach The purpose of this pilot was two-fold: 1) examining the reliability of the group assessment tool to measure the construct of early grade reading; 2) determining whether or not the group assessment could be used as a partial replacement for the individual EGRA.

In order to address the first purpose, this study uses the conventional measure of internal consistency. Internal consistency is measured based on the correlations among the different test items in order to determine whether or not the items appear to be measuring the same general construct. In this case, the underlying construct is reading ability.

For the second research aim, it is necessary to examine the correlation between the group assessment and the individual EGRA via concurrent validity. Concurrent validity is a measure of how well a particular test correlates with a previously validated measure. In this case, the previously validated measure is the individual EGRA assessment and the new test is the pilot group assessment. This analysis was possible due to the fact that all sampled students took both the group assessment and the individual EGRA.

Results Overall, students performed well on both the individual and group assessments. This was anticipated as all sample schools were asked to provide high-level Grade 3 readers for this pilot. However, average scores on the group assessment were significantly higher than their corresponding individual EGRA subtasks. Furthermore, there was little variability in the scores across subtasks on the group assessment. The high scores provide evidence that the group test was too easy (and/or that administration was flawed) and the lack of variability made it difficult to test for concurrent validity.

viii EdData II: Egypt Grade 3 Early Grade Reading – Pilot Group Assessment

Internal Consistency Internal consistency was tested for both the individual EGRA and the group assessment. As expected, the overall test reliability for the previously validated individual EGRA was high (0.83). By contrast, the internal consistency results for the group assessment produced a Cronbach’s alpha of 0.56, which is below the conventional internal consistency threshold of .70. This weak internal consistency provides evidence that the piloted group assessment was not reliably measuring student reading ability.

Concurrent Validity Concurrent validity was tested for this pilot by examining subtask correlations across assessments. Only one half of the total cross-assessment correlations were found to be statistically significant. Even in direct subtask comparisons (e.g. group letter sounds to individual letter sounds) only half of the correlations were significant (i.e. ORF, reading comprehension and maze). Furthermore, Maze was the only subtask from the group assessment that was significantly correlated with all subtasks from the individual assessment. With regard to correlation strength, only two correlations on the entire table would be considered moderate—0.43 (group Maze – individual Listening Comprehension) and 0.50 (group Maze – individual Maze)—while all others were weak to null. This shows that the concurrent validity for the group and individual assessments is very weak and that the group test should not be used in lieu of the individual assessment without significant revisions.

Summary Conclusions The high scores across all subtasks on the group assessment provide evidence that the instrument piloted in Egypt was flawed and that revisions and test redevelopment are necessary for future administrations. This is further supported by the low internal consistency of the test, as well as the weak concurrent validity for the group and individual assessments. The lack of variability in scores within subtasks also made it difficult to appropriately discriminate among high and low scorers. Ultimately, there is no evidence to suggest that the piloted version of the group assessment should be used in lieu of (or even as a complement to) the individual EGRA.

Implications and Recommendations Field observations and the results of this pilot identified three main challenges in designing/implementing this pilot group assessment: 1) the opportunity for students to guess answers; 2) the opportunity for students to copy the answers of classmates; and 3) the difficulty in measuring reading fluency and opportunity for students to exaggerate their reading performance.

In order to address these challenges, we suggest a range of recommendations, as follows: 1) substitute open-response questions for multiple choice items (or increase the number of response options and consider importance of distractors); 2) investigate the most appropriate length of a reading passage for group administered exams (with recognition that it is currently not possible to measure oral reading fluency); 3) pilot

EdData II: Egypt Grade 3 Early Grade Reading – Pilot Group Assessment ix

future instruments with random (and representative) samples of students; 4) assess the utility of group assessments by subtask and grade; 5) clearly specify the objectives of group administered assessments for future use.

Ultimately, by overcoming key challenges and implementing these much needed revisions to both the design and administration of the group assessment, there is reason to believe that the proposed benefits of a group administered early grade reading assessment could be realized in future administrations.

EdData II: Egypt Grade 3 Early Grade Reading Assessment Group Baseline 1

Methodology The 2nd national EGRA for Grade 3 in Egypt included the pilot implementation of a group administered assessment of early reading skills. The purpose of the pilot was to field test a group administered assessment instrument by comparing its implementation and results with the individual EGRA in a typical school environment. This was the first known implementation of a group administered assessment protocol in Arabic for early grade reading.

The pilot implementation planned for 200 Grade 3 students to be assessed using both a standard individual EGRA form and a group assessment instrument. Each student’s standard EGRA form and their individual answer sheet for the group assessment were identically numbered so that the performance of each student on each sub-task in the two assessments could be compared. The sub-task content for the individual and group assessments was different but intended to be comparable in level of difficulty. To ensure no systematic bias from the order of student’s testing, half of the sample students completed their individual EGRA before participating in the group assessment; the other half after the group assessment.

Implementation of the 7 group assessment administrations – one in each of 7 schools – was closely monitored and timed. Participating assessors were debriefed to measure and qualitatively appraise the group methodology experience with the individual EGRAs. The group assessment in one school was professionally videotaped to provide a visual record. Six of the seven group assessments were conducted in a school classroom; the seventh was conducted in a large school amphitheater. The individual EGRAs were conducted in 3-4 empty classrooms in each school, with up to 2 assessors per classroom

In implementing each group assessment with the planned 35 Grade 3 students, a single Group Assessment Leader orally instructed the students before conducting each successive sub-task. Students were instructed on how that sub-task would be conducted and how to correctly mark their answers to the verbal questions and multiple choice items on their individual answer sheet. Only the Group Assessment Leader spoke. Students were instructed to remain silent throughout the group assessment: no answering aloud, no reading aloud, no questions, and no comments were permitted. For the sub-tasks on oral reading fluency and Maze comprehension, the group assessment substituted silent reading for reading aloud in the individual EGRA. During implementation of the group assessment, each sub-task was timed by a supporting assessor. Other assessors served as room proctors to observe and help instruct and monitor the students.

The Schools and Student Sample To complete both individual EGRA and the group assessment with the same students within a school day, the planned sample of 200 Grade 3 students was distributed over 7 schools. In each of 6 schools, 30-35 students participated in both assessments. To minimize the number of different participating assessors for greater comparability of

2 EdData II: Egypt Grade 3 Early Grade Reading – Pilot Group Assessment

results and implementation, two teams of 6 assessors each were used to complete the assessments in 2 schools per day. Twelve students were tested in a final 7th school to make up a shortfall in total student numbers from the previous six schools.

For this pilot implementation, neither the schools nor the students were randomly selected. The participating schools, all in Greater Cairo, were selected by the MOE for the availability of empty classrooms or auditorium in which to conduct the assessment. Schools and students were purposefully selected to ensure that the sample included only intermediate to strong readers. Comparing the results of nonreaders would not have aided comparison of reading results for specific skills. The purpose of the pilot was to test the group assessment protocol, not the reading proficiency of students. None of the 7 MOE primary schools selected for the group assessment were included in the stratified random sample of 200 MOE primary schools chosen for the national EGRA.

The Group Assessment Instrument The design of the Group Assessment instrument applied in this Egypt pilot was informed by RTI research and pilot experiments in Ghana and Zambia. The tool was developed to assess most of the same skills tested by the individual EGRA instrument. Letter, word, and non-word lists and reading passages from the EGRA were adapted for use in this group assessment tool. The group assessment tool was developed to be similar in level of difficulty and follow the same sequence of sub-tasks. The individual EGRA tool used for this pilot was also used for the 2014 2nd national EGRA for Grade 3. Table 1 is an overview and comparison of tasks for both assessments. The individual EGRA instrument and the Assessor tool and student answer sheet (Arabic) for the group assessment are all included in Annex 1.

Table 1. Assessment Tasks: EGRA and Group Assessment Tools

Task EGRA Tasks Definition Group Assessment Tasks

Letters reading Children were asked to name letters

Assessor would read the word, students were asked to circle the initial letter of the word read or to circle the letter that was contained in the word regardless of where it was.

Word reading Children were asked to read the words out loud

Assessor would say the sound which would be found in a given word that was included on student sheets. Students would circle the word that contained the sound read.

Non-word reading

Children were asked to read the non-words out loud

Assessor would say the sound which would be found in a given non-word that was included on student sheets. Students would circle the non-word that contained the sound read.

Reading and comprehension

Children were asked to read a story silently and answer comprehension questions related to this story. This was a timed task. Students were given 1 minute.

Students were asked to read a story silently. Students were then asked to circle the correct answers using the multiple choices to the questions asked by the assessor. This was a timed task. Students were allowed 1 minute to read the story.


Task EGRA Tasks Definition Group Assessment Tasks

Maze Students were asked to read the story silently and pick the correct choice for the missing words in the sentences that they were reading.

Students were asked to read the story silently and pick the correct choice for the missing words in the sentences that they were reading.

Listening comprehension

Children were read to and were asked to answer comprehension question related to this story.

Children were read to, and then asked to circle the correct answer to each comprehension question from the multiple choice option provided on their student sheets.

Training, Data Collection and Data Entry To maximize the comparability of results and test implementation, just 2 Group Assessment Leaders conducted all 7 of the group assessments.1 Both Leaders were thoroughly familiar with the individual EGRA and its sub-tasks from previous EGRAs and EGRA assessor training. Through role-play and practice delivery of the group assessment sub-tasks together, both Leaders prepared themselves for the pilot implementation. Additionally, they were observed by the EGRA Field Director to have capably and comparably implemented the group assessments in the sample schools.

Experienced EGRA assessors – 5 to 6 on each of two teams – conducted the individual EGRAs. The group assessments were conducted immediately after completing the 2nd national EGRA in 200 schools. All trained EGRA assessors who participated in the group assessment had each conducted at least 50 individual assessments during the national EGRA before implementing this pilot. All 7 schools and 200 students were tested over one week.

For each student, the individual EGRA form and student answer sheet were identically numbered to compare their individual results. All sub-task data from the 200 individual EGRA forms and 200 group assessment answer sheets were entered into an Excel data set with separate worksheets for individual and group assessment data on the 200 student records. Data entry was closely checked and validated to ensure accuracy and data integrity. Data and statistical analyses of the data set were performed by Dr. Jonathan Stern of RTI International.

Results Overall, students performed well on both the individual and group assessments. This was anticipated as all sample schools were asked to provide high-level Grade 3 readers for this pilot. As previously noted, the purpose of this pilot was to examine the group assessment methodology and comparability with the individual EGRA.

1 The Group Assessment Leaders were Dr. Samir Shafik Habib of RTI and Dr. Fath al-Bab Rashaad of the MOE. Both Leaders had extensive professional experience implementing individual EGRAs and were well versed in Arabic language instruction and early grade reading.


Average scores on the group assessment were significantly higher than their corresponding individual EGRA sub-tasks. The comparative results and the reasons for this sharp discrepancy are presented in this and the following section. The high scores provide evidence that the group test was too easy and that revisions and test redevelopment are necessary for future administrations.

Results from the group assessments across each of the six subtasks are displayed in Table 2. On average, students were able to correctly answer more than 95% of questions on four of the six subtasks tested (as seen in second column). Furthermore, the “median” column shows that for all but the Maze subtask, at least half of the students were able to answer every question correctly (or to read the entire 55 word passage for the Oral Reading Fluency section). There were no zero scores on the entire assessment. In other words, every student was able to correctly answer at least one question on each subtask. Ultimately, while is it encouraging that students did so well on this assessment, the lack of variability in these scores makes test validation difficult and leads to the assumption that either the tested material was too basic for average grade 3 students tested or that the test design and implementation were flawed.

Table 2. Descriptives for Group Assessment

Subtask Mean Minimum Maximum 25th

Percentile Median 75th

Percentile

Letter Sounds 99.69% 90.00% 100.00% 100.00% 100.00% 100.00%

Non-word Reading 95.33% 70.00% 100.00% 100.00% 100.00% 100.00%

Oral Reading Fluency (Words per minute)

51 (92.80%)

25 (45.45%)

55 (100%)

49 (89.09%)

55 (100%)

55 (100%)

Reading Comprehension 97.78% 33.33% 100.00% 100.00% 100.00% 100.00%

Listening Comprehension 96.34% 57.14% 100.00% 100.00% 100.00% 100.00%

Maze 72.01% 7.14% 100.00% 50.00% 78.57% 92.86%

As a basis for comparison, Table 3 provides results for the same 195 students on the same six subtasks for the individually administered EGRA. In direct contrast to the group assessment, the average scores for all subtasks were below 80%. As a matter of fact, the lowest average on the group assessment (Maze: 72%) was higher than the average for four of the six subtasks on the individual assessment. The most extreme difference came from Non-word Reading, with an average score of 95% on the group assessment and 35% on the individual assessment.


Table 3. Descriptives for Individual Assessment

Subtask Mean Minimum Maximum 25th

Percentile Median 75th

Percentile

Letter Sounds 62.03% 5.00% 100.00% 52.50% 62.50% 72.50%

Non-word Reading 34.75% 0.00% 76.00% 26.00% 34.00% 44.00%

Oral Reading Fluency (Words per minute)

45.8 (78.97%)

8 (13.79%)

58 (100%)

38 (65.52%)

50 (86.21%)

56 (96.55%)

Reading Comprehension 67.01% 0.00% 100.00% 50.00% 66.67% 83.33%

Listening Comprehension 61.10% 0.00% 100.00% 42.86% 71.43% 85.71%

Maze 73.11% 0.00% 100.00% 57.14% 78.57% 92.86%

Another way to examine the difference across assessments is to calculate distributions for each given subtask2. Accordingly, Figure 1 displays the distribution of scores for the Non-word Reading subtask for both the individual and the group assessment. This figure shows that while the scores on the individual assessment were distributed normally, more than 75% of students scored perfectly on the group test and nearly all students scored higher on the group assessment than even the highest achieving students on the individually administered test. Considering that the same group of students took both exams, this provides evidence that the two tests were not equitable.

2 All other subtask distributions are available upon request.

Implementing the group assessment in Shaykh Zayed, al-Giza – April 2014


Figure 1. Individual versus Group Assessment: Distribution of Non-Word Reading

Internal Consistency One of the most important features of any test is its internal consistency or reliability. Internal consistency is measured based on the correlations among the different test items in order to determine whether or not the items appear to be measuring the same general construct. In this case, the underlying construct is reading ability. Since the individually administered EGRA was previously validated, it is expected that the internal consistency will be high. Confirming this assumption, Table 4 shows that the overall test reliability is 0.83 (as shown in the final cell of the table). This high Cronbach’s alpha provides evidence that the test is reliably measuring reading ability. Furthermore, the “Item-Rest Correlation” column shows the correlation between each variable and the rest of the test if that item were to be removed from the test. The item-rest correlations are moderate and acceptable. No subtasks stand out as being inherently problematic.

Table 4. Internal Consistency for Individual Assessment

Item-Rest Correlation Reliability

Letter Sounds 0.56 0.81

Non-words 0.60 0.81

ORF 0.68 0.79



Reading Comprehension 0.71 0.78

Listening Comprehension 0.49 0.83

Maze 0.69 0.78

Scale 0.83

By contrast, the internal consistency results for the group assessment are presented in Table 5. The overall test scale shows a Cronbach’s alpha of 0.56, which is below the conventional internal consistency threshold of .70. Additionally, the item-rest correlations are consistently weak (with only the Maze measure bordering on acceptable). Letter Sounds is particularly problematic, as it has a correlation of only 0.14 with the rest of the group assessment. This table ultimately shows that the internal consistency is weak and that the group assessment does not reliably measure student reading ability. This is a key finding.

Table 5. Internal Consistency for Group Assessment


Letter Sounds 0.14 0.58

Non-words 0.35 0.51

ORF 0.47 0.43

Reading Comprehension 0.34 0.52

Listening Comprehension 0.33 0.52

Maze 0.53 0.48

Scale 0.56

Concurrent Validity Although the results of the previous two sections point to concerns about the group assessment, the most direct measure for determining whether or not the group test could be used as an appropriate substitute for the individual assessment is to test the concurrent validity. Concurrently validity is a measure of how well a particular test correlates with a previously validated measure. In this case, the previously validated measure is the individual assessment and the new test is the group assessment. Beginning with an examination of pairwise correlations of subtasks within tests, Table 6 shows the subtask correlations of the individual assessment. Across subtasks, every correlation is statistically significant and the majority range from moderate to


strong (0.41 to 0.69). The weakest overall correlations are for listening comprehension (0.31-0.46). These results are consistent with the previous internal consistency analyses.

Table 6. Pairwise Correlations of Subtasks for Individual Assessment

Individual Assessment

Letter

Sounds Non-

Words ORF Reading Comp

Listening Comp Maze

Letter Sounds 1

Non-Words 0.69* 1

ORF 0.46* 0.50* 1

Reading Comp 0.41* 0.42* 0.69* 1

Listening Comp 0.31* 0.33* 0.31* 0.46* 1

Maze 0.41* 0.45* 0.59* 0.61* 0.48* 1

For the group assessment, results are displayed in Table 7. Although there is one moderate correlation of 0.45 (Maze and ORF), the majority of them are weak. Furthermore, Letter Sounds is only significantly correlated with two other subtasks (non-words and reading comprehension), thus bolstering the claim that Letter Sounds (as designed) does not necessarily belong in this assessment.

Table 7. Pairwise Correlations of Subtasks for Group Assessment

Group Assessment

Letter

Sounds Non-


Listening Comp Maze

Letter Sounds 1

Non-Words 0.24* 1

ORF 0.07 0.19* 1

Reading Comp 0.21* 0.26* 0.27* 1

Listening Comp 0.03 0.14* 0.23* 0.15* 1


Group Assessment

Letter

Sounds Non-


Listening Comp Maze

Maze 0.04 0.31* 0.45* 0.24* 0.31* 1

As for the direct measurement of concurrent validity, Table 8 provides the results of the subtask correlations across assessments. Each column represents a subtask from the group test and each row refers to the subtasks from the individual assessment. What stands out in this table is the fact that only one half of the cross-assessment correlations are statistically significant. Even in direct subtask comparisons (e.g. group letter sounds to individual letter sounds) only half of the correlations are significant (i.e. ORF, reading comprehension and maze). Furthermore, Maze is the only subtask from the group assessment that is significantly correlated with all subtasks from the individual assessment. With regard to correlation strength, only two correlations on the entire table would be considered moderate—0.43 (group Maze – individual Listening Comprehension) and 0.50 (group Maze – individual Maze). All other correlations are weak to null.

Table 8. Concurrent Validity for Group and Individual Assessments

Group Assessment

Letter

Sounds Non-


Listening Comp Maze

Indi

vidu

al A

sses

smen

t

Letter Sounds

-0.08 -0.02 0.14* 0.01 0.10 0.31*

Non-Words -0.01 0.05 0.13 0.07 0.15* 0.38*

ORF 0.06 0.14 0.32* 0.09 0.14 0.36*

Reading Comp

0.08 0.15* 0.28* 0.26* 0.20* 0.33*

Listening Comp

0.09 0.06 0.30* 0.26* 0.08 0.43*

Maze 0.04 0.24* 0.30* 0.25* 0.12 0.50*

Ultimately, this table shows that the concurrent validity for the group and individual assessments is very weak and that the group test should not be used in lieu of the individual assessment without significant revisions.

Individual Item Assessment Individual items on both assessments are examined in this section in order to identify what changes might enhance the validity of the group assessment. Beginning with the


individual assessment, Figure 2 displays box and whisker plots of each of the six subtasks. Each box represents the 25th to the 75th percentile of the proportion of correct scores for a given subtask (a.k.a. the interquartile range (IQR)). The line in the middle of each box represents the median. Dots represent outliers, which are calculated as being more than 1.5*IQR above the 75th percentile (the top line on the whiskers) or 1.5*IQR below the 25th percentile (the bottom line on the whiskers). This figure shows that all subtasks provide information on a wide range of student abilities and that there is no evidence of ceiling or floor effects. Although improvements could be made, these plots provide an example of a successfully designed early grade reading assessment.

Figure 2. Individual Assessment Subtask Box and Whisker Plots

Conversely, the box and whisker plots for the group assessment subtasks are displayed in Figure 3. Unlike the individual assessment plots, which showed a wide range of scores, the group assessment plots show that score ranges were constrained. As a matter of fact, only two of the boxes are even visible on the figure because four of the six subtasks are subject to such high proportions of correct responses. This lack of variability impacts both the reliability of the test as well as its concurrent validity with the individual assessment. Ultimately, it appears that only the Maze subtask is able to appropriately discriminate among high and low scorers—although the scores on this subtask are still higher than the majority of subtasks on the individual assessment.


Figure 3. Group Assessment Subtask Box and Whisker Plots

The final two figures of this section display the interaction between ORF and reading comprehension on both the individual and group assessment. Since these two measures are core measures of reading proficiency, it is important to ensure that they function properly within the assessment. By design, ORF scores should be highly correlated with reading comprehension scores. This is due to the fact that the reading comprehension questions are asked based on how far into the passage a child can read. For example, students who only read the first sentence of the reading passage would only be asked the first reading comprehension question (i.e. the only question that could be answered from reading that sentence). Such students would have low ORF scores (because they only read the first sentence) and even if they answered the reading comprehension question correctly, their overall score could not be higher than 17% (because they’ve only answered one of six questions). Accordingly, Figure 4 shows that the relationship between ORF scores and reading comprehension is strongly positive (with a correlation of 0.69). In other words, students with low ORF scores tend to do poorly on the reading comprehension section, while those with high ORF scores tend to do well. This is what is expected based on the test design.


Figure 4. ORF and Reading Comprehension – Individual Assessment

It should be noted, however, that the approach used in the administration of the individual EGRA was not the approach used in the group assessment. In the group assessment, all students were asked all reading comprehension questions regardless of how far into the passage they read. While this does provide an opportunity for students to guess, it is still expected that students with low ORF scores would, on average, score well below their high ORF counterparts. However, Figure 5 shows that there is almost no relationship at all between ORF and reading comprehension (correlation = 0.27). Although this is due in part to the fact that ORF scores were relatively high on the group assessment, even students who read small amounts of the passage were able to answer most if not all of the reading comprehension questions correctly. For example, seven students read 30 or fewer words in the passage but five of them were still able to answer all six questions correctly and the other two answered 5 out of 6 questions correctly. Once again, these results do not directly explain why students scored so high on the reading comprehension despite their low ORF scores but it does provide further evidence that the group assessment was flawed and that both the ORF and reading comprehension sections need to be revised for any future administrations.


Figure 5. ORF and Reading Comprehension – Group Assessment

Summary Conclusions The high scores across all subtasks on the group assessment provide evidence that the instrument piloted in Egypt was flawed and that revisions and test redevelopment are necessary for future administrations. This is further supported by the low internal consistency of the test, as well as the weak concurrent validity for the group and individual assessments. The lack of variability in scores within subtasks also made it difficult to appropriately discriminate among high and low scorers. Ultimately, there is no evidence to suggest that the current version of the group assessment should be used in lieu of (or even as a complement to) the individual EGRA.

Limitations, Recommendations and Further Research

Key Challenges to the Group Assessment The high student scores on the group assessment might suggest that subtasks were too easy. However, the reading passages, questions and items used for the oral reading fluency, reading comprehension, listening comprehension and Maze comprehension sub-tasks were taken directly from the 2013 EGRA baseline for Grade 3. In that baseline, only 13% of sample Grade 3 students read the ORF passage at benchmark level (50 cwpm); less than 9% of students correctly answered five of the six reading comprehension questions; and just 18% of students correctly answered six of the


seven listening comprehension questions. Therefore, the subtask content itself does not explain the high scores in the group assessment (see Recommendations).

The Grade 3 students who participated in this group assessment were, on average, significantly better readers than the national random sample of Grade 3 students tested for that 2013 baseline. However, the same students were tested on both the individual and group assessment, so this does not explain the higher scores on the group assessment (relative to the individual).

The high scores chiefly reflect three major challenges in implementing this pilot Group Assessment instrument: i) the opportunity for students to guess answers; ii) the opportunity for students to copy the answers of classmates; and iii) the opportunity for students to exaggerate their reading performance. All three practices were common in this pilot implementation:

• Guessing Answers: For this pilot assessment, the letter sounds, non-words, reading comprehension and listening comprehension sub-tasks were multiple-choice exercises. For most items, students had 3 choices to select from. For a number of items, the choices were quite distinct, such that a student might readily guess the correct answer from just a general knowledge of the story without having to listen carefully to the question. Indeed, more than a few students were observed marking the answers to questions after hearing the listening comprehension passage or completing the reading comprehension story – but before hearing the questions asked by the Group Assessment Leader.

• Copying Answers: The group assessment was administered with quality controls established in many schools and students were tested in well-managed environments. Clear instructions were given to students and the classrooms were organized in a way that sought to maximize privacy for students. Nevertheless, attempts by students to copy answers were observed at one of the schools. This may be attributed to the layout and size of a typical classroom with 35 students, and with two of them sitting together at a single bench desk and with limited physical space in between them. Also, the pacing of the questions allowed for additional time during which a number of students at this school were observed to be looking at other students’ papers. Finally, the layout of the student answer sheet and the multiple-choice answers arranged on a single page for each subtask may have contributed to student attempts to copy answers at this school. This is an important lesson learned from the pilot and will directly inform the future design of group assessments.

• Exaggerating Performance: For the passage reading subtask, students were asked to mark the final word read at the end of the allotted time. The large majority of students marked the final word of the passage. Students silently sight read the passage—and sight reading is significantly faster than oral reading. Unfortunately, the actual reading performance of students cannot be verified by an observer. Therefore, simply reporting the final word circled by students likely overestimates the actual reading fluency and comprehension levels of students taking the group assessment.

Other serious limitations of the group assessment method, some mentioned previously, include:


• Reading Comprehension: All students are asked all reading comprehension questions, regardless of how far they read in the passage in the one minute. The group assessment cannot distinguish which questions are answered incorrectly simply because the student did not read that part of the passage.

• Oral Reading Fluency: The group assessment cannot measure ORF. An assessment leader cannot know how many words a student read silently in one minute. Reading accuracy, a key dimension of fluency, is also impossible to judge in silent reading. The group assessment cannot test students’ reading fluency and pronunciation.

Finally, group assessments cannot be conducted electronically. Paper answer sheets must be scored and entered manually into data sets for analysis.

Benefits of the Group Assessment Although there were challenges with both the design and administration of the group assessment, there are also benefits relative to the individual administration of early grade reading assessments. The benefits of the group assessment include:

• Reduction in assessor error: One limitation of the group assessment is also a benefit. With no oral reading there is no assessor error in scoring a specific word as read correctly or incorrectly with all diacritics. Moreover, the group assessment significantly reduces assessor variance and error in their sub-task instructions to students. Assessor reliability (inter-rater reliability) in scoring non-word and oral reading fluency sub-tasks and instructing students for each sub-task are significant issues when non-professional assessors implement EGRAs.

• Reduced constraints to student reading performance: Not having to read aloud, students are typically more confident and less nervous in group assessments. Reading silently without concern for proper pronunciation, intermediate and strong readers will read much more quickly. They can also: i) re-read previous sections of reading passages to be sure that they correctly understood what they read, and ii) re-check previous selections on the Maze comprehension. Students were observed doing both in this pilot assessment. But in individual EGRAs, students can only read linearly. They cannot return to previous sentences or Maze selections. If reading passages are not too long, students participating in group assessments should score significantly higher on reading comprehension.

• Fewer assessors and reduced training: The group assessment needs only a single trained assessor—and the technical skills required to competently lead a group assessment are far fewer. Training a group assessor who is already experienced implementing an individual EGRA can be done in just 1-2 days.

• Economies of scale: Implementing individual EGRAs with more than 20-25 students in one school in one day is difficult and exceptional. Assuming an EGRA of typical length (5-6 sub-tasks), three assessors would be needed to individually test this number of students. But a single assessor could assess 120-130 students – implementing up to 5 group assessments of 25 students each – in one school. A group assessment needs just 35-45 minutes to complete, from start to finish. Alternatively, a group assessor could test 40-60 students in each of two proximate schools,


implementing 2 group assessments in each school. A national assessment does not need 20-25 students tested per school. But if muderiyas, idaras and schools are keen to assess Grade 3 reading performance in individual schools, perhaps to compare their reading performance with other schools, then a minimum random sample of 40 students are needed in each school. This number of students calls for group assessment.

These benefits of the group assessment, and possible solutions to key challenges, merit further research and pilot implementation of group assessment methodologies.

Recommendations The development of reliable and valid assessment tools requires time and repeated trial implementations. This pilot was the first iteration of a group assessment instrument and methodology in Arabic. The lessons gleaned from this first group assessment will significantly inform the next iteration. This was the purpose of this 2014 pilot.

The benefits of a group assessment method justify further research, development and testing. A group assessment cannot fully substitute for individual EGRAs. But it can be a valid, timely and low-cost complement methodology that adds significant assessment input to the monitoring and evaluation of reading proficiency.

Several technical recommendations can be drawn from this first pilot assessment: • Substitute open response questions for multiple-choice items: Requiring students

to write their answer to each question and not choose from a selection of multiple-choice items will significantly reduce both guessing and ease of copying. It will, however, increase the time and decision-making required to score these answers. Additionally, the required responses must be carefully thought out in order to ensure that the test continues to assess reading as opposed to writing skills.

• In a final group assessment of 12 students, this approach was tried. We replaced the multiple-choice answer sheet with empty lines for students to write their answers for the listening and reading comprehension sections. The results from the revised pilot are are displayed in Figure 6. This figure shows that average and spread of the reading and comprehension scores on the revised assessment were more appropriate than the original group assessment (Figure 3) and much closer to the characteristics of the individual assessment (Figure 2). Whereas students averaged 98% and 96% correct on the reading and listening comprehension subtasks on the original group test, the revised form showed averages of 68% and 85% for reading and listening, respectively. Although this does not fully answer the question of why students universally performed so well on the original group assessment, it does provide promising evidence for how the test can be revised in order to more closely align with scores on the individual assessment.


Figure 6. Results of a Revised Group Assessment Design for 12 Students

Additionally, it may be sufficient to simply provide a greater number of response options, with more appropriate distractors (i.e. choices that might be chosen by a respondent who didn’t fully comprehend the passage). Lastly, providing some sort of divider between students (perhaps in the form of a folder or notebook) could also help reduce the potential for copying in crowded classrooms. These approaches, however, were not tested in our pilot.

• Research the appropriate length for reading passages in silent reading assessments: For most students performing above the level of struggling reader, silent reading is much faster than oral reading aloud. A reading passage of 57-60 words is currently satisfactory for oral reading by all levels of Grade 3 reading proficiency in Egypt. But it is much too short for silent reading by intermediate to strong Grade 3 readers in one minute. Applied research is needed to determine the appropriate length for passages in Modern Standard Arabic – with diacritics – read silently. This may be 100-150 words. This length will also vary by grade.

• Apply group assessments to random samples of students: The next iteration of group assessment instruments should be tested on random samples of students demonstrating the full range of reading abilities. These same students might usefully be tested with individual EGRAs too, so as to compare the range and variability of scores on comparable sub-tasks across tools. These pilot group assessments should also determine the maximum number and optimum physical arrangement of students for a group assessment conducted in a typical MOE classroom in order to minimize the opportunities for copying.

• Rigorously research the utility of group assessments by subtask and Grade: The group assessment methodology may be more valid, accurate and useful for certain


sub-tasks than for others. It may, for example, be an excellent substitute for individual EGRAs in assessing student comprehension but not reading fluency. And it may be more valid and practical for testing student reading in Grades 3 and above than for testing Grades 1 and 2. Younger children in groups may have more difficulty correctly following sub-task instructions delivered by group assessors than when individually directed and guided one-on-one by an EGRA assessor. The next iterations of group assessment methodologies should systematically appraised for their utility and accuracy by sub-task and grade.

• Specify the objectives and use of the group assessment – then test the methodology’s utility for that objective / use: Individual EGRAs benefit from years of applied research, rigorous appraisal and professional guidance on their appropriate application, objectives and utility. This is currently lacking for group administered assessments for early grade reading. What is the specific objective of implementing group assessments? Is it to substitute for individual EGRAs as a system reading diagnostic and national assessment methodology for a ministry? Or is it to facilitate the testing and comparison of student reading proficiency between individual schools or between idaras and muderiyas? Is the group assessment to be implemented by minimally-trained teachers or local supervisors in their own schools or idaras? Or will it to be conducted by MOE assessment professionals in sample schools?

• The objectives and use of the group assessment methodology must be specified for future pilots so that the conditions of trial assessment are consistent with the assessment objectives and the utility of the assessment can be appraised against specific objectives.

While this investigation provides limited evidence on the validity and reliability of the piloted group assessment in Egypt, by overcoming some key challenges and implementing some much needed changes to both the instrument and administration of the group assessment, there is reason to believe that the benefits of a group administered early grade reading assessment could be realized in future administrations.


Annex 1. 2014 EGRA Assessor Tool and Student Answer Sheet; Group Assessor Tool and Student Answer Sheet


The 2014 2nd National EGRA Instrument – Used for Individual Assessment


The 2014 2nd National EGRA Instrument – Letter Sounds Knowledge


The 2014 2nd National EGRA Instrument – Nonwords Reading


The 2014 2nd National EGRA Instrument – Oral Reading Fluency and Reading

Comprehension Subtasks


The 2014 2nd National EGRA Instrument – Listening Comprehension


The 2014 2nd National EGRA Instrument – Maze Comprehension – Instructions


The 2014 2nd National EGRA Instrument – Maze Comprehension – Subtask


The Pilot Group Assessment Instrument – Assessor’s Form


The Pilot Group Assessment Instrument – Student Worksheets

EdData II Task Order 27: Egypt Grade 3 Early Grade Reading ... · EdData II: Egypt Grade 3 Early...

Documents

Transcript of EdData II Task Order 27: Egypt Grade 3 Early Grade Reading ... · EdData II: Egypt Grade 3 Early...