1 Review of Toolkit Amber Gove RTI International Session 3.4.
-
Upload
dayna-anthony -
Category
Documents
-
view
220 -
download
0
Transcript of 1 Review of Toolkit Amber Gove RTI International Session 3.4.
2
Potential Purpose(s)
1. How is the system doing? Raise government/public awareness of a national or regional problem
2. How is the project doing? Evaluate discrete interventions
3. How are students doing? Help teachers track student progress in the lowest primary levels, before students have problems and drop out
In all cases, the end goal is to provide resources for interventions that clearly address the problem.
3
2. RemediateUse evidence-based
instructional approachesand support to teachers
to improve student learning.
3. MonitorDevelop progress
monitoring tools and foster accountability
for meeting goals
1. DiagnoseUse EGRA to identify instructional needs, raise awareness,
and set system goals
The continuous cycle of improving student learning.EGRA and EGRA-based assessments can be used to diagnose, remediate and monitor progress toward improving student learning outcomes.
2. RemediateUse evidence-based
instructional approachesand support to teachers
to improve student learning.
3. MonitorDevelop progress
monitoring tools and foster accountability
for meeting goals
1. DiagnoseUse EGRA to identify instructional needs, raise awareness,
and set system goals
2. RemediateUse evidence-based
instructional approachesand support to teachers
to improve student learning.
3. MonitorDevelop progress
monitoring tools and foster accountability
for meeting goals
1. DiagnoseUse EGRA to identify instructional needs, raise awareness,
and set system goals
The continuous cycle of improving student learning.EGRA and EGRA-based assessments can be used to diagnose, remediate and monitor progress toward improving student learning outcomes.
4
Lessons from Pilots: Phases of EGRA
1. Planning and Development
2. Adaptation and Training
3. Field Work
4. Data Entry and Analysis
5. Dissemination and Reporting
5
Planning and Development
Review purpose with funder and Ministry
Involve/consult with teachers, ministry staff, parents, survey team, curriculum experts
Develop a detailed workplan, clarify roles and responsibilities
Determine use of results and dissemination plan
Carefully develop sample to fit purpose, budget
6
ACTIVITIES Responsible Other actors AG SLT
Adaptation of EGRA instrumentsPrepare materials and presentations Gove Linan-Thompson 3 1 JumeMeet with Honduran specialists to review objectives
World BankGove;Linan-Thompson, Honduran specialists
2 1 June
Adapt existing instruments to Honduran Gove Honduran specialists TBD 2 2 JunePre-test of instruments in 2 schools, ~30 children, revise
Honduran specialistsGove;Linan-Thompson, Honduran specialists
1 2 June
Identification of pilot and baseline samples Honduran specialists Gove (review) 1 SeptPilot fieldwork arrangements (transportation, equipment, contact w/ pilot sites)
Honduran fieldwork team TBD
Gove (review) 1 Sept
Pilot with EGRA Instrument, Training and Training of enumerators Gove Gove; Honduran team 3 SeptPiloting of instruments in 6 schools with ~ 180 children
Honduran team Gove 2 Sept
Pilot study data entry and management Gove Honduran team 1 SeptAnalysis of results and modification of instruments as needed
Gove Gove; Honduran team 1 Sept
Validation of final version of instruments Gove Honduran team 1 SeptIdentification of baseline sample of 60 schools
Honduran specialists Gove 1 Sept
Trip report Gove 1 Oct
DateLOE
7
ACTIVITIES Responsible Other actors AG SLT
Baseline assessment of EGRABaseline fieldwork arrangements (transportation, equipment, sites)
Honduran team Oct
Fieldwork in 60 schools with ~ 1800 children (30 children / school)
Honduran teamGove (long distance trouble shooting)
2 Oct
Baseline assessment analysis and Data entry and management Honduran team Gove (review) 1 NovAnalysis of results, including instrument psychometrics
GoveLinan-Thompson; Honduran specialists
4 Nov
Preparation of draft Ministry report (~10 pages)
GoveLinan-Thompson; Honduran specialists
5 3 Nov
Preparation of draft World Bank report (~15 pages)
GoveLinan-Thompson; Honduran specialists
7 3 Nov
Report review and feedback World Bank Honduran specialists Nov
Preparation of revised, final reports GoveLinan-Thompson; Honduran specialists
5 5 Nov
Production of final report (editing, RTI Gove (review) 0.5 NovWorld Bank presentation World Bank Gove; Honduran specialists 0.5 Nov
Total LOE : 45 17
DateLOE
8
Adaptation and Training
Start with complete, but draft version
Adapt with teachers, survey specialists, curriculum experts (bring textbooks, word lists, sample stories)
Balance economy and desire to include everything
Pre-test and Pilots serve both training and instrument improvement opportunities (stopwatch practice; 0 vs. missing)
Changes must be evidence-based
Assess Inter-Rater Reliability using audio recordings
9
Monday Tuesday Wednesday Thursday Friday
IntroductionsLogisticsReview EGRA development processReview Research underlying EGRA
Review draft instrument and student questionnaireRevise draft instrument
Application training and practice
Pre-test in 2-6 schoolsEnter data as a practice exerciseConduct simple analyses using excel
Discuss resultsDiscuss instructional implicationsRevise draft instrument based on results
10
Monday Tuesday Wednesday Thursday Friday
Review of underlying principlesReview draft instrumentReview draft teacher questionnaireApplication training and practice
Application training and practiceReview roles and responsibilities of supervisors and enumerators
Pilot in 3 to 6 schools (determined by the number of teams being trained) Enter dataAnalyze results and modify instrument
Print final version of the instrumentApplication training and practice
Prepare and pack materialsFinalize and review logistics
11
Field Work
1 Supervisor plus 3 to 4 Enumerators
Timely transport is critical: arrive before school starts and meet, greet, thank director and teachers
Avoid false negatives: be nice to the children
Sample across all classrooms using teacher lists; start with youngest (1st grade) students
Prior to leaving the school: verify completeness and avoid missing data (waste of everyone’s time)
12
Data Entry and Analysis
Involve supervisors in data entry if possible
Develop and test data entry program in advance
Clarify difference between 0 and missing
If using excel, ensure consistency of entry
Use analysis as opportunity to work with ministry staff
Develop strategies for analysis in excel
13
Variable name Variable description Coding notes Clread Correct letters read Blank means that the task
was not continued Clseconds Time in seconds to correctly read
letters Blank means that the task was not continued
Clpm Fluency in correct letters read per minute (created variable)
Zero means that no words were read as the task was not continued
Cwctread Correct words in connected text read
Blank means that the task was not continued
Cwctseconds Time in seconds to read connected text
Blank means that the task was not continued
Cwpmct Correct words per minute in connected text (created variable)
Zero means that no words were read as the task was not continued
14
Grade Variable 1 2 3 Correct letters per minute 12.8 25.4 36.1 Correct letters per minute, excluding non-readers 19.8 28.6 37.7 Correct familiar words per minute 1.2 2.3 4.3 Correct familiar words per minute, excluding non-readers 6.2 8.3 9.2 Correct words per minute, connected text 2.2 4.0 9.2 Correct words per minute, connected text, excluding non-readers 11.0 11.6 17.3
15
Grade 1 2 3 Count 419 389 392 Average correct letters per minute 12.8 25.4 36.1 Standard deviation 18.2 21.9 23.2 Standard error 0.89 1.11 1.17
16
Dissemination and Reporting
Develop different reports for different audiences (parents, teachers, ministry staff, testing specialists)
Recall purpose is to improve, not blame; use results accordingly
Discuss next steps, teacher training
Provide solutions to the challenges identified
Use results to mobilize for improvement
17
1 Not even tracked or benchmarked past kindergarten, on the assumption that this skill is dominated in Kindergarten. Note the low level of the average, for Grade 2, for the case in the table.
Example of possible benchmarking exercise
Average across children in a given country, middle of
year, Grade 2
Maximum school-level average in the
same country, Grade 2
Developed-country
benchmarks for comparison purposes
Letter recognition fluency 22.7 41 40 by end of
year kindergarten1
Nonsense word fluency 7.5 25 50 in middle of
Grade 1
Connected text fluency 11.4 36 20 in middle of
Grade 1 Comprehension score 0.4 2 NA